Adaptive User Interfaces (AUIs) have been promoted as a solution for context variability due to their ability to automatically adapt to the context-of-use at runtime. For the acceptance of AUIs, usability plays a crucial role. Classical usability evaluation methods like usability tests, interviews or cognitive walkthroughs are not sufficient for evaluating end-user satisfaction of AUIs at runtime. The main reason is that the UI and context-of-use are dynamically changing and acceptance of UI adaptation features has to be evaluated in the moment and context-of-use when the adaptations occur. To address this challenge, we present an on-the-fly usability evaluation solution that combines continuous context monitoring together with collection of instant user feedback to assess end-user satisfaction of UI adaptation features. We demonstrate the usefulness and benefit of our solution based on a mobile Android mail app, which served as basis for conducting a usability study.