To objectively evaluate the success of alterations to existing Business Intelligence (BI) environments, we need a way to compare measures from altered and unaltered versions of applications. The focus of this paper is on producing an evaluation tool which can be used to measure the success of amendments or updates made to existing BI solutions to support improved BI reporting. We define what we understand by success in this context, we elicit appropriate clusters of measurements together with the factors to be used for measuring success, and we develop an evaluation tool to be used by relevant stakeholders to measure success. We validate the evaluation tool with relevant domain experts and key users and make suggestions for future work.