Over the last dozen years, I have been an external evaluator for social and education programs. The honor of working with so many honest, intelligent, and hard-working people is tempered by the underlying desire not to have me on the project. Many people see me as the interloper or the added expense. Because of this, I often come in on the second year of the project. In this post, I explain why.
The role of the external evaluator is much like a financial auditor. The auditor makes sure the money is spent appropriately. The external evaluator checks to see if the money is spent effectively. Evaluation reports inform the team, suggests an improvement, and justifies the use of resources.
Many people involved in education or social initiatives try to evaluate the programs themselves. Justifiably, program coordinators feel that an evaluation report is a natural extension of the implementation process. Program developers are comfortable and experienced with the necessary variables. They often enjoy a deep understanding of the qualitative and unmeasured elements of the program. Also, external evaluators require more time, cost, and data security issues. In general, I accept the argument to be made for self-evaluation but also find it incomplete and, in the end, more often misguided.
There are two reasons that an external evaluation is an essential step in the development of effective interventions. An external evaluation can first improve report quality and second external acceptance.
First, people who implement are rarely very good at evaluation. Data collection and standardized measures are often a problem. While the implementation team may understand their guiding measures, they are just as often less comfortable in their bottom line or summative measures. For example, teachers need to work with letter naming fluency and phenomes but less comfortable with the summative score. Implementation teams tend to use everything possible to ensure the success of the effort, so summative measures are naturally reductive.
The external evaluator can add rigor by standardizing data collection and choosing the best variables. Sometimes the logical measure does not work because of poor quality or incomplete ability to differentiate between groups. For example, “attendance” is always a logical variable. You can not receive an effect of the program if you do not attend the program. At the same time, what is attendance? How late can a subject be and still attend? Is there a makeup option? Do you accept excuses? In the end, attendance is often very high. Making a decision on differential effects based on a 5% nonattendance rate is difficult in the largest of samples. Thus, some variables that are logical and have strong face validity just to not work as wel.
An effective evaluator can align measures with accepted standards so that results can be compared to other studies. Different groups can be equated through techniques such as statistical weighing. A good statistician can demonstrate relationships that are simply not clear with simple proportions and can separate accidental from true differences. Program administrators are not specialist in aligning standards across studies and identifying real differences from random chance.
Second, there is a matter of trust in the results. Changes in rules, curriculum, or methods can be uncomfortable. People question the need or effectiveness of the changes – sometimes in loud, public meetings. The motivations of program directors and their supervisors are questioned. I have talked to people that maintain programs exist for commercial or even religious reasons. Convincing speakers claim damage to their educational objectives, privacy, and faith.
Program providers can be too close. The program provider’s assessment is often seen as a cover-up or an excuse. Also, when evaluating the program provider is more likely to forgive deviation because they see what is supposed to happen. Variables that become challenging may be replaced rather than improved.
An external evaluator’s report can show the public that every effort has been made to assess and improve the quality of political leaders and funding sources. The report can be more than yes/good or no/bad. A good report can identify strengths and weakness for program improvement by looking at the program through a fresh set of eyes. Past evaluations have shown effectiveness in one area but not in another, gap closure, or other areas of improvement.
In the end, external evaluation can provide valuable, independent insight into the program and target areas for improvement. A quality evaluation identifies possible management and problems and areas were content experts can enhance for better outcomes. The external evaluation is key to improving and justifying a program.