A practical guide to evaluation

For the past 20 years I have been extensively involved in the training of evaluation practitioners and scholars. My experiences have allowed me to engage in critical discussions about the role and significance of evaluation. As part of my ongoing work in the area of learning and development, I recently took up a role as an instructor on the United Nations System Staff College’s “Evaluation in the Secretariat” course. Being part of this course and meeting highly knowledgeable professionals from across UN led to a brief reflection on the many enriching years of my career in evaluation.

I realized that there are unique aspects and perspectives that can help evaluators gain the skills and knowledge that they need to be effective in their roles. I think these insights are particularly important for prospective UN evaluation practitioners who are considering this interesting field.

1. Evaluation is guided by a set of norms and standards

I have come across a number of people who confuse evaluation for basic research or consulting. That is why it is important to understand that evaluation is a field on its own. It draws from different areas of knowledge which include economics, psychology, education, and sociology. It is however a function which has its own set of standards and practices that should be adhered to in order to increase the potential of its usefulness and impact. Evaluation also has a set of unique networks, guiding principles, research-based practices, and informational resources. In the UN, the UNEG Norms and Standards guides how UN evaluations are conducted. It provides a reference framework for UN evaluation practices. This rich set of resources can help new evaluators gain a sense of evaluation as a discipline. I have found the them to be extremely beneficial for those who wish to understand accepted ethical evaluation practices that can support their work and help them to avoid common pitfalls or errors in practice.

2. Evaluation learning requires a strong practice component

Like any other skill that we take up, making a regular effort to practice the skill is the key to being successful. In evaluation, learning from examples and case studies about evaluation is an important first step. Having said this, evaluation training requires real-world experiences to help translate the knowledge gained in formal settings to more complex real-life contexts. The skills that you need to respond to changes on the ground, deal with different stakeholders who have varying interests and communicate the implications of evaluation findings can only be developed through experience that is built on formal knowledge. It has therefore been encouraging to see many of the participants in the Evaluation in the Secretariat course working on their first evaluations.

3. Evaluation is not a technocratic exercise

It is sometimes difficult to convey that evaluation is not technocratic, particularly during training sessions. I find it best to discuss specific cases to illustrate this point. I let participants know that evaluation often encompasses a myriad of different considerations. One has to take a few things into consideration - the interests of stakeholders in the evaluation results, determining who gets to be involved in deciding the focus and questions of the evaluation, understanding what the optimal or most feasible methodologies are in a specific context, and the best way to render a judgment or recommendation from the evaluation results.

One element that we focus on in Evaluation in the Secretariat course is teaching participants to think about evaluation holistically. We remind them that evaluation processes require constant reflection and consideration for other influencing factors. Thinking about evaluation holistically helps to increase the potential that an evaluation project is valid, credible, and useful.

4. Understanding what valuing means in evaluation

The concept of valuing is at the heart of evaluation but it also tends to be one of the most challenging to train or teach on given the complexities associated with it. One way to help people understand the valuing process is to provide multiple examples of the two main methods for establishing values. They include the norm-reference method and the criterion-based method. The norm-reference method compares the performance on a measure (let us say a math test) to other measures – (let us say the number of people who completed the math test). If one person did better than most people on the math test then we can say that they are good or prepared for the math test.

The criterion-based method compares performance to an established set of standards, meaning that it does not matter how well you perform compared to others, the only thing that matters is how you perform compared to the criterion. In the case of the math example, we can say that a criterion is getting at least 85% of the questions correct which means that you are prepared or good at math. These two approaches to valuing often get confused and tangled when people think about how to make sense of evaluation results. I frequently encourage training participants to determine which valuing method will be adopted within any evaluation project.

5. Determine the purpose of the evaluation, and remember it throughout the process

Sometimes evaluations can take many months or years to complete, and it's important to remember why the evaluation is being conducted. This can determine how the evaluation is designed and how the findings can be used. Broadly speaking evaluations serve two purposes, firstly they can be used for accountability purposes, meaning that the evaluation project is meant to determine if something is effective or not. The knowledge can potentially be used to decide if something should continue being funded (this is often referred to as summative evaluation). Secondly evaluations can be used as an opportunity for learning and program development purposes. In this instance the findings are often used to help improve program implementation and offer some insights about the potential impacts of the program (this is often referred to as formative evaluation). From an evaluation perspective, it is critical to have some clarity about the purpose(s) of the evaluation. Sometimes it is purely formative or summative, and sometimes it starts as a formative exercise then transitions to summative one.

The Administrative Instruction (AI) on Evaluation in the United Nations Secretariat (ST/AI/2021/3) requires each Secretariat entity to have adequate evaluation capacity. If you are interested in sharpening your skills and knowledge on evaluation, I invite you to look into our upcoming edition of Evaluation in the Secretariat course (26 Sep-21 Oct).