You Call This Exemplary? Lessons from an Unsung International Evaluation.

Saved in:

Bibliographic Details
Title:	You Call This Exemplary? Lessons from an Unsung International Evaluation.
Authors:	Horton, Douglas¹
Source:	Journal of MultiDisciplinary Evaluation. 2017, Vol. 13 Issue 28, p41-52. 12p.
Subject Terms:	Stakeholders, Professional standards, Disciplinary infractions
Abstract:	Background: Program evaluators have reached broad agreement on principles for planning and conducting evaluations and on standards for judging their quality. However, many evaluation stakeholders, including key intended users, may judge evaluations on criteria that differ sharply from the professional standards and the criteria we commonly employ in meta-evaluations. Purpose: This paper highlights the role of academic discipline and epistemic community in judging what is an "exemplary" evaluation, by examining the case of an evaluation that was considered exemplary by professional program evaluators, but methodologically flawed by professionals from different disciplinary traditions. Setting: The evaluation in question was carried out within the Consultative Group on International Agricultural Research (CGIAR), a research network with a rich tradition of economic impact assessment. Intervention: NA Research Design: This is a case study that combines participatory action research and historical analysis. Data Collection and Analysis: The study is based on the author's personal involvement in the evaluation and on a review of publications and unpublished documents related to the case. Findings: A team of experienced evaluators applied what are generally considered to be good practices in the program evaluation community. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. In contrast, within the CGIAR, both the evaluation and the meta-evaluation study were considered to be methodologically flawed and biased. Three lessons related to exemplary evaluation are formulated and elaborated upon: Lesson 1. Being exemplary is in the eyes of the beholder. Lesson 2. Epistemic communities are hard nuts to crack. Lesson 3. You can't win them all. While the early efforts with program evaluation analyzed here were experienced as failures, a number of subsequent developments have led to greater understanding of diverse evaluation approaches, and some movement toward agreement on what constitutes exemplary evaluation in the CGIAR. Nevertheless, there is still a considerable way to go. [ABSTRACT FROM AUTHOR]
	Copyright of Journal of MultiDisciplinary Evaluation is the property of Western Michigan University, The Evaluation Center and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database:	Education Research Complete

Description
Abstract:	Background: Program evaluators have reached broad agreement on principles for planning and conducting evaluations and on standards for judging their quality. However, many evaluation stakeholders, including key intended users, may judge evaluations on criteria that differ sharply from the professional standards and the criteria we commonly employ in meta-evaluations. Purpose: This paper highlights the role of academic discipline and epistemic community in judging what is an "exemplary" evaluation, by examining the case of an evaluation that was considered exemplary by professional program evaluators, but methodologically flawed by professionals from different disciplinary traditions. Setting: The evaluation in question was carried out within the Consultative Group on International Agricultural Research (CGIAR), a research network with a rich tradition of economic impact assessment. Intervention: NA Research Design: This is a case study that combines participatory action research and historical analysis. Data Collection and Analysis: The study is based on the author's personal involvement in the evaluation and on a review of publications and unpublished documents related to the case. Findings: A team of experienced evaluators applied what are generally considered to be good practices in the program evaluation community. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. In contrast, within the CGIAR, both the evaluation and the meta-evaluation study were considered to be methodologically flawed and biased. Three lessons related to exemplary evaluation are formulated and elaborated upon: Lesson 1. Being exemplary is in the eyes of the beholder. Lesson 2. Epistemic communities are hard nuts to crack. Lesson 3. You can't win them all. While the early efforts with program evaluation analyzed here were experienced as failures, a number of subsequent developments have led to greater understanding of diverse evaluation approaches, and some movement toward agreement on what constitutes exemplary evaluation in the CGIAR. Nevertheless, there is still a considerable way to go. [ABSTRACT FROM AUTHOR]
ISSN:	15568180
DOI:	10.56645/jmde.v13i28.469