You Call This Exemplary? Lessons from an Unsung International Evaluation.
Saved in:
| Title: | You Call This Exemplary? Lessons from an Unsung International Evaluation. |
|---|---|
| Authors: | Horton, Douglas1 |
| Source: | Journal of MultiDisciplinary Evaluation. 2017, Vol. 13 Issue 28, p41-52. 12p. |
| Subject Terms: | Stakeholders, Professional standards, Disciplinary infractions |
| Abstract: | Background: Program evaluators have reached broad agreement on principles for planning and conducting evaluations and on standards for judging their quality. However, many evaluation stakeholders, including key intended users, may judge evaluations on criteria that differ sharply from the professional standards and the criteria we commonly employ in meta-evaluations. Purpose: This paper highlights the role of academic discipline and epistemic community in judging what is an "exemplary" evaluation, by examining the case of an evaluation that was considered exemplary by professional program evaluators, but methodologically flawed by professionals from different disciplinary traditions. Setting: The evaluation in question was carried out within the Consultative Group on International Agricultural Research (CGIAR), a research network with a rich tradition of economic impact assessment. Intervention: NA Research Design: This is a case study that combines participatory action research and historical analysis. Data Collection and Analysis: The study is based on the author's personal involvement in the evaluation and on a review of publications and unpublished documents related to the case. Findings: A team of experienced evaluators applied what are generally considered to be good practices in the program evaluation community. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. In contrast, within the CGIAR, both the evaluation and the meta-evaluation study were considered to be methodologically flawed and biased. Three lessons related to exemplary evaluation are formulated and elaborated upon: Lesson 1. Being exemplary is in the eyes of the beholder. Lesson 2. Epistemic communities are hard nuts to crack. Lesson 3. You can't win them all. While the early efforts with program evaluation analyzed here were experienced as failures, a number of subsequent developments have led to greater understanding of diverse evaluation approaches, and some movement toward agreement on what constitutes exemplary evaluation in the CGIAR. Nevertheless, there is still a considerable way to go. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of MultiDisciplinary Evaluation is the property of Western Michigan University, The Evaluation Center and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Links: – Type: pdflink Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 123506354 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: You Call This Exemplary? Lessons from an Unsung International Evaluation. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Horton%2C+Douglas%22">Horton, Douglas</searchLink><relatesTo>1</relatesTo> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Journal+of+MultiDisciplinary+Evaluation%22">Journal of MultiDisciplinary Evaluation</searchLink>. 2017, Vol. 13 Issue 28, p41-52. 12p. – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Stakeholders%22">Stakeholders</searchLink><br /><searchLink fieldCode="DE" term="%22Professional+standards%22">Professional standards</searchLink><br /><searchLink fieldCode="DE" term="%22Disciplinary+infractions%22">Disciplinary infractions</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Background: Program evaluators have reached broad agreement on principles for planning and conducting evaluations and on standards for judging their quality. However, many evaluation stakeholders, including key intended users, may judge evaluations on criteria that differ sharply from the professional standards and the criteria we commonly employ in meta-evaluations. Purpose: This paper highlights the role of academic discipline and epistemic community in judging what is an "exemplary" evaluation, by examining the case of an evaluation that was considered exemplary by professional program evaluators, but methodologically flawed by professionals from different disciplinary traditions. Setting: The evaluation in question was carried out within the Consultative Group on International Agricultural Research (CGIAR), a research network with a rich tradition of economic impact assessment. Intervention: NA Research Design: This is a case study that combines participatory action research and historical analysis. Data Collection and Analysis: The study is based on the author's personal involvement in the evaluation and on a review of publications and unpublished documents related to the case. Findings: A team of experienced evaluators applied what are generally considered to be good practices in the program evaluation community. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. In contrast, within the CGIAR, both the evaluation and the meta-evaluation study were considered to be methodologically flawed and biased. Three lessons related to exemplary evaluation are formulated and elaborated upon: Lesson 1. Being exemplary is in the eyes of the beholder. Lesson 2. Epistemic communities are hard nuts to crack. Lesson 3. You can't win them all. While the early efforts with program evaluation analyzed here were experienced as failures, a number of subsequent developments have led to greater understanding of diverse evaluation approaches, and some movement toward agreement on what constitutes exemplary evaluation in the CGIAR. Nevertheless, there is still a considerable way to go. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Journal of MultiDisciplinary Evaluation is the property of Western Michigan University, The Evaluation Center and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=123506354 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.56645/jmde.v13i28.469 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 12 StartPage: 41 Subjects: – SubjectFull: Stakeholders Type: general – SubjectFull: Professional standards Type: general – SubjectFull: Disciplinary infractions Type: general Titles: – TitleFull: You Call This Exemplary? Lessons from an Unsung International Evaluation. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Horton, Douglas IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Text: 2017 Type: published Y: 2017 Identifiers: – Type: issn-print Value: 15568180 Numbering: – Type: volume Value: 13 – Type: issue Value: 28 Titles: – TitleFull: Journal of MultiDisciplinary Evaluation Type: main |
| ResultId | 1 |
