The Use of Program Theory in Mathematics Education Evaluation Research
Saved in:
| Title: | The Use of Program Theory in Mathematics Education Evaluation Research |
|---|---|
| Language: | English |
| Authors: | Munter, Charles, Cobb, Paul, Shekell, Calli, Society for Research on Educational Effectiveness (SREE) |
| Source: | Society for Research on Educational Effectiveness. 2014. |
| Availability: | Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org |
| Peer Reviewed: | Y |
| Page Count: | 8 |
| Publication Date: | 2014 |
| Document Type: | Reports - Research |
| Education Level: | Elementary Education Secondary Education |
| Descriptors: | Educational Research, Program Evaluation, Theories, Mathematics Education, Clearinghouses, Intervention, Educational Policy, Randomized Controlled Trials, Research Design, Fidelity, Elementary School Mathematics, Secondary School Mathematics |
| Abstract: | One purpose of education research is to develop and rigorously evaluate the effectiveness of programs for supporting students' learning and achievement. The Institute of Education Sciences has amplified that purpose (Shadish & Cook, 2009) and attempted to improve the methodological standards for conducting such work--primarily through the What Works Clearinghouse (WWC), which, since 2002, has supported an ongoing effort to synthesize research on the effectiveness of educational interventions, programs, and policies. According to its stringent, methodological standards, only "well-designed and well-implemented" randomized controlled trials and studies employing quasi-experimental designs with equating or matching are included in the WWC's 15 topical syntheses, one of which is mathematics. Despite repeated calls for increased attention to theory in program evaluation for more than 25 years, recent reviews of the literature suggest that the field has seen little change (Confrey & Stohl, 2004; Coryn, Noakes, Westine, & Schröter, 2010; Weiss, 1997). The requirement that evaluators link research designs to the theories underlying the programs they evaluate is not included in the WWC's standards for rigorous evaluations. Given the considerable differences in mathematical goals and theories of learning from which mathematics programs are designed, and the high stakes for students' mathematics learning and academic futures (as well as for the fortunes of program developers), it is important to ask whether the WWC's methodological specifications are sufficient, or whether program evaluation has once again lost (or perhaps never gained) sight of the role of theory in evaluation design and implementation (Bickman, 1987), resulting in overly-constrained and uninterpretable syntheses of otherwise methodologically strong evaluation research (Schoenfeld, 2006). The purpose of this paper is to determine whether calls for theory-based evaluation research have had an impact on the extent to which evaluators of mathematics programs attend to program theory in their design, implementation, and reporting of studies. For each report that met stringent methodological standards (i.e., WWC evidence standards), the authors asked: (1) What type of program theory was articulated (none, sub-theoretical, or theoretical; Lipsey et al., 1985); (2) What was the quality of evaluators' articulation of program theory (entirely implicit, drawn from a limited number of resources such as developer or publisher descriptions, or drawn from multiple resources and situated in the research literature); (3) How was the articulated program theory used in the evaluation; and (4) To what extent does attention to program theory vary by characteristics of the program and evaluator(s), such as the nature of the program's mathematics learning goals and instruction (i.e., back-to-basics, typical, inquiry-based, or blended); type of publication (e.g., peer-reviewed journal or technical report); funding source (e.g., Federal/state agency or publisher/developer); and timing of analysis (i.e., primary or secondary analysis)? Modeled after the WWC's process, every report was examined and coded independently by two individuals employing the same coding scheme, and research questions listed here. The findings of evaluation studies guide the decisions of policy makers at every level, including the adoption of both curriculum materials and intervention programs. These decisions are consequential for students' mathematics learning and academic futures. It is therefore crucial that evaluators "get it right" when assessing the effectiveness of such programs. This analysis indicates that WWC's methodological specifications are inadequate because they overlook understanding and using theory in evaluation design and implementation. In general, evaluation research of mathematics programs needs to improve in its attention to and use of program theory. Tables and figures are appended. |
| Abstractor: | ERIC |
| Number of References: | 13 |
| Entry Date: | 2016 |
| Accession Number: | ED563007 |
| Database: | ERIC |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED563007 Name: ERIC Full Text Category: fullText Text: Full Text from ERIC |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: ED563007 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: The Use of Program Theory in Mathematics Education Evaluation Research – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Munter%2C+Charles%22">Munter, Charles</searchLink><br /><searchLink fieldCode="AR" term="%22Cobb%2C+Paul%22">Cobb, Paul</searchLink><br /><searchLink fieldCode="AR" term="%22Shekell%2C+Calli%22">Shekell, Calli</searchLink><br /><searchLink fieldCode="AR" term="%22Society+for+Research+on+Educational+Effectiveness+%28SREE%29%22">Society for Research on Educational Effectiveness (SREE)</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Society+for+Research+on+Educational+Effectiveness%22"><i>Society for Research on Educational Effectiveness</i></searchLink>. 2014. – Name: Avail Label: Availability Group: Avail Data: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 8 – Name: DatePubCY Label: Publication Date Group: Date Data: 2014 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Reports - Research – Name: Audience Label: Education Level Group: Audnce Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink> – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Theories%22">Theories</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Education%22">Mathematics Education</searchLink><br /><searchLink fieldCode="DE" term="%22Clearinghouses%22">Clearinghouses</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Policy%22">Educational Policy</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Design%22">Research Design</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Mathematics%22">Elementary School Mathematics</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Mathematics%22">Secondary School Mathematics</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: One purpose of education research is to develop and rigorously evaluate the effectiveness of programs for supporting students' learning and achievement. The Institute of Education Sciences has amplified that purpose (Shadish & Cook, 2009) and attempted to improve the methodological standards for conducting such work--primarily through the What Works Clearinghouse (WWC), which, since 2002, has supported an ongoing effort to synthesize research on the effectiveness of educational interventions, programs, and policies. According to its stringent, methodological standards, only "well-designed and well-implemented" randomized controlled trials and studies employing quasi-experimental designs with equating or matching are included in the WWC's 15 topical syntheses, one of which is mathematics. Despite repeated calls for increased attention to theory in program evaluation for more than 25 years, recent reviews of the literature suggest that the field has seen little change (Confrey & Stohl, 2004; Coryn, Noakes, Westine, & Schröter, 2010; Weiss, 1997). The requirement that evaluators link research designs to the theories underlying the programs they evaluate is not included in the WWC's standards for rigorous evaluations. Given the considerable differences in mathematical goals and theories of learning from which mathematics programs are designed, and the high stakes for students' mathematics learning and academic futures (as well as for the fortunes of program developers), it is important to ask whether the WWC's methodological specifications are sufficient, or whether program evaluation has once again lost (or perhaps never gained) sight of the role of theory in evaluation design and implementation (Bickman, 1987), resulting in overly-constrained and uninterpretable syntheses of otherwise methodologically strong evaluation research (Schoenfeld, 2006). The purpose of this paper is to determine whether calls for theory-based evaluation research have had an impact on the extent to which evaluators of mathematics programs attend to program theory in their design, implementation, and reporting of studies. For each report that met stringent methodological standards (i.e., WWC evidence standards), the authors asked: (1) What type of program theory was articulated (none, sub-theoretical, or theoretical; Lipsey et al., 1985); (2) What was the quality of evaluators' articulation of program theory (entirely implicit, drawn from a limited number of resources such as developer or publisher descriptions, or drawn from multiple resources and situated in the research literature); (3) How was the articulated program theory used in the evaluation; and (4) To what extent does attention to program theory vary by characteristics of the program and evaluator(s), such as the nature of the program's mathematics learning goals and instruction (i.e., back-to-basics, typical, inquiry-based, or blended); type of publication (e.g., peer-reviewed journal or technical report); funding source (e.g., Federal/state agency or publisher/developer); and timing of analysis (i.e., primary or secondary analysis)? Modeled after the WWC's process, every report was examined and coded independently by two individuals employing the same coding scheme, and research questions listed here. The findings of evaluation studies guide the decisions of policy makers at every level, including the adoption of both curriculum materials and intervention programs. These decisions are consequential for students' mathematics learning and academic futures. It is therefore crucial that evaluators "get it right" when assessing the effectiveness of such programs. This analysis indicates that WWC's methodological specifications are inadequate because they overlook understanding and using theory in evaluation design and implementation. In general, evaluation research of mathematics programs needs to improve in its attention to and use of program theory. Tables and figures are appended. – Name: AbstractInfo Label: Abstractor Group: Ab Data: ERIC – Name: Ref Label: Number of References Group: RefInfo Data: 13 – Name: DateEntry Label: Entry Date Group: Date Data: 2016 – Name: AN Label: Accession Number Group: ID Data: ED563007 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED563007 |
| RecordInfo | BibRecord: BibEntity: Languages: – Text: English PhysicalDescription: Pagination: PageCount: 8 Subjects: – SubjectFull: Educational Research Type: general – SubjectFull: Program Evaluation Type: general – SubjectFull: Theories Type: general – SubjectFull: Mathematics Education Type: general – SubjectFull: Clearinghouses Type: general – SubjectFull: Intervention Type: general – SubjectFull: Educational Policy Type: general – SubjectFull: Randomized Controlled Trials Type: general – SubjectFull: Research Design Type: general – SubjectFull: Fidelity Type: general – SubjectFull: Elementary School Mathematics Type: general – SubjectFull: Secondary School Mathematics Type: general Titles: – TitleFull: The Use of Program Theory in Mathematics Education Evaluation Research Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Society for Research on Educational Effectiveness (SREE) – PersonEntity: Name: NameFull: Munter, Charles – PersonEntity: Name: NameFull: Cobb, Paul – PersonEntity: Name: NameFull: Shekell, Calli IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2014 Titles: – TitleFull: Society for Research on Educational Effectiveness Type: main |
| ResultId | 1 |
