View in EDS

How Methodological Features Affect Effect Sizes in Education

Saved in:
Bibliographic Details
Title: How Methodological Features Affect Effect Sizes in Education
Language: English
Authors: Cheung, Alan, Slavin, Robert, Society for Research on Educational Effectiveness (SREE)
Source: Society for Research on Educational Effectiveness. 2016.
Availability: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org
Peer Reviewed: Y
Page Count: 6
Publication Date: 2016
Intended Audience: Policymakers; Researchers
Document Type: Reports - Research
Education Level: Elementary Secondary Education
Early Childhood Education
Descriptors: Effect Size, Evidence Based Practice, Educational Change, Educational Policy, Research Design, Educational Experiments, Program Evaluation, Publications, Sample Size, Literature Reviews, Best Practices, Educational Research, Elementary Secondary Education, Mathematics Education, Science Education, Reading Instruction, Reading Programs, Early Childhood Education, Technology Uses in Education, Meta Analysis, Computer Software, Statistical Analysis, Q Methodology, Quasiexperimental Design, Randomized Controlled Trials
Abstract: As evidence-based reform becomes increasingly important in educational policy, it is becoming essential to understand how research design might contribute to reported effect sizes in experiments evaluating educational programs. The purpose of this study was to examine how methodological features such as types of publication, sample sizes, and research designs affect effect sizes in experiments. In order to investigate the relationships between study methodological features and effect sizes, the authors analyzed 645 studies that met the standards of inclusion for any of 12 reviews written for the Best Evidence Encyclopedia and (in most cases) published in review journals. The reviews cover programs in elementary and secondary math, elementary and secondary science, and elementary and secondary reading, as well as a review of elementary reading programs for struggling readers and a review of early childhood education. Studies included in reviews focusing on technology applications in reading and math were also included. Comprehensive Meta-Analysis software Version 2 (Borenstein, Hedges, Higgins, & Rothstein, 2005) was used to carry out all statistical analyses such as Q statistics and overall effect sizes. The findings suggest that effect sizes are roughly twice as large for published articles, small-scale trials, and experimenter-made measures, than for unpublished reports, large-scale studies, and independent measures, respectively. In addition, effect sizes are significantly higher in quasi-experiments than in randomized experiments. Based on the findings of the analyses, it is clear that researchers as well as policy makers need to take into account research design, sample size, measures, and type of publication before comparing effect sizes from program evaluations. Some specific recommendations are provided.
Abstractor: As Provided
Number of References: 12
Entry Date: 2016
Accession Number: ED567237
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED567237
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED567237
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: How Methodological Features Affect Effect Sizes in Education
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Cheung%2C+Alan%22">Cheung, Alan</searchLink><br /><searchLink fieldCode="AR" term="%22Slavin%2C+Robert%22">Slavin, Robert</searchLink><br /><searchLink fieldCode="AR" term="%22Society+for+Research+on+Educational+Effectiveness+%28SREE%29%22">Society for Research on Educational Effectiveness (SREE)</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Society+for+Research+on+Educational+Effectiveness%22"><i>Society for Research on Educational Effectiveness</i></searchLink>. 2016.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 6
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2016
– Name: Audience
  Label: Intended Audience
  Group: Audnce
  Data: Policymakers; Researchers
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Effect+Size%22">Effect Size</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence+Based+Practice%22">Evidence Based Practice</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Change%22">Educational Change</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Policy%22">Educational Policy</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Design%22">Research Design</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Experiments%22">Educational Experiments</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Publications%22">Publications</searchLink><br /><searchLink fieldCode="DE" term="%22Sample+Size%22">Sample Size</searchLink><br /><searchLink fieldCode="DE" term="%22Literature+Reviews%22">Literature Reviews</searchLink><br /><searchLink fieldCode="DE" term="%22Best+Practices%22">Best Practices</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Education%22">Mathematics Education</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Education%22">Science Education</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Instruction%22">Reading Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Programs%22">Reading Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Meta+Analysis%22">Meta Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Software%22">Computer Software</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Q+Methodology%22">Q Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Quasiexperimental+Design%22">Quasiexperimental Design</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: As evidence-based reform becomes increasingly important in educational policy, it is becoming essential to understand how research design might contribute to reported effect sizes in experiments evaluating educational programs. The purpose of this study was to examine how methodological features such as types of publication, sample sizes, and research designs affect effect sizes in experiments. In order to investigate the relationships between study methodological features and effect sizes, the authors analyzed 645 studies that met the standards of inclusion for any of 12 reviews written for the Best Evidence Encyclopedia and (in most cases) published in review journals. The reviews cover programs in elementary and secondary math, elementary and secondary science, and elementary and secondary reading, as well as a review of elementary reading programs for struggling readers and a review of early childhood education. Studies included in reviews focusing on technology applications in reading and math were also included. Comprehensive Meta-Analysis software Version 2 (Borenstein, Hedges, Higgins, & Rothstein, 2005) was used to carry out all statistical analyses such as Q statistics and overall effect sizes. The findings suggest that effect sizes are roughly twice as large for published articles, small-scale trials, and experimenter-made measures, than for unpublished reports, large-scale studies, and independent measures, respectively. In addition, effect sizes are significantly higher in quasi-experiments than in randomized experiments. Based on the findings of the analyses, it is clear that researchers as well as policy makers need to take into account research design, sample size, measures, and type of publication before comparing effect sizes from program evaluations. Some specific recommendations are provided.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 12
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2016
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED567237
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED567237
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 6
    Subjects:
      – SubjectFull: Effect Size
        Type: general
      – SubjectFull: Evidence Based Practice
        Type: general
      – SubjectFull: Educational Change
        Type: general
      – SubjectFull: Educational Policy
        Type: general
      – SubjectFull: Research Design
        Type: general
      – SubjectFull: Educational Experiments
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
      – SubjectFull: Publications
        Type: general
      – SubjectFull: Sample Size
        Type: general
      – SubjectFull: Literature Reviews
        Type: general
      – SubjectFull: Best Practices
        Type: general
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Mathematics Education
        Type: general
      – SubjectFull: Science Education
        Type: general
      – SubjectFull: Reading Instruction
        Type: general
      – SubjectFull: Reading Programs
        Type: general
      – SubjectFull: Early Childhood Education
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Meta Analysis
        Type: general
      – SubjectFull: Computer Software
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Q Methodology
        Type: general
      – SubjectFull: Quasiexperimental Design
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
    Titles:
      – TitleFull: How Methodological Features Affect Effect Sizes in Education
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Society for Research on Educational Effectiveness (SREE)
      – PersonEntity:
          Name:
            NameFull: Cheung, Alan
      – PersonEntity:
          Name:
            NameFull: Slavin, Robert
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2016
          Titles:
            – TitleFull: Society for Research on Educational Effectiveness
              Type: main
ResultId 1