View in EDS

Experimental Education Research: Clarifying Why, How and When to Use Random Assignment. EdWorkingPaper No. 23-821

Saved in:
Bibliographic Details
Title: Experimental Education Research: Clarifying Why, How and When to Use Random Assignment. EdWorkingPaper No. 23-821
Language: English
Authors: Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues, Ben Styles, Ben Weidmann, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2023.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 27
Publication Date: 2023
Sponsoring Agency: National Foundation for Educational Research (NFER) (United Kingdom)
University College London (UCL) (United Kingdom), Centre for Education Policy & Equalising Opportunities (CEPEO)
Economic and Social Research Council (United Kingdom)
UK Research and Innovation (UKRI) (United Kingdom), Economic and Social Research Council (ESRC)
Document Type: Reports - Descriptive
Descriptors: Educational Research, Research Methodology, Randomized Controlled Trials, Program Effectiveness, Effect Size, Quasiexperimental Design, Accuracy, Educational Theories, Models, Decision Making, Schemata (Cognition), Educational Experiments
Abstract: Over the last twenty years, education researchers have increasingly conducted randomised experiments with the goal of informing the decisions of educators and policymakers. Such experiments have generally employed broad, consequential, standardised outcome measures in the hope that this would allow decisionmakers to compare effectiveness of different approaches. However, a combination of small effect sizes, wide confidence intervals, and treatment effect heterogeneity means that researchers have largely failed to achieve this goal. We argue that quasiexperimental methods and multi-site trials will often be superior for informing educators' decisions on the grounds that they can achieve greater precision and better address heterogeneity. Experimental research remains valuable in applied education research. However, it should primarily be used to test theoretical models, which can in turn inform educators' mental models, rather than attempting to directly inform decision making. Since comparable effect size estimates are not of interest when testing educational theory, researchers can and should improve the power of theory-informing experiments by using more closely aligned (i.e., valid) outcome measures. We argue that this approach would reduce wasteful research spending and make the research that does go ahead more statistically informative, thus improving the return on investment in educational research. [This work was partially supported by Research England.]
Abstractor: As Provided
Entry Date: 2025
Accession Number: ED672356
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED672356
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED672356
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Experimental Education Research: Clarifying Why, How and When to Use Random Assignment. EdWorkingPaper No. 23-821
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sam+Sims%22">Sam Sims</searchLink><br /><searchLink fieldCode="AR" term="%22Jake+Anders%22">Jake Anders</searchLink><br /><searchLink fieldCode="AR" term="%22Matthew+Inglis%22">Matthew Inglis</searchLink><br /><searchLink fieldCode="AR" term="%22Hugues+Lortie-Forgues%22">Hugues Lortie-Forgues</searchLink><br /><searchLink fieldCode="AR" term="%22Ben+Styles%22">Ben Styles</searchLink><br /><searchLink fieldCode="AR" term="%22Ben+Weidmann%22">Ben Weidmann</searchLink><br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2023.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 27
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Foundation for Educational Research (NFER) (United Kingdom)<br />University College London (UCL) (United Kingdom), Centre for Education Policy & Equalising Opportunities (CEPEO)<br />Economic and Social Research Council (United Kingdom)<br />UK Research and Innovation (UKRI) (United Kingdom), Economic and Social Research Council (ESRC)
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Effect+Size%22">Effect Size</searchLink><br /><searchLink fieldCode="DE" term="%22Quasiexperimental+Design%22">Quasiexperimental Design</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Theories%22">Educational Theories</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+Making%22">Decision Making</searchLink><br /><searchLink fieldCode="DE" term="%22Schemata+%28Cognition%29%22">Schemata (Cognition)</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Experiments%22">Educational Experiments</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Over the last twenty years, education researchers have increasingly conducted randomised experiments with the goal of informing the decisions of educators and policymakers. Such experiments have generally employed broad, consequential, standardised outcome measures in the hope that this would allow decisionmakers to compare effectiveness of different approaches. However, a combination of small effect sizes, wide confidence intervals, and treatment effect heterogeneity means that researchers have largely failed to achieve this goal. We argue that quasiexperimental methods and multi-site trials will often be superior for informing educators' decisions on the grounds that they can achieve greater precision and better address heterogeneity. Experimental research remains valuable in applied education research. However, it should primarily be used to test theoretical models, which can in turn inform educators' mental models, rather than attempting to directly inform decision making. Since comparable effect size estimates are not of interest when testing educational theory, researchers can and should improve the power of theory-informing experiments by using more closely aligned (i.e., valid) outcome measures. We argue that this approach would reduce wasteful research spending and make the research that does go ahead more statistically informative, thus improving the return on investment in educational research. [This work was partially supported by Research England.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED672356
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED672356
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 27
    Subjects:
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Effect Size
        Type: general
      – SubjectFull: Quasiexperimental Design
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Educational Theories
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Decision Making
        Type: general
      – SubjectFull: Schemata (Cognition)
        Type: general
      – SubjectFull: Educational Experiments
        Type: general
    Titles:
      – TitleFull: Experimental Education Research: Clarifying Why, How and When to Use Random Assignment. EdWorkingPaper No. 23-821
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Sam Sims
      – PersonEntity:
          Name:
            NameFull: Jake Anders
      – PersonEntity:
          Name:
            NameFull: Matthew Inglis
      – PersonEntity:
          Name:
            NameFull: Hugues Lortie-Forgues
      – PersonEntity:
          Name:
            NameFull: Ben Styles
      – PersonEntity:
          Name:
            NameFull: Ben Weidmann
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2023
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1