View in EDS HTML Full Text PDF Full Text

Should We account for classrooms? Analyzing online experimental data with student-level randomization.

Saved in:
Bibliographic Details
Title: Should We account for classrooms? Analyzing online experimental data with student-level randomization.
Authors: Closser, Avery H. (AUTHOR), Sales, Adam (AUTHOR), Botelho, Anthony F. (AUTHOR)
Source: Educational Technology Research & Development. Oct2024, Vol. 72 Issue 5, p2865-2894. 30p.
Subjects: Data collection platforms, Learning, Causal inference, Education research, Consolidated financial statements, Educational technology, Multilevel models
Abstract: Emergent technologies present platforms for educational researchers to conduct randomized controlled trials (RCTs) and collect rich data to study students' performance, behavior, learning processes, and outcomes in authentic learning environments. As educational research increasingly uses methods and data collection from such platforms, it is necessary to consider the most appropriate ways to analyze this data to draw causal inferences from RCTs. Here, we examine whether and how analysis results are impacted by accounting for multilevel variance in samples from RCTs with student-level randomization within one platform. We propose and demonstrate a method that leverages auxiliary non-experimental "remnant" data collected within a learning platform to inform analysis decisions. Specifically, we compare five commonly-applied analysis methods to estimate treatment effects while accounting for, or ignoring, class-level factors and observed measures of confidence and accuracy to identify best practices under real-world conditions. We find that methods that account for groups as either fixed effects or random effects consistently outperform those that ignore group-level factors, even though randomization was applied at the student level. However, we found no meaningful differences between the use of fixed or random effects as a means to account for groups. We conclude that analyses of online experiments should account for the naturally-nested structure of students within classes, despite the notion that student-level randomization may alleviate group-level differences. Further, we demonstrate how to use remnant data to identify appropriate methods for analyzing experiments. These findings provide practical guidelines for researchers conducting RCTs in similar educational technologies to make more informed decisions when approaching analyses. [ABSTRACT FROM AUTHOR]
Copyright of Educational Technology Research & Development is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Psychology and Behavioral Sciences Collection
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: pbh
DbLabel: Psychology and Behavioral Sciences Collection
An: 180830889
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Should We account for classrooms? Analyzing online experimental data with student-level randomization.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Closser%2C+Avery+H%2E%22">Closser, Avery H.</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Sales%2C+Adam%22">Sales, Adam</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Botelho%2C+Anthony+F%2E%22">Botelho, Anthony F.</searchLink> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Educational+Technology+Research+%26+Development%22">Educational Technology Research & Development</searchLink>. Oct2024, Vol. 72 Issue 5, p2865-2894. 30p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Data+collection+platforms%22">Data collection platforms</searchLink><br /><searchLink fieldCode="DE" term="%22Learning%22">Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Causal+inference%22">Causal inference</searchLink><br /><searchLink fieldCode="DE" term="%22Education+research%22">Education research</searchLink><br /><searchLink fieldCode="DE" term="%22Consolidated+financial+statements%22">Consolidated financial statements</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+technology%22">Educational technology</searchLink><br /><searchLink fieldCode="DE" term="%22Multilevel+models%22">Multilevel models</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Emergent technologies present platforms for educational researchers to conduct randomized controlled trials (RCTs) and collect rich data to study students' performance, behavior, learning processes, and outcomes in authentic learning environments. As educational research increasingly uses methods and data collection from such platforms, it is necessary to consider the most appropriate ways to analyze this data to draw causal inferences from RCTs. Here, we examine whether and how analysis results are impacted by accounting for multilevel variance in samples from RCTs with student-level randomization within one platform. We propose and demonstrate a method that leverages auxiliary non-experimental "remnant" data collected within a learning platform to inform analysis decisions. Specifically, we compare five commonly-applied analysis methods to estimate treatment effects while accounting for, or ignoring, class-level factors and observed measures of confidence and accuracy to identify best practices under real-world conditions. We find that methods that account for groups as either fixed effects or random effects consistently outperform those that ignore group-level factors, even though randomization was applied at the student level. However, we found no meaningful differences between the use of fixed or random effects as a means to account for groups. We conclude that analyses of online experiments should account for the naturally-nested structure of students within classes, despite the notion that student-level randomization may alleviate group-level differences. Further, we demonstrate how to use remnant data to identify appropriate methods for analyzing experiments. These findings provide practical guidelines for researchers conducting RCTs in similar educational technologies to make more informed decisions when approaching analyses. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Educational Technology Research & Development is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=pbh&AN=180830889
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s11423-023-10325-x
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 30
        StartPage: 2865
    Subjects:
      – SubjectFull: Data collection platforms
        Type: general
      – SubjectFull: Learning
        Type: general
      – SubjectFull: Causal inference
        Type: general
      – SubjectFull: Education research
        Type: general
      – SubjectFull: Consolidated financial statements
        Type: general
      – SubjectFull: Educational technology
        Type: general
      – SubjectFull: Multilevel models
        Type: general
    Titles:
      – TitleFull: Should We account for classrooms? Analyzing online experimental data with student-level randomization.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Closser, Avery H.
      – PersonEntity:
          Name:
            NameFull: Sales, Adam
      – PersonEntity:
          Name:
            NameFull: Botelho, Anthony F.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: Oct2024
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 10421629
          Numbering:
            – Type: volume
              Value: 72
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Educational Technology Research & Development
              Type: main
ResultId 1