View in EDS HTML Full Text PDF Full Text

Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study.

Saved in:
Bibliographic Details
Title: Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study.
Authors: Sales, Adam C.1 (AUTHOR), Pane, John F.2 (AUTHOR)
Source: Journal of Research on Educational Effectiveness. Jan-Mar2021, Vol. 14 Issue 1, p241-269. 29p.
Subject Terms: *Educational technology, *Educational evaluation, *Mediation, *Case studies, *Students, Data logging
Abstract: Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations. [ABSTRACT FROM AUTHOR]
Copyright of Journal of Research on Educational Effectiveness is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: ehh
DbLabel: Education Research Complete
An: 149920084
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sales%2C+Adam+C%2E%22">Sales, Adam C.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Pane%2C+John+F%2E%22">Pane, John F.</searchLink><relatesTo>2</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Journal+of+Research+on+Educational+Effectiveness%22">Journal of Research on Educational Effectiveness</searchLink>. Jan-Mar2021, Vol. 14 Issue 1, p241-269. 29p.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *<searchLink fieldCode="DE" term="%22Educational+technology%22">Educational technology</searchLink><br />*<searchLink fieldCode="DE" term="%22Educational+evaluation%22">Educational evaluation</searchLink><br />*<searchLink fieldCode="DE" term="%22Mediation%22">Mediation</searchLink><br />*<searchLink fieldCode="DE" term="%22Case+studies%22">Case studies</searchLink><br />*<searchLink fieldCode="DE" term="%22Students%22">Students</searchLink><br /><searchLink fieldCode="DE" term="%22Data+logging%22">Data logging</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Journal of Research on Educational Effectiveness is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=149920084
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2020.1823538
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 29
        StartPage: 241
    Subjects:
      – SubjectFull: Educational technology
        Type: general
      – SubjectFull: Educational evaluation
        Type: general
      – SubjectFull: Mediation
        Type: general
      – SubjectFull: Case studies
        Type: general
      – SubjectFull: Students
        Type: general
      – SubjectFull: Data logging
        Type: general
    Titles:
      – TitleFull: Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sales, Adam C.
      – PersonEntity:
          Name:
            NameFull: Pane, John F.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Text: Jan-Mar2021
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-print
              Value: 19345747
          Numbering:
            – Type: volume
              Value: 14
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Research on Educational Effectiveness
              Type: main
ResultId 1