Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
Saved in:
| Title: | Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study |
|---|---|
| Language: | English |
| Authors: | Sales, Adam C. (ORCID |
| Source: | Journal of Research on Educational Effectiveness. 2021 14(1):241-269. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 29 |
| Publication Date: | 2021 |
| Sponsoring Agency: | National Science Foundation (NSF) |
| Contract Number: | 1420374 |
| Document Type: | Journal Articles Reports - Descriptive |
| Descriptors: | Educational Technology, Use Studies, Randomized Controlled Trials, Mathematics Curriculum, Curriculum Evaluation, Algebra, Intelligent Tutoring Systems, Cues, Problem Solving, Statistical Analysis, Causal Models |
| DOI: | 10.1080/19345747.2020.1823538 |
| ISSN: | 1934-5747 |
| Abstract: | Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations. |
| Abstractor: | As Provided |
| Entry Date: | 2021 |
| Accession Number: | EJ1293487 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations. |
|---|---|
| ISSN: | 1934-5747 |
| DOI: | 10.1080/19345747.2020.1823538 |
