Problem-Solving Behavior and EdTech Effectiveness: A Model for Exploratory Causal Analysis
Saved in:
| Title: | Problem-Solving Behavior and EdTech Effectiveness: A Model for Exploratory Causal Analysis |
|---|---|
| Language: | English |
| Authors: | Adam C. Sales, Kirk P. Vanacore, Hyeon-Ah Kang, Tiffany A. Whittaker |
| Source: | International Educational Data Mining Society. 2024. |
| Availability: | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/ |
| Peer Reviewed: | Y |
| Page Count: | 11 |
| Publication Date: | 2024 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R305D210036 |
| Document Type: | Speeches/Meeting Papers Reports - Research |
| Education Level: | High Schools Secondary Education Junior High Schools Middle Schools |
| Descriptors: | Educational Technology, Problem Solving, Electronic Learning, Instructional Effectiveness, Learner Engagement, Technology Uses in Education, Algebra, Mathematics Instruction, High School Students, Middle School Students |
| Abstract: | The gold-standard evaluation of an educational technology product is a randomized study comparing students randomized to use a computer-based learning platform (CBLP) to students assigned to a "business as usual" condition, such as pencil-and-paper work, and estimating average treatment effects. However, not everyone uses the same CBLP in the same way--indeed, an individual may engage with CBLP in multiple different ways over the course of a study--and the platform's effectiveness may depend on how students are using it. This paper introduces a model that serves two aims: classifying different modes of problem-solving or engagement among CBLP users, and estimating varying program effectiveness with varying usage patterns. The model uses mixed-type problem-level variables, such as time spent, the number of errors committed, and the number of hints requested, to cluster each problem attempt by each student into one of a number of categories, using a model-based, probabilistic, latent profile model. Students differ from each other based on their probabilities of working on problems in each of the identified modes. Finally, the model uses a fully latent principal stratification approach to estimate varying treatment effects as a function of those probabilities. In this paper, we describe the model and estimation in detail and illustrate application using data from two large randomized field trials, one evaluating Cognitive Tutor Algebra I, and the other evaluating ASSISTments. [For the complete proceedings, see ED675485.] |
| Abstractor: | As Provided |
| Notes: | https://osf.io/r3nf2 |
| IES Funded: | Yes |
| Entry Date: | 2025 |
| Accession Number: | ED675575 |
| Database: | ERIC |
| Abstract: | The gold-standard evaluation of an educational technology product is a randomized study comparing students randomized to use a computer-based learning platform (CBLP) to students assigned to a "business as usual" condition, such as pencil-and-paper work, and estimating average treatment effects. However, not everyone uses the same CBLP in the same way--indeed, an individual may engage with CBLP in multiple different ways over the course of a study--and the platform's effectiveness may depend on how students are using it. This paper introduces a model that serves two aims: classifying different modes of problem-solving or engagement among CBLP users, and estimating varying program effectiveness with varying usage patterns. The model uses mixed-type problem-level variables, such as time spent, the number of errors committed, and the number of hints requested, to cluster each problem attempt by each student into one of a number of categories, using a model-based, probabilistic, latent profile model. Students differ from each other based on their probabilities of working on problems in each of the identified modes. Finally, the model uses a fully latent principal stratification approach to estimate varying treatment effects as a function of those probabilities. In this paper, we describe the model and estimation in detail and illustrate application using data from two large randomized field trials, one evaluating Cognitive Tutor Algebra I, and the other evaluating ASSISTments. [For the complete proceedings, see ED675485.] |
|---|
