Effective Evaluation of Online Learning Interventions with Surrogate Measures
Saved in:
| Title: | Effective Evaluation of Online Learning Interventions with Surrogate Measures |
|---|---|
| Language: | English |
| Authors: | Prihar, Ethan, Vanacore, Kirk, Sales, Adam, Heffernan, Neil |
| Source: | International Educational Data Mining Society. 2023. |
| Availability: | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/ |
| Peer Reviewed: | Y |
| Page Count: | 7 |
| Publication Date: | 2023 |
| Sponsoring Agency: | National Science Foundation (NSF) Institute of Education Sciences (ED) Office of Elementary and Secondary Education (OESE) (ED), Education Innovation and Research (EIR) Office of Naval Research (ONR) (DOD) Federal Highway Administration (FHWA), National Highway Institute (NHI) |
| Contract Number: | 2118725 2118904 1950683 1917808 1931523 1940236 1917713 1903304 1822830 1759229 1724889 1636782 1535428 R305N210049 R305D210031 R305A170137 R305A170243 R305A180401 R305A120125 U411B190024 S411B210024 N000141812768 R44GM146483 |
| Document Type: | Speeches/Meeting Papers Reports - Research |
| Descriptors: | Electronic Learning, Intervention, Instructional Effectiveness, Data Collection, Models, Measurement |
| Abstract: | There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized experiments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, distal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional methods. To help discern the effectiveness of instructional interventions, this work uses data from 26,060 clickstream sequences of students across 31 different online educational experiments exploring 51 different research questions and the students' posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next-problem correctness was determined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next-problem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problem-level experiments. [For the complete proceedings, see ED630829. Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN).] |
| Abstractor: | As Provided |
| Notes: | https://osf.io/uj48v |
| IES Funded: | Yes |
| Entry Date: | 2023 |
| Accession Number: | ED630844 |
| Database: | ERIC |
| Abstract: | There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized experiments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, distal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional methods. To help discern the effectiveness of instructional interventions, this work uses data from 26,060 clickstream sequences of students across 31 different online educational experiments exploring 51 different research questions and the students' posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next-problem correctness was determined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next-problem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problem-level experiments. [For the complete proceedings, see ED630829. Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN).] |
|---|
