Using Big Data to Sharpen Design-Based Inference in A/B Tests
Saved in:
| Title: | Using Big Data to Sharpen Design-Based Inference in A/B Tests |
|---|---|
| Language: | English |
| Authors: | Sales, Adam C., Botelho, Anthony, Patikorn, Thanaporn, Heffernan, Neil T. |
| Source: | International Educational Data Mining Society. 2018. |
| Availability: | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: http://www.educationaldatamining.org |
| Peer Reviewed: | Y |
| Page Count: | 7 |
| Publication Date: | 2018 |
| Sponsoring Agency: | National Science Foundation (NSF) Institute of Education Sciences (ED) |
| Contract Number: | IIS1636782 ACI1440753 DRL1252297 DRL1109483 DRL1316736 DGE1535428 DRL1031398 R305A120125 R305C100024 |
| Document Type: | Reports - Research Speeches/Meeting Papers |
| Descriptors: | Courseware, Data Analysis, Causal Models, Prediction, Outcomes of Education, Evaluation Methods, Mastery Learning, Skill Development, Intelligent Tutoring Systems, Statistical Bias, Randomized Controlled Trials, Artificial Intelligence |
| Abstract: | Randomized A/B tests in educational software are not run in a vacuum: often, reams of historical data are available alongside the data from a randomized trial. This paper proposes a method to use this historical data--often highdimensional and longitudinal--to improve causal estimates from A/B tests. The method proceeds in two steps: first, fit a machine learning model to the historical data predicting students' outcomes as a function of their covariates. Then, use that model to predict the outcomes of the randomized students in the A/B test. Finally, use design-based methods to estimate the treatment effect in the A/B test, using prediction errors in place of outcomes. This method retains all of the advantages of design-based inference, while, under certain conditions, yielding more precise estimators. This paper will give a theoretical condition under which the method improves statistical precision, and demonstrates it using a deep learning algorithm to help estimate effects in a set of experiments run inside ASSISTments. [For the full proceedings, see ED593090.] |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2019 |
| Accession Number: | ED593197 |
| Database: | ERIC |
| Abstract: | Randomized A/B tests in educational software are not run in a vacuum: often, reams of historical data are available alongside the data from a randomized trial. This paper proposes a method to use this historical data--often highdimensional and longitudinal--to improve causal estimates from A/B tests. The method proceeds in two steps: first, fit a machine learning model to the historical data predicting students' outcomes as a function of their covariates. Then, use that model to predict the outcomes of the randomized students in the A/B test. Finally, use design-based methods to estimate the treatment effect in the A/B test, using prediction errors in place of outcomes. This method retains all of the advantages of design-based inference, while, under certain conditions, yielding more precise estimators. This paper will give a theoretical condition under which the method improves statistical precision, and demonstrates it using a deep learning algorithm to help estimate effects in a set of experiments run inside ASSISTments. [For the full proceedings, see ED593090.] |
|---|
