View in EDS

More Powerful A/B Testing Using Auxiliary Data and Deep Learning

Saved in:
Bibliographic Details
Title: More Powerful A/B Testing Using Auxiliary Data and Deep Learning
Language: English
Authors: Sales, Adam C., Prihar, Ethan, Gagnon-Bartsch, Johann, Gurung, Ashish, Heffernan, Neil T.
Source: Grantee Submission. 2022.
Peer Reviewed: Y
Page Count: 4
Publication Date: 2022
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305D210031
Document Type: Speeches/Meeting Papers
Reports - Research
Descriptors: Research Methodology, Educational Experiments, Causal Models, Computation, Electronic Learning, Statistical Inference, Data Analysis, Accuracy
DOI: 10.1007/978-3-031-11647-6_107
Abstract: Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and unbiased without additional assumptions, regardless of the deep-learning model's quality. In this dataset, incorporating auxiliary data improves precision consistently and, in some cases, substantially.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2023
Accession Number: ED627066
Database: ERIC
Description
Abstract:Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and unbiased without additional assumptions, regardless of the deep-learning model's quality. In this dataset, incorporating auxiliary data improves precision consistently and, in some cases, substantially.
DOI:10.1007/978-3-031-11647-6_107