View in EDS

Using Big Data to Sharpen Design-Based Inference in A/B Tests

Saved in:
Bibliographic Details
Title: Using Big Data to Sharpen Design-Based Inference in A/B Tests
Language: English
Authors: Sales, Adam C., Botelho, Anthony, Patikorn, Thanaporn, Heffernan, Neil T.
Source: International Educational Data Mining Society. 2018.
Availability: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: http://www.educationaldatamining.org
Peer Reviewed: Y
Page Count: 7
Publication Date: 2018
Sponsoring Agency: National Science Foundation (NSF)
Institute of Education Sciences (ED)
Contract Number: IIS1636782
ACI1440753
DRL1252297
DRL1109483
DRL1316736
DGE1535428
DRL1031398
R305A120125
R305C100024
Document Type: Reports - Research
Speeches/Meeting Papers
Descriptors: Courseware, Data Analysis, Causal Models, Prediction, Outcomes of Education, Evaluation Methods, Mastery Learning, Skill Development, Intelligent Tutoring Systems, Statistical Bias, Randomized Controlled Trials, Artificial Intelligence
Abstract: Randomized A/B tests in educational software are not run in a vacuum: often, reams of historical data are available alongside the data from a randomized trial. This paper proposes a method to use this historical data--often highdimensional and longitudinal--to improve causal estimates from A/B tests. The method proceeds in two steps: first, fit a machine learning model to the historical data predicting students' outcomes as a function of their covariates. Then, use that model to predict the outcomes of the randomized students in the A/B test. Finally, use design-based methods to estimate the treatment effect in the A/B test, using prediction errors in place of outcomes. This method retains all of the advantages of design-based inference, while, under certain conditions, yielding more precise estimators. This paper will give a theoretical condition under which the method improves statistical precision, and demonstrates it using a deep learning algorithm to help estimate effects in a set of experiments run inside ASSISTments. [For the full proceedings, see ED593090.]
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2019
Accession Number: ED593197
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED593197
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED593197
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Using Big Data to Sharpen Design-Based Inference in A/B Tests
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sales%2C+Adam+C%2E%22">Sales, Adam C.</searchLink><br /><searchLink fieldCode="AR" term="%22Botelho%2C+Anthony%22">Botelho, Anthony</searchLink><br /><searchLink fieldCode="AR" term="%22Patikorn%2C+Thanaporn%22">Patikorn, Thanaporn</searchLink><br /><searchLink fieldCode="AR" term="%22Heffernan%2C+Neil+T%2E%22">Heffernan, Neil T.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Educational+Data+Mining+Society%22"><i>International Educational Data Mining Society</i></searchLink>. 2018.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: http://www.educationaldatamining.org
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 7
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2018
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)<br />Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: IIS1636782<br />ACI1440753<br />DRL1252297<br />DRL1109483<br />DRL1316736<br />DGE1535428<br />DRL1031398<br />R305A120125<br />R305C100024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research<br />Speeches/Meeting Papers
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Courseware%22">Courseware</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Analysis%22">Data Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Causal+Models%22">Causal Models</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Outcomes+of+Education%22">Outcomes of Education</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Mastery+Learning%22">Mastery Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Skill+Development%22">Skill Development</searchLink><br /><searchLink fieldCode="DE" term="%22Intelligent+Tutoring+Systems%22">Intelligent Tutoring Systems</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Bias%22">Statistical Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Randomized A/B tests in educational software are not run in a vacuum: often, reams of historical data are available alongside the data from a randomized trial. This paper proposes a method to use this historical data--often highdimensional and longitudinal--to improve causal estimates from A/B tests. The method proceeds in two steps: first, fit a machine learning model to the historical data predicting students' outcomes as a function of their covariates. Then, use that model to predict the outcomes of the randomized students in the A/B test. Finally, use design-based methods to estimate the treatment effect in the A/B test, using prediction errors in place of outcomes. This method retains all of the advantages of design-based inference, while, under certain conditions, yielding more precise estimators. This paper will give a theoretical condition under which the method improves statistical precision, and demonstrates it using a deep learning algorithm to help estimate effects in a set of experiments run inside ASSISTments. [For the full proceedings, see ED593090.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2019
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED593197
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED593197
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 7
    Subjects:
      – SubjectFull: Courseware
        Type: general
      – SubjectFull: Data Analysis
        Type: general
      – SubjectFull: Causal Models
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Outcomes of Education
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Mastery Learning
        Type: general
      – SubjectFull: Skill Development
        Type: general
      – SubjectFull: Intelligent Tutoring Systems
        Type: general
      – SubjectFull: Statistical Bias
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
    Titles:
      – TitleFull: Using Big Data to Sharpen Design-Based Inference in A/B Tests
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sales, Adam C.
      – PersonEntity:
          Name:
            NameFull: Botelho, Anthony
      – PersonEntity:
          Name:
            NameFull: Patikorn, Thanaporn
      – PersonEntity:
          Name:
            NameFull: Heffernan, Neil T.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Type: published
              Y: 2018
          Titles:
            – TitleFull: International Educational Data Mining Society
              Type: main
ResultId 1