View in EDS

Effective Evaluation of Online Learning Interventions with Surrogate Measures

Saved in:
Bibliographic Details
Title: Effective Evaluation of Online Learning Interventions with Surrogate Measures
Language: English
Authors: Prihar, Ethan, Vanacore, Kirk, Sales, Adam, Heffernan, Neil
Source: International Educational Data Mining Society. 2023.
Availability: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Peer Reviewed: Y
Page Count: 7
Publication Date: 2023
Sponsoring Agency: National Science Foundation (NSF)
Institute of Education Sciences (ED)
Office of Elementary and Secondary Education (OESE) (ED), Education Innovation and Research (EIR)
Office of Naval Research (ONR) (DOD)
Federal Highway Administration (FHWA), National Highway Institute (NHI)
Contract Number: 2118725
2118904
1950683
1917808
1931523
1940236
1917713
1903304
1822830
1759229
1724889
1636782
1535428
R305N210049
R305D210031
R305A170137
R305A170243
R305A180401
R305A120125
U411B190024
S411B210024
N000141812768
R44GM146483
Document Type: Speeches/Meeting Papers
Reports - Research
Descriptors: Electronic Learning, Intervention, Instructional Effectiveness, Data Collection, Models, Measurement
Abstract: There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized experiments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, distal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional methods. To help discern the effectiveness of instructional interventions, this work uses data from 26,060 clickstream sequences of students across 31 different online educational experiments exploring 51 different research questions and the students' posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next-problem correctness was determined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next-problem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problem-level experiments. [For the complete proceedings, see ED630829. Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN).]
Abstractor: As Provided
Notes: https://osf.io/uj48v
IES Funded: Yes
Entry Date: 2023
Accession Number: ED630844
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED630844
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED630844
AccessLevel: 3
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Effective Evaluation of Online Learning Interventions with Surrogate Measures
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Prihar%2C+Ethan%22">Prihar, Ethan</searchLink><br /><searchLink fieldCode="AR" term="%22Vanacore%2C+Kirk%22">Vanacore, Kirk</searchLink><br /><searchLink fieldCode="AR" term="%22Sales%2C+Adam%22">Sales, Adam</searchLink><br /><searchLink fieldCode="AR" term="%22Heffernan%2C+Neil%22">Heffernan, Neil</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Educational+Data+Mining+Society%22"><i>International Educational Data Mining Society</i></searchLink>. 2023.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 7
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)<br />Institute of Education Sciences (ED)<br />Office of Elementary and Secondary Education (OESE) (ED), Education Innovation and Research (EIR)<br />Office of Naval Research (ONR) (DOD)<br />Federal Highway Administration (FHWA), National Highway Institute (NHI)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 2118725<br />2118904<br />1950683<br />1917808<br />1931523<br />1940236<br />1917713<br />1903304<br />1822830<br />1759229<br />1724889<br />1636782<br />1535428<br />R305N210049<br />R305D210031<br />R305A170137<br />R305A170243<br />R305A180401<br />R305A120125<br />U411B190024<br />S411B210024<br />N000141812768<br />R44GM146483
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Speeches/Meeting Papers<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Electronic+Learning%22">Electronic Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Instructional+Effectiveness%22">Instructional Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Collection%22">Data Collection</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement%22">Measurement</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized experiments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, distal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional methods. To help discern the effectiveness of instructional interventions, this work uses data from 26,060 clickstream sequences of students across 31 different online educational experiments exploring 51 different research questions and the students' posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next-problem correctness was determined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next-problem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problem-level experiments. [For the complete proceedings, see ED630829. Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN).]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/uj48v
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED630844
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED630844
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 7
    Subjects:
      – SubjectFull: Electronic Learning
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Instructional Effectiveness
        Type: general
      – SubjectFull: Data Collection
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Measurement
        Type: general
    Titles:
      – TitleFull: Effective Evaluation of Online Learning Interventions with Surrogate Measures
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Prihar, Ethan
      – PersonEntity:
          Name:
            NameFull: Vanacore, Kirk
      – PersonEntity:
          Name:
            NameFull: Sales, Adam
      – PersonEntity:
          Name:
            NameFull: Heffernan, Neil
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2023
          Titles:
            – TitleFull: International Educational Data Mining Society
              Type: main
ResultId 1