View in EDS

Problem-Solving Behavior and EdTech Effectiveness: A Model for Exploratory Causal Analysis

Saved in:
Bibliographic Details
Title: Problem-Solving Behavior and EdTech Effectiveness: A Model for Exploratory Causal Analysis
Language: English
Authors: Adam C. Sales, Kirk P. Vanacore, Hyeon-Ah Kang, Tiffany A. Whittaker
Source: International Educational Data Mining Society. 2024.
Availability: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Peer Reviewed: Y
Page Count: 11
Publication Date: 2024
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305D210036
Document Type: Speeches/Meeting Papers
Reports - Research
Education Level: High Schools
Secondary Education
Junior High Schools
Middle Schools
Descriptors: Educational Technology, Problem Solving, Electronic Learning, Instructional Effectiveness, Learner Engagement, Technology Uses in Education, Algebra, Mathematics Instruction, High School Students, Middle School Students
Abstract: The gold-standard evaluation of an educational technology product is a randomized study comparing students randomized to use a computer-based learning platform (CBLP) to students assigned to a "business as usual" condition, such as pencil-and-paper work, and estimating average treatment effects. However, not everyone uses the same CBLP in the same way--indeed, an individual may engage with CBLP in multiple different ways over the course of a study--and the platform's effectiveness may depend on how students are using it. This paper introduces a model that serves two aims: classifying different modes of problem-solving or engagement among CBLP users, and estimating varying program effectiveness with varying usage patterns. The model uses mixed-type problem-level variables, such as time spent, the number of errors committed, and the number of hints requested, to cluster each problem attempt by each student into one of a number of categories, using a model-based, probabilistic, latent profile model. Students differ from each other based on their probabilities of working on problems in each of the identified modes. Finally, the model uses a fully latent principal stratification approach to estimate varying treatment effects as a function of those probabilities. In this paper, we describe the model and estimation in detail and illustrate application using data from two large randomized field trials, one evaluating Cognitive Tutor Algebra I, and the other evaluating ASSISTments. [For the complete proceedings, see ED675485.]
Abstractor: As Provided
Notes: https://osf.io/r3nf2
IES Funded: Yes
Entry Date: 2025
Accession Number: ED675575
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED675575
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED675575
AccessLevel: 3
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Problem-Solving Behavior and EdTech Effectiveness: A Model for Exploratory Causal Analysis
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Adam+C%2E+Sales%22">Adam C. Sales</searchLink><br /><searchLink fieldCode="AR" term="%22Kirk+P%2E+Vanacore%22">Kirk P. Vanacore</searchLink><br /><searchLink fieldCode="AR" term="%22Hyeon-Ah+Kang%22">Hyeon-Ah Kang</searchLink><br /><searchLink fieldCode="AR" term="%22Tiffany+A%2E+Whittaker%22">Tiffany A. Whittaker</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Educational+Data+Mining+Society%22"><i>International Educational Data Mining Society</i></searchLink>. 2024.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 11
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305D210036
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Speeches/Meeting Papers<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Educational+Technology%22">Educational Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+Solving%22">Problem Solving</searchLink><br /><searchLink fieldCode="DE" term="%22Electronic+Learning%22">Electronic Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Instructional+Effectiveness%22">Instructional Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Learner+Engagement%22">Learner Engagement</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Algebra%22">Algebra</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Instruction%22">Mathematics Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The gold-standard evaluation of an educational technology product is a randomized study comparing students randomized to use a computer-based learning platform (CBLP) to students assigned to a "business as usual" condition, such as pencil-and-paper work, and estimating average treatment effects. However, not everyone uses the same CBLP in the same way--indeed, an individual may engage with CBLP in multiple different ways over the course of a study--and the platform's effectiveness may depend on how students are using it. This paper introduces a model that serves two aims: classifying different modes of problem-solving or engagement among CBLP users, and estimating varying program effectiveness with varying usage patterns. The model uses mixed-type problem-level variables, such as time spent, the number of errors committed, and the number of hints requested, to cluster each problem attempt by each student into one of a number of categories, using a model-based, probabilistic, latent profile model. Students differ from each other based on their probabilities of working on problems in each of the identified modes. Finally, the model uses a fully latent principal stratification approach to estimate varying treatment effects as a function of those probabilities. In this paper, we describe the model and estimation in detail and illustrate application using data from two large randomized field trials, one evaluating Cognitive Tutor Algebra I, and the other evaluating ASSISTments. [For the complete proceedings, see ED675485.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/r3nf2
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED675575
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED675575
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
    Subjects:
      – SubjectFull: Educational Technology
        Type: general
      – SubjectFull: Problem Solving
        Type: general
      – SubjectFull: Electronic Learning
        Type: general
      – SubjectFull: Instructional Effectiveness
        Type: general
      – SubjectFull: Learner Engagement
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Algebra
        Type: general
      – SubjectFull: Mathematics Instruction
        Type: general
      – SubjectFull: High School Students
        Type: general
      – SubjectFull: Middle School Students
        Type: general
    Titles:
      – TitleFull: Problem-Solving Behavior and EdTech Effectiveness: A Model for Exploratory Causal Analysis
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Adam C. Sales
      – PersonEntity:
          Name:
            NameFull: Kirk P. Vanacore
      – PersonEntity:
          Name:
            NameFull: Hyeon-Ah Kang
      – PersonEntity:
          Name:
            NameFull: Tiffany A. Whittaker
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Titles:
            – TitleFull: International Educational Data Mining Society
              Type: main
ResultId 1