View in EDS

Impact of Problem Type on Learning in Computer-Based Learning Platforms: Results from a Large-Scale Experiment

Saved in:
Bibliographic Details
Title: Impact of Problem Type on Learning in Computer-Based Learning Platforms: Results from a Large-Scale Experiment
Language: English
Authors: Kirk Vanacore, Ashish Gurung, Adam Sales, Neil Heffernan, Society for Research on Educational Effectiveness (SREE)
Source: Society for Research on Educational Effectiveness. 2024.
Availability: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Peer Reviewed: Y
Publication Date: 2024
Document Type: Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
Descriptors: Computer Assisted Instruction, Learning Management Systems, Questioning Techniques, Grading, Multiple Choice Tests, Learning Activities, Instructional Effectiveness, Performance, Middle School Mathematics, Mathematics Instruction, Mastery Tests, Mathematics Tests
Abstract: Background: The proliferation of computer-based learning platforms (CBLPs) has caused an increased focus on understanding how to build scalable systems that optimize learning. Thus, CBLPs often rely on close response questions (e.g., multiple choice questions, "select all that apply," "arrange in the correct order," etc.) for learning tasks as they are easy to grade automatically. Furthermore, students' selection of distractors may be leveraged to provide automated feedback based on students' likely misconceptions. Yet, educators often believe that closed-response questions may lead to shallow learning (Wang et al., 2021). Alternatively, open-response questions are often considered more rigorous (Magliano et al., 2007). However, open-response questions are harder to grade automatically, and therefore, it is harder to provide timely and targeted feedback based on students' responses. Fill-in problems, in which students must provide a specific response in a text box, represent a middle ground for some content areas by requiring students to produce answers independently. Furthermore, responses to fill-in problems can be used to provide automated targeted feedback based on what the student submitted (Gurung, Baral, et al., 2023; Gurung, Lee, et al., 2023). Considering the mixed results from prior work on the learning and assessment value of closed and open-response questions (Funk & Dickson, 2011; Magliano et al., 2007; Sugrue et al., 1998), fill-in problems provide a potential alternative problem type that may combine the advantages of closed and open-response questions. Current Study: Using an experimental design, we evaluate the impact of the fill-in problems compared to multiple-choice questions (MCQs) on students' performance during a learning activity (RQ1) and its effect on their learning as measured by the post-test (RQ2). Furthermore, we must ensure that all students share any learning benefits of the features of CBLPs. Thus, following the guidance of Kizilcec & Lee (2022), we evaluate whether the feature does not widen the gap between advantaged and disadvantaged students based on students' performance before the activity (RQ3). Design: This study used mastery learning activities through ASSISTments (Heffernan & Heffernan, 2014), a math-focused CBLP. The experiment involved two middle school math mastery-based activities. Students were randomized to either receive Fill-in problems or MCQs throughout the activity. The problem order during the activity was also random. Figure 1 shows the randomization scheme, and Figure 2 presents the differences in the fill-in, MQC, and post-test. Notably, the post-test consisted of more complex transfer items. Students only took the post-test if they mastered the activity. Data: The data was collected across five school years in the United States (2017-18, 2018-19, 2019-20, 2020-21, 2021-22). Middle school teachers who used ASSISTments as an instructional tool assigned mastery learning activities to their students as part of their lessons. 192 teachers assigned the two problem sets to 383 classes. 6774 students participated in the experiment. Method: Our analysis required four steps. First, we ran a series of regression models with robust standard errors (one model for problem sequence) predicting students' accuracy (Y[subscript ij]) within the master learning activity by whether students were randomized to treatment (Fill-In[subscript i]) with fixed effects for the activities' problems (P[subscript j]) (Equation 1). This evaluates how problem type influences students' performance across the activity. Second, we assessed whether attrition differed across conditions. Third, we use a multi-level logistic regression to estimate whether problem type affects students' learning by regressing treatment assignments on students' accuracy on post-test items accounting for the student ([mu][subscript i]), class ([mu][subscript c]) and problem ([mu][subscript j]) (Equation 2). Finally, we added an interaction to Equation 2 to between Fill-In[subscript i] and students' prior performance in ASSISTments as a measure of their prior mathematical knowledge. Results: (RQ1) On the first problem sequence, students in the MCQ condition outperformed those in the Fill-In condition by an estimated five percentage points ([beta][subscript 1] = -0.05, SE = 0.012, p < 0.0001). Figure 2 displays the average performance (lines) and samples (shading) by condition across the first ten problems in the mastery learning component of the experiment. The pattern of performance differences across the activity suggests that the difficulty experienced early on by the students in the Fill-In condition potentially benefited the students later in the activity. Nevertheless, it is important to acknowledge a potential confound to this explanation as the samples differed across conditions on problem sequences due to differing mastery and attrition rates. (RQ2) Table 1 details the experiment's attrition rates and the balance test statistics. Students could attrit in multiple ways: not masting, not stating the post-test, and not completing the post-test. Overall, there weren't significant differences in attrition across conditions, but the balance between mastery rates was only marginally non-significant. Table 2 presents the model used to estimate the effect of problem type on post-test performance. Students who engaged with Fill-In problem sets were significantly more likely to provide correct responses in the post-test than those who worked through MCQs ([gamma]1 = 0.23, SE = 0.06, p > 0.001). Students in the MCQ condition had a 27% probability of getting either of the transfer problems correct. In contrast, students in the Fill-In condition had a 31% probability of getting the transfer item correct. As a robustness check, we ran this model, giving all students who attired incorrect responses, and the effect was still positive and significant. (RQ3) Table 3 displays the results of the interaction model. The interaction between the prior performance and the Fill-In problem set is significant and positive ([gamma]3 = 0.20, SE = 0.10, p = 0.042). Figure 4 plots the interaction, showing that while students with higher prior performance experienced a positive effect from fill-in problems, those with lower knowledge likely benefited from MQCs. Conclusion: Overall, this study's findings present causal evidence that problem types influence how and whether students learn. We observed that, on average, students had better learning outcomes when using mastery-based assignments with Fill-In problems compared to MCQs, but this effect may vary by students' prior knowledge. Notably, prior performance may not be a perfect measure of prior knowledge; thus, this analysis points to the need for further investigation into how problems are presented to ensure all students are learning.
Abstractor: As Provided
Entry Date: 2025
Access URL: https://www.sree.org/2024-conference
Accession Number: ED663539
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: ED663539
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Impact of Problem Type on Learning in Computer-Based Learning Platforms: Results from a Large-Scale Experiment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Kirk+Vanacore%22&quot;&gt;Kirk Vanacore&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Ashish+Gurung%22&quot;&gt;Ashish Gurung&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Adam+Sales%22&quot;&gt;Adam Sales&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Neil+Heffernan%22&quot;&gt;Neil Heffernan&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Society+for+Research+on+Educational+Effectiveness+%28SREE%29%22&quot;&gt;Society for Research on Educational Effectiveness (SREE)&lt;/searchLink&gt;
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;SO&quot; term=&quot;%22Society+for+Research+on+Educational+Effectiveness%22&quot;&gt;&lt;i&gt;Society for Research on Educational Effectiveness&lt;/i&gt;&lt;/searchLink&gt;. 2024.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: &lt;searchLink fieldCode=&quot;EL&quot; term=&quot;%22Junior+High+Schools%22&quot;&gt;Junior High Schools&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;EL&quot; term=&quot;%22Middle+Schools%22&quot;&gt;Middle Schools&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;EL&quot; term=&quot;%22Secondary+Education%22&quot;&gt;Secondary Education&lt;/searchLink&gt;
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Computer+Assisted+Instruction%22&quot;&gt;Computer Assisted Instruction&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Learning+Management+Systems%22&quot;&gt;Learning Management Systems&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Questioning+Techniques%22&quot;&gt;Questioning Techniques&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Grading%22&quot;&gt;Grading&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Multiple+Choice+Tests%22&quot;&gt;Multiple Choice Tests&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Learning+Activities%22&quot;&gt;Learning Activities&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Instructional+Effectiveness%22&quot;&gt;Instructional Effectiveness&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Performance%22&quot;&gt;Performance&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Middle+School+Mathematics%22&quot;&gt;Middle School Mathematics&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Mathematics+Instruction%22&quot;&gt;Mathematics Instruction&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Mastery+Tests%22&quot;&gt;Mastery Tests&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Mathematics+Tests%22&quot;&gt;Mathematics Tests&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: The proliferation of computer-based learning platforms (CBLPs) has caused an increased focus on understanding how to build scalable systems that optimize learning. Thus, CBLPs often rely on close response questions (e.g., multiple choice questions, &quot;select all that apply,&quot; &quot;arrange in the correct order,&quot; etc.) for learning tasks as they are easy to grade automatically. Furthermore, students&#39; selection of distractors may be leveraged to provide automated feedback based on students&#39; likely misconceptions. Yet, educators often believe that closed-response questions may lead to shallow learning (Wang et al., 2021). Alternatively, open-response questions are often considered more rigorous (Magliano et al., 2007). However, open-response questions are harder to grade automatically, and therefore, it is harder to provide timely and targeted feedback based on students&#39; responses. Fill-in problems, in which students must provide a specific response in a text box, represent a middle ground for some content areas by requiring students to produce answers independently. Furthermore, responses to fill-in problems can be used to provide automated targeted feedback based on what the student submitted (Gurung, Baral, et al., 2023; Gurung, Lee, et al., 2023). Considering the mixed results from prior work on the learning and assessment value of closed and open-response questions (Funk &amp; Dickson, 2011; Magliano et al., 2007; Sugrue et al., 1998), fill-in problems provide a potential alternative problem type that may combine the advantages of closed and open-response questions. Current Study: Using an experimental design, we evaluate the impact of the fill-in problems compared to multiple-choice questions (MCQs) on students&#39; performance during a learning activity (RQ1) and its effect on their learning as measured by the post-test (RQ2). Furthermore, we must ensure that all students share any learning benefits of the features of CBLPs. Thus, following the guidance of Kizilcec &amp; Lee (2022), we evaluate whether the feature does not widen the gap between advantaged and disadvantaged students based on students&#39; performance before the activity (RQ3). Design: This study used mastery learning activities through ASSISTments (Heffernan &amp; Heffernan, 2014), a math-focused CBLP. The experiment involved two middle school math mastery-based activities. Students were randomized to either receive Fill-in problems or MCQs throughout the activity. The problem order during the activity was also random. Figure 1 shows the randomization scheme, and Figure 2 presents the differences in the fill-in, MQC, and post-test. Notably, the post-test consisted of more complex transfer items. Students only took the post-test if they mastered the activity. Data: The data was collected across five school years in the United States (2017-18, 2018-19, 2019-20, 2020-21, 2021-22). Middle school teachers who used ASSISTments as an instructional tool assigned mastery learning activities to their students as part of their lessons. 192 teachers assigned the two problem sets to 383 classes. 6774 students participated in the experiment. Method: Our analysis required four steps. First, we ran a series of regression models with robust standard errors (one model for problem sequence) predicting students&#39; accuracy (Y[subscript ij]) within the master learning activity by whether students were randomized to treatment (Fill-In[subscript i]) with fixed effects for the activities&#39; problems (P[subscript j]) (Equation 1). This evaluates how problem type influences students&#39; performance across the activity. Second, we assessed whether attrition differed across conditions. Third, we use a multi-level logistic regression to estimate whether problem type affects students&#39; learning by regressing treatment assignments on students&#39; accuracy on post-test items accounting for the student ([mu][subscript i]), class ([mu][subscript c]) and problem ([mu][subscript j]) (Equation 2). Finally, we added an interaction to Equation 2 to between Fill-In[subscript i] and students&#39; prior performance in ASSISTments as a measure of their prior mathematical knowledge. Results: (RQ1) On the first problem sequence, students in the MCQ condition outperformed those in the Fill-In condition by an estimated five percentage points ([beta][subscript 1] = -0.05, SE = 0.012, p &lt; 0.0001). Figure 2 displays the average performance (lines) and samples (shading) by condition across the first ten problems in the mastery learning component of the experiment. The pattern of performance differences across the activity suggests that the difficulty experienced early on by the students in the Fill-In condition potentially benefited the students later in the activity. Nevertheless, it is important to acknowledge a potential confound to this explanation as the samples differed across conditions on problem sequences due to differing mastery and attrition rates. (RQ2) Table 1 details the experiment&#39;s attrition rates and the balance test statistics. Students could attrit in multiple ways: not masting, not stating the post-test, and not completing the post-test. Overall, there weren&#39;t significant differences in attrition across conditions, but the balance between mastery rates was only marginally non-significant. Table 2 presents the model used to estimate the effect of problem type on post-test performance. Students who engaged with Fill-In problem sets were significantly more likely to provide correct responses in the post-test than those who worked through MCQs ([gamma]1 = 0.23, SE = 0.06, p &gt; 0.001). Students in the MCQ condition had a 27% probability of getting either of the transfer problems correct. In contrast, students in the Fill-In condition had a 31% probability of getting the transfer item correct. As a robustness check, we ran this model, giving all students who attired incorrect responses, and the effect was still positive and significant. (RQ3) Table 3 displays the results of the interaction model. The interaction between the prior performance and the Fill-In problem set is significant and positive ([gamma]3 = 0.20, SE = 0.10, p = 0.042). Figure 4 plots the interaction, showing that while students with higher prior performance experienced a positive effect from fill-in problems, those with lower knowledge likely benefited from MQCs. Conclusion: Overall, this study&#39;s findings present causal evidence that problem types influence how and whether students learn. We observed that, on average, students had better learning outcomes when using mastery-based assignments with Fill-In problems compared to MCQs, but this effect may vary by students&#39; prior knowledge. Notably, prior performance may not be a perfect measure of prior knowledge; thus, this analysis points to the need for further investigation into how problems are presented to ensure all students are learning.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: URL
  Label: Access URL
  Group: URL
  Data: &lt;link linkTarget=&quot;URL&quot; linkTerm=&quot;https://www.sree.org/2024-conference&quot; linkWindow=&quot;_blank&quot;&gt;https://www.sree.org/2024-conference&lt;/link&gt;
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED663539
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED663539
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    Subjects:
      – SubjectFull: Computer Assisted Instruction
        Type: general
      – SubjectFull: Learning Management Systems
        Type: general
      – SubjectFull: Questioning Techniques
        Type: general
      – SubjectFull: Grading
        Type: general
      – SubjectFull: Multiple Choice Tests
        Type: general
      – SubjectFull: Learning Activities
        Type: general
      – SubjectFull: Instructional Effectiveness
        Type: general
      – SubjectFull: Performance
        Type: general
      – SubjectFull: Middle School Mathematics
        Type: general
      – SubjectFull: Mathematics Instruction
        Type: general
      – SubjectFull: Mastery Tests
        Type: general
      – SubjectFull: Mathematics Tests
        Type: general
    Titles:
      – TitleFull: Impact of Problem Type on Learning in Computer-Based Learning Platforms: Results from a Large-Scale Experiment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Society for Research on Educational Effectiveness (SREE)
      – PersonEntity:
          Name:
            NameFull: Kirk Vanacore
      – PersonEntity:
          Name:
            NameFull: Ashish Gurung
      – PersonEntity:
          Name:
            NameFull: Adam Sales
      – PersonEntity:
          Name:
            NameFull: Neil Heffernan
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 21
              M: 09
              Type: published
              Y: 2024
          Titles:
            – TitleFull: Society for Research on Educational Effectiveness
              Type: main
ResultId 1