View in EDS

The role of rapid guessing and test‐taking persistence in modelling test‐taking engagement.

Saved in:
Bibliographic Details
Title: The role of rapid guessing and test‐taking persistence in modelling test‐taking engagement.
Authors: Nagy, Gabriel, Ulitzsch, Esther, Lindner, Marlit Annalena
Source: Journal of Computer Assisted Learning. Jun2023, Vol. 39 Issue 3, p751-766. 16p.
Subjects: Behavioral assessment, Computer software, Computer assisted testing (Education), Self-evaluation, Mathematical models, Regression analysis, Academic achievement, Decision making, Theory, Teaching aids, Descriptive statistics, Research funding, Psychology of school children, Student attitudes, Reaction time, Statistical sampling, Maximum likelihood statistics, Science, Probability theory, Pamphlets, Algorithms
Abstract: Background: Item response times in computerized assessments are frequently used to identify rapid guessing behaviour as a manifestation of response disengagement. However, non‐rapid responses (i.e., with longer response times) are not necessarily engaged, which means that response‐time‐based procedures could overlook disengaged responses. Therefore, the identification of disengaged responses could be improved by considering additional indicators of disengagement. We investigated the extent to which decreases in individuals' item solution probabilities over the course of a test reflect disengaged response behaviour. Objectives: To disentangle different types of possibly disengaged responses and better understand non‐effortful test‐taking behaviour, we augmented responses‐time‐based procedures for identifying rapid guessing with strategies for detecting disengaged responses on the basis of performance declines in non‐rapid responses. Methods: We combined item response theory (IRT) models for rapid guessing and test‐taking persistence to examine the capability of response times and item positions to capture response disengagement. We used a computerized assessment in which science items were randomly distributed across positions for each student. This allowed us to estimate individual differences in test‐taking persistence (i.e., the duration for which the initial level of performance is maintained) while accounting for rapid responses. Results and Conclusions: Response times did not fully explain disengagement; item responses reflected test‐taking persistence even when rapid responses were accounted for. This interpretation was supported by a strong correlation of test‐taking persistence with decreases in self‐reported test‐taking effort. Furthermore, our results suggest that IRT models for test‐taking persistence can effectively account for the undesirable impact of low test‐taking effort even when response times are unavailable. Practitioner Notes: Assessments of proficiencies that attempt to quantify what individuals know and can do lead to biased results when individuals provide disengaged responses.Item response times are frequently used to identify rapid guessing behaviour as a manifestation of response disengagement, but response‐time‐based procedures could overlook disengaged responses.To disentangle different types of possibly disengaged responses and better understand non‐effortful test‐taking behaviour, we augmented responses‐time‐based procedures for identifying rapid guessing with strategies for detecting disengaged responses on the basis of performance declines in non‐rapid responses.In a sample of fifth and sixth graders, we found that response times did not fully explain disengagement, as many students showed performance declines in non‐rapid item responses.Our results suggest that item response theory models for test‐taking persistence can effectively account for the undesirable impact of low test‐taking effort even when response times are unavailable. [ABSTRACT FROM AUTHOR]
Copyright of Journal of Computer Assisted Learning is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Psychology and Behavioral Sciences Collection
FullText Text:
  Availability: 0
Header DbId: pbh
DbLabel: Psychology and Behavioral Sciences Collection
An: 163886536
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The role of rapid guessing and test‐taking persistence in modelling test‐taking engagement.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Nagy%2C+Gabriel%22">Nagy, Gabriel</searchLink><br /><searchLink fieldCode="AR" term="%22Ulitzsch%2C+Esther%22">Ulitzsch, Esther</searchLink><br /><searchLink fieldCode="AR" term="%22Lindner%2C+Marlit+Annalena%22">Lindner, Marlit Annalena</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Journal+of+Computer+Assisted+Learning%22">Journal of Computer Assisted Learning</searchLink>. Jun2023, Vol. 39 Issue 3, p751-766. 16p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Behavioral+assessment%22">Behavioral assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+software%22">Computer software</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+assisted+testing+%28Education%29%22">Computer assisted testing (Education)</searchLink><br /><searchLink fieldCode="DE" term="%22Self-evaluation%22">Self-evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+models%22">Mathematical models</searchLink><br /><searchLink fieldCode="DE" term="%22Regression+analysis%22">Regression analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+achievement%22">Academic achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+making%22">Decision making</searchLink><br /><searchLink fieldCode="DE" term="%22Theory%22">Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+aids%22">Teaching aids</searchLink><br /><searchLink fieldCode="DE" term="%22Descriptive+statistics%22">Descriptive statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Research+funding%22">Research funding</searchLink><br /><searchLink fieldCode="DE" term="%22Psychology+of+school+children%22">Psychology of school children</searchLink><br /><searchLink fieldCode="DE" term="%22Student+attitudes%22">Student attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Reaction+time%22">Reaction time</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+sampling%22">Statistical sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Maximum+likelihood+statistics%22">Maximum likelihood statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Science%22">Science</searchLink><br /><searchLink fieldCode="DE" term="%22Probability+theory%22">Probability theory</searchLink><br /><searchLink fieldCode="DE" term="%22Pamphlets%22">Pamphlets</searchLink><br /><searchLink fieldCode="DE" term="%22Algorithms%22">Algorithms</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Item response times in computerized assessments are frequently used to identify rapid guessing behaviour as a manifestation of response disengagement. However, non‐rapid responses (i.e., with longer response times) are not necessarily engaged, which means that response‐time‐based procedures could overlook disengaged responses. Therefore, the identification of disengaged responses could be improved by considering additional indicators of disengagement. We investigated the extent to which decreases in individuals' item solution probabilities over the course of a test reflect disengaged response behaviour. Objectives: To disentangle different types of possibly disengaged responses and better understand non‐effortful test‐taking behaviour, we augmented responses‐time‐based procedures for identifying rapid guessing with strategies for detecting disengaged responses on the basis of performance declines in non‐rapid responses. Methods: We combined item response theory (IRT) models for rapid guessing and test‐taking persistence to examine the capability of response times and item positions to capture response disengagement. We used a computerized assessment in which science items were randomly distributed across positions for each student. This allowed us to estimate individual differences in test‐taking persistence (i.e., the duration for which the initial level of performance is maintained) while accounting for rapid responses. Results and Conclusions: Response times did not fully explain disengagement; item responses reflected test‐taking persistence even when rapid responses were accounted for. This interpretation was supported by a strong correlation of test‐taking persistence with decreases in self‐reported test‐taking effort. Furthermore, our results suggest that IRT models for test‐taking persistence can effectively account for the undesirable impact of low test‐taking effort even when response times are unavailable. Practitioner Notes: Assessments of proficiencies that attempt to quantify what individuals know and can do lead to biased results when individuals provide disengaged responses.Item response times are frequently used to identify rapid guessing behaviour as a manifestation of response disengagement, but response‐time‐based procedures could overlook disengaged responses.To disentangle different types of possibly disengaged responses and better understand non‐effortful test‐taking behaviour, we augmented responses‐time‐based procedures for identifying rapid guessing with strategies for detecting disengaged responses on the basis of performance declines in non‐rapid responses.In a sample of fifth and sixth graders, we found that response times did not fully explain disengagement, as many students showed performance declines in non‐rapid item responses.Our results suggest that item response theory models for test‐taking persistence can effectively account for the undesirable impact of low test‐taking effort even when response times are unavailable. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Journal of Computer Assisted Learning is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=pbh&AN=163886536
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jcal.12719
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
        StartPage: 751
    Subjects:
      – SubjectFull: Behavioral assessment
        Type: general
      – SubjectFull: Computer software
        Type: general
      – SubjectFull: Computer assisted testing (Education)
        Type: general
      – SubjectFull: Self-evaluation
        Type: general
      – SubjectFull: Mathematical models
        Type: general
      – SubjectFull: Regression analysis
        Type: general
      – SubjectFull: Academic achievement
        Type: general
      – SubjectFull: Decision making
        Type: general
      – SubjectFull: Theory
        Type: general
      – SubjectFull: Teaching aids
        Type: general
      – SubjectFull: Descriptive statistics
        Type: general
      – SubjectFull: Research funding
        Type: general
      – SubjectFull: Psychology of school children
        Type: general
      – SubjectFull: Student attitudes
        Type: general
      – SubjectFull: Reaction time
        Type: general
      – SubjectFull: Statistical sampling
        Type: general
      – SubjectFull: Maximum likelihood statistics
        Type: general
      – SubjectFull: Science
        Type: general
      – SubjectFull: Probability theory
        Type: general
      – SubjectFull: Pamphlets
        Type: general
      – SubjectFull: Algorithms
        Type: general
    Titles:
      – TitleFull: The role of rapid guessing and test‐taking persistence in modelling test‐taking engagement.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Nagy, Gabriel
      – PersonEntity:
          Name:
            NameFull: Ulitzsch, Esther
      – PersonEntity:
          Name:
            NameFull: Lindner, Marlit Annalena
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Text: Jun2023
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 02664909
          Numbering:
            – Type: volume
              Value: 39
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Journal of Computer Assisted Learning
              Type: main
ResultId 1