View in EDS

Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems

Saved in:
Bibliographic Details
Title: Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems
Language: English
Authors: Alexander Kwako (ORCID 0000-0002-6603-4874), Susan Lottridge, Christopher Ormerod
Source: Educational Measurement: Issues and Practice. 2026 45(2).
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 14
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Descriptors: Models, Confidence Testing, Scores, Scoring, Computer Assisted Testing, Man Machine Systems, Automation, Item Response Theory
DOI: 10.1111/emip.70019
ISSN: 0731-1745
1745-3992
Abstract: In large-scale assessments, constructed response items are often scored using hybrid scoring systems, which combine human and automated scores. In this study, we augment automated scoring with confidence modeling to strategically route difficult-to-score responses for human review. We utilize "hybrid performance curves" to visualize the impact of routing on performance. Additionally, we propose several "hybrid scoring policies" for selecting optimal routing thresholds given practical constraints. Our findings reveal that hybrid scoring systems can achieve an overall performance that exceeds that of human- and automated-only systems. Moreover, the superior performance of the hybrid system is less expensive than a human-only system. These findings highlight the complementarity of human raters and automated scoring engines. Although current standards focus on the performance of human raters and automated scoring engines in isolation, we recommend that practitioners also report on the performance of the hybrid scoring system as a whole.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1506976
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1506976
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Alexander+Kwako%22">Alexander Kwako</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6603-4874">0000-0002-6603-4874</externalLink>)<br /><searchLink fieldCode="AR" term="%22Susan+Lottridge%22">Susan Lottridge</searchLink><br /><searchLink fieldCode="AR" term="%22Christopher+Ormerod%22">Christopher Ormerod</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Measurement%3A+Issues+and+Practice%22"><i>Educational Measurement: Issues and Practice</i></searchLink>. 2026 45(2).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 14
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Confidence+Testing%22">Confidence Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Man+Machine+Systems%22">Man Machine Systems</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/emip.70019
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0731-1745<br />1745-3992
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In large-scale assessments, constructed response items are often scored using hybrid scoring systems, which combine human and automated scores. In this study, we augment automated scoring with confidence modeling to strategically route difficult-to-score responses for human review. We utilize "hybrid performance curves" to visualize the impact of routing on performance. Additionally, we propose several "hybrid scoring policies" for selecting optimal routing thresholds given practical constraints. Our findings reveal that hybrid scoring systems can achieve an overall performance that exceeds that of human- and automated-only systems. Moreover, the superior performance of the hybrid system is less expensive than a human-only system. These findings highlight the complementarity of human raters and automated scoring engines. Although current standards focus on the performance of human raters and automated scoring engines in isolation, we recommend that practitioners also report on the performance of the hybrid scoring system as a whole.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1506976
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1506976
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/emip.70019
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 14
    Subjects:
      – SubjectFull: Models
        Type: general
      – SubjectFull: Confidence Testing
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Man Machine Systems
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
    Titles:
      – TitleFull: Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Alexander Kwako
      – PersonEntity:
          Name:
            NameFull: Susan Lottridge
      – PersonEntity:
          Name:
            NameFull: Christopher Ormerod
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0731-1745
            – Type: issn-electronic
              Value: 1745-3992
          Numbering:
            – Type: volume
              Value: 45
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Educational Measurement: Issues and Practice
              Type: main
ResultId 1