View in EDS

Parameter Estimation in Comparative Judgment under Random and Adaptive Scheduling Schemes

Saved in:
Bibliographic Details
Title: Parameter Estimation in Comparative Judgment under Random and Adaptive Scheduling Schemes
Language: English
Authors: Ian Hamilton (ORCID 0000-0002-3991-9693), Nick Tawn
Source: Journal of Educational Measurement. 2026 63(1).
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 28
Publication Date: 2026
Document Type: Journal Articles
Reports - Descriptive
Descriptors: Comparative Analysis, Evaluation Methods, Computation, Scheduling, Statistical Bias, Simulation, Statistical Analysis
DOI: 10.1111/jedm.70022
ISSN: 0022-0655
1745-3984
Abstract: Comparative judgment is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed to ensure finiteness of estimates. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty in Comparative Judgment is not the best-performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimation. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered, including identifying a preferred alternative. Further, we propose a novel approach based on a parametric bootstrap. It is found to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions. The work allows for more efficient implementations of comparative judgment.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1501252
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1501252
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Parameter Estimation in Comparative Judgment under Random and Adaptive Scheduling Schemes
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Ian+Hamilton%22">Ian Hamilton</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-3991-9693">0000-0002-3991-9693</externalLink>)<br /><searchLink fieldCode="AR" term="%22Nick+Tawn%22">Nick Tawn</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2026 63(1).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 28
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Computation%22">Computation</searchLink><br /><searchLink fieldCode="DE" term="%22Scheduling%22">Scheduling</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Bias%22">Statistical Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Simulation%22">Simulation</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.70022
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Comparative judgment is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed to ensure finiteness of estimates. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty in Comparative Judgment is not the best-performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimation. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered, including identifying a preferred alternative. Further, we propose a novel approach based on a parametric bootstrap. It is found to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions. The work allows for more efficient implementations of comparative judgment.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1501252
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1501252
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.70022
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 28
    Subjects:
      – SubjectFull: Comparative Analysis
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Computation
        Type: general
      – SubjectFull: Scheduling
        Type: general
      – SubjectFull: Statistical Bias
        Type: general
      – SubjectFull: Simulation
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
    Titles:
      – TitleFull: Parameter Estimation in Comparative Judgment under Random and Adaptive Scheduling Schemes
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Ian Hamilton
      – PersonEntity:
          Name:
            NameFull: Nick Tawn
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 63
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1