Parameter Estimation in Comparative Judgment under Random and Adaptive Scheduling Schemes
Saved in:
| Title: | Parameter Estimation in Comparative Judgment under Random and Adaptive Scheduling Schemes |
|---|---|
| Language: | English |
| Authors: | Ian Hamilton (ORCID |
| Source: | Journal of Educational Measurement. 2026 63(1). |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 28 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Descriptive |
| Descriptors: | Comparative Analysis, Evaluation Methods, Computation, Scheduling, Statistical Bias, Simulation, Statistical Analysis |
| DOI: | 10.1111/jedm.70022 |
| ISSN: | 0022-0655 1745-3984 |
| Abstract: | Comparative judgment is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed to ensure finiteness of estimates. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty in Comparative Judgment is not the best-performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimation. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered, including identifying a preferred alternative. Further, we propose a novel approach based on a parametric bootstrap. It is found to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions. The work allows for more efficient implementations of comparative judgment. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1501252 |
| Database: | ERIC |
| Abstract: | Comparative judgment is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley-Terry model. Likelihood penalization is often employed to ensure finiteness of estimates. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty in Comparative Judgment is not the best-performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimation. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered, including identifying a preferred alternative. Further, we propose a novel approach based on a parametric bootstrap. It is found to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions. The work allows for more efficient implementations of comparative judgment. |
|---|---|
| ISSN: | 0022-0655 1745-3984 |
| DOI: | 10.1111/jedm.70022 |
