Using Item Parameter Predictions for Reducing Calibration Sample Requirements--A Case Study Based on a High-Stakes Admission Test
Saved in:
| Title: | Using Item Parameter Predictions for Reducing Calibration Sample Requirements--A Case Study Based on a High-Stakes Admission Test |
|---|---|
| Language: | English |
| Authors: | Esther Ulitzsch, Dmitry Belov, Oliver Lüdtke, Alexander Robitzsch |
| Source: | Journal of Educational Measurement. 2026 63(1). |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 52 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | High Stakes Tests, Test Items, Difficulty Level, Computation, Bayesian Statistics, Maximum Likelihood Statistics, Sample Size, Accuracy, Prediction |
| DOI: | 10.1111/jedm.12426 |
| ISSN: | 0022-0655 1745-3984 |
| Abstract: | In item difficulty modeling (IDM), item parameters are predicted from the items' linguistic features, aiming to ultimately render item calibration redundant. Current IDM applications, however, commonly do not yield the required prediction accuracy. To immediately exploit even somewhat inaccurate IDM predictions, we blend IDM with established Bayesian estimation techniques. We propose a two-step approach where (1) IDM predictions are obtained and (2) employed to construct informative prior distributions. We evaluate the approach in a case study on small-sample calibration of the 3PL in a high-stakes test. First, concerning implementation, we find computationally efficient penalized maximum likelihood estimation to be comparable to the best-performing MCMC-based approach. Second, we investigate sample size reductions achievable with state-of-the-art IDM predictions, finding negligible gains compared to merely considering the historical distribution of parameters. Third, we evaluate the prediction accuracy required for a targeted sample size reduction by gradually increasing simulated IDM prediction accuracies. We find that required accuracies can counterbalance each other, allowing calibration sample size to be reduced when either high-quality item difficulty predictions or good predictions of item discriminations and pseudo-guessing parameters are available. We argue that these evaluations provide new, portable IDM benchmarks quantifying performance in terms of achievable sample size reductions. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1501461 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | In item difficulty modeling (IDM), item parameters are predicted from the items' linguistic features, aiming to ultimately render item calibration redundant. Current IDM applications, however, commonly do not yield the required prediction accuracy. To immediately exploit even somewhat inaccurate IDM predictions, we blend IDM with established Bayesian estimation techniques. We propose a two-step approach where (1) IDM predictions are obtained and (2) employed to construct informative prior distributions. We evaluate the approach in a case study on small-sample calibration of the 3PL in a high-stakes test. First, concerning implementation, we find computationally efficient penalized maximum likelihood estimation to be comparable to the best-performing MCMC-based approach. Second, we investigate sample size reductions achievable with state-of-the-art IDM predictions, finding negligible gains compared to merely considering the historical distribution of parameters. Third, we evaluate the prediction accuracy required for a targeted sample size reduction by gradually increasing simulated IDM prediction accuracies. We find that required accuracies can counterbalance each other, allowing calibration sample size to be reduced when either high-quality item difficulty predictions or good predictions of item discriminations and pseudo-guessing parameters are available. We argue that these evaluations provide new, portable IDM benchmarks quantifying performance in terms of achievable sample size reductions. |
|---|---|
| ISSN: | 0022-0655 1745-3984 |
| DOI: | 10.1111/jedm.12426 |
