The Precision and Bias of Cut Score Estimates from the Beuk Standard Setting Method
Saved in:
| Title: | The Precision and Bias of Cut Score Estimates from the Beuk Standard Setting Method |
|---|---|
| Language: | English |
| Authors: | Joseph H. Grochowalski (ORCID |
| Source: | Journal of Educational Measurement. 2025 62(4):687-717. |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 31 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Cutting Scores, Standard Setting, Accuracy, Statistical Bias, Simulation, Interrater Reliability |
| DOI: | 10.1111/jedm.70007 |
| ISSN: | 0022-0655 1745-3984 |
| Abstract: | The Beuk standard setting method derives cut scores through expert judgment that balances content and normative perspectives. This study developed a method to estimate confidence intervals for Beuk settings and assessed their accuracy via simulations. Simulations varied SME panel size, expert agreement, cut score locations, score distributions, and decision alignment. Panels with 20+ participants provided precise and accurate cut score estimates if strongly agreed upon. Larger panels did not improve precision significantly. Cut score location influenced confidence interval widths, highlighting its importance in planning. Real data showed SME disagreement increased bias and variance of Beuk estimates. Use Beuk cut scores cautiously with small panels, flat score distributions, or significant expert disagreement. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1491384 |
| Database: | ERIC |
| Abstract: | The Beuk standard setting method derives cut scores through expert judgment that balances content and normative perspectives. This study developed a method to estimate confidence intervals for Beuk settings and assessed their accuracy via simulations. Simulations varied SME panel size, expert agreement, cut score locations, score distributions, and decision alignment. Panels with 20+ participants provided precise and accurate cut score estimates if strongly agreed upon. Larger panels did not improve precision significantly. Cut score location influenced confidence interval widths, highlighting its importance in planning. Real data showed SME disagreement increased bias and variance of Beuk estimates. Use Beuk cut scores cautiously with small panels, flat score distributions, or significant expert disagreement. |
|---|---|
| ISSN: | 0022-0655 1745-3984 |
| DOI: | 10.1111/jedm.70007 |
