The Biggest Equating Study in the World… Ever. Research Report
Saved in:
| Title: | The Biggest Equating Study in the World… Ever. Research Report |
|---|---|
| Language: | English |
| Authors: | Tom Benton, Matthew Carroll, Cambridge University Press and Assessment (United Kingdom) |
| Source: | Cambridge University Press & Assessment. 2025. |
| Availability: | Cambridge University Press & Assessment. Shaftesbury Road Cambridge CB2 8EA. Tel: 44-1223-553311; e-mail: directcs@cambridge.org; Web site: https://www.cambridgeassessment.org.uk/ |
| Peer Reviewed: | N |
| Page Count: | 95 |
| Publication Date: | 2025 |
| Document Type: | Reports - Research |
| Descriptors: | Testing, Equated Scores, Item Response Theory, Statistical Analysis, Evaluation Methods, Simulation, High Stakes Tests, Educational Research, Methods Research |
| Abstract: | What is this report about? This report presents the largest empirical evaluation of test equating methods ever conducted. Equating is the process of identifying scores on different test forms that can be treated as equivalent. Whilst various equating methods are available, there remains no consensus on which are most effective under which conditions. This study aims to provide practical guidance by comparing 69 techniques across 60 scenarios using real data from 1,000 high-stakes tests. What did we do? We conducted a large-scale simulation using item-level data from 1,000 real examinations administered by Cambridge OCR and Cambridge International. Each test was split into two pseudo-forms with shared anchor items, and candidates were split into two groups, one taking each pseudo-form. Equating methods were applied to estimate score equivalencies between forms, and their accuracy was assessed against a criterion equating function derived from the full dataset. The simulation varied three key conditions: sample size (50 to 2,000), anchor test length (18% to 40% test form overlap), and group ability differences (random, medium, and maximum). Accuracy was primarily evaluated using weighted mean absolute error (MAE). Two new methods were developed: odds-transform equating, and transformed linear equating. Both are described in the report and have been made available for practitioners. What did we find? (1) Best-performing methods varied by scenario. For small samples ([less than or equal to]100 candidates per test form), Rasch PCM true score equating (with JML estimation) and odds-transform methods were most effective. For large samples ([greater than or equal to]500 candidates per test form), IRT observed score equating using GRM or GPCM models performed best with medium or large group differences, whilst Frequency Estimation Equipercentile with kernel smoothing performed best with random groups. (2) Classical methods like kernel-smoothed equipercentile equating worked well when group differences were small and sample sizes were large. Circle-arc and odds-transform equating methods, originally designed for small samples, performed surprisingly well even with larger samples and large group differences. (3) Performance of different methods was often broadly similar meaning that although there were clear "best" approaches, if equating has been carried out using other methods, the results should still be acceptable. (4) Several methods were consistently outperformed by at least one other method, so could be excluded from consideration in most cases. These include identity equating, unsmoothed equipercentile equating, and certain IRT variants. (5) For IRT equating, model fit had some impact, but the methods remained robust even when fit indices were below recommended thresholds. What are the implications? This study provides evidence-based recommendations for practitioners selecting equating methods. It narrows the field to a core set of reliable techniques and identifies when each should be used. Odds-transform equating emerges as a promising classical method due to its simplicity, robustness, and performance across diverse scenarios. Furthermore, given good performance in a range of scenarios, equating using the Rasch PCM (ideally with concurrent estimation) could be considered a good "default" approach. Findings highlight the importance of considering sample size and group differences when selecting methods. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | ED678108 |
| Database: | ERIC |
| Abstract: | What is this report about? This report presents the largest empirical evaluation of test equating methods ever conducted. Equating is the process of identifying scores on different test forms that can be treated as equivalent. Whilst various equating methods are available, there remains no consensus on which are most effective under which conditions. This study aims to provide practical guidance by comparing 69 techniques across 60 scenarios using real data from 1,000 high-stakes tests. What did we do? We conducted a large-scale simulation using item-level data from 1,000 real examinations administered by Cambridge OCR and Cambridge International. Each test was split into two pseudo-forms with shared anchor items, and candidates were split into two groups, one taking each pseudo-form. Equating methods were applied to estimate score equivalencies between forms, and their accuracy was assessed against a criterion equating function derived from the full dataset. The simulation varied three key conditions: sample size (50 to 2,000), anchor test length (18% to 40% test form overlap), and group ability differences (random, medium, and maximum). Accuracy was primarily evaluated using weighted mean absolute error (MAE). Two new methods were developed: odds-transform equating, and transformed linear equating. Both are described in the report and have been made available for practitioners. What did we find? (1) Best-performing methods varied by scenario. For small samples ([less than or equal to]100 candidates per test form), Rasch PCM true score equating (with JML estimation) and odds-transform methods were most effective. For large samples ([greater than or equal to]500 candidates per test form), IRT observed score equating using GRM or GPCM models performed best with medium or large group differences, whilst Frequency Estimation Equipercentile with kernel smoothing performed best with random groups. (2) Classical methods like kernel-smoothed equipercentile equating worked well when group differences were small and sample sizes were large. Circle-arc and odds-transform equating methods, originally designed for small samples, performed surprisingly well even with larger samples and large group differences. (3) Performance of different methods was often broadly similar meaning that although there were clear "best" approaches, if equating has been carried out using other methods, the results should still be acceptable. (4) Several methods were consistently outperformed by at least one other method, so could be excluded from consideration in most cases. These include identity equating, unsmoothed equipercentile equating, and certain IRT variants. (5) For IRT equating, model fit had some impact, but the methods remained robust even when fit indices were below recommended thresholds. What are the implications? This study provides evidence-based recommendations for practitioners selecting equating methods. It narrows the field to a core set of reliable techniques and identifies when each should be used. Odds-transform equating emerges as a promising classical method due to its simplicity, robustness, and performance across diverse scenarios. Furthermore, given good performance in a range of scenarios, equating using the Rasch PCM (ideally with concurrent estimation) could be considered a good "default" approach. Findings highlight the importance of considering sample size and group differences when selecting methods. |
|---|
