View in EDS

The Biggest Equating Study in the World… Ever. Research Report

Saved in:
Bibliographic Details
Title: The Biggest Equating Study in the World… Ever. Research Report
Language: English
Authors: Tom Benton, Matthew Carroll, Cambridge University Press and Assessment (United Kingdom)
Source: Cambridge University Press & Assessment. 2025.
Availability: Cambridge University Press & Assessment. Shaftesbury Road Cambridge CB2 8EA. Tel: 44-1223-553311; e-mail: directcs@cambridge.org; Web site: https://www.cambridgeassessment.org.uk/
Peer Reviewed: N
Page Count: 95
Publication Date: 2025
Document Type: Reports - Research
Descriptors: Testing, Equated Scores, Item Response Theory, Statistical Analysis, Evaluation Methods, Simulation, High Stakes Tests, Educational Research, Methods Research
Abstract: What is this report about? This report presents the largest empirical evaluation of test equating methods ever conducted. Equating is the process of identifying scores on different test forms that can be treated as equivalent. Whilst various equating methods are available, there remains no consensus on which are most effective under which conditions. This study aims to provide practical guidance by comparing 69 techniques across 60 scenarios using real data from 1,000 high-stakes tests. What did we do? We conducted a large-scale simulation using item-level data from 1,000 real examinations administered by Cambridge OCR and Cambridge International. Each test was split into two pseudo-forms with shared anchor items, and candidates were split into two groups, one taking each pseudo-form. Equating methods were applied to estimate score equivalencies between forms, and their accuracy was assessed against a criterion equating function derived from the full dataset. The simulation varied three key conditions: sample size (50 to 2,000), anchor test length (18% to 40% test form overlap), and group ability differences (random, medium, and maximum). Accuracy was primarily evaluated using weighted mean absolute error (MAE). Two new methods were developed: odds-transform equating, and transformed linear equating. Both are described in the report and have been made available for practitioners. What did we find? (1) Best-performing methods varied by scenario. For small samples ([less than or equal to]100 candidates per test form), Rasch PCM true score equating (with JML estimation) and odds-transform methods were most effective. For large samples ([greater than or equal to]500 candidates per test form), IRT observed score equating using GRM or GPCM models performed best with medium or large group differences, whilst Frequency Estimation Equipercentile with kernel smoothing performed best with random groups. (2) Classical methods like kernel-smoothed equipercentile equating worked well when group differences were small and sample sizes were large. Circle-arc and odds-transform equating methods, originally designed for small samples, performed surprisingly well even with larger samples and large group differences. (3) Performance of different methods was often broadly similar meaning that although there were clear "best" approaches, if equating has been carried out using other methods, the results should still be acceptable. (4) Several methods were consistently outperformed by at least one other method, so could be excluded from consideration in most cases. These include identity equating, unsmoothed equipercentile equating, and certain IRT variants. (5) For IRT equating, model fit had some impact, but the methods remained robust even when fit indices were below recommended thresholds. What are the implications? This study provides evidence-based recommendations for practitioners selecting equating methods. It narrows the field to a core set of reliable techniques and identifies when each should be used. Odds-transform equating emerges as a promising classical method due to its simplicity, robustness, and performance across diverse scenarios. Furthermore, given good performance in a range of scenarios, equating using the Rasch PCM (ideally with concurrent estimation) could be considered a good "default" approach. Findings highlight the importance of considering sample size and group differences when selecting methods.
Abstractor: As Provided
Entry Date: 2026
Accession Number: ED678108
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED678108
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED678108
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Biggest Equating Study in the World… Ever. Research Report
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Tom+Benton%22">Tom Benton</searchLink><br /><searchLink fieldCode="AR" term="%22Matthew+Carroll%22">Matthew Carroll</searchLink><br /><searchLink fieldCode="AR" term="%22Cambridge+University+Press+and+Assessment+%28United+Kingdom%29%22">Cambridge University Press and Assessment (United Kingdom)</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Cambridge+University+Press+%26+Assessment%22"><i>Cambridge University Press & Assessment</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Cambridge University Press & Assessment. Shaftesbury Road Cambridge CB2 8EA. Tel: 44-1223-553311; e-mail: directcs@cambridge.org; Web site: https://www.cambridgeassessment.org.uk/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 95
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Testing%22">Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Equated+Scores%22">Equated Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Simulation%22">Simulation</searchLink><br /><searchLink fieldCode="DE" term="%22High+Stakes+Tests%22">High Stakes Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Methods+Research%22">Methods Research</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: What is this report about? This report presents the largest empirical evaluation of test equating methods ever conducted. Equating is the process of identifying scores on different test forms that can be treated as equivalent. Whilst various equating methods are available, there remains no consensus on which are most effective under which conditions. This study aims to provide practical guidance by comparing 69 techniques across 60 scenarios using real data from 1,000 high-stakes tests. What did we do? We conducted a large-scale simulation using item-level data from 1,000 real examinations administered by Cambridge OCR and Cambridge International. Each test was split into two pseudo-forms with shared anchor items, and candidates were split into two groups, one taking each pseudo-form. Equating methods were applied to estimate score equivalencies between forms, and their accuracy was assessed against a criterion equating function derived from the full dataset. The simulation varied three key conditions: sample size (50 to 2,000), anchor test length (18% to 40% test form overlap), and group ability differences (random, medium, and maximum). Accuracy was primarily evaluated using weighted mean absolute error (MAE). Two new methods were developed: odds-transform equating, and transformed linear equating. Both are described in the report and have been made available for practitioners. What did we find? (1) Best-performing methods varied by scenario. For small samples ([less than or equal to]100 candidates per test form), Rasch PCM true score equating (with JML estimation) and odds-transform methods were most effective. For large samples ([greater than or equal to]500 candidates per test form), IRT observed score equating using GRM or GPCM models performed best with medium or large group differences, whilst Frequency Estimation Equipercentile with kernel smoothing performed best with random groups. (2) Classical methods like kernel-smoothed equipercentile equating worked well when group differences were small and sample sizes were large. Circle-arc and odds-transform equating methods, originally designed for small samples, performed surprisingly well even with larger samples and large group differences. (3) Performance of different methods was often broadly similar meaning that although there were clear "best" approaches, if equating has been carried out using other methods, the results should still be acceptable. (4) Several methods were consistently outperformed by at least one other method, so could be excluded from consideration in most cases. These include identity equating, unsmoothed equipercentile equating, and certain IRT variants. (5) For IRT equating, model fit had some impact, but the methods remained robust even when fit indices were below recommended thresholds. What are the implications? This study provides evidence-based recommendations for practitioners selecting equating methods. It narrows the field to a core set of reliable techniques and identifies when each should be used. Odds-transform equating emerges as a promising classical method due to its simplicity, robustness, and performance across diverse scenarios. Furthermore, given good performance in a range of scenarios, equating using the Rasch PCM (ideally with concurrent estimation) could be considered a good "default" approach. Findings highlight the importance of considering sample size and group differences when selecting methods.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED678108
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED678108
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 95
    Subjects:
      – SubjectFull: Testing
        Type: general
      – SubjectFull: Equated Scores
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Simulation
        Type: general
      – SubjectFull: High Stakes Tests
        Type: general
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Methods Research
        Type: general
    Titles:
      – TitleFull: The Biggest Equating Study in the World… Ever. Research Report
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Cambridge University Press and Assessment (United Kingdom)
      – PersonEntity:
          Name:
            NameFull: Tom Benton
      – PersonEntity:
          Name:
            NameFull: Matthew Carroll
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Cambridge University Press & Assessment
              Type: main
ResultId 1