View in EDS

An Examination of the Performance of Variance Estimators in International Large-Scale Assessments. Final Report

Saved in:
Bibliographic Details
Title: An Examination of the Performance of Variance Estimators in International Large-Scale Assessments. Final Report
Language: English
Authors: Umut Atasever, Sabine Meinck, Diego Cortes, International Association for the Evaluation of Educational Achievement (IEA) (Netherlands)
Source: International Association for the Evaluation of Educational Achievement. 2025.
Availability: International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands. Tel: +31-20-625-3625; Fax: +31-20-420-7136; e-mail: department@iea.nl; Web site: http://www.iea.nl
Peer Reviewed: N
Page Count: 93
Publication Date: 2025
Document Type: Reports - Research
Education Level: Elementary Secondary Education
Descriptors: Achievement Tests, Elementary Secondary Education, Foreign Countries, International Assessment, Research Methodology, Statistical Analysis, Error of Measurement, Sampling, Monte Carlo Methods, Statistical Inference, Test Bias, Research Problems, Benchmarking, Replication (Evaluation)
Abstract: The primary objective of this study is to examine the performance of the most widely used sampling variance estimators in international large-scale assessments (ILSAs): the "Balanced Repeated Replication" (BRR) method and the "Jackknife Repeated Replication" (JK2) method (both "Half" and "Full" variants). Additionally, we investigate the impact of Fay modification factors (10%, 30%, and 50%) on both BRR and JK2 methods in estimating sampling variance. We also evaluate Bootstrapping as an alternative variance estimator for clustered sampling in ILSAs. Aiming for comprehensive results, we examine different conditions reflecting common scenarios in ILSAs. We vary school sample sizes, namely looking at samples of 30, 50, 100, and 150 schools. The study further explores the treatment of Primary Sampling Units (PSUs) in variance strata formation under school non-response scenarios, and the occurrence of odd numbers of PSUs in given explicit strata. A Monte Carlo simulation is conducted, mirroring a TIMSS Grade 4 student population with realistic distributions of achievement scores, standard deviation, intraclass correlation coefficient (ICC), and background characteristics. Probability samples are repeatedly drawn following a two-stage stratified cluster sampling design, where schools are PSUs and classes are secondary sampling units. One thousand samples are selected for each sample scenario. For each sample, the population parameter of interest is estimated, and for each sample scenario, its sampling variance is investigated. The "true" sampling error is approximated from the variability of estimates across the sample iterations as the standard deviation of the sampling distribution. The performance of each variance estimator is assessed by comparing the respective estimated sampling error, computed as the average of the square roots of the estimated sampling variances across all samples, to the (approximated) true sampling error. Results are summarized using "Relative Bias" to assess accuracy by quantifying over- or under-estimation and "Stability" to measure precision. Results indicate that "Bootstrapping", "JK2" and "BRR without Fay" modification yield the most accurate sampling variance estimates for smooth statistics such as means, with JK2 demonstrating the highest precision across conditions. For nonsmooth statistics such as percentiles, "Bootstrapping" and "BRR" outperform "JK2" in terms of accuracy and precision. Notably, the "JK2 Half" and "Full" variants perform virtually identically in terms of variance estimation accuracy and precision. "Fay-modified JK2" does not improve estimation precision, while "BRR with a Fay factor" improves performance for non-smooth statistics, particularly at 10% and 30%, compared to the traditional 50% "Fay factor." Under non-response conditions, specific methods currently in use for contemporary large-scale assessments consistently under- or overestimate sampling variance, suggesting that unadjusted variance strata introduce systematic bias. This study advances discussions on variance estimation in ILSAs, offering insights into the optimal application of resampling methods, including the trade-offs of "BRR", "JK2" and "Bootstrapping" under varying statistical conditions and sampling designs.
Abstractor: As Provided
Entry Date: 2026
Accession Number: ED679606
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED679606
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED679606
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: An Examination of the Performance of Variance Estimators in International Large-Scale Assessments. Final Report
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Umut+Atasever%22">Umut Atasever</searchLink><br /><searchLink fieldCode="AR" term="%22Sabine+Meinck%22">Sabine Meinck</searchLink><br /><searchLink fieldCode="AR" term="%22Diego+Cortes%22">Diego Cortes</searchLink><br /><searchLink fieldCode="AR" term="%22International+Association+for+the+Evaluation+of+Educational+Achievement+%28IEA%29+%28Netherlands%29%22">International Association for the Evaluation of Educational Achievement (IEA) (Netherlands)</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Association+for+the+Evaluation+of+Educational+Achievement%22"><i>International Association for the Evaluation of Educational Achievement</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands. Tel: +31-20-625-3625; Fax: +31-20-420-7136; e-mail: department@iea.nl; Web site: http://www.iea.nl
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 93
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Error+of+Measurement%22">Error of Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Sampling%22">Sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Monte+Carlo+Methods%22">Monte Carlo Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Inference%22">Statistical Inference</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Problems%22">Research Problems</searchLink><br /><searchLink fieldCode="DE" term="%22Benchmarking%22">Benchmarking</searchLink><br /><searchLink fieldCode="DE" term="%22Replication+%28Evaluation%29%22">Replication (Evaluation)</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The primary objective of this study is to examine the performance of the most widely used sampling variance estimators in international large-scale assessments (ILSAs): the "Balanced Repeated Replication" (BRR) method and the "Jackknife Repeated Replication" (JK2) method (both "Half" and "Full" variants). Additionally, we investigate the impact of Fay modification factors (10%, 30%, and 50%) on both BRR and JK2 methods in estimating sampling variance. We also evaluate Bootstrapping as an alternative variance estimator for clustered sampling in ILSAs. Aiming for comprehensive results, we examine different conditions reflecting common scenarios in ILSAs. We vary school sample sizes, namely looking at samples of 30, 50, 100, and 150 schools. The study further explores the treatment of Primary Sampling Units (PSUs) in variance strata formation under school non-response scenarios, and the occurrence of odd numbers of PSUs in given explicit strata. A Monte Carlo simulation is conducted, mirroring a TIMSS Grade 4 student population with realistic distributions of achievement scores, standard deviation, intraclass correlation coefficient (ICC), and background characteristics. Probability samples are repeatedly drawn following a two-stage stratified cluster sampling design, where schools are PSUs and classes are secondary sampling units. One thousand samples are selected for each sample scenario. For each sample, the population parameter of interest is estimated, and for each sample scenario, its sampling variance is investigated. The "true" sampling error is approximated from the variability of estimates across the sample iterations as the standard deviation of the sampling distribution. The performance of each variance estimator is assessed by comparing the respective estimated sampling error, computed as the average of the square roots of the estimated sampling variances across all samples, to the (approximated) true sampling error. Results are summarized using "Relative Bias" to assess accuracy by quantifying over- or under-estimation and "Stability" to measure precision. Results indicate that "Bootstrapping", "JK2" and "BRR without Fay" modification yield the most accurate sampling variance estimates for smooth statistics such as means, with JK2 demonstrating the highest precision across conditions. For nonsmooth statistics such as percentiles, "Bootstrapping" and "BRR" outperform "JK2" in terms of accuracy and precision. Notably, the "JK2 Half" and "Full" variants perform virtually identically in terms of variance estimation accuracy and precision. "Fay-modified JK2" does not improve estimation precision, while "BRR with a Fay factor" improves performance for non-smooth statistics, particularly at 10% and 30%, compared to the traditional 50% "Fay factor." Under non-response conditions, specific methods currently in use for contemporary large-scale assessments consistently under- or overestimate sampling variance, suggesting that unadjusted variance strata introduce systematic bias. This study advances discussions on variance estimation in ILSAs, offering insights into the optimal application of resampling methods, including the trade-offs of "BRR", "JK2" and "Bootstrapping" under varying statistical conditions and sampling designs.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED679606
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED679606
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 93
    Subjects:
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Error of Measurement
        Type: general
      – SubjectFull: Sampling
        Type: general
      – SubjectFull: Monte Carlo Methods
        Type: general
      – SubjectFull: Statistical Inference
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: Research Problems
        Type: general
      – SubjectFull: Benchmarking
        Type: general
      – SubjectFull: Replication (Evaluation)
        Type: general
    Titles:
      – TitleFull: An Examination of the Performance of Variance Estimators in International Large-Scale Assessments. Final Report
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: International Association for the Evaluation of Educational Achievement (IEA) (Netherlands)
      – PersonEntity:
          Name:
            NameFull: Umut Atasever
      – PersonEntity:
          Name:
            NameFull: Sabine Meinck
      – PersonEntity:
          Name:
            NameFull: Diego Cortes
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 07
              M: 07
              Type: published
              Y: 2025
          Titles:
            – TitleFull: International Association for the Evaluation of Educational Achievement
              Type: main
ResultId 1