An Examination of the Performance of Variance Estimators in International Large-Scale Assessments. Final Report
Saved in:
| Title: | An Examination of the Performance of Variance Estimators in International Large-Scale Assessments. Final Report |
|---|---|
| Language: | English |
| Authors: | Umut Atasever, Sabine Meinck, Diego Cortes, International Association for the Evaluation of Educational Achievement (IEA) (Netherlands) |
| Source: | International Association for the Evaluation of Educational Achievement. 2025. |
| Availability: | International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands. Tel: +31-20-625-3625; Fax: +31-20-420-7136; e-mail: department@iea.nl; Web site: http://www.iea.nl |
| Peer Reviewed: | N |
| Page Count: | 93 |
| Publication Date: | 2025 |
| Document Type: | Reports - Research |
| Education Level: | Elementary Secondary Education |
| Descriptors: | Achievement Tests, Elementary Secondary Education, Foreign Countries, International Assessment, Research Methodology, Statistical Analysis, Error of Measurement, Sampling, Monte Carlo Methods, Statistical Inference, Test Bias, Research Problems, Benchmarking, Replication (Evaluation) |
| Abstract: | The primary objective of this study is to examine the performance of the most widely used sampling variance estimators in international large-scale assessments (ILSAs): the "Balanced Repeated Replication" (BRR) method and the "Jackknife Repeated Replication" (JK2) method (both "Half" and "Full" variants). Additionally, we investigate the impact of Fay modification factors (10%, 30%, and 50%) on both BRR and JK2 methods in estimating sampling variance. We also evaluate Bootstrapping as an alternative variance estimator for clustered sampling in ILSAs. Aiming for comprehensive results, we examine different conditions reflecting common scenarios in ILSAs. We vary school sample sizes, namely looking at samples of 30, 50, 100, and 150 schools. The study further explores the treatment of Primary Sampling Units (PSUs) in variance strata formation under school non-response scenarios, and the occurrence of odd numbers of PSUs in given explicit strata. A Monte Carlo simulation is conducted, mirroring a TIMSS Grade 4 student population with realistic distributions of achievement scores, standard deviation, intraclass correlation coefficient (ICC), and background characteristics. Probability samples are repeatedly drawn following a two-stage stratified cluster sampling design, where schools are PSUs and classes are secondary sampling units. One thousand samples are selected for each sample scenario. For each sample, the population parameter of interest is estimated, and for each sample scenario, its sampling variance is investigated. The "true" sampling error is approximated from the variability of estimates across the sample iterations as the standard deviation of the sampling distribution. The performance of each variance estimator is assessed by comparing the respective estimated sampling error, computed as the average of the square roots of the estimated sampling variances across all samples, to the (approximated) true sampling error. Results are summarized using "Relative Bias" to assess accuracy by quantifying over- or under-estimation and "Stability" to measure precision. Results indicate that "Bootstrapping", "JK2" and "BRR without Fay" modification yield the most accurate sampling variance estimates for smooth statistics such as means, with JK2 demonstrating the highest precision across conditions. For nonsmooth statistics such as percentiles, "Bootstrapping" and "BRR" outperform "JK2" in terms of accuracy and precision. Notably, the "JK2 Half" and "Full" variants perform virtually identically in terms of variance estimation accuracy and precision. "Fay-modified JK2" does not improve estimation precision, while "BRR with a Fay factor" improves performance for non-smooth statistics, particularly at 10% and 30%, compared to the traditional 50% "Fay factor." Under non-response conditions, specific methods currently in use for contemporary large-scale assessments consistently under- or overestimate sampling variance, suggesting that unadjusted variance strata introduce systematic bias. This study advances discussions on variance estimation in ILSAs, offering insights into the optimal application of resampling methods, including the trade-offs of "BRR", "JK2" and "Bootstrapping" under varying statistical conditions and sampling designs. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | ED679606 |
| Database: | ERIC |
| Abstract: | The primary objective of this study is to examine the performance of the most widely used sampling variance estimators in international large-scale assessments (ILSAs): the "Balanced Repeated Replication" (BRR) method and the "Jackknife Repeated Replication" (JK2) method (both "Half" and "Full" variants). Additionally, we investigate the impact of Fay modification factors (10%, 30%, and 50%) on both BRR and JK2 methods in estimating sampling variance. We also evaluate Bootstrapping as an alternative variance estimator for clustered sampling in ILSAs. Aiming for comprehensive results, we examine different conditions reflecting common scenarios in ILSAs. We vary school sample sizes, namely looking at samples of 30, 50, 100, and 150 schools. The study further explores the treatment of Primary Sampling Units (PSUs) in variance strata formation under school non-response scenarios, and the occurrence of odd numbers of PSUs in given explicit strata. A Monte Carlo simulation is conducted, mirroring a TIMSS Grade 4 student population with realistic distributions of achievement scores, standard deviation, intraclass correlation coefficient (ICC), and background characteristics. Probability samples are repeatedly drawn following a two-stage stratified cluster sampling design, where schools are PSUs and classes are secondary sampling units. One thousand samples are selected for each sample scenario. For each sample, the population parameter of interest is estimated, and for each sample scenario, its sampling variance is investigated. The "true" sampling error is approximated from the variability of estimates across the sample iterations as the standard deviation of the sampling distribution. The performance of each variance estimator is assessed by comparing the respective estimated sampling error, computed as the average of the square roots of the estimated sampling variances across all samples, to the (approximated) true sampling error. Results are summarized using "Relative Bias" to assess accuracy by quantifying over- or under-estimation and "Stability" to measure precision. Results indicate that "Bootstrapping", "JK2" and "BRR without Fay" modification yield the most accurate sampling variance estimates for smooth statistics such as means, with JK2 demonstrating the highest precision across conditions. For nonsmooth statistics such as percentiles, "Bootstrapping" and "BRR" outperform "JK2" in terms of accuracy and precision. Notably, the "JK2 Half" and "Full" variants perform virtually identically in terms of variance estimation accuracy and precision. "Fay-modified JK2" does not improve estimation precision, while "BRR with a Fay factor" improves performance for non-smooth statistics, particularly at 10% and 30%, compared to the traditional 50% "Fay factor." Under non-response conditions, specific methods currently in use for contemporary large-scale assessments consistently under- or overestimate sampling variance, suggesting that unadjusted variance strata introduce systematic bias. This study advances discussions on variance estimation in ILSAs, offering insights into the optimal application of resampling methods, including the trade-offs of "BRR", "JK2" and "Bootstrapping" under varying statistical conditions and sampling designs. |
|---|
