View in EDS

Reassessing Weights in Large-Scale Assessments and Multilevel Models

Saved in:
Bibliographic Details
Title: Reassessing Weights in Large-Scale Assessments and Multilevel Models
Language: English
Authors: Umut Atasever, Francis L. Huang, Leslie Rutkowski
Source: Large-scale Assessments in Education. 2025 13.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 27
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Secondary Education
Secondary Education
Descriptors: Mathematics Tests, International Assessment, Elementary Secondary Education, Foreign Countries, Mathematics Achievement, Science Tests, Science Achievement, Achievement Tests, Secondary School Students, Sampling, Probability, Statistical Bias, Scaling, Hierarchical Linear Modeling, Monte Carlo Methods
Assessment and Survey Identifiers: Trends in International Mathematics and Science Study, Program for International Student Assessment
DOI: 10.1186/s40536-025-00245-y
ISSN: 2196-0739
Abstract: When analyzing large-scale assessments (LSAs) that use complex sampling designs, it is important to account for probability sampling using weights. However, the use of these weights in multilevel models has been widely debated, particularly regarding their application at different levels of the model. Yet, no consensus has been reached on the best method to apply weights. To address this, we conducted a Monte Carlo simulation, modeling a typical LSA population with known true values for the variables of interest. Using repeated sampling from this population, we generated weights using a stratified two-stage cluster design, where clusters (schools) were selected using probability proportional to size (PPS) sampling from designated explicit strata. We examined both class-level and student-level sampling structures and applied a nonresponse model at both the school and student levels. For each sample drawn, we assessed bias and coverage rates across models that applied weights at two levels, only at level 2, only at level 1, and without weights. Our findings show that applying only level-2 weights produced the most precise estimates, while models with no weights or only rescaled level-1 weights led to the highest bias. Using both level-1 and level-2 weights together was acceptable, although variance components were slightly underestimated. However, scaling level-1 weights would mirror using only the level-2 weights in datasets where there is no variation of weights within clusters. An applied example using TIMSS data supports these findings. This study contributes to the literature by explaining the least biased weight methods with complex sampling scenarios and offering practical guidance on using weights in multilevel models. We provide the R syntax for both the simulation and the applied example for reproducibility.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1464974
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1464974
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Reassessing Weights in Large-Scale Assessments and Multilevel Models
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Umut+Atasever%22">Umut Atasever</searchLink><br /><searchLink fieldCode="AR" term="%22Francis+L%2E+Huang%22">Francis L. Huang</searchLink><br /><searchLink fieldCode="AR" term="%22Leslie+Rutkowski%22">Leslie Rutkowski</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Large-scale+Assessments+in+Education%22"><i>Large-scale Assessments in Education</i></searchLink>. 2025 13.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 27
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Tests%22">Science Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Achievement%22">Science Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Sampling%22">Sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Probability%22">Probability</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Bias%22">Statistical Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Scaling%22">Scaling</searchLink><br /><searchLink fieldCode="DE" term="%22Hierarchical+Linear+Modeling%22">Hierarchical Linear Modeling</searchLink><br /><searchLink fieldCode="DE" term="%22Monte+Carlo+Methods%22">Monte Carlo Methods</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Trends+in+International+Mathematics+and+Science+Study%22">Trends in International Mathematics and Science Study</searchLink><br /><searchLink fieldCode="SU" term="%22Program+for+International+Student+Assessment%22">Program for International Student Assessment</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1186/s40536-025-00245-y
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2196-0739
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: When analyzing large-scale assessments (LSAs) that use complex sampling designs, it is important to account for probability sampling using weights. However, the use of these weights in multilevel models has been widely debated, particularly regarding their application at different levels of the model. Yet, no consensus has been reached on the best method to apply weights. To address this, we conducted a Monte Carlo simulation, modeling a typical LSA population with known true values for the variables of interest. Using repeated sampling from this population, we generated weights using a stratified two-stage cluster design, where clusters (schools) were selected using probability proportional to size (PPS) sampling from designated explicit strata. We examined both class-level and student-level sampling structures and applied a nonresponse model at both the school and student levels. For each sample drawn, we assessed bias and coverage rates across models that applied weights at two levels, only at level 2, only at level 1, and without weights. Our findings show that applying only level-2 weights produced the most precise estimates, while models with no weights or only rescaled level-1 weights led to the highest bias. Using both level-1 and level-2 weights together was acceptable, although variance components were slightly underestimated. However, scaling level-1 weights would mirror using only the level-2 weights in datasets where there is no variation of weights within clusters. An applied example using TIMSS data supports these findings. This study contributes to the literature by explaining the least biased weight methods with complex sampling scenarios and offering practical guidance on using weights in multilevel models. We provide the R syntax for both the simulation and the applied example for reproducibility.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1464974
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1464974
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1186/s40536-025-00245-y
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 27
    Subjects:
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: Science Tests
        Type: general
      – SubjectFull: Science Achievement
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Sampling
        Type: general
      – SubjectFull: Probability
        Type: general
      – SubjectFull: Statistical Bias
        Type: general
      – SubjectFull: Scaling
        Type: general
      – SubjectFull: Hierarchical Linear Modeling
        Type: general
      – SubjectFull: Monte Carlo Methods
        Type: general
      – SubjectFull: Trends in International Mathematics and Science Study
        Type: general
      – SubjectFull: Program for International Student Assessment
        Type: general
    Titles:
      – TitleFull: Reassessing Weights in Large-Scale Assessments and Multilevel Models
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Umut Atasever
      – PersonEntity:
          Name:
            NameFull: Francis L. Huang
      – PersonEntity:
          Name:
            NameFull: Leslie Rutkowski
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 2196-0739
          Numbering:
            – Type: volume
              Value: 13
          Titles:
            – TitleFull: Large-scale Assessments in Education
              Type: main
ResultId 1