View in EDS

Evaluating Uncertainty: The Impact of the Sampling and Assessment Design on Statistical Inference in the Context of ILSA

Saved in:
Bibliographic Details
Title: Evaluating Uncertainty: The Impact of the Sampling and Assessment Design on Statistical Inference in the Context of ILSA
Language: English
Authors: Diego Cortes (ORCID 0009-0001-3501-7676), Dirk Hastedt, Sabine Meinck
Source: Large-scale Assessments in Education. 2025 13.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 21
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Education
Grade 4
Intermediate Grades
Descriptors: Sampling, Research Design, Educational Assessment, Statistical Inference, International Assessment, Misconceptions, Achievement Tests, Foreign Countries, Grade 4, Reading Tests, Reading Achievement
Assessment and Survey Identifiers: Progress in International Reading Literacy Study
DOI: 10.1186/s40536-025-00246-x
ISSN: 2196-0739
Abstract: This paper informs users of data collected in international large-scale assessments (ILSA), by presenting argumentsunderlining the importance of considering two design features employed in these studies. We examine a commonmisconception stating that the uncertainty arising from the assessment design is negligible compared with that arisingfrom the sampling design. This misconception can lead to the erroneous conclusion that there is always a relatively lowrisk of ignoring the uncertainty arising from the assessment design when reporting estimates of population parameters. We use the design effect framework to assess the impact that the sampling and the assessment design have on theestimation. We first evaluate the loss in efficiency in the estimation of a population parameter attributable to each ofthe designs. We then examine whether knowledge about the effect of one design feature can justify any belief about theeffect of the other design feature. We repeat this examination across different parameters characterizing theachievement distribution in a population. We provide empirical results using data collected for PIRLS 2016. Our empirical results can be summarized in two general findings. First, when estimating mean achievement, the effectof the sampling design is often substantially larger than that of the assessment design. This finding might explain themisconception we try to address. However, we show that this is not true in all instances, and the magnitude of thedifference between both design effects is context dependent and hence not generalizable. Second, differences in designeffects become less predictable when estimating other parameters, e.g. the proportion of students reaching a certainthreshold in the achievement scale (i.e., benchmarks), or an association estimated using linear regression. This contribution underlines that accounting for all sources of uncertainty in the estimation is of paramount importanceto obtain credible inferences. We conclude that it is difficult to justify a priori the belief that the effect of the samplingdesign in the estimation is always greater than that of the assessment design.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1469000
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1469000
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evaluating Uncertainty: The Impact of the Sampling and Assessment Design on Statistical Inference in the Context of ILSA
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Diego+Cortes%22">Diego Cortes</searchLink> (ORCID <externalLink term="http://orcid.org/0009-0001-3501-7676">0009-0001-3501-7676</externalLink>)<br /><searchLink fieldCode="AR" term="%22Dirk+Hastedt%22">Dirk Hastedt</searchLink><br /><searchLink fieldCode="AR" term="%22Sabine+Meinck%22">Sabine Meinck</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Large-scale+Assessments+in+Education%22"><i>Large-scale Assessments in Education</i></searchLink>. 2025 13.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 21
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="EL" term="%22Intermediate+Grades%22">Intermediate Grades</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Sampling%22">Sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Design%22">Research Design</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Assessment%22">Educational Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Inference%22">Statistical Inference</searchLink><br /><searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Misconceptions%22">Misconceptions</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Tests%22">Reading Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Achievement%22">Reading Achievement</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Progress+in+International+Reading+Literacy+Study%22">Progress in International Reading Literacy Study</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1186/s40536-025-00246-x
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2196-0739
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper informs users of data collected in international large-scale assessments (ILSA), by presenting argumentsunderlining the importance of considering two design features employed in these studies. We examine a commonmisconception stating that the uncertainty arising from the assessment design is negligible compared with that arisingfrom the sampling design. This misconception can lead to the erroneous conclusion that there is always a relatively lowrisk of ignoring the uncertainty arising from the assessment design when reporting estimates of population parameters. We use the design effect framework to assess the impact that the sampling and the assessment design have on theestimation. We first evaluate the loss in efficiency in the estimation of a population parameter attributable to each ofthe designs. We then examine whether knowledge about the effect of one design feature can justify any belief about theeffect of the other design feature. We repeat this examination across different parameters characterizing theachievement distribution in a population. We provide empirical results using data collected for PIRLS 2016. Our empirical results can be summarized in two general findings. First, when estimating mean achievement, the effectof the sampling design is often substantially larger than that of the assessment design. This finding might explain themisconception we try to address. However, we show that this is not true in all instances, and the magnitude of thedifference between both design effects is context dependent and hence not generalizable. Second, differences in designeffects become less predictable when estimating other parameters, e.g. the proportion of students reaching a certainthreshold in the achievement scale (i.e., benchmarks), or an association estimated using linear regression. This contribution underlines that accounting for all sources of uncertainty in the estimation is of paramount importanceto obtain credible inferences. We conclude that it is difficult to justify a priori the belief that the effect of the samplingdesign in the estimation is always greater than that of the assessment design.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1469000
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1469000
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1186/s40536-025-00246-x
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 21
    Subjects:
      – SubjectFull: Sampling
        Type: general
      – SubjectFull: Research Design
        Type: general
      – SubjectFull: Educational Assessment
        Type: general
      – SubjectFull: Statistical Inference
        Type: general
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Misconceptions
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Grade 4
        Type: general
      – SubjectFull: Reading Tests
        Type: general
      – SubjectFull: Reading Achievement
        Type: general
      – SubjectFull: Progress in International Reading Literacy Study
        Type: general
    Titles:
      – TitleFull: Evaluating Uncertainty: The Impact of the Sampling and Assessment Design on Statistical Inference in the Context of ILSA
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Diego Cortes
      – PersonEntity:
          Name:
            NameFull: Dirk Hastedt
      – PersonEntity:
          Name:
            NameFull: Sabine Meinck
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 2196-0739
          Numbering:
            – Type: volume
              Value: 13
          Titles:
            – TitleFull: Large-scale Assessments in Education
              Type: main
ResultId 1