View in EDS

Embedding Embedded Standard Setting: An Application of Cross-Classified Item Response Theory. CRESST Report 876

Saved in:
Bibliographic Details
Title: Embedding Embedded Standard Setting: An Application of Cross-Classified Item Response Theory. CRESST Report 876
Language: English
Authors: Yun-Kyung Kim, Li Cai, National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
Source: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 2025.
Availability: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 300 Charles E Young Drive N, GSE&IS Building 3rd Floor, Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532; Fax: 310-825-3883; Web site: http://www.cresst.org
Peer Reviewed: N
Page Count: 25
Publication Date: 2025
Sponsoring Agency: Department of Education (ED)
Contract Number: S368A190007
Document Type: Reports - Research
Descriptors: Standard Setting (Scoring), Item Response Theory, Test Items, Difficulty Level, Alternative Assessment, English Learners
Abstract: This paper introduces an application of cross-classified item response theory (IRT) modeling to an assessment utilizing the embedded standard setting (ESS) method (Lewis & Cook). The cross-classified IRT model is used to treat both item and person effects as random, where the item effects are regressed on the target performance levels (target PLs) assigned by item writers. The resulting regression coefficients reflect the alignment between the target PLs and item difficulty levels, a critical factor in evaluating the efficacy of the ESS method. A simulation study confirmed a successful recovery of the regression coefficients. We applied the model to an alternate English language proficiency assessment (Alt ELPA) developed for English learners with the most significant cognitive disabilities. The empirical example showcases the model's ability to provide validity and technical adequacy information for an assessment program serving a particularly small population, under heavy constraints on data collection design and reporting requirements.
Abstractor: As Provided
Entry Date: 2025
Accession Number: ED672695
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED672695
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED672695
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Embedding Embedded Standard Setting: An Application of Cross-Classified Item Response Theory. CRESST Report 876
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Yun-Kyung+Kim%22">Yun-Kyung Kim</searchLink><br /><searchLink fieldCode="AR" term="%22Li+Cai%22">Li Cai</searchLink><br /><searchLink fieldCode="AR" term="%22National+Center+for+Research+on+Evaluation%2C+Standards%2C+and+Student+Testing+%28CRESST%29%22">National Center for Research on Evaluation, Standards, and Student Testing (CRESST)</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22National+Center+for+Research+on+Evaluation%2C+Standards%2C+and+Student+Testing+%28CRESST%29%22"><i>National Center for Research on Evaluation, Standards, and Student Testing (CRESST)</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 300 Charles E Young Drive N, GSE&IS Building 3rd Floor, Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532; Fax: 310-825-3883; Web site: http://www.cresst.org
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 25
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Department of Education (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: S368A190007
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Standard+Setting+%28Scoring%29%22">Standard Setting (Scoring)</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Difficulty+Level%22">Difficulty Level</searchLink><br /><searchLink fieldCode="DE" term="%22Alternative+Assessment%22">Alternative Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22English+Learners%22">English Learners</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper introduces an application of cross-classified item response theory (IRT) modeling to an assessment utilizing the embedded standard setting (ESS) method (Lewis & Cook). The cross-classified IRT model is used to treat both item and person effects as random, where the item effects are regressed on the target performance levels (target PLs) assigned by item writers. The resulting regression coefficients reflect the alignment between the target PLs and item difficulty levels, a critical factor in evaluating the efficacy of the ESS method. A simulation study confirmed a successful recovery of the regression coefficients. We applied the model to an alternate English language proficiency assessment (Alt ELPA) developed for English learners with the most significant cognitive disabilities. The empirical example showcases the model's ability to provide validity and technical adequacy information for an assessment program serving a particularly small population, under heavy constraints on data collection design and reporting requirements.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED672695
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED672695
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 25
    Subjects:
      – SubjectFull: Standard Setting (Scoring)
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Difficulty Level
        Type: general
      – SubjectFull: Alternative Assessment
        Type: general
      – SubjectFull: English Learners
        Type: general
    Titles:
      – TitleFull: Embedding Embedded Standard Setting: An Application of Cross-Classified Item Response Theory. CRESST Report 876
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
      – PersonEntity:
          Name:
            NameFull: Yun-Kyung Kim
      – PersonEntity:
          Name:
            NameFull: Li Cai
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Titles:
            – TitleFull: National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
              Type: main
ResultId 1