View in EDS

Empirical vs. Expected IRT-Based Reliability Estimation in Computerized Multistage Testing (MST)

Saved in:
Bibliographic Details
Title: Empirical vs. Expected IRT-Based Reliability Estimation in Computerized Multistage Testing (MST)
Language: English
Authors: Zhang, Yanwei, Breithaupt, Krista, Tessema, Aster, Chuah, David
Source: Online Submission. 2006Paper presented at the Annual Conference of the National Council of Measurement in Education (San Francisco, CA, Apr 2006).
Peer Reviewed: N
Page Count: 21
Publication Date: 2006
Document Type: Reports - Evaluative
Speeches/Meeting Papers
Descriptors: Individual Testing, Test Reliability, Programming, Error of Measurement, Programming Languages, Adaptive Testing, Scores
Abstract: Two IRT-based procedures to estimate test reliability for a certification exam that used both adaptive (via a MST model) and non-adaptive design were considered in this study. Both procedures rely on calibrated item parameters to estimate error variance. In terms of score variance, one procedure (Method 1) uses the empirical ability distribution from a particular sample of examinees, and the other procedure (Method 2) assumes a normal distribution of ability and is sample-free. Due to the problem of sampling restriction in adaptive tests, Method 1 was modified (Method 1 extension) to "beef up" the sample and estimate reliability for each testlet in a MST panel before aggregating the estimates into an overall estimate for a test form or route. Overall, results imply that Method 1 and Method 2 tend to produce similar results for both adaptive and non-adaptive tests on the panel level and the test section level. Method 1 should not be applied to individual test forms in adaptive tests by a MST design. In the latter case, the modified or extended procedure can be used to alleviate the problem of restricted sample. The algorithms of the discussed procedures can be implemented in common statistical programming language such as SAS and SPSS as flexible alternatives to the theoretical and empirical reliability estimates computed, for example, by the BILOG-MG software. (Contains 6 tables and 1 figure.)
Abstractor: Author
Number of References: 7
Entry Date: 2007
Accession Number: ED498534
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED498534
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED498534
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Empirical vs. Expected IRT-Based Reliability Estimation in Computerized Multistage Testing (MST)
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Zhang%2C+Yanwei%22">Zhang, Yanwei</searchLink><br /><searchLink fieldCode="AR" term="%22Breithaupt%2C+Krista%22">Breithaupt, Krista</searchLink><br /><searchLink fieldCode="AR" term="%22Tessema%2C+Aster%22">Tessema, Aster</searchLink><br /><searchLink fieldCode="AR" term="%22Chuah%2C+David%22">Chuah, David</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Online+Submission%22"><i>Online Submission</i></searchLink>. 2006Paper presented at the Annual Conference of the National Council of Measurement in Education (San Francisco, CA, Apr 2006).
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 21
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2006
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Evaluative<br />Speeches/Meeting Papers
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Individual+Testing%22">Individual Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Programming%22">Programming</searchLink><br /><searchLink fieldCode="DE" term="%22Error+of+Measurement%22">Error of Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Programming+Languages%22">Programming Languages</searchLink><br /><searchLink fieldCode="DE" term="%22Adaptive+Testing%22">Adaptive Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Two IRT-based procedures to estimate test reliability for a certification exam that used both adaptive (via a MST model) and non-adaptive design were considered in this study. Both procedures rely on calibrated item parameters to estimate error variance. In terms of score variance, one procedure (Method 1) uses the empirical ability distribution from a particular sample of examinees, and the other procedure (Method 2) assumes a normal distribution of ability and is sample-free. Due to the problem of sampling restriction in adaptive tests, Method 1 was modified (Method 1 extension) to "beef up" the sample and estimate reliability for each testlet in a MST panel before aggregating the estimates into an overall estimate for a test form or route. Overall, results imply that Method 1 and Method 2 tend to produce similar results for both adaptive and non-adaptive tests on the panel level and the test section level. Method 1 should not be applied to individual test forms in adaptive tests by a MST design. In the latter case, the modified or extended procedure can be used to alleviate the problem of restricted sample. The algorithms of the discussed procedures can be implemented in common statistical programming language such as SAS and SPSS as flexible alternatives to the theoretical and empirical reliability estimates computed, for example, by the BILOG-MG software. (Contains 6 tables and 1 figure.)
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: Author
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 7
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2007
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED498534
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED498534
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 21
    Subjects:
      – SubjectFull: Individual Testing
        Type: general
      – SubjectFull: Test Reliability
        Type: general
      – SubjectFull: Programming
        Type: general
      – SubjectFull: Error of Measurement
        Type: general
      – SubjectFull: Programming Languages
        Type: general
      – SubjectFull: Adaptive Testing
        Type: general
      – SubjectFull: Scores
        Type: general
    Titles:
      – TitleFull: Empirical vs. Expected IRT-Based Reliability Estimation in Computerized Multistage Testing (MST)
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Zhang, Yanwei
      – PersonEntity:
          Name:
            NameFull: Breithaupt, Krista
      – PersonEntity:
          Name:
            NameFull: Tessema, Aster
      – PersonEntity:
          Name:
            NameFull: Chuah, David
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2006
          Titles:
            – TitleFull: Online Submission
              Type: main
ResultId 1