View in EDS

From Investigating the Alignment of a Priori Item Characteristics Based on the CTT and Four-Parameter Logistic (4-PL) IRT Models to Further Exploring the Comparability of the Two Models

Saved in:
Bibliographic Details
Title: From Investigating the Alignment of a Priori Item Characteristics Based on the CTT and Four-Parameter Logistic (4-PL) IRT Models to Further Exploring the Comparability of the Two Models
Language: English
Authors: Agus Santoso, Heri Retnawati, Timbul Pardede, Ibnu Rafi, Munaya Nikma Rosyada, Gulzhaina K. Kassymova, Xu Wenxin
Source: Practical Assessment, Research & Evaluation. 2024 29.
Availability: University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/
Peer Reviewed: Y
Page Count: 28
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Foreign Countries, Undergraduate Students, Business English, Test Construction, Test Content, Difficulty Level, Item Response Theory, Item Banks, Test Items, Comparative Testing, Test Reviews, Guessing (Tests)
Geographic Terms: Indonesia
ISSN: 1531-7714
Abstract: The test blueprint is important in test development, where it guides the test item writer in creating test items according to the desired objectives and specifications or characteristics (so-called a priori item characteristics), such as the level of item difficulty in the category and the distribution of items based on their difficulty level. Given that the difficulty level of the test items (easy, medium, or hard) created is influenced by the perceptions, knowledge, and experience of the item writer, item analysis based on empirical data using a specific measurement framework needs to be conducted, in addition to evaluation based on expert judgment, to ensure that the test items and the test itself have appropriate characteristics. The present study investigated the extent to which the a priori characteristics (i.e., item difficulty) of the items of the Business English test taken by 4,836 Universitas Terbuka (UT) students aligned with their characteristics when estimated under classical test theory (CTT) and four-parameter logistic (4-PL) IRT models based on empirical data. In light of the two measurement models used, CTT and 4-PL, we extended this study to exploring the comparability of the two models based on the yielded item difficulty and discrimination estimates and the relationship between pseudo-guessing and carelessness parameters. Our study suggested insufficient support for asserting that the characteristics of the items used in the Business English test align with the characteristics expected by the test developers. The exploration of the comparability of the CTT and 4-PL models demonstrated that while the two models were comparable in terms of the item difficulty estimates yielded, they were not comparable for the item discrimination estimates. Our study also did not find a linear association of the pseudo-guessing and carelessness parameters estimated under the 4-PL model. Further findings of our study and their implications, especially on test development practices, are discussed.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1452992
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1452992
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1452992
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: From Investigating the Alignment of a Priori Item Characteristics Based on the CTT and Four-Parameter Logistic (4-PL) IRT Models to Further Exploring the Comparability of the Two Models
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Agus+Santoso%22">Agus Santoso</searchLink><br /><searchLink fieldCode="AR" term="%22Heri+Retnawati%22">Heri Retnawati</searchLink><br /><searchLink fieldCode="AR" term="%22Timbul+Pardede%22">Timbul Pardede</searchLink><br /><searchLink fieldCode="AR" term="%22Ibnu+Rafi%22">Ibnu Rafi</searchLink><br /><searchLink fieldCode="AR" term="%22Munaya+Nikma+Rosyada%22">Munaya Nikma Rosyada</searchLink><br /><searchLink fieldCode="AR" term="%22Gulzhaina+K%2E+Kassymova%22">Gulzhaina K. Kassymova</searchLink><br /><searchLink fieldCode="AR" term="%22Xu+Wenxin%22">Xu Wenxin</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Practical+Assessment%2C+Research+%26+Evaluation%22"><i>Practical Assessment, Research & Evaluation</i></searchLink>. 2024 29.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 28
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Business+English%22">Business English</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Content%22">Test Content</searchLink><br /><searchLink fieldCode="DE" term="%22Difficulty+Level%22">Difficulty Level</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Banks%22">Item Banks</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Testing%22">Comparative Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reviews%22">Test Reviews</searchLink><br /><searchLink fieldCode="DE" term="%22Guessing+%28Tests%29%22">Guessing (Tests)</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Indonesia%22">Indonesia</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1531-7714
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The test blueprint is important in test development, where it guides the test item writer in creating test items according to the desired objectives and specifications or characteristics (so-called a priori item characteristics), such as the level of item difficulty in the category and the distribution of items based on their difficulty level. Given that the difficulty level of the test items (easy, medium, or hard) created is influenced by the perceptions, knowledge, and experience of the item writer, item analysis based on empirical data using a specific measurement framework needs to be conducted, in addition to evaluation based on expert judgment, to ensure that the test items and the test itself have appropriate characteristics. The present study investigated the extent to which the a priori characteristics (i.e., item difficulty) of the items of the Business English test taken by 4,836 Universitas Terbuka (UT) students aligned with their characteristics when estimated under classical test theory (CTT) and four-parameter logistic (4-PL) IRT models based on empirical data. In light of the two measurement models used, CTT and 4-PL, we extended this study to exploring the comparability of the two models based on the yielded item difficulty and discrimination estimates and the relationship between pseudo-guessing and carelessness parameters. Our study suggested insufficient support for asserting that the characteristics of the items used in the Business English test align with the characteristics expected by the test developers. The exploration of the comparability of the CTT and 4-PL models demonstrated that while the two models were comparable in terms of the item difficulty estimates yielded, they were not comparable for the item discrimination estimates. Our study also did not find a linear association of the pseudo-guessing and carelessness parameters estimated under the 4-PL model. Further findings of our study and their implications, especially on test development practices, are discussed.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1452992
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1452992
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 28
    Subjects:
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Business English
        Type: general
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Test Content
        Type: general
      – SubjectFull: Difficulty Level
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Item Banks
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Comparative Testing
        Type: general
      – SubjectFull: Test Reviews
        Type: general
      – SubjectFull: Guessing (Tests)
        Type: general
      – SubjectFull: Indonesia
        Type: general
    Titles:
      – TitleFull: From Investigating the Alignment of a Priori Item Characteristics Based on the CTT and Four-Parameter Logistic (4-PL) IRT Models to Further Exploring the Comparability of the Two Models
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Agus Santoso
      – PersonEntity:
          Name:
            NameFull: Heri Retnawati
      – PersonEntity:
          Name:
            NameFull: Timbul Pardede
      – PersonEntity:
          Name:
            NameFull: Ibnu Rafi
      – PersonEntity:
          Name:
            NameFull: Munaya Nikma Rosyada
      – PersonEntity:
          Name:
            NameFull: Gulzhaina K. Kassymova
      – PersonEntity:
          Name:
            NameFull: Xu Wenxin
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-electronic
              Value: 1531-7714
          Numbering:
            – Type: volume
              Value: 29
          Titles:
            – TitleFull: Practical Assessment, Research & Evaluation
              Type: main
ResultId 1