View in EDS

Development and Use of a Multiple-Choice Item Writing Flaws Evaluation Instrument in the Context of General Chemistry

Saved in:
Bibliographic Details
Title: Development and Use of a Multiple-Choice Item Writing Flaws Evaluation Instrument in the Context of General Chemistry
Language: English
Authors: Breakall, Jar (ORCID 0000-0001-9631-7771), Randles, Christopher, Tasker, Roy
Source: Chemistry Education Research and Practice. Apr 2019 20(2):369-382.
Availability: Royal Society of Chemistry. Thomas Graham House, Science Park, Milton Road, Cambridge, CB4 0WF, UK. Tel: +44-1223 420066; Fax: +44-1223 423623; e-mail: cerp@rsc.org; Web site: http://www.rsc.org/cerp
Peer Reviewed: Y
Page Count: 14
Publication Date: 2019
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Science Instruction, Chemistry, College Science, Undergraduate Study, Science Tests, Multiple Choice Tests, Test Construction, Test Items
DOI: 10.1039/c8rp00262b
ISSN: 1756-1108
Abstract: Multiple-choice (MC) exams are common in undergraduate general chemistry courses in the United States and are known for being difficult to construct. With their extensive use in the general chemistry classroom, it is important to ensure that these exams are valid measures of what chemistry students know and can do. One threat to MC exam validity is the presence of flaws, known as item writing flaws, that can falsely inflate or deflate a student's performance on an exam, independent of their chemistry knowledge. Such flaws can disadvantage (or falsely advantage) students in their exam performance. Additionally, these flaws can introduce unwanted noise into exam data. With the numerous possible flaws that can be made during MC exam creation, it can be difficult to recognize (and avoid) these flaws when creating MC general chemistry exams. In this study a rubric, known as the Item Writing Flaws Evaluation Instrument (IWFEI), has been created that can be used to identify item writing flaws in MC exams. The instrument was developed based on a review of the item writing literature and was tested for inter-rater reliability using general chemistry exam items. The instrument was found to have a high degree of inter-rater reliability with an overall percent agreement of 91.8% and a Krippendorff Alpha of 0.836. Using the IWFEI in an analysis of 1019 general chemistry MC exam items, it was found that 83% of items contained at least one item writing flaw with the most common flaw being the inclusion of implausible distractors. From the results of this study, an instrument has been developed that can be used in both research and teaching settings. As the IWFEI is used in these settings we envision an improvement in MC exam development practice and quality.
Abstractor: As Provided
Entry Date: 2019
Accession Number: EJ1211566
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1211566
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Development and Use of a Multiple-Choice Item Writing Flaws Evaluation Instrument in the Context of General Chemistry
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Breakall%2C+Jar%22">Breakall, Jar</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-9631-7771">0000-0001-9631-7771</externalLink>)<br /><searchLink fieldCode="AR" term="%22Randles%2C+Christopher%22">Randles, Christopher</searchLink><br /><searchLink fieldCode="AR" term="%22Tasker%2C+Roy%22">Tasker, Roy</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Chemistry+Education+Research+and+Practice%22"><i>Chemistry Education Research and Practice</i></searchLink>. Apr 2019 20(2):369-382.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Royal Society of Chemistry. Thomas Graham House, Science Park, Milton Road, Cambridge, CB4 0WF, UK. Tel: +44-1223 420066; Fax: +44-1223 423623; e-mail: cerp@rsc.org; Web site: http://www.rsc.org/cerp
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 14
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2019
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Science+Instruction%22">Science Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Chemistry%22">Chemistry</searchLink><br /><searchLink fieldCode="DE" term="%22College+Science%22">College Science</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Study%22">Undergraduate Study</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Tests%22">Science Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Multiple+Choice+Tests%22">Multiple Choice Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1039/c8rp00262b
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1756-1108
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Multiple-choice (MC) exams are common in undergraduate general chemistry courses in the United States and are known for being difficult to construct. With their extensive use in the general chemistry classroom, it is important to ensure that these exams are valid measures of what chemistry students know and can do. One threat to MC exam validity is the presence of flaws, known as item writing flaws, that can falsely inflate or deflate a student's performance on an exam, independent of their chemistry knowledge. Such flaws can disadvantage (or falsely advantage) students in their exam performance. Additionally, these flaws can introduce unwanted noise into exam data. With the numerous possible flaws that can be made during MC exam creation, it can be difficult to recognize (and avoid) these flaws when creating MC general chemistry exams. In this study a rubric, known as the Item Writing Flaws Evaluation Instrument (IWFEI), has been created that can be used to identify item writing flaws in MC exams. The instrument was developed based on a review of the item writing literature and was tested for inter-rater reliability using general chemistry exam items. The instrument was found to have a high degree of inter-rater reliability with an overall percent agreement of 91.8% and a Krippendorff Alpha of 0.836. Using the IWFEI in an analysis of 1019 general chemistry MC exam items, it was found that 83% of items contained at least one item writing flaw with the most common flaw being the inclusion of implausible distractors. From the results of this study, an instrument has been developed that can be used in both research and teaching settings. As the IWFEI is used in these settings we envision an improvement in MC exam development practice and quality.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2019
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1211566
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1211566
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1039/c8rp00262b
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 14
        StartPage: 369
    Subjects:
      – SubjectFull: Science Instruction
        Type: general
      – SubjectFull: Chemistry
        Type: general
      – SubjectFull: College Science
        Type: general
      – SubjectFull: Undergraduate Study
        Type: general
      – SubjectFull: Science Tests
        Type: general
      – SubjectFull: Multiple Choice Tests
        Type: general
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Test Items
        Type: general
    Titles:
      – TitleFull: Development and Use of a Multiple-Choice Item Writing Flaws Evaluation Instrument in the Context of General Chemistry
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Breakall, Jar
      – PersonEntity:
          Name:
            NameFull: Randles, Christopher
      – PersonEntity:
          Name:
            NameFull: Tasker, Roy
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2019
          Identifiers:
            – Type: issn-electronic
              Value: 1756-1108
          Numbering:
            – Type: volume
              Value: 20
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Chemistry Education Research and Practice
              Type: main
ResultId 1