View in EDS

An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures

Saved in:
Bibliographic Details
Title: An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures
Language: English
Authors: Lorié, William A.
Source: Online Submission. 2013.
Peer Reviewed: N
Page Count: 34
Publication Date: 2013
Document Type: Reports - Research
Descriptors: Numeracy, Mathematical Concepts, Mathematical Logic, Difficulty Level, Test Items, Task Analysis, Engineering Technology, Human Factors Engineering, Item Sampling, Cognitive Measurement, Equated Scores, Construct Validity, Intelligence, Mathematical Applications, Mathematical Models, Statistical Analysis
Assessment and Survey Identifiers: Program for International Student Assessment
Abstract: A reverse engineering approach to automatic item generation (AIG) was applied to a figure-based publicly released test item from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) mathematical literacy cognitive instrument as part of a proof of concept. The author created an item template from which three items were randomly generated from within each of six types defined by a feature deemed to be most likely to affect item difficulty, for a total of eighteen distinct items. To assess their equivalence, these items were embedded in otherwise identical test forms and administered to human intelligence task workers on the Amazon Mechanical Turk system. One level of the type-defining feature appeared to affect item difficulty systematically. The author provides a task requirement rationale for removing this level. Implications for AIG theory and practice are discussed. An appendix presents a sample HTT.
Abstractor: As Provided
Number of References: 17
Entry Date: 2014
Accession Number: ED545460
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED545460
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED545460
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Lorié%2C+William+A%2E%22">Lorié, William A.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Online+Submission%22"><i>Online Submission</i></searchLink>. 2013.
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 34
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2013
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Numeracy%22">Numeracy</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+Concepts%22">Mathematical Concepts</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+Logic%22">Mathematical Logic</searchLink><br /><searchLink fieldCode="DE" term="%22Difficulty+Level%22">Difficulty Level</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Task+Analysis%22">Task Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Engineering+Technology%22">Engineering Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Human+Factors+Engineering%22">Human Factors Engineering</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Sampling%22">Item Sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Cognitive+Measurement%22">Cognitive Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Equated+Scores%22">Equated Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Construct+Validity%22">Construct Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Intelligence%22">Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+Applications%22">Mathematical Applications</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+Models%22">Mathematical Models</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Program+for+International+Student+Assessment%22">Program for International Student Assessment</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: A reverse engineering approach to automatic item generation (AIG) was applied to a figure-based publicly released test item from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) mathematical literacy cognitive instrument as part of a proof of concept. The author created an item template from which three items were randomly generated from within each of six types defined by a feature deemed to be most likely to affect item difficulty, for a total of eighteen distinct items. To assess their equivalence, these items were embedded in otherwise identical test forms and administered to human intelligence task workers on the Amazon Mechanical Turk system. One level of the type-defining feature appeared to affect item difficulty systematically. The author provides a task requirement rationale for removing this level. Implications for AIG theory and practice are discussed. An appendix presents a sample HTT.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 17
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2014
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED545460
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED545460
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 34
    Subjects:
      – SubjectFull: Numeracy
        Type: general
      – SubjectFull: Mathematical Concepts
        Type: general
      – SubjectFull: Mathematical Logic
        Type: general
      – SubjectFull: Difficulty Level
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Task Analysis
        Type: general
      – SubjectFull: Engineering Technology
        Type: general
      – SubjectFull: Human Factors Engineering
        Type: general
      – SubjectFull: Item Sampling
        Type: general
      – SubjectFull: Cognitive Measurement
        Type: general
      – SubjectFull: Equated Scores
        Type: general
      – SubjectFull: Construct Validity
        Type: general
      – SubjectFull: Intelligence
        Type: general
      – SubjectFull: Mathematical Applications
        Type: general
      – SubjectFull: Mathematical Models
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Program for International Student Assessment
        Type: general
    Titles:
      – TitleFull: An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Lorié, William A.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2013
          Titles:
            – TitleFull: Online Submission
              Type: main
ResultId 1