View in EDS

An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures

Saved in:
Bibliographic Details
Title: An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures
Language: English
Authors: Lorié, William A.
Source: Online Submission. 2013.
Peer Reviewed: N
Page Count: 34
Publication Date: 2013
Document Type: Reports - Research
Descriptors: Numeracy, Mathematical Concepts, Mathematical Logic, Difficulty Level, Test Items, Task Analysis, Engineering Technology, Human Factors Engineering, Item Sampling, Cognitive Measurement, Equated Scores, Construct Validity, Intelligence, Mathematical Applications, Mathematical Models, Statistical Analysis
Assessment and Survey Identifiers: Program for International Student Assessment
Abstract: A reverse engineering approach to automatic item generation (AIG) was applied to a figure-based publicly released test item from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) mathematical literacy cognitive instrument as part of a proof of concept. The author created an item template from which three items were randomly generated from within each of six types defined by a feature deemed to be most likely to affect item difficulty, for a total of eighteen distinct items. To assess their equivalence, these items were embedded in otherwise identical test forms and administered to human intelligence task workers on the Amazon Mechanical Turk system. One level of the type-defining feature appeared to affect item difficulty systematically. The author provides a task requirement rationale for removing this level. Implications for AIG theory and practice are discussed. An appendix presents a sample HTT.
Abstractor: As Provided
Number of References: 17
Entry Date: 2014
Accession Number: ED545460
Database: ERIC
Description
Abstract:A reverse engineering approach to automatic item generation (AIG) was applied to a figure-based publicly released test item from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) mathematical literacy cognitive instrument as part of a proof of concept. The author created an item template from which three items were randomly generated from within each of six types defined by a feature deemed to be most likely to affect item difficulty, for a total of eighteen distinct items. To assess their equivalence, these items were embedded in otherwise identical test forms and administered to human intelligence task workers on the Amazon Mechanical Turk system. One level of the type-defining feature appeared to affect item difficulty systematically. The author provides a task requirement rationale for removing this level. Implications for AIG theory and practice are discussed. An appendix presents a sample HTT.