View in EDS

Ordered or Scrambled: How the Forward Sequencing of Multiple Choice Questions Affects Test Item Scores

Saved in:
Bibliographic Details
Title: Ordered or Scrambled: How the Forward Sequencing of Multiple Choice Questions Affects Test Item Scores
Language: English
Authors: Cornelis W. Haasnoot (ORCID 0000-0002-4907-4348), Charan van Krevel (ORCID 0000-0002-7870-5866), Albert de Vaal (ORCID 0000-0002-7538-1421)
Source: Assessment & Evaluation in Higher Education. 2026 51(2):247-262.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 16
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Foreign Countries, Multiple Choice Tests, Sequential Approach, Sequential Learning, Economics Education, Microeconomics, Paper and Pencil Tests, Exit Examinations, Higher Education, Test Results, Scores, Test Bias, Evaluation Methods
Geographic Terms: Netherlands
DOI: 10.1080/02602938.2025.2568187
ISSN: 0260-2938
1469-297X
Abstract: We analyse the impact on student performance of scrambling the order of multiple choice questions (MCQs) in examinations. We test if students perform better on a question if it is preceded by a question that relates to either the same or the preceding lecture in the course (forward sequenced). Previous research has been unable to find clear evidence of a forward sequencing effect, which we argue could be related to their focus on test level outcomes. By considering test-item level results instead, using a sample of 16,127 MCQ scores from 12 Microeconomics examinations at Radboud University, the Netherlands, we find strong evidence that the forward sequencing of MCQs on examinations positively impacts student performance at the test-item level. An average student has a higher likelihood of answering correctly a MCQ that is in sequence than one that is not. This result holds after controlling for student ability, question difficulty, and a range of other determinants of student performance. The positive effect of forward sequencing builds up in case of a streak of sequenced MCQs. We conclude that the scrambling of MCQs on examinations is not innocuous and we call for consistent scrambling practices across examination versions.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1499463
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1499463
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Ordered or Scrambled: How the Forward Sequencing of Multiple Choice Questions Affects Test Item Scores
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Cornelis+W%2E+Haasnoot%22">Cornelis W. Haasnoot</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4907-4348">0000-0002-4907-4348</externalLink>)<br /><searchLink fieldCode="AR" term="%22Charan+van+Krevel%22">Charan van Krevel</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7870-5866">0000-0002-7870-5866</externalLink>)<br /><searchLink fieldCode="AR" term="%22Albert+de+Vaal%22">Albert de Vaal</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7538-1421">0000-0002-7538-1421</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Assessment+%26+Evaluation+in+Higher+Education%22"><i>Assessment & Evaluation in Higher Education</i></searchLink>. 2026 51(2):247-262.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 16
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Multiple+Choice+Tests%22">Multiple Choice Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Sequential+Approach%22">Sequential Approach</searchLink><br /><searchLink fieldCode="DE" term="%22Sequential+Learning%22">Sequential Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Economics+Education%22">Economics Education</searchLink><br /><searchLink fieldCode="DE" term="%22Microeconomics%22">Microeconomics</searchLink><br /><searchLink fieldCode="DE" term="%22Paper+and+Pencil+Tests%22">Paper and Pencil Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Exit+Examinations%22">Exit Examinations</searchLink><br /><searchLink fieldCode="DE" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Results%22">Test Results</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Netherlands%22">Netherlands</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/02602938.2025.2568187
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0260-2938<br />1469-297X
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: We analyse the impact on student performance of scrambling the order of multiple choice questions (MCQs) in examinations. We test if students perform better on a question if it is preceded by a question that relates to either the same or the preceding lecture in the course (forward sequenced). Previous research has been unable to find clear evidence of a forward sequencing effect, which we argue could be related to their focus on test level outcomes. By considering test-item level results instead, using a sample of 16,127 MCQ scores from 12 Microeconomics examinations at Radboud University, the Netherlands, we find strong evidence that the forward sequencing of MCQs on examinations positively impacts student performance at the test-item level. An average student has a higher likelihood of answering correctly a MCQ that is in sequence than one that is not. This result holds after controlling for student ability, question difficulty, and a range of other determinants of student performance. The positive effect of forward sequencing builds up in case of a streak of sequenced MCQs. We conclude that the scrambling of MCQs on examinations is not innocuous and we call for consistent scrambling practices across examination versions.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1499463
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1499463
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/02602938.2025.2568187
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
        StartPage: 247
    Subjects:
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Multiple Choice Tests
        Type: general
      – SubjectFull: Sequential Approach
        Type: general
      – SubjectFull: Sequential Learning
        Type: general
      – SubjectFull: Economics Education
        Type: general
      – SubjectFull: Microeconomics
        Type: general
      – SubjectFull: Paper and Pencil Tests
        Type: general
      – SubjectFull: Exit Examinations
        Type: general
      – SubjectFull: Higher Education
        Type: general
      – SubjectFull: Test Results
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Netherlands
        Type: general
    Titles:
      – TitleFull: Ordered or Scrambled: How the Forward Sequencing of Multiple Choice Questions Affects Test Item Scores
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Cornelis W. Haasnoot
      – PersonEntity:
          Name:
            NameFull: Charan van Krevel
      – PersonEntity:
          Name:
            NameFull: Albert de Vaal
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0260-2938
            – Type: issn-electronic
              Value: 1469-297X
          Numbering:
            – Type: volume
              Value: 51
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Assessment & Evaluation in Higher Education
              Type: main
ResultId 1