View in EDS HTML Full Text PDF Full Text

Item-Writing Guidelines on Response Option Placement: A Systematic Review

Saved in:
Bibliographic Details
Title: Item-Writing Guidelines on Response Option Placement: A Systematic Review
Language: English
Authors: Séverin Lions (ORCID 0000-0003-0697-7974), María Paz Blanco, Pablo Dartnell, Carlos Monsalve, Gabriel Ortega, Julie Lemarié
Source: Applied Measurement in Education. 2024 37(4):289-304.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 16
Publication Date: 2024
Document Type: Journal Articles
Information Analyses
Descriptors: Test Construction, Multiple Choice Tests, Guidelines, Test Items, Responses, Placement
DOI: 10.1080/08957347.2024.2424551
ISSN: 0895-7347
1532-4818
Abstract: Multiple-choice items are universally used in formal education. Since they should assess learning, not test-wiseness or guesswork, they must be constructed following the highest possible standards. Hundreds of item-writing guides have provided guidelines to help test developers adopt appropriate strategies to define the distribution and sequence of correct answers across tests and, thus, not inadvertently provide clues to correct responses. In this study, these option placement guidelines were systematically reviewed. Consensual recommendations and recommendations from the most theoretically sound guides were identified. Additionally, results from empirical studies evaluating the effect of option position on examinees' responses and research reports on where test developers place correct answers in tests were considered to better examine these recommendations' relevance and suitability. Consistent and easy-to-follow recommendations about where and how to place options in items and tests are finally provided so that developers can gauge the adequacy of their option placement strategies.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1454681
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEaaNw7BUcOnmB6zKfStAGPAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDOnQ8V8kX-odU4x5IAIBEICBm_Gjz0nPCrxdJnrXEOe2Red7HdiDGuIVbrbq1gHjoNgrT-Zs-hICP_pT7Oo4ajk2jkHMHwewGnXoBpL8niDnJOdMvHVKZ0MiSGS7MayZ0dmMKQqUmubC8nJqPw655HEqhvxlH5yAFlcT02KzhFzLhMmls_rXjNvqdaRSFCYc61tq9zMynCD5rCBqWOzBPrc4htyDjn_cwgppr8AI
Text:
  Availability: 1
  Value: <anid>AN0181729311;7lg01oct.24;2024Dec20.04:46;v2.2.500</anid> <title id="AN0181729311-1">Item-Writing Guidelines on Response Option Placement: A Systematic Review </title> <sbt id="AN0181729311-2">1. Introduction</sbt> <p>Multiple-choice items are universally used in formal education. Since they should assess learning, not test-wiseness or guesswork, they must be constructed following the highest possible standards. Hundreds of item-writing guides have provided guidelines to help test developers adopt appropriate strategies to define the distribution and sequence of correct answers across tests and, thus, not inadvertently provide clues to correct responses. In this study, these option placement guidelines were systematically reviewed. Consensual recommendations and recommendations from the most theoretically sound guides were identified. Additionally, results from empirical studies evaluating the effect of option position on examinees' responses and research reports on where test developers place correct answers in tests were considered to better examine these recommendations' relevance and suitability. Consistent and easy-to-follow recommendations about where and how to place options in items and tests are finally provided so that developers can gauge the adequacy of their option placement strategies.</p> <p>Multiple-choice format is probably the most common assessment format in educational contexts. It is regularly used both in teacher-made tests and standardized tests to assess learning in many disciplines. Thus, performance on multiple-choice tests is highly consequential for students, both academically and professionally. A considerable number of item-writing guides have been published providing standards to build items and tests. Two groups of researchers (led by Haladyna and Moreno) have made a particularly significant contribution to guides' design (Haladyna & Downing, [<reflink idref="bib44" id="ref1">44</reflink>]; Haladyna & Rodriguez, [<reflink idref="bib46" id="ref2">46</reflink>]; Haladyna et al., [<reflink idref="bib45" id="ref3">45</reflink>]; Moreno et al., [<reflink idref="bib78" id="ref4">78</reflink>], [<reflink idref="bib79" id="ref5">79</reflink>], [<reflink idref="bib77" id="ref6">77</reflink>]).</p> <p>Even if item-writing guides are available for most disciplines, items still commonly contain construction flaws, such as clues to correct responses (Lions et al., [<reflink idref="bib63" id="ref7">63</reflink>]). Item-writing flaws have been found in teacher-made tests and even in standardized tests, impacting students' scores and pass/fail outcomes (DeVore et al., [<reflink idref="bib27" id="ref8">27</reflink>]; Downing, [<reflink idref="bib28" id="ref9">28</reflink>]; Tarrant & Ware, [<reflink idref="bib99" id="ref10">99</reflink>]). Addressing these flaws is important because they might compromise multiple-choice test results' validity and fairness (Haladyna & Rodriguez, [<reflink idref="bib46" id="ref11">46</reflink>]).</p> <p>Some item-writing flaws, such as lengthier correct option as compared with distractors or lexical overlap between correct option and stem, can be exploited by testees (Casu & García-García, [<reflink idref="bib19" id="ref12">19</reflink>]). Examinees are expected to solve items based solely on content knowledge and skills, but sometimes they pick an option "at random" or select responses by guessing (McKenna, [<reflink idref="bib68" id="ref13">68</reflink>]). The ability to successfully use guessing strategies and discover which option is correct based on some item/test characteristic is known as test-wiseness (Millman et al., [<reflink idref="bib75" id="ref14">75</reflink>]). Test developers should be aware that examinees know test-wiseness rules and that these rules, when mastered, may lead to higher scores, irrespective of content knowledge (Bailey et al., [<reflink idref="bib5" id="ref15">5</reflink>]).</p> <p>One test-wiseness strategy consists of selecting an option based on its position in the option list (e.g., A, B, C, or D in four-option items, Rubinstein & Salant, [<reflink idref="bib90" id="ref16">90</reflink>]; Sarnacki, [<reflink idref="bib91" id="ref17">91</reflink>]). Such location-based response strategies allow the selection of correct answers in the absence of relevant knowledge, the distribution or sequence of answer keys potentially acting as a clue (Carnegie, [<reflink idref="bib17" id="ref18">17</reflink>]). For instance, correct answers might be selected by avoiding choosing the same answer location for consecutive items (Lee, [<reflink idref="bib61" id="ref19">61</reflink>]).</p> <p>Guidelines on how to arrange options and distribute answer keys have been provided to help test developers remove unwanted clues from their tests (e.g., Haladyna & Downing, [<reflink idref="bib44" id="ref20">44</reflink>]; Haladyna & Rodriguez, [<reflink idref="bib46" id="ref21">46</reflink>]; Haladyna et al., [<reflink idref="bib45" id="ref22">45</reflink>]). Still, test-developing practices continue to be suboptimal in terms of option placement strategies. For instance, in many tests, answer keys are more frequently placed at the middle options than at the edges (Lions et al., [<reflink idref="bib63" id="ref23">63</reflink>]). This keying middle bias has been observed for decades (Metfessel & Sax, [<reflink idref="bib71" id="ref24">71</reflink>]), even in standardized tests (Attali & Bar‐Hillel, [<reflink idref="bib4" id="ref25">4</reflink>]), and remains common (DeVore et al., [<reflink idref="bib27" id="ref26">27</reflink>]). Examinees might benefit from this keying bias using a widespread test-wise strategy known as "When in doubt, choose C" (Carter, [<reflink idref="bib18" id="ref27">18</reflink>]).</p> <p>Non-professional developers receive little training on item writing and may not know item-writing guidelines (Tarrant et al., [<reflink idref="bib98" id="ref28">98</reflink>]). Developers who do regularly consult guides (such as standardized test developers) may find that recommendations about option placement are not precise (Hohensinn & Baghaei, [<reflink idref="bib49" id="ref29">49</reflink>]). Since item-writing training and guides can help develop high-quality items (Lachiver, [<reflink idref="bib59" id="ref30">59</reflink>]; Przymuszała et al., [<reflink idref="bib88" id="ref31">88</reflink>]), this review systematically analyzed the option placement guidelines provided by different existing guides in order to provide a comprehensive and consistent set of recommendations on the topic.</p> <hd id="AN0181729311-3">1.1. Item-Writing Guidelines Regarding the Placement of Response Options</hd> <p>Option placement guidelines are generally based on authors' accumulated expertise and conventional wisdom rather than on evidence. These guidelines encourage test developers to either arrange items' options in a particular order or monitor the distribution of answer keys across a test. Guidelines on ordering response options include scrambling option order or arranging options in some consistent order whenever possible, based on some common trait(s). Numbers can be arranged in ascending or descending order; text options can be ordered alphabetically or by length. Guidelines on positioning answer keys include varying the position of correct responses by randomization or practicing key balancing, i.e., placing the same number of answer keys in each option position across the test. The rationale for all these guidelines is that the position/order of options should never be of help in selecting correct answers (Haladyna & Rodriguez, [<reflink idref="bib46" id="ref32">46</reflink>]; McDonald, [<reflink idref="bib67" id="ref33">67</reflink>]) and that option placement should promote the validity of test score interpretation and use (Albano & Rodriguez, [<reflink idref="bib2" id="ref34">2</reflink>]).</p> <p>Since different, sometimes conflicting option placement guidelines co-exist in literature, several research syntheses have been conducted to determine consensual recommendations (Balaha, [<reflink idref="bib6" id="ref35">6</reflink>]; Coffman et al., [<reflink idref="bib21" id="ref36">21</reflink>]; Cox, [<reflink idref="bib24" id="ref37">24</reflink>]; Ellsworth et al., [<reflink idref="bib30" id="ref38">30</reflink>]; Frey et al., [<reflink idref="bib31" id="ref39">31</reflink>]; Gierl et al., [<reflink idref="bib36" id="ref40">36</reflink>]; Haladyna & Downing, [<reflink idref="bib44" id="ref41">44</reflink>]; Haladyna et al., [<reflink idref="bib45" id="ref42">45</reflink>]; Tarrant et al., [<reflink idref="bib98" id="ref43">98</reflink>]; Trump & Haggerty, [<reflink idref="bib102" id="ref44">102</reflink>]). However, even these allegedly consensual guidelines present inconsistencies. For instance, some reviews recommend tests to have the same number of answer keys in each option position, that is, a balanced distribution of answer keys (Haladyna & Downing, [<reflink idref="bib44" id="ref45">44</reflink>]; Haladyna et al., [<reflink idref="bib45" id="ref46">45</reflink>]; Tarrant et al., [<reflink idref="bib98" id="ref47">98</reflink>]). Other reviews urge developers to randomize the key position (Cox, [<reflink idref="bib24" id="ref48">24</reflink>]; Ellsworth et al., [<reflink idref="bib30" id="ref49">30</reflink>]; Trump & Haggerty, [<reflink idref="bib102" id="ref50">102</reflink>]), which rarely generates an even distribution.</p> <p>Given all this, option placement guidelines need to be systematically reviewed and discussed to provide test developers with clear-cut and easy-to-follow recommendations. To this date, not even the most comprehensive review on the topic (Gierl et al., [<reflink idref="bib36" id="ref51">36</reflink>]) has run a systematic research synthesis such as the one presented here. In this systematic review, consensual recommendations and recommendations from the most theoretically sound guides are identified and discussed in light of results from empirical studies evaluating the effect of option position on examinees' responses and research reports focusing on where test developers place correct answers in tests. The main goal is to consistently synthesize available guides' recommendations on where and how to place response options in multiple-choice achievement tests to subsequently discuss how adequate they are. This synthesis will be of interest to test developers and instructors working in the education field. Additionally, the identified pool of item-writing guides may become a most useful source of information for scholars willing to systematically review other item-writing guidelines. Research questions may be summarized as follows: Are guidelines on response option placement usually found in item-writing guides? Which option placement strategies are most frequently recommended? Which strategies are recommended in the most theoretically sound guides? Do guidelines vary depending on domain? Are some guidelines outdated? Which guidelines should test developers follow, why, and how are they to be implemented?</p> <hd id="AN0181729311-4">2. Method</hd> <p>Bibliography search, document identification, document inclusion, and coding procedures are described with enough detail to comply with standards for systematic reviews (Cooper, [<reflink idref="bib23" id="ref52">23</reflink>]; Moher et al., [<reflink idref="bib76" id="ref53">76</reflink>]; Page et al., [<reflink idref="bib85" id="ref54">85</reflink>]).</p> <hd id="AN0181729311-5">2.1. Literature Search and Identification of Relevant Documents</hd> <p>The implemented search code was (<emph>directions OR recommendations OR guidelines OR rules OR principles OR suggestions OR guide OR guiding) AND (writing OR constructing OR developing OR formulating OR preparing OR designing) AND (question OR item OR test OR examination OR exam OR assessment OR quiz OR questionnaire) AND (MCQ OR multiple-choice OR multiple-answer</emph>).</p> <p>Terms used in this code were identified in item-writing guides that were found in preliminary non-systematic searches. This search code was used to run advanced searches throughout the two most extensive academic databases (Web of Science and Scopus) on April 2, 2020. Searches were run within titles, abstracts, and keywords throughout entire data collections (i.e., research articles and reviews, books and book chapters, conference papers, technical reports, theses), all languages, and all years included. Duplicates were detected and merged automatically or manually using Mendeley reference management software. Search alerts were created so that results for this search that were freshly incorporated into WOS or Scopus databases could be collected and checked for inclusion. The last update for this search was done on November 10, 2023.</p> <hd id="AN0181729311-6">2.2. Screening and Inclusion Criteria</hd> <p>A pilot screening with a randomly selected 10% of the sample was implemented by the first and second authors. Inter-researcher agreement was 100%; therefore, the remaining screening was conducted solely by the second author. Figure 1 shows the screening process in detail. In the first step (eligibility), titles and abstracts were reviewed to identify eligible records, i.e., records complying with the following inclusion criteria: (a) guidelines about item writing were provided, (b) they were conceived to develop achievement tests, (c) they were advanced as an overall framework to design items and/or options; (d) they provided directions to write multiple-choice items, and (e) they addressed the writing of response options. In the second step (inclusion), the full text of eligible records was examined to determine inclusion. Excluded records were studies that (a) presented empirical data about just one item-writing guideline, (b) discussed issues regarding just a few item-writing guidelines, (c) provided general recommendations about the different test development steps, (d) provided recommendations about item analysis only, (e) provided recommendations about psychological test development, (f) provided recommendations on writing open-ended items, (g) evaluated the quality of multiple-choice items or tests through analyses of item psychometric properties or item-writing flaws frequency, (g) presented the development or validation of a test, (h) evaluated teachers' assessment literacy, (i) assessed the effectiveness of a teacher training program for assessment literacy or item-writing skills improvement.</p> <p>Graph: Figure 1. Flow diagram.</p> <hd id="AN0181729311-7">2.3. Eligible and Included Records</hd> <p>The search output contained 1,440 WOS and 342 Scopus results. With 207 duplicates identified, the initial selection process was finally conducted on 1,575 references. Additionally, 790 references from search alerts entered this process. Fifty-six eligible records were identified. All were suited for inclusion and were thus included as primary sources. Additional eligible records were searched using two complementary strategies: checking the complete reference list of all included records and checking the documents presented in Google Scholar as citing the included records (for all those cited by less than 1,575 documents). These two strategies were recursively implemented until no new eligible records emerged. Three hundred seventy-four new eligible records were thus identified. They were finally included as secondary sources (see Figure 1).</p> <p>Out of the 430 identified sources, 93 could not be included because access to full text was restricted, 15 were excluded because they presented the same data presented in some other record, and seven could not be encoded because they were written in languages other than English, Spanish, Portuguese, or French (no translation services were used). Therefore, this review finally considered 315 records (henceforth called item-writing guides or guides, see Figure 1). Twelve were non-English records.</p> <hd id="AN0181729311-8">2.4. Variable Coding</hd> <p>Seven variables of interest were encoded and analyzed: (<reflink idref="bib1" id="ref55">1</reflink>) publication type (e.g., journal article, book, etc.), (<reflink idref="bib2" id="ref56">2</reflink>) domain (e.g., medicine), (<reflink idref="bib3" id="ref57">3</reflink>) year of publication (e.g., 2015), (<reflink idref="bib4" id="ref58">4</reflink>) number of relevant cited sources (well-referenced, others), (<reflink idref="bib5" id="ref59">5</reflink>) inclusion of guideline(s) addressing option ordering or key positioning (for each category: yes, no), (<reflink idref="bib6" id="ref60">6</reflink>) type of guideline(s) regarding option ordering (logical, numerical, alphabetical, random, by length, other), and (<reflink idref="bib7" id="ref61">7</reflink>) type of guideline(s) regarding key positioning (key randomization, approximative key global balancing, exact key global balancing, key variation, local key balancing, other). Regarding the number of relevant cited sources, item-writing guides were labeled "well-referenced" when they were based on a solid theoretical background. The "well-referenced" guides could be identified because they reviewed several other guides and/or relevant empirical studies focusing on the effect of option position on examinees' responses or on where test developers place correct answers in tests. They also employed a clear methodology for their bibliography search. On the contrary, the remaining guides did not adopt clear bibliography search criteria, cited a limited number of other item-writing guides, and hardly discussed results from empirical studies on response option placement.</p> <p>The coding process was independently conducted by two raters (the first and second authors of this study) in Microsoft Excel. Agreement level between the two raters on the coding of each variable was calculated by dividing the number of single-cell-coding agreements by the total number of coded spreadsheet cells related to the variable. This agreement was moderate to very high (range 63–96%), being high on average (81%). All disagreements were solved by discussing them until reaching consensus. Third-party involvement was never needed.</p> <hd id="AN0181729311-9">2.5. Data Analysis</hd> <p>A descriptive analysis was first conducted to give an overview of the amount and origin of data considered in the review. Then, the number of each option placement guideline was calculated, along with an aggregated total number of guidelines. A distinction was made between option ordering guidelines and key positioning guidelines for clarity. Item-writing guides were inspected both aggregately and based on publication type, domain, and year of publication. The number of relevant cited sources was used to reanalyze the data, restricting the analysis to well-referenced guides. This last analysis aimed to identify guidelines that were consensual but also deemed as appropriate by leading researchers renowned in the field.</p> <hd id="AN0181729311-10">3. Results</hd> <p>The oldest identified item-writing guide was published in 1926. From then on, guides were present in all decades. However, more than 50% of the 315 considered guides were published in the twenty-first century, which reveals that most guidelines have been provided recently (Table 1). About 80% of the guides were published as journal articles, books, or book chapters, which means that they are, in principle, high-standard documents (Table 2). More than 80% of the guides came from two fields: education/evaluation and medicine/other health sciences (Table 3). Since guides from these two domains presented similar guidelines (see Supplementary Results and Table S1) and involved different communication channels (most journal articles were guides from health sciences, while most books or book chapters were guides from education/evaluation), option placement recommendations were aggregated without distinguishing domains or publication types.</p> <p>Table 1. Description of the included records, sorted by publication date.</p> <p> <ephtml> <table><thead><tr><td>Decade</td><td>Number (percentage) of records</td></tr></thead><tbody><tr><td>1920s</td><td>2 (0.6%)</td></tr><tr><td>1930s</td><td>5 (1.6%)</td></tr><tr><td>1940s</td><td>5 (1.6%)</td></tr><tr><td>1950s</td><td>9 (2.9%)</td></tr><tr><td>1960s</td><td>12 (3.8%)</td></tr><tr><td>1970s</td><td>28 (8.9%)</td></tr><tr><td>1980s</td><td>30 (9.5%)</td></tr><tr><td>1990s</td><td>44 (14.0%)</td></tr><tr><td>2000s</td><td>57 (18.1%)</td></tr><tr><td>2010s</td><td>99 (31.4%)</td></tr><tr><td>2020s</td><td>20 (6.3%)</td></tr><tr><td>Unknown</td><td>4 (1.3%)</td></tr><tr><td><bold>Total</bold></td><td><bold>315 (100%)</bold></td></tr></tbody></table> </ephtml> </p> <p>Table 2. Description of the included records, sorted by publication type.</p> <p> <ephtml> <table><thead><tr><td>Publication type</td><td>Number (percentage) of records</td></tr></thead><tbody><tr><td>Journal article</td><td>132 (41.9%)</td></tr><tr><td>Book (or book chapter)</td><td>117 (37.1%)</td></tr><tr><td>University guide</td><td>29 (9.2%)</td></tr><tr><td>Scientific society guide</td><td>15 (4.8%)</td></tr><tr><td>Technical report</td><td>10 (3.2%)</td></tr><tr><td>Conference</td><td>10 (3.2%)</td></tr><tr><td>Theses</td><td>2 (0.6%)</td></tr><tr><td><bold>Total</bold></td><td><bold>315 (100%)</bold></td></tr></tbody></table> </ephtml> </p> <p>Table 3. Description of the included records, sorted by domain.</p> <p> <ephtml> <table><thead><tr><td>Field</td><td>Number (percentage) of records</td></tr></thead><tbody><tr><td>Education</td><td>157 (49.8%)</td></tr><tr><td>Medicine</td><td>77 (24.4%)</td></tr><tr><td>Nursing</td><td>24 (7.6%)</td></tr><tr><td>Pharmacy</td><td>6 (1.9%)</td></tr><tr><td>Science</td><td>5 (1.6%)</td></tr><tr><td>Engineering</td><td>4 (1.3%)</td></tr><tr><td>Finance, accounting, and auditing</td><td>4 (1.3%)</td></tr><tr><td>Language</td><td>4 (1.3%)</td></tr><tr><td>Language (L2)</td><td>4 (1.3%)</td></tr><tr><td>Physics</td><td>4 (1.3%)</td></tr><tr><td>Chemistry</td><td>4 (1.3%)</td></tr><tr><td>Human resources</td><td>3 (1.0%)</td></tr><tr><td>Law</td><td>3 (1.0%)</td></tr><tr><td>Psychology</td><td>3 (1.0%)</td></tr><tr><td>Information technology</td><td>3 (1.0%)</td></tr><tr><td>Sport</td><td>2 (0.7%)</td></tr><tr><td>Army</td><td>1 (0.3%)</td></tr><tr><td>Biology</td><td>1 (0.3%)</td></tr><tr><td>Ecology</td><td>1 (0.3%)</td></tr><tr><td>Geoscience</td><td>1 (0.3%)</td></tr><tr><td>Marketing</td><td>1 (0.3%)</td></tr><tr><td>Mathematics</td><td>1 (0.3%)</td></tr><tr><td>Technology</td><td>1 (0.3%)</td></tr><tr><td>Therapy-rehab</td><td>1 (0.3%)</td></tr><tr><td><bold>Total</bold></td><td><bold>315 (100%)</bold></td></tr></tbody></table> </ephtml> </p> <p>Three-quarters of all inspected guides (75.6%, <emph>n</emph> = 238) did include guidelines about the placement of response options or keys, suggesting that most authors consider this a relevant issue. More than 50% of the guides stated that response options need to be ordered in some way (55.3%, <emph>n</emph> = 174) and that the distribution of answer keys must be controlled (58.1%, <emph>n</emph> = 183), suggesting that these issues have similar importance in literature. One hundred nineteen of the 238 guides discussing option placement (50.0%) provided recommendations both for option ordering and for key positioning, with the remaining 119 (50.0%) considering only either one of these two variables (Table 4).</p> <p>Table 4. Inclusion of option placement guidelines in item-writing guides.</p> <p> <ephtml> <table><thead><tr><td>Inclusion of option placement guidelines</td><td>Number (percentage of records)</td></tr></thead><tbody><tr><td>Do not include any</td><td>77 (24.4%)</td></tr><tr><td>About response order only</td><td>55 (17.5%)</td></tr><tr><td>About key position only</td><td>64 (20.3%)</td></tr><tr><td>For both response order and key position</td><td>119 (37.8%)</td></tr><tr><td><bold>Total</bold></td><td><bold>315 (100%)</bold></td></tr></tbody></table> </ephtml> </p> <p>No single option placement guideline was found to be universally advanced. The most consensual recommendation consisted of arranging options either in ascending or descending order (if they presented numbers) or in some logical order (whenever possible, given the options' particular traits, Table 5). Some other frequent recommendations were randomizing key position, presenting a balanced distribution of answer keys at test level, and arranging options alphabetically. All these guidelines were individually or collectively provided by guides published during the twenty-first century; none of them seems thus to be obsolete (Figure S1), which calls for a thorough discussion of each.</p> <p>Table 5. Frequency of option placement guidelines observed in item-writing guides.</p> <p> <ephtml> <table><thead><tr><td /><td>Number (percentage) of guides</td></tr><tr><td>Guideline</td><td>All guides</td><td>Well-referenced guides</td></tr></thead><tbody><tr><td><bold>Option ordering</bold></td><td /><td /></tr><tr><td>Numerical</td><td>143 (60.1%)</td><td>11 (73.3%)</td></tr><tr><td>Logical</td><td>122 (51.3%)</td><td>11 (73.3%)</td></tr><tr><td>Alphabetical</td><td>87 (36.6%)</td><td>3 (20.0%)</td></tr><tr><td>Random</td><td>26 (10.9%)</td><td>1 (6.7%)</td></tr><tr><td>By Length</td><td>14 (5.9%)</td><td>0 (0.0%)</td></tr><tr><td><bold>Key positioning</bold></td><td /><td /></tr><tr><td>Randomization</td><td>101 (42.4%)</td><td>7 (46.7%)</td></tr><tr><td>Global Balancing</td><td>97 (40.8%)</td><td>5 (33.3%)</td></tr><tr><td>Variation</td><td>52 (21.8%)</td><td>1 (6.7%)</td></tr><tr><td>Local Balancing</td><td>10 (4.2%)</td><td>0 (0.0%)</td></tr><tr><td>Other</td><td>2 (0.8%)</td><td>1 (6.7%)</td></tr><tr><td><bold>Total</bold></td><td><bold>238 (100%)</bold></td><td><bold>15 (100%)</bold></td></tr></tbody></table> </ephtml> </p> <hd id="AN0181729311-11">3.1. Guidelines for the Ordering of Response Options: Logical Ordering vs. Scrambling</hd> <p>Guidelines about ordering of response options and their observed frequency in item-writing guides are presented in Table 5. The most consensual guideline about ordering was that numerical options should be presented in an ascending or descending order. This guideline applied to mathematical and science items with numerical options but also to items with dates as options (chronological ordering, here included under the "numerical ordering" label). For non-numerical options, many guides suggested alphabetical order, although this guideline was less frequent than numerical ordering.</p> <p>More generally, most guides recommended options to be arranged "logically" each time they could be so organized, based on some common trait(s). Numerical and alphabetical orders were classified as logical orders by some (but not all) authors, revealing that logical ordering, in a broad sense, was the most popular option ordering guideline. Two other ordering guidelines were found: random ordering (also called scrambling) and ordering based on options length (number of characters or words), but they were both much less frequently recommended.</p> <p>Logical ordering criterion was the most commonly advanced by guides, even if not all authors agreed on its adequacy. One guide stated that neither alphabetical ordering nor ordering by length should be implemented (ARRT Examination Requirements and Psychometrics, [<reflink idref="bib3" id="ref62">3</reflink>]). Another guide discouraged using an alphabetical ordering criterion because it requires option order to be checked every time modifications are made during the options writing process (McDonald, [<reflink idref="bib67" id="ref63">67</reflink>]).</p> <p>The main rationale for presenting options in a consistent sequential order (whatever that might be) was that this allows answer keys to be positioned without following any patterns. Thus, any test developers' bias to place answer keys somewhat systematically in a particular position would be removed and no clues to the correct responses provided to examinees, just as if key randomization was applied instead. Additionally, but less frequently, guide authors mentioned that arranging options in a logical order could help test takers explore options more effectively.</p> <p>In sum, most item-writing guides considered ordering options to be an appropriate strategy to remove potential clues to correct responses. Among the various ordering strategies, ordering options following some logical criteria was more frequently recommended than scrambling. It is also worth mentioning that some guides encourage using a combination of ordering techniques (e.g., ordering options numerically when possible and randomizing when not, Gierl et al., [<reflink idref="bib36" id="ref64">36</reflink>]).</p> <hd id="AN0181729311-12">3.2. Guidelines for the Positioning of Answer Keys: Balancing Vs. Randomization</hd> <p>Guidelines regarding the positioning of answer keys and their observed frequency in item-writing guides are presented in Table 5. These guidelines could be grouped into two categories: balancing and randomization, the most frequent ones being key randomization and key global balancing. Key randomization involves generating a random distribution and/or sequence of answer keys at test level. Concretely, it means that key position is randomly assigned to each item or that the sequence of key positions across the test is scrambled after the number of answer keys for each option position has been fixed. A kindred but much less frequently recommended strategy is key variation, which consists of arbitrarily changing key positions to prevent the emergence of too obvious global and/or local patterns of answer keys.</p> <p>On the other hand, key global balancing involves placing the same number of answer keys (exact balancing) or at least a similar number of keys (approximative balancing) in each option position across the test so that no option positions are noticeably over keyed or under keyed. Guides generally promoted approximative balancing (31.1%, <emph>n</emph> = 74) more than exact balancing (9.7%, <emph>n</emph> = 23). Key balancing can also be applied locally, i.e., at item series level. However, local key balancing, mostly aimed at removing long runs of keys located at the same position, is rarely recommended.</p> <p>The rationale for key positioning and option ordering strategies is similar: controlling for the possible emergence of clues to correct responses. Key global balancing was conceived to ensure that examinees who repeatedly and strategically select the same option position do not achieve a higher score than the one they would have obtained by random guessing. Randomizing key position makes the sequence of answer keys unpredictable, removing the emergence of any systematic, exploitable patterns. Key global balancing and randomization are also advanced as useful in removing test developers' tendency to over-key middle options.</p> <p>No authors presented arguments against key randomization, but two guides argued that key global balancing should not be used because examinees might take advantage of the balancing procedure by practicing balanced guessing, i.e., filling up missing responses by selecting positions they did not frequently select before (Haladyna & Rodriguez, [<reflink idref="bib46" id="ref65">46</reflink>]; Schuwirth & Pearce, [<reflink idref="bib93" id="ref66">93</reflink>]). Some authors considered that key global balancing and key randomization should be applied together so that the final number of answer keys in each option position is similar but with a random sequence of keys (Broom, [<reflink idref="bib13" id="ref67">13</reflink>]; Conoley & O'Neil, [<reflink idref="bib22" id="ref68">22</reflink>]; Gerberich et al., [<reflink idref="bib35" id="ref69">35</reflink>]; Greene et al., [<reflink idref="bib39" id="ref70">39</reflink>]; Herman, [<reflink idref="bib48" id="ref71">48</reflink>]; Jacobs & Chase, [<reflink idref="bib55" id="ref72">55</reflink>]; McBeath, [<reflink idref="bib66" id="ref73">66</reflink>]; Miller et al., [<reflink idref="bib72" id="ref74">72</reflink>]; Nelson, [<reflink idref="bib80" id="ref75">80</reflink>]; Noll, [<reflink idref="bib81" id="ref76">81</reflink>]; Popham, [<reflink idref="bib87" id="ref77">87</reflink>]; Thorndike & Hagen, [<reflink idref="bib101" id="ref78">101</reflink>]; Wick, [<reflink idref="bib105" id="ref79">105</reflink>]; Wiersma & Jurs, [<reflink idref="bib106" id="ref80">106</reflink>]).</p> <p>However, many authors suggest that key positioning procedures are not necessary and should thus not be adopted if ordering procedures have been implemented (e.g., Davis, [<reflink idref="bib25" id="ref81">25</reflink>]; Gay, [<reflink idref="bib33" id="ref82">33</reflink>]; Suski & Banta, [<reflink idref="bib97" id="ref83">97</reflink>]). Some guides claimed that balancing is not necessary when some ordering procedure (or key randomization) has been applied, since these procedures already lead to a balanced distribution of answer keys (Ebel, [<reflink idref="bib29" id="ref84">29</reflink>]; Green, [<reflink idref="bib38" id="ref85">38</reflink>]; Gronlund, [<reflink idref="bib42" id="ref86">42</reflink>]; Heaton, [<reflink idref="bib47" id="ref87">47</reflink>]; Hopkins & Antes, [<reflink idref="bib52" id="ref88">52</reflink>]; Jacobs, [<reflink idref="bib54" id="ref89">54</reflink>]; Ory, [<reflink idref="bib84" id="ref90">84</reflink>]; Wesman, [<reflink idref="bib104" id="ref91">104</reflink>]). Other guides suggested that key randomization is unnecessary if ordering strategies have been used (Miller & Erickson, [<reflink idref="bib73" id="ref92">73</reflink>], [<reflink idref="bib74" id="ref93">74</reflink>]) because the key position would be already randomized (or at least be varied) when alphabetical, numerical, and/or logical ordering has been implemented (Abdul Rahim et al., [<reflink idref="bib1" id="ref94">1</reflink>]; Balaha, [<reflink idref="bib6" id="ref95">6</reflink>]; Begum, [<reflink idref="bib9" id="ref96">9</reflink>]; Bloom et al., [<reflink idref="bib10" id="ref97">10</reflink>]; Brown, [<reflink idref="bib14" id="ref98">14</reflink>]; Brown, [<reflink idref="bib15" id="ref99">15</reflink>]; Burton et al., [<reflink idref="bib16" id="ref100">16</reflink>]; Clegg & Cashin, [<reflink idref="bib20" id="ref101">20</reflink>]; Gareis & Grant, [<reflink idref="bib32" id="ref102">32</reflink>]; Gronlund, [<reflink idref="bib40" id="ref103">40</reflink>], [<reflink idref="bib41" id="ref104">41</reflink>], [<reflink idref="bib42" id="ref105">42</reflink>]; Holland & Stevens, [<reflink idref="bib50" id="ref106">50</reflink>]; Kranz et al., [<reflink idref="bib58" id="ref107">58</reflink>]; Masters et al., [<reflink idref="bib65" id="ref108">65</reflink>]; Meredith, [<reflink idref="bib70" id="ref109">70</reflink>]; Oermann & Gaberson, [<reflink idref="bib82" id="ref110">82</reflink>]; Rieck, [<reflink idref="bib89" id="ref111">89</reflink>]; Scannell & Tracy, [<reflink idref="bib92" id="ref112">92</reflink>]; Stevens et al., [<reflink idref="bib96" id="ref113">96</reflink>]). Some authors added that key randomization makes no sense if the order of options has been scrambled (Bloom et al., [<reflink idref="bib10" id="ref114">10</reflink>]).</p> <hd id="AN0181729311-13">3.3. Guidelines from the Most Theoretically Sound Guides</hd> <p>The analysis of the number of relevant cited sources variable allowed the identification of 16 item-writing guides that were more theoretically sound than the rest. These "well-referenced" guides analyzed frequency of the guidelines advanced in literature (Balaha, [<reflink idref="bib6" id="ref115">6</reflink>]; Coffman et al., [<reflink idref="bib21" id="ref116">21</reflink>]; Cox, [<reflink idref="bib24" id="ref117">24</reflink>]; Ellsworth et al., [<reflink idref="bib30" id="ref118">30</reflink>]; Frey et al., [<reflink idref="bib31" id="ref119">31</reflink>]; Gierl et al., [<reflink idref="bib36" id="ref120">36</reflink>]; Haladyna & Downing, [<reflink idref="bib44" id="ref121">44</reflink>]; Haladyna et al., [<reflink idref="bib45" id="ref122">45</reflink>]; Tarrant et al., [<reflink idref="bib98" id="ref123">98</reflink>]; Trump & Haggerty, [<reflink idref="bib102" id="ref124">102</reflink>]), reviewed many item-writing guides and empirical studies about option position (Breakall et al., [<reflink idref="bib12" id="ref125">12</reflink>]; Dell & Wantuch, [<reflink idref="bib26" id="ref126">26</reflink>]; Tarrant & Ware, [<reflink idref="bib100" id="ref127">100</reflink>]), or performed a formal analysis of the guidelines' taxonomy (Moreno et al., [<reflink idref="bib78" id="ref128">78</reflink>], [<reflink idref="bib79" id="ref129">79</reflink>], [<reflink idref="bib77" id="ref130">77</reflink>]). Note that three guides fulfilled the first two criteria (Gierl et al., [<reflink idref="bib36" id="ref131">36</reflink>]; Haladyna & Downing, [<reflink idref="bib44" id="ref132">44</reflink>]; Trump & Haggerty, [<reflink idref="bib102" id="ref133">102</reflink>]).</p> <p>As observed for the whole database of guides, most well-referenced guides were published during the twenty-first century (<emph>n</emph> = 13), were journal articles (<emph>n</emph> = 14), and came from the education field or health sciences (<emph>n</emph> = 13). Most well-referenced guides provided recommendations for option ordering or key positioning (<emph>n</emph> = 15), confirming that response option placement is considered a relevant issue in test development. Consistent with findings for the whole database, the most consensual recommendations were numerical and logical ordering, with key randomization, key balancing, and alphabetical ordering advanced as frequent recommendations. Nonetheless, two differences emerged. On the one hand, by-length ordering, key variation, and local key balancing were never (or seldom) mentioned in well-referenced guides, suggesting that these recommendations are suboptimal. On the other hand, key balancing was less frequently recommended than key randomization, suggesting that randomization might be a better option placement strategy (Table 5).</p> <hd id="AN0181729311-14">4. Discussion</hd> <p>In this review, most item-writing guides available in literature were identified by means of a systematic search, and recommendations on the placement of response options were systematically analyzed. Six hundred fifty-four recommendations from 238 guides were identified. This impressive amount of information suggests that response option placement in multiple-choice tests is relevant in educational assessment and makes this research synthesis potentially useful to distinguish which guidelines should be adhered to. Two overarching approaches to response option placement were observed: key positioning and option ordering. These approaches aim at preventing position-based test-taking strategies, a test-wiseness tool that examinees may use, from being effective. The appropriateness of recommendations advanced in the reviewed literature is now discussed in light of results from a systematic review on where test developers place correct answers in tests (Lions et al., [<reflink idref="bib63" id="ref134">63</reflink>]) and results from empirical studies evaluating the effect of option position on examinees' responses (e.g., see Lions et al., [<reflink idref="bib64" id="ref135">64</reflink>]).</p> <hd id="AN0181729311-15">4.1. Should the Distribution And/Or Sequence of Answer Keys Be Manipulated or Checked?</hd> <p>Although some guide authors claim that there is no need to manipulate key position when option ordering is implemented (e.g., Balaha, [<reflink idref="bib6" id="ref136">6</reflink>]), many item-writing guides recommend test developers to use specific key positioning strategies. Key randomization and key global balancing were the most frequently recommended, with arbitrary placement of keys (also labeled key variation) advanced only by a handful of guides. Since one goal of option placement guidelines is masking the location of the correct answer within the option lists, it seems only natural that many guides provide direct recommendations on how to distribute answer keys in multiple-choice tests.</p> <p>As mentioned by a few guides (Haladyna & Rodriguez, [<reflink idref="bib46" id="ref137">46</reflink>]; Schuwirth & Pearce, [<reflink idref="bib93" id="ref138">93</reflink>]), empirical results show that global key balancing should actually be avoided. Examinees could take advantage of a balanced distribution of answer keys and practice balanced guessing to identify the correct option for items they cannot answer based on their content knowledge (Bar-Hillel & Attali, [<reflink idref="bib7" id="ref139">7</reflink>]). Similarly, local key balancing is also not recommended because empirical studies have shown that examinees avoid repeatedly selecting one option position, expecting infrequent key repetition in tests (Lee, [<reflink idref="bib61" id="ref140">61</reflink>]). As for arbitrary manipulations of key position, authors of empirical studies have mentioned that they are suboptimal because they do not control for the possible emergence of test developers' keying bias (Joseph, [<reflink idref="bib56" id="ref141">56</reflink>]). Key randomization would thus be the best key-positioning strategy. Authors of empirical studies have clarified that it rarely leads to an even distribution of keys and has been adopted by several institutions in the testing industry (Attali & Bar‐Hillel, [<reflink idref="bib4" id="ref142">4</reflink>]). Consistently, the most theoretically sound guides identified in this review do not include local balancing and rarely include key variation among recommendations. They recommend key randomization more frequently than key balancing.</p> <hd id="AN0181729311-16">4.2. Should Response Options Be Ordered? If So, Which Criteria Should Be Followed?</hd> <p>Most item-writing guides advance that response options should be arranged in some order and that ordering techniques effectively eliminate position clues to the correct option. Guides generally favor logical ordering of options over random ordering. However, this issue is far from settled, with some authors of empirical studies suggesting that scrambling option order might be a more acceptable strategy than ordering them "logically" (McLeod et al., [<reflink idref="bib69" id="ref143">69</reflink>]).</p> <p>Surprisingly, the benefits of logical ordering criteria advanced by guides do not seem to be supported by empirical studies. More precisely, most guides do not either cite or provide evidence that logical ordering regularly entails key balancing or key randomization (see Gierl et al., [<reflink idref="bib36" id="ref144">36</reflink>] for a related discussion). When a comprehensive search of empirical studies supporting the alleged benefits of logical ordering was conducted in literature other than the guides themselves (Lions et al., [<reflink idref="bib63" id="ref145">63</reflink>]), almost no empirical evidence backing this placement strategy was found. One study related to this topic (Shantikumar et al., [<reflink idref="bib94" id="ref146">94</reflink>]) reports that arranging options in alphabetical order leads to obtaining an adequate distribution of answer keys (approximately balanced distribution). Alphabetical order, however, is not universally considered logical, and there is no debate about alphabetical ordering actually leading to randomly distributed answer keys. When considering numerical ordering, things are different. There seems to be no evidence supporting that this procedure involves randomizing keys. Moreover, the divergent view that the correct option of numerical items is usually located at a middle option rather than at the edges, advanced by several authors of empirical studies (Geiger & Higgins, [<reflink idref="bib34" id="ref147">34</reflink>]; Langer et al., [<reflink idref="bib60" id="ref148">60</reflink>]; Oluwafemiv & Ifedayo, [<reflink idref="bib83" id="ref149">83</reflink>]; Walsh et al., [<reflink idref="bib103" id="ref150">103</reflink>]), is also unsupported.</p> <p>As for the claim that logical ordering makes options' text exploration easier, it does not seem to be rooted in a solid body of empirical evidence either, with a remarkable absence of studies assessing the effects of option ordering either on reading efficiency or on eye-tracking measures during reading. One empirical study addressing the issue (Huntley & Welch, [<reflink idref="bib53" id="ref151">53</reflink>]) inspected examinees' performance to items with numerical options, comparing options presented in ascending/descending order against options in random order, and found no overall statistically significant differences between option ordering procedures.</p> <p>Providing methodologically sound evidence of the general lack of empirical studies supporting some very usual claims regarding logical ordering is beyond the scope of the present article. Therefore, no emphatic claims can be made here in terms of logical ordering criteria being unsubstantiated. However, the fact remains that many guides do claim that logical ordering entails key randomization and makes items easier to read without providing any robust empirical evidence whatsoever. Given the prominence of logical ordering criteria in literature, it might be worth pondering these recommendations with care.</p> <p>In contrast to logical ordering, randomizing the option order of every test item unquestionably leads to key randomization at test level (see Joseph, [<reflink idref="bib56" id="ref152">56</reflink>]; Xu et al., [<reflink idref="bib107" id="ref153">107</reflink>]). Thus, scrambling options seem to be a more reliable strategy to guarantee that key position will not provide any clues to correct responses. Additional advantages of randomizing option order are that it potentially minimizes item exposure effects (Gottlieb et al., [<reflink idref="bib37" id="ref154">37</reflink>]) and can be automatically implemented with some computing devices. Therefore, it is an effortless procedure, in contrast with logical ordering procedures that very often require human intervention every time options are modified.</p> <hd id="AN0181729311-17">4.3. Recommendations on Response Option Placement: A Proposal</hd> <p>Any option placement strategies that do not provide clues to correct responses will be appropriate. The most effective strategy might be arranging options in random order for each item, given the fact that it is an effortless procedure that guarantees that both the distribution of keys and their sequence are almost unpredictable. On the contrary, arranging options in a logical order has not been shown to offer such a guarantee and might be time-consuming because it requires checking and possibly revising the order every time an option's content is modified. Randomizing response order seems to be preferable to key randomization because not all test-design applications (e.g., learning management system online quiz tool) allow specifically randomizing key position and because key randomization does not control for the potential effects of distractors' position on item responses, which might sometimes lead to unintended side effects (Authors, [<reflink idref="bib62" id="ref155">62</reflink>], [<reflink idref="bib64" id="ref156">64</reflink>]; Kiat et al., [<reflink idref="bib57" id="ref157">57</reflink>]; Shin et al., [<reflink idref="bib95" id="ref158">95</reflink>]).</p> <p>Additionally, scrambling options for each test form or for each examinee when several forms containing the same items are administered is not recommended. It is a potentially effective strategy to reduce cheating (Gierl et al., [<reflink idref="bib36" id="ref159">36</reflink>]), but empirical studies have shown that the same item used in different test forms with key position modifications will not necessarily be equivalent (Hagenmüller, [<reflink idref="bib43" id="ref160">43</reflink>]). The order of options had better remain the same for all examinees, at least when stakes are high, because empirical results show it could impact item difficulty and might thus inject unacceptable test inequity (Bolt et al., [<reflink idref="bib11" id="ref161">11</reflink>]; Holzknecht et al., [<reflink idref="bib51" id="ref162">51</reflink>]). Consistently, the option order of items administered in computer adaptive mode (CAT) should be held fixed. Moreover, the result of the randomization process should not be corrected to avoid key repetition or balance answer keys' distribution because such corrections could alter the unpredictable nature of the obtained distribution and sequence of answer keys. All these recommendations apply to both standardized testing and classroom assessment.</p> <p>Item-writing guidelines should be worded in the most straightforward possible way and should not require any supplementary information for users to implement them. An adequate recommendation on this topic would be the following one: "<emph>Randomize the order of response options for each item using a computing device. Administer the items with the same order of options for all examinees. Make sure to let examinees know what you did and explain to them through a narrative and examples adapted to their age what randomization is, clearly stating that position-based strategies will be useless. Use the same order of options if readministering the items</emph>." This recommendation includes giving clear, straightforward information to examinees since they should be encouraged to respond based on knowledge without resorting to content-free strategies (Moreno et al., [<reflink idref="bib77" id="ref163">77</reflink>]). It also includes explaining what random and randomization exactly mean because people typically fail to fully understand these concepts (Bar-Hillel & Wagenaar, [<reflink idref="bib8" id="ref164">8</reflink>]; Paul et al., [<reflink idref="bib86" id="ref165">86</reflink>]). Further empirical research, such as eye-tracking studies, might be of help to either challenge or confirm the adequacy of this recommendation.</p> <p>This review allowed us to detect some imprecisions or misconceptions about option placement. For instance, the term logical was used to label different kinds of ordering. Guides might consider refraining from using this term or at least they should carefully define it, providing a list of examples of what is meant. If numerical ordering is recommended, it is important to explicitly state whether this only applies to integers (0, 1, 2, 3, 4 ...) or to other kinds of numbers as well, such as negative numbers, decimals, or fractions. Similarly, guides recommending alphabetical ordering or ordering by length need to specify whether this applies to whole sentences or to single words only. More generally, guides providing several option placement guidelines should make sure that these guidelines do not conflict with each other and should provide details on the involved placement procedures, so that developers may put guidelines into practice as straightforwardly as possible. Details on how option ordering or key positioning strategies can be implemented are rarely given by guide authors but can certainly be valuable to optimize the test-design processes.</p> <hd id="AN0181729311-18">5. Conclusion</hd> <p>Constructing multiple-choice tests with a high level of validity and fairness is a crucial task to measure learning. It is also a delicate art. There is no doubt that item-writing guides are and have been helpful. However, over the years, so many guides have been published that item-writing literature may overwhelm test developers. In this context, comprehensive reviews are valuable in identifying which guidelines should be preferentially used. This review sheds light on how to address one item-construction issue, namely the response option placement, and will hopefully encourage researchers willing to contribute to the item-writing literature to study other issues based on the database of guides provided in this study (see supplementary file).</p> <hd id="AN0181729311-19">Acknowledgments</hd> <p>We sincerely thank Camilo Quezada Gaponov for editing the manuscript.</p> <hd id="AN0181729311-20">Disclosure Statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <hd id="AN0181729311-21">Author Contributions</hd> <p>SL, MPB, PD, and CM developed the study concept; SL and MPB performed data collection, encoding, and analyses; GO and JL provided crucial information about the study design and results' presentation. SL drafted the manuscript. All the other authors provided critical revisions and have approved the final version of the article.</p> <hd id="AN0181729311-22">Data Availability Statement</hd> <p>The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials. Raw data are available from the corresponding author, SL, upon request.</p> <hd id="AN0181729311-23">Supplementary Material</hd> <p>Supplemental data for this article can be accessed online at https://doi.org/10.1080/08957347.2024.2424551</p> <ref id="AN0181729311-24"> <title> References </title> <blist> <bibl id="bib1" idref="ref55" type="bt">1</bibl> <bibtext> Abdul Rahim, A. F., Simok, A. A., & Abdull Wahab, S. F. (2022). A guide for writing single best answer questions to assess higher-order thinking skills based on learning outcomes. Education in Medicine Journal, 14 (2), 111 – 124. https://doi.org/10.21315/eimj2022.14.2.9</bibtext> </blist> <blist> <bibl id="bib2" idref="ref34" type="bt">2</bibl> <bibtext> Albano, A. D., & Rodriguez, M. C. (2018). Item development research and practice. In S. N. Elliott, R. J. Kettler, P. A. Beddow, & A. Kurz (Eds.), Handbook of accessible instruction and testing practices: Issues, innovations, and applications (pp. 181 – 198). Springer Publishing.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref57" type="bt">3</bibl> <bibtext> ARRT Examination Requirements and Psychometrics. (2019). Item writing manual. The American Registry of Radiologic Technologists. https://assets-us-01.kc-usercontent.com/406ac8c6-58e8-00b3-e3c1-0c312965deb2/56195351-3925-4890-a745-fd87c2b1a110/Item%20Writing%20Manual.pdf •.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref25" type="bt">4</bibl> <bibtext> Attali, Y., & Bar‐Hillel, M. (2003). Guess where: The position of correct answers in multiple‐choice test items as a psychometric variable. Journal of Educational Measurement, 40 (2), 109 – 128. https://doi.org/10.1111/j.1745-3984.2003.tb01099.x</bibtext> </blist> <blist> <bibl id="bib5" idref="ref15" type="bt">5</bibl> <bibtext> Bailey, C. D., Briggs, J. W., & Irving, J. H. (2022). Test-wiseness and multiple-choice accounting questions: Implications for instructors. Issues in Accounting Education, 37 (2), 1 – 14. https://doi.org/10.2308/ISSUES-2021-025</bibtext> </blist> <blist> <bibl id="bib6" idref="ref35" type="bt">6</bibl> <bibtext> Balaha, M. H. (2019). Simplified guidelines for multiple-choice question writing to increase faculty compliance and ensure valid student results. Education in Medicine Journal, 11 (4), 1 – 17. https://doi.org/10.21315/eimj2019.11.4.1 •.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref61" type="bt">7</bibl> <bibtext> Bar-Hillel, M., & Attali, Y. (2002). Seek whence: Answer sequences and their consequences in key-balanced multiple-choice tests. American Statistician, 56 (4), 299 – 303. https://doi.org/10.1198/000313002623</bibtext> </blist> <blist> <bibl id="bib8" idref="ref164" type="bt">8</bibl> <bibtext> Bar-Hillel, M., & Wagenaar, W. A. (1991). The perception of randomness. Advances in Applied Mathematics, 12 (4), 428 – 454. https://doi.org/10.1016/0196-8858(91)90029-I</bibtext> </blist> <blist> <bibl id="bib9" idref="ref96" type="bt">9</bibl> <bibtext> Begum, T. (2012). A guideline on developing effective multiple choice questions and construction of single best answer format. Journal of Bangladesh College of Physicians & Surgeons, 30 (3), 159 – 166. https://doi.org/10.3329/jbcps.v30i3.12466 •.</bibtext> </blist> <blist> <bibtext> Bloom, B. S., Madaus, G. F., & Hastings, J. T. (1981). Evaluation to improve learning. McGraw-Hill •.</bibtext> </blist> <blist> <bibtext> Bolt, D. M., Kim, N., Wollack, J., Pan, Y., Eckerly, C., & Sowles, J. (2020). A psychometric model for discrete-option multiple-choice items. Applied Psychological Measurement, 44 (1), 33 – 48. https://doi.org/10.1177/0146621619835499</bibtext> </blist> <blist> <bibtext> Breakall, J., Randles, C., & Tasker, R. (2019). Development and use of a multiple-choice item writing flaws evaluation instrument in the context of general chemistry. Chemistry Education Research and Practice, 20 (2), 369 – 382. https://doi.org/10.1039/c8rp00262b •.</bibtext> </blist> <blist> <bibtext> Broom, M. E. (1939). Educational measurements in the elementary school. McGraw-Hill •.</bibtext> </blist> <blist> <bibtext> Brown, F. G. (1981). Measuring classroom achievement. Harcourt School •.</bibtext> </blist> <blist> <bibtext> Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. McGraw Hill •.</bibtext> </blist> <blist> <bibtext> Burton, S. J., Sudweeks, R. R., Merrill, P. F., & Wood, B. (1991). How to prepare better multiple-choice test items: Guidelines for university faculty (Testing Service Report). Department of Instructional Science, Brigham Young University. https://testing.byu.edu/handbooks/betteritems.pdf •.</bibtext> </blist> <blist> <bibtext> Carnegie, J. A. (2017). Does correct answer distribution influence student choices when writing multiple choice examinations? Canadian Journal for the Scholarship of Teaching and Learning, 8 (1), article 11. https://doi.org/10.5206/cjsotl-rcacea.2017.1.11</bibtext> </blist> <blist> <bibtext> Carter, K. (1986). Test‐wiseness for teachers and students. Educational Measurement Issues & Practice, 5 (4), 20 – 23. https://doi.org/10.1111/j.1745-3992.1986.tb00495.x</bibtext> </blist> <blist> <bibtext> Casu, G., & García-García, C. (2019). Differential length and overlap with the stem in multiple-choice item options: A pilot experiment. Psicología Educativa, 25 (1), 43 – 48. https://doi.org/10.5093/psed2018a20</bibtext> </blist> <blist> <bibtext> Clegg, V. L., & Cashin, W. E. (1986). Improving multiple-choice tests (ED298150). ERIC. Center for Faculty Evaluation and Development, Kansas State University. https://eric.ed.gov/?id=ED298150 •.</bibtext> </blist> <blist> <bibtext> Coffman, J., Rencis, J. J., Jensen, D. J., Brown, A. O., White, C., Liu, J., & Kaufman, K. (2010, June 20–23). Structured process for writing, revising, and assessing multiple-choice quizzes. [Paper presentation]. American Society for Engineering Education Annual Conference and Exposition, Louisville, Kentucky, US. https://peer.asee.org/structured-process-for-writing-revising-and-assessing-multiple-choice-quizzes.pdf •.</bibtext> </blist> <blist> <bibtext> Conoley, J., & O'Neil, H. (1979). A prime for developing test items. In H. O'Neil (Ed.), Procedures for instructional systems development (pp. 95 – 128). Academic Press •.</bibtext> </blist> <blist> <bibtext> Cooper, H. (2017). Research synthesis and meta-analysis: A step-by-step approach (Vol. 2). Sage publications.</bibtext> </blist> <blist> <bibtext> Cox, C. W. (2019). Best practice tips for the assessment of learning of undergraduate nursing students via multiple-choice questions. Nursing Education Perspectives, 40 (4), 228 – 230. https://doi.org/10.1097/01.nep.0000000000000456 •.</bibtext> </blist> <blist> <bibtext> Davis, B. G. (1993). Multiple-choice and matching tests. In B. G. Davis (Ed.), Tools for teaching (pp. 343 – 419). Jossey-Bass Inc •.</bibtext> </blist> <blist> <bibtext> Dell, K. A., & Wantuch, G. A. (2017). How-to-guide for writing multiple choice questions for the pharmacy instructor. Currents in Pharmacy Teaching & Learning, 9 (1), 137 – 144. https://doi.org/10.1016/j.cptl.2016.08.036 •.</bibtext> </blist> <blist> <bibtext> DeVore, S., Stewart, J., & Stewart, G. (2016). Examining the effects of testwiseness in conceptual physics evaluations. Physical Review Physics Education Research, 12 (2), 020138. https://doi.org/10.1103/PhysRevPhysEducRes.12.020138</bibtext> </blist> <blist> <bibtext> Downing, S. M. (2005). The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences Education, 10 (2), 133 – 143. https://doi.org/10.1007/s10459-004-4019-5</bibtext> </blist> <blist> <bibtext> Ebel, R. L. (1951). Writing the test item. In E. F. Lindquist (Ed.), Educational measurement (1st ed. pp., 185 – 249). American Council on Education •.</bibtext> </blist> <blist> <bibtext> Ellsworth, R. A., Dunnell, P., & Duell, O. K. (1990). Multiple-choice test items: What are textbook authors telling teachers? Journal of Educational Research, 83 (5), 289 – 293. https://doi.org/10.1080/00220671.1990.10885972 •.</bibtext> </blist> <blist> <bibtext> Frey, B. B., Petersen, S., Edwards, L., Pedrotti, J., & Peyton, V. (2005). Item-writing rules: Collective wisdom. Teaching & Teacher Education, 21 (4), 357 – 364. https://doi.org/10.1016/j.tate.2005.01.008 •.</bibtext> </blist> <blist> <bibtext> Gareis, C. R., & Grant, L. W. (2015). Teacher-made assessments: How to connect curriculum, instruction, and student learning (2nd ed.). Routledge. https://doi.org/10.4324/9781315773414 •.</bibtext> </blist> <blist> <bibtext> Gay, L. R. (1980). Educational evaluation and measurement: Competencies for analysis and application. Merrill Publishing Co •.</bibtext> </blist> <blist> <bibtext> Geiger, M. A., & Higgins, M. M. (1997). On the answer-arrangement bias of professional certification examinations in accounting. Accounting Educators' Journal, 9 (2), 89. https://scholarship.richmond.edu/cgi/viewcontent.cgi?article=1003&context=accounting-faculty-publications</bibtext> </blist> <blist> <bibtext> Gerberich, J. R., Greene, H. A., & Jorgensen, A. N. (1962). Measurement and evaluation in the modern school. McKay Co •.</bibtext> </blist> <blist> <bibtext> Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87 (6), 1082 – 1116. https://doi.org/10.3102/0034654317726529 •.</bibtext> </blist> <blist> <bibtext> Gottlieb, M., Bailitz, J., Fix, M., Shappell, E., & Wagner, M. J. (2023). Educator's blueprint: A how‐to guide for developing high‐quality multiple‐choice questions. AEM Education and Training, 7 (1), e10836. https://doi.org/10.1002/aet2.10836 •.</bibtext> </blist> <blist> <bibtext> Green, J. A. (1963). Teacher-made tests. Harper & Row •.</bibtext> </blist> <blist> <bibtext> Greene, H. A., Jorgersen, A. N., & Ooirberich, J. R. (1943). Measurement and evaluation in the secondary school. Longmans, Green and Co •.</bibtext> </blist> <blist> <bibtext> Gronlund, N. E. (1977). Constructing achievement tests (2nd ed.). Prentice-Hall, Inc •.</bibtext> </blist> <blist> <bibtext> Gronlund, N. E. (1981). Measurement and evaluation in teaching (4th ed.). Macmillan Publishing Company, Inc •.</bibtext> </blist> <blist> <bibtext> Gronlund, N. E. (1998). Assessment of student achievement (6th ed.). Allyn & Bacon Publishing •.</bibtext> </blist> <blist> <bibtext> Hagenmüller, B. (2021). On the impact of the response options' position on item difficulty in multiple-choice-items. European Journal of Psychological Assessment, 37 (4), 290 – 299. https://doi.org/10.1027/1015-5759/a000615</bibtext> </blist> <blist> <bibtext> Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple choice item-writing Rules. Applied Measurement in Education, 2 (1), 37 – 50. https://doi.org/10.1207/s15324818ame0201%5f3 •.</bibtext> </blist> <blist> <bibtext> Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15 (3), 309 – 334. https://doi.org/10.1207/s15324818ame1503%5f5 •.</bibtext> </blist> <blist> <bibtext> Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating multiple-choice test items (4th ed.). Routledge •.</bibtext> </blist> <blist> <bibtext> Heaton, J. B. (1977). Writing English language tests. Longman Group UK limited •.</bibtext> </blist> <blist> <bibtext> Herman, J. (1984). A practical approach to local test development: Research into practice project (ED252563). ERIC. Center for the Study of Evaluation, California University. https://files.eric.ed.gov/fulltext/ED252563.pdf •.</bibtext> </blist> <blist> <bibtext> Hohensinn, C., & Baghaei, P. (2017). Does the position of response options in multiple-choice tests matter? Psicológica, 38 (1), 93 – 109. https://files.eric.ed.gov/fulltext/EJ1125979.pdf</bibtext> </blist> <blist> <bibtext> Holland, J., & Stevens, N. (2021). Guidelines for the development of multiple choice items & assessments. Royal College of Surgeons in Ireland. University of Medicine and Health Sciences. https://doi.org/10.25419/rcsi.13947164.v1 •.</bibtext> </blist> <blist> <bibtext> Holzknecht, F., McCray, G., Eberharter, K., Kremmel, B., Zehentner, M., Spiby, R., & Dunlea, J. (2020). The effect of response order on candidate viewing behaviour and item difficulty in a multiple-choice listening test. Language Testing, 38 (1), 41 – 61. https://doi.org/10.1177/0265532220917316</bibtext> </blist> <blist> <bibtext> Hopkins, C. D., & Antes, R. L. (1979). Classroom testing: Construction. Peacock •.</bibtext> </blist> <blist> <bibtext> Huntley, R. M., & Welch, C. J. (1993, April 12–16). Numerical answer options: Logical or random order? [Paper presentation]. Annual Meeting of the American Educational Research Association, Atlanta, GA.</bibtext> </blist> <blist> <bibtext> Jacobs, L. C. (2004). How to write better tests: A handbook for improving test construction skills. Bloomington Evaluation Services & Testing, Indiana University. https://hdl.handle.net/10333/9104 •.</bibtext> </blist> <blist> <bibtext> Jacobs, L. C., & Chase, C. I. (1992). Developing and using tests effectively: A guide for faculty. Jossey-Bass •.</bibtext> </blist> <blist> <bibtext> Joseph, D. C. (2019). Randomize it. Fair procedures when constructing multiple-choice test-keys. Journal of Effective Teaching in Higher Education, 2 (1), 80 – 92. https://doi.org/10.36021/jethe.v2i1.26</bibtext> </blist> <blist> <bibtext> Kiat, J. E., Ong, A. R., & Ganesan, A. (2018). The influence of distractor strength and response order on MCQ responding. Educational Psychology, 38 (3), 368 – 380. https://doi.org/10.1080/01443410.2017.1349877</bibtext> </blist> <blist> <bibtext> Kranz, C., Love, A., Roche, C., Billings, D. M., & Kowalski, K. (2019). How to write a good test question: Nine tips for novice nurse educators. Journal of Continuing Education in Nursing, 50 (1), 12 – 14. https://doi.org/10.3928/00220124-20190102-04 •.</bibtext> </blist> <blist> <bibtext> Lachiver, É. V. (2017). L'utilisation de consignes de rédaction de questions d'examens écrits en médecine et l'effet sur les propriétés psychométriques de celles-ci [The use of writing guidelines for written examination questions in medicine and the effect on the psychometric properties of these] [ Master Thesis ]. University of Sherbrooke Repository. https://savoirs.usherbrooke.ca/handle/11143/10570 •.</bibtext> </blist> <blist> <bibtext> Langer, G., Wark, D., & Johnson, S. (1973). Test-wiseness in objective tests. In P. L. Nacke (Ed.), Diversity in mature reading: Theory and research, 22nd Yearbook of the National Reading Conference (pp. 280 – 287). National Reading Conference, Milwaukee, Wisconsin.</bibtext> </blist> <blist> <bibtext> Lee, C. J. (2019). The test taker's fallacy: How students guess answers on multiple‐choice tests. Journal of Behavioral Decision Making, 32 (2), 140 – 151. https://doi.org/10.1002/bdm.2101</bibtext> </blist> <blist> <bibtext> Lions, S., Monsalve, C., Dartnell, P., Godoy, M.I., Córdova, N., Jiménez, D., Blanco, M.P., Ortega, G., & Lemarié, J. (2021). The position of distractors in multiple-choice test Items: The strongest precede the weakest. Frontiers in Education, 6,731763. https://doi.org/10.3389/feduc.2021.731763</bibtext> </blist> <blist> <bibtext> Lions, S., Monsalve, C., Dartnell, P., Blanco, MP., Ortega, G., & Lemarié, J. (2022). Does the response options placement provide clues to the correct answers in multiple-choice tests? A systematic review. Applied Measurement in Education, 35 (2),133-152. https://doi.org/10.1080/08957347.2022.2067539</bibtext> </blist> <blist> <bibtext> Lions, S., Dartnell, P., Godoy, MI., Córdova, N., Jiménez, D., & Lemarié, J. (2023). Position of correct option and distractors impacts responses to multiple-choice items: evidence from a national test. Educational and Psychological Measurement, 83 (5), 861–884. https://doi.org/10.1177/00131644221132335</bibtext> </blist> <blist> <bibtext> Masters, J. C., Hulsmeyer, B. S., Pike, M. E., Leichty, K., Miller, M. T., & Verst, A. L. (2001). Assessment of multiple-choice questions in selected test banks companying text books used in nursing education. The Journal of Nursing Education, 40 (1), 25 – 32. https://doi.org/10.3928/0148-4834-20010101-07 •.</bibtext> </blist> <blist> <bibtext> McBeath, R. J. (1992). Instructing and evaluating in higher education: Guide book for planning learning outcome. Educational Technology Publications •.</bibtext> </blist> <blist> <bibtext> McDonald, M. E. (2007). The nurse educator's guide to assessing learning outcomes (2nd ed.). Jones and Bartlett Learning •.</bibtext> </blist> <blist> <bibtext> McKenna, P. (2019). Multiple choice questions: Answering correctly and knowing the answer. Interactive Technology & Smart Education, 16 (1), 59 – 73. https://doi.org/10.1108/ITSE-09-2018-0071</bibtext> </blist> <blist> <bibtext> McLeod, I., Zhang, Y., & Yu, H. (2003). Multiple-choice randomization. Journal of Statistics Education, 11 (1). <ulink href="http://ww2.amstat.org/publications/jse/v11n1/mcleod.html">http://ww2.amstat.org/publications/jse/v11n1/mcleod.html</ulink></bibtext> </blist> <blist> <bibtext> Meredith, D. B. (2012, June 10–13). Test preparation and test quality assessment: What I wish someone had told me in the beginning. [Paper presentation]. American Society for Engineering Education Annual Conference, San Antonio, Texas, US •.</bibtext> </blist> <blist> <bibtext> Metfessel, N. S., & Sax, G. (1958). Systematic biases in the keying of correct responses on certain standardized tests. Educational and Psychological Measurement, 18 (4), 787 – 790. https://doi.org/10.1177/001316445801800411</bibtext> </blist> <blist> <bibtext> Miller, M. D., Linn, R. L., & Gronlund, N. E. (2009). Measurement and assessment in teaching (10th ed.). Prentice Hall •.</bibtext> </blist> <blist> <bibtext> Miller, P. W., & Erickson, H. E. (1985). Teacher-written student tests: A guide for planning, creating, administering, and assessing. National Education Association •.</bibtext> </blist> <blist> <bibtext> Miller, P. W., & Erickson, H. E. (1990). How to write tests for students (ED322157). ERIC. National Education Association. https://files.eric.ed.gov/fulltext/ED322157.pdf •.</bibtext> </blist> <blist> <bibtext> Millman, J., Bishop, C. H., & Ebel, R. (1965). An analysis of test-wiseness. Educational and Psychological Measurement, 25 (3), 707 – 726. https://doi.org/10.1177/001316446502500304</bibtext> </blist> <blist> <bibtext> Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & Prisma Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ, 339 (11), 1 – 9. https://doi.org/10.1136/bmj.b2535</bibtext> </blist> <blist> <bibtext> Moreno, R., Martinez, R. J., & Muniz, J. (2015). Guidelines based on validity criteria for the development of multiple choice items. Psicothema, 27 (4), 388 – 394. https://doi.org/10.7334/psicothema2015.110 •.</bibtext> </blist> <blist> <bibtext> Moreno, R., Martínez, R. J., & Muñiz, J. (2004). Directrices para la construcción de ítems de elección múltiple [Guidelines for the construction of multiple-choice items]. Psicothema, 16 (3), 490 – 497 •.</bibtext> </blist> <blist> <bibtext> Moreno, R., Martínez, R. J., & Muñiz, J. (2006). New guidelines for developing multiple-choice items. Methodology, 2 (2), 65 – 72. https://doi.org/10.1027/1614-2241.2.2.65 •.</bibtext> </blist> <blist> <bibtext> Nelson, C. H. (1970). Measurement and evaluation in the classroom. Macmillan Publishing Co •.</bibtext> </blist> <blist> <bibtext> Noll, V. H. (1965). Introduction to educational measurement (2nd ed.). Houghton Mifflin company •.</bibtext> </blist> <blist> <bibtext> Oermann, M. H., & Gaberson, K. B. (2021). Evaluation and testing in nursing education (6th ed.). Springer Publishing •.</bibtext> </blist> <blist> <bibtext> Oluwafemiv, A. T., & Ifedayo, A. E. R. (2013). A study of item response and anchor bias in economics objective tests among senior secondary school students in Osun State. Academic Journal of Interdisciplinary Studies, 2 (3), 173 – 179. https://doi.org/10.5901/ajis.2013.v2n3p173</bibtext> </blist> <blist> <bibtext> Ory, J. C. (1983). Improving your test questions (ED285499). ERIC. Office of Instructional and Management Services, University of Illinois, Urbana-Champaign. https://files.eric.ed.gov/fulltext/ED285499.pdf •.</bibtext> </blist> <blist> <bibtext> Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., & Mulrow, C. D., ... & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372 (71), 1 – 9. https://doi.org/10.1136/bmj.n71</bibtext> </blist> <blist> <bibtext> Paul, S. T., Monda, S., Olausson, S. M., & Reed-Daley, B. (2014). Effects of apophenia on multiple-choice exam performance. SAGE Open, 4 (4), 1 – 7. https://doi.org/10.1177/2158244014556628</bibtext> </blist> <blist> <bibtext> Popham, W. J. (2012). Selected-response tests: Building and bettering (12th ed.). Pearson •.</bibtext> </blist> <blist> <bibtext> Przymuszała, P., Piotrowska, K., Lipski, D., Marciniak, R., & Cerbin-Koczorowska, M. (2020). Guidelines on writing multiple choice questions: A well-received and effective faculty development intervention. SAGE Open, 10 (3), 1 – 12. https://doi.org/10.1177/2158244020947432</bibtext> </blist> <blist> <bibtext> Rieck, W. A. (2002). Putting assessment to the test. The Science Teacher, 69 (2), 46 – 49 •.</bibtext> </blist> <blist> <bibtext> Rubinstein, A., & Salant, Y. (2006). A model of choice from lists. Theoretical Economics, 1 (1), 3 – 17. https://econtheory.org/ojs/index.php/te/article/viewFile/20060003/444/8</bibtext> </blist> <blist> <bibtext> Sarnacki, R. E. (1979). An examination of test-wiseness in the cognitive test domain. Review of Educational Research, 49 (2), 252 – 279. https://doi.org/10.3102/00346543049002252</bibtext> </blist> <blist> <bibtext> Scannell, D. P., & Tracy, D. B. (1975). Testing and measurement in the classroom. Houghton Mifflin Company •.</bibtext> </blist> <blist> <bibtext> Schuwirth, L., & Pearce, J. (2014). Determining the quality of assessment items in collaborations: Aspects to discuss to reach agreement. Department of Education, Australian Government. https://research.acer.edu.au/higher%5feducation/42 •.</bibtext> </blist> <blist> <bibtext> Shantikumar, S., Watson, M., & Handa, A. (2010). Does alphabetisation ensure randomisation of single best answer questions? The Clinical Teacher, 7 (2), 136 – 137. https://doi.org/10.1111/j.1743-498X.2010.00365.x</bibtext> </blist> <blist> <bibtext> Shin, J., Bulut, O., & Gierl, M. J. (2019). The effect of the most-attractive-distractor location on multiple-choice item difficulty. The Journal of Experimental Education, 88 (4), 643 – 659. https://doi.org/10.1080/00220973.2019.1629577</bibtext> </blist> <blist> <bibtext> Stevens, S. P., Palocsay, S. W., & Novoa, L. J. (2022). Practical guidance for writing multiple-choice test questions in introductory analytics courses. INFORMS Transactions on Education, 1 – 19. https://doi.org/10.1287/ited.2022.0274</bibtext> </blist> <blist> <bibtext> Suski, L., & Banta, T. (2009). Assessing student learning: A common sense guide. Jossey-Bass •.</bibtext> </blist> <blist> <bibtext> Tarrant, M., Knierim, A., Hayes, S. K., & Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education Today, 26 (8), 662 – 671. https://doi.org/10.1016/j.nedt.2006.07.006 •.</bibtext> </blist> <blist> <bibtext> Tarrant, M., & Ware, J. (2008). Impact of item‐writing flaws in multiple‐choice questions on student achievement in high‐stakes nursing assessments. Medical Education, 42 (2), 198 – 206. https://doi.org/10.1111/j.1365-2923.2007.02957.x</bibtext> </blist> <blist> <bibtext> Tarrant, M., & Ware, J. (2012). A framework for improving the quality of multiple-choice assessments. Nurse Educator, 37 (3), 98 – 104. https://doi.org/10.1097/NNE.0b013e31825041d0 •.</bibtext> </blist> <blist> <bibtext> Thorndike, R. L., & Hagen, E. (1961). Measurement and evaluation in psychology and education (2nd ed.). Wiley •.</bibtext> </blist> <blist> <bibtext> Trump, J. B., & Haggerty, H. R. (1952). Test construction manual: Basic principles in achievement test item construction. Department of the Army •.</bibtext> </blist> <blist> <bibtext> Walsh, J. L., Harris, B. H. L., & Smith, P. E. (2017). Single best answer question-writing tips for clinicians. Postgraduate Medical Journal, 93 (1096), 76 – 81. https://doi.org/10.1136/postgradmedj-2015-133893 •.</bibtext> </blist> <blist> <bibtext> Wesman, A. G. (1971). Writing the test item. In R. L. Thorndike (Ed.), Educational measurement (2nd ed. pp., 81 – 129). American Council on Education •.</bibtext> </blist> <blist> <bibtext> Wick, J. W. (1973). Educational measurement: Where are we going and how will we know when we get there. Merrell Publishing •.</bibtext> </blist> <blist> <bibtext> Wiersma, W., & Jurs, S. G. (1990). Educational measurement and testing (2nd ed.). Allyn and Bacon •.</bibtext> </blist> <blist> <bibtext> Xu, X., Kauer, S., & Tupy, S. (2016). Multiple-choice questions: Tips for optimizing assessment in-seat and online. Scholarship of Teaching and Learning in Psychology, 2 (2), 147 – 158. https://doi.org/10.1037/stl0000062 •.</bibtext> </blist> </ref> <aug> <p>By Séverin Lions; María Paz Blanco; Pablo Dartnell; Carlos Monsalve; Gabriel Ortega and Julie Lemarié</p> <p>Reported by Author; Author; Author; Author; Author; Author</p> <p></p> <p>S. Lions holds a master's degree in Biology from the University of Paris VI, France, and a PhD in Neuroscience from the University of Valparaíso, Chile. He is an associate researcher at the Center for Advanced Research in Education, University of Chile. His research fields are Cognition and Education, with focus on the cognitive processes associated with reading, reading comprehension, and multiple-choice questions solving.</p> <p>M.P. Blanco (psychologist), C. Monsalve (psychologist), and G. Ortega (mathematical engineer) are research assistants at the Center for Advanced Research in Education.</p> <p>P. Dartnell holds an engineering degree in Applied Mathematics from the University of Chile and a PhD in Mathematics from Yale University, USA. He is a researcher at the Center for Advanced Research in Education, an Adjunct Professor at the Mathematical Engineering Department and at the Institute of Education, and a researcher at the Education Area of the Center for Mathematical Modeling, University of Chile. His research fields are Cognition and Mathematics Education, with focus on the cognitive origin of the learning difficulties on key topics in school mathematics, mathematical modeling of learning and mathematical cognition.</p> <p>J. Lemarié holds a master's degree in Cognitive Science from the University of Paris VIII and a PhD in Psychology from the University of Toulouse, France. She is a researcher and Associate Professor at the UT2, CLLE (CNRS-UT2), TMBI (University of Toulouse), and works in collaboration with the Cognition Team of the Center for Advanced Research in Education. Her research fields are Cognition, Learning and Ergonomics.</p> </aug> <nolink nlid="nl1" bibid="bib44" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib46" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib45" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib78" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib79" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib77" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib63" firstref="ref7"></nolink> <nolink nlid="nl8" bibid="bib27" firstref="ref8"></nolink> <nolink nlid="nl9" bibid="bib28" firstref="ref9"></nolink> <nolink nlid="nl10" bibid="bib99" firstref="ref10"></nolink> <nolink nlid="nl11" bibid="bib19" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib68" firstref="ref13"></nolink> <nolink nlid="nl13" bibid="bib75" firstref="ref14"></nolink> <nolink nlid="nl14" bibid="bib90" firstref="ref16"></nolink> <nolink nlid="nl15" bibid="bib91" firstref="ref17"></nolink> <nolink nlid="nl16" bibid="bib17" firstref="ref18"></nolink> <nolink nlid="nl17" bibid="bib61" firstref="ref19"></nolink> <nolink nlid="nl18" bibid="bib71" firstref="ref24"></nolink> <nolink nlid="nl19" bibid="bib18" firstref="ref27"></nolink> <nolink nlid="nl20" bibid="bib98" firstref="ref28"></nolink> <nolink nlid="nl21" bibid="bib49" firstref="ref29"></nolink> <nolink nlid="nl22" bibid="bib59" firstref="ref30"></nolink> <nolink nlid="nl23" bibid="bib88" firstref="ref31"></nolink> <nolink nlid="nl24" bibid="bib67" firstref="ref33"></nolink> <nolink nlid="nl25" bibid="bib21" firstref="ref36"></nolink> <nolink nlid="nl26" bibid="bib24" firstref="ref37"></nolink> <nolink nlid="nl27" bibid="bib30" firstref="ref38"></nolink> <nolink nlid="nl28" bibid="bib31" firstref="ref39"></nolink> <nolink nlid="nl29" bibid="bib36" firstref="ref40"></nolink> <nolink nlid="nl30" bibid="bib102" firstref="ref44"></nolink> <nolink nlid="nl31" bibid="bib23" firstref="ref52"></nolink> <nolink nlid="nl32" bibid="bib76" firstref="ref53"></nolink> <nolink nlid="nl33" bibid="bib85" firstref="ref54"></nolink> <nolink nlid="nl34" bibid="bib93" firstref="ref66"></nolink> <nolink nlid="nl35" bibid="bib13" firstref="ref67"></nolink> <nolink nlid="nl36" bibid="bib22" firstref="ref68"></nolink> <nolink nlid="nl37" bibid="bib35" firstref="ref69"></nolink> <nolink nlid="nl38" bibid="bib39" firstref="ref70"></nolink> <nolink nlid="nl39" bibid="bib48" firstref="ref71"></nolink> <nolink nlid="nl40" bibid="bib55" firstref="ref72"></nolink> <nolink nlid="nl41" bibid="bib66" firstref="ref73"></nolink> <nolink nlid="nl42" bibid="bib72" firstref="ref74"></nolink> <nolink nlid="nl43" bibid="bib80" firstref="ref75"></nolink> <nolink nlid="nl44" bibid="bib81" firstref="ref76"></nolink> <nolink nlid="nl45" bibid="bib87" firstref="ref77"></nolink> <nolink nlid="nl46" bibid="bib101" firstref="ref78"></nolink> <nolink nlid="nl47" bibid="bib105" firstref="ref79"></nolink> <nolink nlid="nl48" bibid="bib106" firstref="ref80"></nolink> <nolink nlid="nl49" bibid="bib25" firstref="ref81"></nolink> <nolink nlid="nl50" bibid="bib33" firstref="ref82"></nolink> <nolink nlid="nl51" bibid="bib97" firstref="ref83"></nolink> <nolink nlid="nl52" bibid="bib29" firstref="ref84"></nolink> <nolink nlid="nl53" bibid="bib38" firstref="ref85"></nolink> <nolink nlid="nl54" bibid="bib42" firstref="ref86"></nolink> <nolink nlid="nl55" bibid="bib47" firstref="ref87"></nolink> <nolink nlid="nl56" bibid="bib52" firstref="ref88"></nolink> <nolink nlid="nl57" bibid="bib54" firstref="ref89"></nolink> <nolink nlid="nl58" bibid="bib84" firstref="ref90"></nolink> <nolink nlid="nl59" bibid="bib104" firstref="ref91"></nolink> <nolink nlid="nl60" bibid="bib73" firstref="ref92"></nolink> <nolink nlid="nl61" bibid="bib74" firstref="ref93"></nolink> <nolink nlid="nl62" bibid="bib10" firstref="ref97"></nolink> <nolink nlid="nl63" bibid="bib14" firstref="ref98"></nolink> <nolink nlid="nl64" bibid="bib15" firstref="ref99"></nolink> <nolink nlid="nl65" bibid="bib16" firstref="ref100"></nolink> <nolink nlid="nl66" bibid="bib20" firstref="ref101"></nolink> <nolink nlid="nl67" bibid="bib32" firstref="ref102"></nolink> <nolink nlid="nl68" bibid="bib40" firstref="ref103"></nolink> <nolink nlid="nl69" bibid="bib41" firstref="ref104"></nolink> <nolink nlid="nl70" bibid="bib50" firstref="ref106"></nolink> <nolink nlid="nl71" bibid="bib58" firstref="ref107"></nolink> <nolink nlid="nl72" bibid="bib65" firstref="ref108"></nolink> <nolink nlid="nl73" bibid="bib70" firstref="ref109"></nolink> <nolink nlid="nl74" bibid="bib82" firstref="ref110"></nolink> <nolink nlid="nl75" bibid="bib89" firstref="ref111"></nolink> <nolink nlid="nl76" bibid="bib92" firstref="ref112"></nolink> <nolink nlid="nl77" bibid="bib96" firstref="ref113"></nolink> <nolink nlid="nl78" bibid="bib12" firstref="ref125"></nolink> <nolink nlid="nl79" bibid="bib26" firstref="ref126"></nolink> <nolink nlid="nl80" bibid="bib100" firstref="ref127"></nolink> <nolink nlid="nl81" bibid="bib64" firstref="ref135"></nolink> <nolink nlid="nl82" bibid="bib56" firstref="ref141"></nolink> <nolink nlid="nl83" bibid="bib69" firstref="ref143"></nolink> <nolink nlid="nl84" bibid="bib94" firstref="ref146"></nolink> <nolink nlid="nl85" bibid="bib34" firstref="ref147"></nolink> <nolink nlid="nl86" bibid="bib60" firstref="ref148"></nolink> <nolink nlid="nl87" bibid="bib83" firstref="ref149"></nolink> <nolink nlid="nl88" bibid="bib103" firstref="ref150"></nolink> <nolink nlid="nl89" bibid="bib53" firstref="ref151"></nolink> <nolink nlid="nl90" bibid="bib107" firstref="ref153"></nolink> <nolink nlid="nl91" bibid="bib37" firstref="ref154"></nolink> <nolink nlid="nl92" bibid="bib62" firstref="ref155"></nolink> <nolink nlid="nl93" bibid="bib57" firstref="ref157"></nolink> <nolink nlid="nl94" bibid="bib95" firstref="ref158"></nolink> <nolink nlid="nl95" bibid="bib43" firstref="ref160"></nolink> <nolink nlid="nl96" bibid="bib11" firstref="ref161"></nolink> <nolink nlid="nl97" bibid="bib51" firstref="ref162"></nolink> <nolink nlid="nl98" bibid="bib86" firstref="ref165"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1454681
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Item-Writing Guidelines on Response Option Placement: A Systematic Review
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Séverin+Lions%22">Séverin Lions</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0697-7974">0000-0003-0697-7974</externalLink>)<br /><searchLink fieldCode="AR" term="%22María+Paz+Blanco%22">María Paz Blanco</searchLink><br /><searchLink fieldCode="AR" term="%22Pablo+Dartnell%22">Pablo Dartnell</searchLink><br /><searchLink fieldCode="AR" term="%22Carlos+Monsalve%22">Carlos Monsalve</searchLink><br /><searchLink fieldCode="AR" term="%22Gabriel+Ortega%22">Gabriel Ortega</searchLink><br /><searchLink fieldCode="AR" term="%22Julie+Lemarié%22">Julie Lemarié</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Applied+Measurement+in+Education%22"><i>Applied Measurement in Education</i></searchLink>. 2024 37(4):289-304.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 16
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Information Analyses
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Multiple+Choice+Tests%22">Multiple Choice Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Guidelines%22">Guidelines</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Responses%22">Responses</searchLink><br /><searchLink fieldCode="DE" term="%22Placement%22">Placement</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/08957347.2024.2424551
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0895-7347<br />1532-4818
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Multiple-choice items are universally used in formal education. Since they should assess learning, not test-wiseness or guesswork, they must be constructed following the highest possible standards. Hundreds of item-writing guides have provided guidelines to help test developers adopt appropriate strategies to define the distribution and sequence of correct answers across tests and, thus, not inadvertently provide clues to correct responses. In this study, these option placement guidelines were systematically reviewed. Consensual recommendations and recommendations from the most theoretically sound guides were identified. Additionally, results from empirical studies evaluating the effect of option position on examinees' responses and research reports on where test developers place correct answers in tests were considered to better examine these recommendations' relevance and suitability. Consistent and easy-to-follow recommendations about where and how to place options in items and tests are finally provided so that developers can gauge the adequacy of their option placement strategies.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1454681
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1454681
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/08957347.2024.2424551
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
        StartPage: 289
    Subjects:
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Multiple Choice Tests
        Type: general
      – SubjectFull: Guidelines
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Responses
        Type: general
      – SubjectFull: Placement
        Type: general
    Titles:
      – TitleFull: Item-Writing Guidelines on Response Option Placement: A Systematic Review
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Séverin Lions
      – PersonEntity:
          Name:
            NameFull: María Paz Blanco
      – PersonEntity:
          Name:
            NameFull: Pablo Dartnell
      – PersonEntity:
          Name:
            NameFull: Carlos Monsalve
      – PersonEntity:
          Name:
            NameFull: Gabriel Ortega
      – PersonEntity:
          Name:
            NameFull: Julie Lemarié
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0895-7347
            – Type: issn-electronic
              Value: 1532-4818
          Numbering:
            – Type: volume
              Value: 37
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Applied Measurement in Education
              Type: main
ResultId 1