View in EDS HTML Full Text PDF Full Text

Does the Format of an Assessment (Closed Book or Open Book) Affect Learning? A Systematic Review of the Literature

Saved in:
Bibliographic Details
Title: Does the Format of an Assessment (Closed Book or Open Book) Affect Learning? A Systematic Review of the Literature
Language: English
Authors: Vahe Permzadian (ORCID 0000-0003-1278-9056), Kit W. Cho (ORCID 0000-0003-0194-9851)
Source: Teaching in Higher Education. 2025 30(4):880-897.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 18
Publication Date: 2025
Document Type: Journal Articles
Information Analyses
Education Level: Higher Education
Postsecondary Education
Descriptors: College Students, Tests, Test Format, Long Term Memory, Academic Achievement, Evaluation Methods, Test Results, Retention (Psychology), Test Selection
DOI: 10.1080/13562517.2023.2289128
ISSN: 1356-2517
1470-1294
Abstract: When administering an in-class exam, a common decision that confronts every instructor is whether the exam format should be closed book or open book. The present review synthesizes research examining the effect of administering closed-book or open-book assessments on long-term learning. Although the overall effect of assessment format on learning was mixed, two robust findings were that closed-book assessments led to a lower rate of forgetting (i.e. the percentage of change in information participants remembered on an initial test compared to that of the final test) relative to open-book assessments and students preferred to take open-book assessments. We also discuss possible moderators (e.g. item overlap) of the effect. Recommendations are provided to instructors who seek guidance regarding the most appropriate assessment format to adopt for their course. We conclude by outlining avenues for future research.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1468349
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEvBYMzQ9sCzQVauFYQy6awAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI4zFcHMc3kzd4zIKQIBEICBm_DnIX8v2h1mi_K1wuWxqJdVagCRcs4YNQAsrEi9h0RvJfNyEQJ9FM0O_eT-DpHNNvAQkXKvFkvgKstS4rc-F20U3SvX7zsDWonj6BPoRR9LVIpjZ-IISKzLPIcUzdI9nvGlMmfEEr4cQrtgpvZDFe7FeF0NNEVCNcDx4GfjVR73pmNmlvy0A90utjUAiRHq8mvk3OQhSuZB2lB_
Text:
  Availability: 1
  Value: <anid>AN0184595461;thd01may.25;2025Apr22.02:22;v2.2.500</anid> <title id="AN0184595461-1">Does the format of an assessment (closed book or open book) affect learning? A systematic review of the literature </title> <p>When administering an in-class exam, a common decision that confronts every instructor is whether the exam format should be closed book or open book. The present review synthesizes research examining the effect of administering closed-book or open-book assessments on long-term learning. Although the overall effect of assessment format on learning was mixed, two robust findings were that closed-book assessments led to a lower rate of forgetting (i.e. the percentage of change in information participants remembered on an initial test compared to that of the final test) relative to open-book assessments and students preferred to take open-book assessments. We also discuss possible moderators (e.g. item overlap) of the effect. Recommendations are provided to instructors who seek guidance regarding the most appropriate assessment format to adopt for their course. We conclude by outlining avenues for future research.</p> <p>Keywords: Closed-book exams; open-book exams; performance; learning; forgetting</p> <p>When determining the format of an in-class exam, college and university faculty typically have a few choices. Should they opt for a closed-book exam or an open-book exam given their options? Importantly, is there a relationship between assessment type (e.g. closed book versus open book) and student learning? These questions are the impetus for the present review. Determining the format of an assessment can be a challenging decision, as seemingly minor changes by instructors can have major implications for learning (Hattie [<reflink idref="bib19" id="ref1">19</reflink>]). Based on our experiences as educators, one of the common questions students typically ask on the first day of class is how assessments will be administered and the format of these assessments. Such questions are warranted given that standard assessments (e.g. tests) typically comprise the majority of students' grades in most classes. Furthermore, in response to the unprecedented emotional impact of COVID-19, some educators have reported implementing pedagogical practices such as administering open-book exams as one method to alleviate student stress (Mathieson, Sutthakorn, and Thomas [<reflink idref="bib24" id="ref2">24</reflink>]). Open-book exams, therefore, could be construed as one strategy to accommodate and meet the needs of individual learning differences, which is one of the central tenets of universal design educational models (Rao, Ok, and Bryant [<reflink idref="bib31" id="ref3">31</reflink>]).</p> <p>A closed-book assessment (CBA) requires students to respond to questions without accessing external materials (Johanns, Dinkens, and Moore [<reflink idref="bib20" id="ref4">20</reflink>]). An open-book assessment (OBA), in comparison, typically allows students to access any class-assigned material deemed necessary (e.g. textbook) to answer exam questions. Although there are other assessment formats (e.g. a CBA with access to a 'cheat sheet,' typically one page of notes prepared by the student), research has largely focused on comparing the effects of these two formats and are therefore the focus of the present review.</p> <p>Performance on these two types of assessment generally requires different cognitive processes. Because there is a greater demand in a CBA to retrieve information from memory, performance can depend on an individual's ability to access and retrieve the correct information or recognize the correct response, particularly if the focus is on assessing retention (i.e. the retrieval of information retained in long-term memory). As such, performance in certain CBAs may simply reflect the ability to memorize information or select a familiar response. Because of this drawback, CBAs that focus on assessing transfer (i.e. the ability to apply retained knowledge in a new context) enable a more rigorous assessment of learning.</p> <p>Performance in an OBA is less dependent on the ability of a test-taker to retrieve information from memory as an individual would have access to the relevant class-assigned materials to respond to questions. This feature of OBAs suggests that testing for retention can lead to a poor indication of learning because correct responses could be based on the test-taker's ability to 'look up' answers rather than a reflection of what they have retained from prior exposure to the material. Thus, as with CBAs, a better use of OBAs is assessing transfer with questions that require the application of knowledge and skills (e.g. 'How you would design an intervention to motivate employees that adopts two theories of work motivation from your readings?'), rather than the recall of knowledge or retention-based questions (e.g. 'Define cognitive dissonance'). Students cannot perform well on application-based questions by relying solely on rote learning or memorization and must adjust their study strategies accordingly (Cahill-Ripley [<reflink idref="bib9" id="ref5">9</reflink>]).</p> <p>Despite these differences, CBAs and OBAs are not mutually exclusive types of assessment. Both formats can be used concurrently by instructors (e.g. an exam with a closed-book section and an open-book section) depending on the assessment's target learning outcomes. Further, providing a correct response in either assessment format will require students to understand a given question and have basic knowledge of the material.</p> <p>Introductory-level undergraduate courses tend to emphasize lower-order cognitive skills and therefore assessments more often focus on testing retention (Momsen et al. [<reflink idref="bib27" id="ref6">27</reflink>]). Although OBAs can be used for this purpose, instructors will typically employ CBAs with multiple-choice questions as they are well-suited for courses with high enrollment (Simkin and Kuechler [<reflink idref="bib39" id="ref7">39</reflink>]). In comparison, advanced-level undergraduate courses and graduate courses tend to emphasize higher-order cognitive skills and therefore assessments are more focused on testing transfer (White [<reflink idref="bib43" id="ref8">43</reflink>]). Although CBAs can serve this objective, instructors are more likely to utilize OBAs with open-ended questions or papers and projects as they are more amenable to courses with lower enrollment (Simkin and Kuechler [<reflink idref="bib39" id="ref9">39</reflink>]).</p> <hd id="AN0184595461-2">The present review</hd> <p>Although there have been recent papers on the effect of various assessment formats (e.g. Bengtsson [<reflink idref="bib5" id="ref10">5</reflink>]; Durning et al. [<reflink idref="bib15" id="ref11">15</reflink>]; Johanns, Dinkens, and Moore [<reflink idref="bib20" id="ref12">20</reflink>]; Larwin, Gorman, and Larwin [<reflink idref="bib23" id="ref13">23</reflink>]), the focus of this work is different from these previous reviews. Specifically, our review focuses on research that is directly applicable to and reflective of the current and standard education setting. Whereas Bengtsson ([<reflink idref="bib5" id="ref14">5</reflink>]) examined take-home exams, this study focuses on 'proctored' (in-class) assessments and therefore excludes take-home assessments. The focus of Larwin, Gorman, and Larwin ([<reflink idref="bib23" id="ref15">23</reflink>]) was mainly on the effect of student-prepared testing aids (i.e. cheat sheets) on performance. This study, however, excludes cheat sheets and only examines the effect of CBAs and OBAs on learning. Whereas Johanns, Dinkens, and Moore ([<reflink idref="bib20" id="ref16">20</reflink>]) examined the effect of assessment format on the development of critical thinking skills (i.e. a specific type of learned skill), this study is concerned with the effect on learning more broadly. Durning et al. ([<reflink idref="bib15" id="ref17">15</reflink>]) conducted a review that compared how CBAs and OBAs affect various outcomes. However, they did not specifically examine the effect on learning and focused on a select population (i.e. students in the healthcare professions). In contrast, the present review focuses on a broader population (i.e. all college or tertiary-level students). Thus, this study makes a different contribution than previous studies that have examined the effect of assessment format, with the present review being more relevant to contemporary educators in higher education. Furthermore, none of the previous reviews assessed learning as a function of forgetting rate, which is important because of its implication for long-term learning. Our review also examines whether students prefer one type of assessment format over another. This information is important because administering an assessment that students are more comfortable with could reduce their levels of test anxiety, which is one of the strongest negative predictors of academic achievement (Schneider and Preckel [<reflink idref="bib37" id="ref18">37</reflink>]). In addition, the present review explores moderators (e.g. study setting) that have ecological relevance for educators and were not discussed in any of the previous reviews. Finally, more than half of the papers we analyzed in the present review were not included in the aforementioned reviews, highlighting the rapid growth of this literature and the need for an updated review.</p> <p>In this review, we define learning as a gain in knowledge, skills, or abilities demonstrated by retention or transfer at least two days after the acquisition process. The acquisition process is the training experience in which individuals engage with the to-be-learned content.[<reflink idref="bib1" id="ref19">1</reflink>] As described by Soderstrom and Bjork ([<reflink idref="bib40" id="ref20">40</reflink>]), performance represents 'temporary fluctuations in behavior or knowledge that can be observed and measured during or immediately after the acquisition process' (<reflink idref="bib176" id="ref21">176</reflink>). Based on these definitions, assessments that occur during or immediately after the acquisition process would measure performance (transient knowledge) rather than learning (enduring knowledge).</p> <p>The distinction between performance and learning is important because performance is sometimes a flawed indicator of learning (Bjork and Bjork [<reflink idref="bib6" id="ref22">6</reflink>]; Soderstrom and Bjork [<reflink idref="bib40" id="ref23">40</reflink>]). Indeed, changes in knowledge and skills during the acquisition process (i.e. an assessment of performance) might simply reflect current accessibility – how easily information can be retrieved from memory at the present time (Clark and Bjork [<reflink idref="bib12" id="ref24">12</reflink>]). Furthermore, empirical evidence shows that learning can occur irrespective of any performance increases (Clark and Bjork [<reflink idref="bib12" id="ref25">12</reflink>]; McDaniel and Butler [<reflink idref="bib25" id="ref26">25</reflink>]).</p> <p>To summarize, the present review focuses on learning among college or tertiary-level students in a controlled (proctored) environment when the information is acquired using a CBA versus an OBA. The focal research questions are:</p> <p>RQ1: Does assessment format (i.e. closed book or open book) affect student learning?</p> <p>RQ2: Do students prefer one type of assessment over the other?</p> <p>RQ3: Does the effect of assessment format on learning differ depending on moderators that pertain to the characteristics of the learning environment (e.g. study setting), testing environment (e.g. retention interval), and the materials on the test (e.g. item overlap)?</p> <hd id="AN0184595461-3">Method</hd> <p></p> <hd id="AN0184595461-4">Literature search</hd> <p>The literature search identified the initial set of records between 1970–2021 by keyword searches of the PsycINFO, PubMed, Scopus, and Google Scholar databases. These searches were conducted by the first author using the following keywords: <emph>open-book exam</emph>, <emph>closed-book exam</emph>, <emph>open-book test</emph>, <emph>closed-book test</emph>, <emph>open-book examination</emph>, <emph>closed-book examination</emph>, <emph>assessment</emph>, <emph>learning</emph>, and <emph>performance</emph>. To narrow the literature search to the most relevant records, titles, abstracts, and keywords were searched (rather than all fields) if the database provided this option (e.g. Scopus). When using Google Scholar, the first 15 pages were screened as the number of potentially eligible papers seemed to decrease (e.g. fewer records from peer-reviewed sources). Per a recent review by Harari et al. ([<reflink idref="bib18" id="ref27">18</reflink>]), of the 13 commonly used databases in systematic reviews and meta-analyses, Google Scholar was by far the most comprehensive, yielding over 98% of all the articles from the searches. Moreover, their analyses showed that Google Scholar captured all the unpublished papers in their search, which included a database specifically dedicated to grey literature (i.e. ProQuest Dissertations & Theses). The references section of Durning et al.'s ([<reflink idref="bib15" id="ref28">15</reflink>]) systematic review was also examined for potential papers. The literature search identified approximately 300 records, excluding any duplicates. Using our educationally applicable inclusion criteria described below led to a relatively limited number of papers included in this review.</p> <hd id="AN0184595461-5">Inclusion criteria</hd> <p>Given our review's focus on learners and instructors in higher education, studies with high school (upper-secondary level) students were not considered. Further, only studies in English from peer-reviewed sources (e.g. journal articles, conference proceedings) were included. Many studies were excluded because they focused on examining whether there were performance (rather than learning) differences between CBAs and OBAs. These studies compared the test scores of participants who were administered a closed-book test with those who were administered an open-book test during the acquisition process (e.g. Block [<reflink idref="bib7" id="ref29">7</reflink>]). Therefore, studies were only included if they met the following criteria:</p> <p></p> <ulist> <item> The test to assess learning was administered two or more days after the acquisition process (i.e. after instruction or training).[<reflink idref="bib2" id="ref30">2</reflink>]</item> <p></p> <item> Learning was measured with a CBA.</item> <p></p> <item> All assessments (whether open book or closed book) were administered in a controlled (proctored) environment (i.e. classroom or laboratory). This requirement reduces the possibility that any differences in learning outcomes might have alternative explanations (i.e. due to confounding variables). Therefore, studies with take-home assessments were excluded (e.g. Krasne et al. [<reflink idref="bib22" id="ref31">22</reflink>]).</item> <p></p> <item> The only difference in study conditions participants experienced was assessment format (i.e. CBA or OBA), as opposed to additional manipulations. This restriction reduces the number of other plausible explanations for any observed differences in study results. For example, Vidya ([<reflink idref="bib42" id="ref32">42</reflink>]) was excluded because participants in the OBA condition also partook in small-group discussions after every exam. In contrast, participants in the CBA condition did not experience this supplemental instruction.</item> </ulist> <hd id="AN0184595461-6">Selection process</hd> <p>The first step of the selection process involved screening all records by reviewing titles and abstracts to determine preliminary eligibility for inclusion and whether a full-text perusal would be warranted. This screening led to a total of 72 papers being selected for further examination. In the next step, the first author examined each paper to determine whether the study met the various criteria for inclusion. To ensure a comprehensive search and selection process, the literature review sections of these papers were also screened to find any potential records that were not already identified in the literature search.</p> <p>After the full-text review of the 72 papers, 63 were excluded as they did not meet at least one of the inclusion criteria. The nine remaining papers satisfied all requirements and were thus included in this review. Given that some of these papers reported the results of multiple studies (e.g. Arnold et al. [<reflink idref="bib3" id="ref33">3</reflink>] [two studies]), a total of 11 studies were the basis for the review (Agarwal et al. [<reflink idref="bib1" id="ref34">1</reflink>] [two studies]; Agarwal and Roediger [<reflink idref="bib2" id="ref35">2</reflink>], study 2; Arnold et al. [<reflink idref="bib3" id="ref36">3</reflink>] [two studies]; Ebersbach [<reflink idref="bib16" id="ref37">16</reflink>]; Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref38">17</reflink>]; Moore and Jensen [<reflink idref="bib28" id="ref39">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref40">30</reflink>]; Roelle and Berthold [<reflink idref="bib34" id="ref41">34</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref42">36</reflink>]).</p> <hd id="AN0184595461-7">Coding process</hd> <p>After determining the papers that would be included, both authors independently read and coded all studies. Information on any variable that could be a potential moderator of assessment format was also coded. Some of this information was not provided and therefore implied if reasonable (e.g. participants in field studies will know a final exam's format [i.e. a test given as part of a course requirement]). Any coding differences between the two authors were resolved and corrected after discussion.</p> <hd id="AN0184595461-8">Results</hd> <p></p> <hd id="AN0184595461-9">Overall effect of assessment format</hd> <p>The first research question focused on the relationship between assessment format (closed book versus open book) and learning. As seen in the first column of Table 1, overall, the effect of assessment format on learning was mixed, with approximately half of the studies showing no advantage for either format (middle portion of table) and approximately the other half (bottom portion of table) showing a CBA advantage. However, as we will discuss below, several studies that have reported a CBA advantage have also reported that participants in the CBA group used more effective study and self-regulated learning strategies (e.g. Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref43">17</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref44">30</reflink>]), which would themselves enhance learning, irrespective of assessment format. Thus, these results should be interpreted with some caution.</p> <p>Table 1. List of studies and potential moderators reviewed.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td>Study</td><td>Learning advantage (1)</td><td>Forgetting rate (2)</td><td>Test format preference (3)</td><td>Retention interval (4)</td><td>Initial test knowledge (5)</td><td>Final test knowledge (6)</td><td>Study setting (7)</td><td>Learning criteria (8)</td><td>Item overlap (9)</td></tr></thead><tbody><tr><td>Agarwal et al. (<xref ref-type="bibr" rid="bibr1">2008</xref>)<sup>S1</sup></td><td>OB</td><td>OB > CB</td><td>–</td><td>1 week</td><td>No</td><td>No</td><td>Lab</td><td>Retention</td><td>Yes</td></tr><tr><td>Agarwal et al. (<xref ref-type="bibr" rid="bibr1">2008</xref>)<sup>S2</sup></td><td>OB</td><td>OB > CB</td><td>–</td><td>1 week</td><td>No</td><td>No</td><td>Lab</td><td>Retention</td><td>Yes</td></tr><tr><td>Agarwal and Roediger (<xref ref-type="bibr" rid="bibr2">2011</xref>)<sup>S2</sup></td><td>=</td><td>OB > CB</td><td>OB</td><td>2 days</td><td>Yes</td><td>Yes</td><td>Lab</td><td>Both<sup>RT</sup></td><td>Yes</td></tr><tr><td>Arnold et al. (<xref ref-type="bibr" rid="bibr3">2021</xref>)<sup>S1</sup></td><td>=</td><td>–</td><td>OB</td><td>2 days</td><td>Yes</td><td>No</td><td>Lab</td><td>Both<sup>RT</sup></td><td>No</td></tr><tr><td>Arnold et al. (<xref ref-type="bibr" rid="bibr3">2021</xref>)<sup>S2</sup></td><td>=</td><td>–</td><td>–</td><td>2 days</td><td>Yes</td><td>No</td><td>Lab</td><td>Both<sup>RT</sup></td><td>No</td></tr><tr><td>Ebersbach (<xref ref-type="bibr" rid="bibr16">2020</xref>)</td><td>=</td><td>OB > CB</td><td>–</td><td>1 week</td><td>Yes</td><td>No</td><td>Field</td><td>Retention</td><td>Yes</td></tr><tr><td>Gharib, Phillips, and Mathew (<xref ref-type="bibr" rid="bibr17">2012</xref>)</td><td>=</td><td>–</td><td>OB</td><td>2 weeks</td><td>Yes</td><td>No</td><td>Field</td><td>Retention</td><td>No</td></tr><tr><td>Moore and Jensen (<xref ref-type="bibr" rid="bibr28">2007</xref>)</td><td>CB</td><td>OB > CB</td><td>OB</td><td>1–2 months</td><td>Yes</td><td>Yes</td><td>Field</td><td>Retention</td><td>No</td></tr><tr><td>Nsor-Ambala (<xref ref-type="bibr" rid="bibr30">2020</xref>)</td><td>CB</td><td>OB > CB</td><td>OB</td><td>1 week</td><td>Yes</td><td>Yes</td><td>Field</td><td>Retention</td><td>No</td></tr><tr><td>Roelle and Berthold (<xref ref-type="bibr" rid="bibr34">2017</xref>)</td><td>CB</td><td>OB > CB</td><td>–</td><td>1 week</td><td>No</td><td>No</td><td>Lab</td><td>Both<sup>RT</sup></td><td>Yes</td></tr><tr><td>Rummer, Schweppe, and Schwede (<xref ref-type="bibr" rid="bibr36">2019</xref>)</td><td>CB</td><td>–</td><td>–</td><td>9 weeks</td><td>Yes</td><td>Yes</td><td>Field</td><td>Retention</td><td>No</td></tr></tbody></table> </ephtml> </p> <p>1 Note. S1 = study 1; S2 = study 2; OB = open book; CB = closed book; RT = retention and transfer; – = data were not collected or not enough information was given.</p> <p>Only two studies (both from Agarwal et al. [<reflink idref="bib1" id="ref45">1</reflink>]), reported an OBA advantage. However, these findings should also be interpreted with some caution. First, performance on the initial test was 9% higher in the OBA group compared to the CBA group (.81 versus.72). As Rowland's ([<reflink idref="bib35" id="ref46">35</reflink>]) meta-analysis noted, initial test performance is strongly correlated with final test performance. As such, when possible, initial test performance for both groups should be equated. Second, as demonstrated in Agarwal et al. ([<reflink idref="bib1" id="ref47">1</reflink>]), when a CBA is supplemented with feedback, there is no longer an OBA advantage. This finding is congruent with the memory and testing effect literature. Providing feedback enhances the benefit of retrieving information from memory, especially when students fail to retrieve the correct information (e.g. Kang, McDermott, and Roediger [<reflink idref="bib21" id="ref48">21</reflink>]).</p> <hd id="AN0184595461-10">Forgetting rate</hd> <p>Although the overall effect of assessment format on learning was mixed, quantifying learning by examining forgetting rate (i.e. the percentage of change in information participants remembered on an initial test compared to that of the final test), reveals a robust CBA advantage. The forgetting rate was computed using the following formula: (initial test performance − final test performance) ÷ initial test performance. Ideally, when computing the forgetting rate, one would want to have the same items and test format for both the initial and final tests (Arnold and McDermott [<reflink idref="bib4" id="ref49">4</reflink>]). In two studies (Moore and Jensen [<reflink idref="bib28" id="ref50">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref51">30</reflink>]) we examined, however, different items were used on the initial and final tests, but the same information was assessed on both tests, thereby permitting a forgetting rate analysis. Studies with different participants completing the initial and final tests (e.g. Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref52">17</reflink>]) were excluded from this analysis due to potential participant-by-item selection artifact. As can be seen in the second column of Table 1, all studies in which we could compute the forgetting rate showed that taking an initial OBA resulted in a greater rate of forgetting relative to a CBA, OBA<subs>Mean</subs>: 34% versus CBA<subs>Mean</subs>: 24%.[<reflink idref="bib3" id="ref53">3</reflink>] These results provide strong evidence that a CBA slows down forgetting relative to an OBA.</p> <p>The finding that a CBA slows down forgetting more than an OBA is especially germane in the education setting where the final exam of a class may be cumulative, or subsequent courses students take may presume they have retained knowledge and skills from their previous classes. The latter scenario is epitomized by courses that have prerequisites. It is thus surprising that many studies in this literature have not evaluated forgetting rate.</p> <p>The mechanism for why a CBA slows down forgetting more than an OBA is congruent with Bjork and Bjork's ([<reflink idref="bib6" id="ref54">6</reflink>]) desirable difficulties framework, which states that learning conditions that make acquiring information difficult promote long-term learning. Of course, the blanket assumption that any condition that makes the acquisition of knowledge difficult will promote learning (e.g. watching TV while studying) is unwarranted. Instead, difficult conditions conducive to learning require, among other things, an individual to engage in the cognitive processes complementary to the material, such as trying to recall the material or information relevant to it from memory (McDaniel and Butler [<reflink idref="bib25" id="ref55">25</reflink>]). Because a CBA that primarily assesses retention necessarily requires a student to retrieve studied information from memory rather than transcribing the correct answer, which is possible when using an OBA that primarily assesses retention and requires considerably less effort, a CBA could be considered a 'desirably difficult' condition for learning.</p> <hd id="AN0184595461-11">Assessment format preference</hd> <p>The second research question focused on students' preferences for the type of assessment they wish to take. This topic is important because administering the test format that matches students' preferences could effectively alleviate their test anxiety (e.g. Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref56">17</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref57">30</reflink>]). Test anxiety is among the strongest (negative) correlates of students' grades because it prevents them from applying acquired knowledge on assessments (Schneider and Preckel [<reflink idref="bib37" id="ref58">37</reflink>]). Test anxiety could also indirectly affect student achievement by encouraging academic self-handicapping (e.g. procrastinating; Schwinger et al. [<reflink idref="bib38" id="ref59">38</reflink>]).</p> <p>In the five studies that asked students which type of test they preferred to take, all five reported that they preferred an OBA (see Table 1, column 3). In studies that assessed students' test anxiety (e.g. Nsor-Ambala [<reflink idref="bib30" id="ref60">30</reflink>]), students reported that taking an OBA produced less anxiety than taking a CBA. This finding dovetails with those reported in similar reviews (e.g. Durning et al. [<reflink idref="bib15" id="ref61">15</reflink>]). Thus, allowing students to take an OBA is one technique that can reduce students' test anxiety levels (Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref62">17</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref63">30</reflink>]).</p> <hd id="AN0184595461-12">Moderators</hd> <p>Although the overall mixed effects of assessment format on learning may imply that there is simply no effect, the variability of the findings (i.e. significant effects in favor of CBAs and OBAs, with some studies reporting no effect) suggests that different variables could moderate the effect of assessment format on learning. Thus, we provide a discussion of the potential moderators that were examined, which addresses the third research question. Table 2 summarizes the research evidence for these moderators.</p> <p>Table 2. Summary of evidence for potential moderators.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td>Moderators</td><td>Results summary</td></tr></thead><tbody><tr><td>Retention interval</td><td>No supporting evidence for variable as moderator</td></tr><tr><td>Initial test knowledge</td><td>No supporting evidence for variable as moderator</td></tr><tr><td>Final test knowledge</td><td>CBA learning advantage with final test knowledge</td></tr><tr><td>Study setting</td><td>No supporting evidence for variable as moderator</td></tr><tr><td>Learning criteria</td><td>No supporting evidence for variable as moderator</td></tr><tr><td>Item overlap</td><td>CBA learning advantage without item overlap</td></tr></tbody></table> </ephtml> </p> <hd id="AN0184595461-13">Retention interval</hd> <p>The retention interval is the length of time between an assessment in the acquisition process (i.e. during instruction) and a learning assessment post-acquisition. Longer retention intervals are more valid assessments of long-term learning than shorter retention intervals, especially in the education setting (Soderstrom and Bjork [<reflink idref="bib40" id="ref64">40</reflink>]). Because current accessibility is a poor predictor of learning (Bjork and Bjork [<reflink idref="bib6" id="ref65">6</reflink>]; Soderstrom and Bjork [<reflink idref="bib40" id="ref66">40</reflink>]), longer retention intervals enable more accurate assessments of learning and thereby provide greater confidence that any gains in knowledge, skills, or abilities are likely to reflect relatively long-term learning.</p> <p>The retention intervals in the studies reviewed ranged from 2 to 63 days (see Table 1, column 4). Examining the pattern of findings in these studies shows that the effect of assessment format on learning does not seem to depend on the retention interval. In the three studies with a relatively short, 2-day retention interval (Agarwal and Roediger [<reflink idref="bib2" id="ref67">2</reflink>], study 2; Arnold et al. [<reflink idref="bib3" id="ref68">3</reflink>] [two studies]), there was no significant difference between a CBA and an OBA. In the five studies with a longer retention interval of 7 days, the results were mixed, as two studies (Nsor-Ambala [<reflink idref="bib30" id="ref69">30</reflink>]; Roelle and Berthold [<reflink idref="bib34" id="ref70">34</reflink>]) found a CBA advantage, two other studies (both from Agarwal et al. [<reflink idref="bib1" id="ref71">1</reflink>]) found an OBA advantage, and another study (Ebersbach [<reflink idref="bib16" id="ref72">16</reflink>]) did not find a significant difference between a CBA and an OBA.</p> <p>In the three studies with the longest retention intervals (mean of 36 days), two studies (Moore and Jensen [<reflink idref="bib28" id="ref73">28</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref74">36</reflink>]) reported a CBA advantage, and the other study (Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref75">17</reflink>]) found that assessment format did not have a significant effect on learning. Given that longer retention intervals are better learning evaluations, the two studies reporting a CBA learning edge might be interpreted as supporting evidence for a CBA. However, Moore and Jensen's ([<reflink idref="bib28" id="ref76">28</reflink>]) paper was a field study that examined study behavior when anticipating different assessment formats and found that students preparing for an OBA engaged in more behaviors detrimental to learning (e.g. not attending class). Thus, the reported CBA learning edge in this study was confounded by the less optimal study behaviors of students taking an OBA. Given this observation, it would be more appropriate to also characterize the results in these three studies with the longest retention intervals as mixed. In conclusion, the effect of assessment format on learning does not appear to depend on the retention interval.</p> <hd id="AN0184595461-14">Initial test knowledge</hd> <p>Initial test knowledge refers to whether an individual knows the format of an upcoming assessment. This variable is relevant because individuals can change their study behaviors and strategies based on the type of assessment they anticipate. For example, previous research has reported that students generally adjust their exam preparation time by studying less for OBAs (Durning et al. [<reflink idref="bib15" id="ref77">15</reflink>]). Decreased study effort when preparing for an OBA might subsequently hinder learning. Given that participants in most field studies and students in classes will often know the format of an upcoming assessment, there would be greater concern regarding potential differences in study behaviors as being an alternative explanation for observed results in studies with a between-groups design. In studies where participants are not aware of the format of an upcoming assessment, there is control for this potential confound. Without initial test knowledge, individuals assume that a CBA is likely (Theophilides and Koutselini [<reflink idref="bib41" id="ref78">41</reflink>]) and prepare with this expectation (Agarwal and Roediger [<reflink idref="bib2" id="ref79">2</reflink>]). Thus, irrespective of the study condition, the likelihood that differences in exam preparation strategies might be an alternative explanation for study findings is diminished.</p> <p>Participants had initial test knowledge in eight studies (Agarwal and Roediger [<reflink idref="bib2" id="ref80">2</reflink>], study 2; Arnold et al. [<reflink idref="bib3" id="ref81">3</reflink>] [two studies]; Ebersbach [<reflink idref="bib16" id="ref82">16</reflink>]; Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref83">17</reflink>]; Moore and Jensen [<reflink idref="bib28" id="ref84">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref85">30</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref86">36</reflink>]). Thus, results in these studies could be confounded by differences in study effort and strategies during the acquisition process. Examining the pattern of findings in all 11 studies shows that the effect of assessment format on learning does not seem to depend on initial test knowledge. In the eight studies that participants were aware of the assessment format, five found that assessment format did not have a significant effect on learning. In contrast, three other studies found a CBA advantage (see Table 1, column 5). Given that individuals usually study less when preparing for an OBA (Durning et al. [<reflink idref="bib15" id="ref87">15</reflink>]), one might hypothesize a CBA learning edge when the test format is known. This hypothesis is supported by Moore and Jensen's ([<reflink idref="bib28" id="ref88">28</reflink>]) finding that students preparing for OBAs attended fewer review sessions and classes. In the three studies that participants were not aware of the assessment format, Roelle and Berthold ([<reflink idref="bib34" id="ref89">34</reflink>]) found a CBA advantage, whereas the two studies in Agarwal et al. ([<reflink idref="bib1" id="ref90">1</reflink>]) found an OBA advantage. Overall, the findings of these three studies lend more support for the conclusion that the results were mixed, and initial test knowledge does not moderate the effect of assessment format on learning.</p> <hd id="AN0184595461-15">Final test knowledge</hd> <p>We also coded whether participants' awareness of the final test's format (i.e. the test administered to assess learning after the acquisition process) moderated the effect of assessment format. Based on previous research (e.g. Moore and Jensen [<reflink idref="bib28" id="ref91">28</reflink>]), when participants are aware that the initial test will be closed book, they will deploy self-regulated learning strategies that are more conducive toward learning (e.g. increased study time, more likely to attend class). Similarly, being aware that the final test will be closed book might also encourage participants to engage in more effective learning strategies such as elaborative processing (e.g. making connections between the studied materials or with related prior knowledge). The latter explanation dovetails with findings in the test-expectancy literature that participants can tailor their encoding strategy which leads to a higher probability of recall given the anticipated testing condition (Cho and Neely [<reflink idref="bib10" id="ref92">10</reflink>]; Rivers and Dunlosky [<reflink idref="bib32" id="ref93">32</reflink>]). Both explanations are congruent with Arnold et al.'s ([<reflink idref="bib3" id="ref94">3</reflink>]) finding that participants perceived an OBA to be much easier to do well on than a CBA. Consequently, participants might be overconfident in their future performance when told that they will be taking OBAs and more likely to engage in maladaptive learning habits.</p> <p>In seven studies (Agarwal et al. [<reflink idref="bib1" id="ref95">1</reflink>] [two studies]; Arnold et al. [<reflink idref="bib3" id="ref96">3</reflink>] [two studies]; Ebersbach [<reflink idref="bib16" id="ref97">16</reflink>]; Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref98">17</reflink>]; Roelle and Berthold [<reflink idref="bib34" id="ref99">34</reflink>]), participants were not aware that the final test's format would be closed book (see Table 1, column 6). The results in these studies were mixed as two studies found an OBA advantage, one study found a CBA advantage, and the other four studies did not find a significant difference between CBAs and OBAs.</p> <p>In four studies (Agarwal and Roediger [<reflink idref="bib2" id="ref100">2</reflink>], study 2; Moore and Jensen [<reflink idref="bib28" id="ref101">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref102">30</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref103">36</reflink>]), participants were aware of the final test's format. In all but one study, there was a CBA advantage (CBAs and OBAs were equivalent in Agarwal and Roediger [<reflink idref="bib2" id="ref104">2</reflink>], study 2). These data provide suggestive evidence that the effect of assessment format on learning appears to be moderated by final test knowledge such that there is a CBA advantage when the format of the final test is known.</p> <hd id="AN0184595461-16">Study setting</hd> <p>Another variable we examined as a potential moderator was study setting (i.e. laboratory or field study). There was no a priori prediction regarding study setting as findings in laboratory studies tend to be highly correlated with findings in field studies (Mitchell [<reflink idref="bib26" id="ref105">26</reflink>]), despite the better-controlled conditions of laboratory studies. Whereas six of the studies in this review were conducted in the laboratory (Agarwal et al. [<reflink idref="bib1" id="ref106">1</reflink>] [two studies]; Agarwal and Roediger [<reflink idref="bib2" id="ref107">2</reflink>], study 2; Arnold et al. [<reflink idref="bib3" id="ref108">3</reflink>] [two studies]; Roelle and Berthold [<reflink idref="bib34" id="ref109">34</reflink>]), the other five were conducted in the field (Ebersbach [<reflink idref="bib16" id="ref110">16</reflink>]; Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref111">17</reflink>]; Moore and Jensen [<reflink idref="bib28" id="ref112">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref113">30</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref114">36</reflink>]).</p> <p>In the six laboratory studies, the results were mixed as two studies found that OBAs were associated with higher learning, one study found that a CBA was associated with higher learning, whereas the other three studies did not find a significant difference. In the five field studies, three studies found a CBA advantage (Moore and Jensen [<reflink idref="bib28" id="ref115">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref116">30</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref117">36</reflink>]), whereas the other two studies did not find a significant difference (see Table 1, column 7). Although these results were also mixed, an OBA was not found to have a larger effect on learning in any field study. One explanation for these findings is that the results are confounded with assessment format knowledge given that participants were aware of the upcoming assessment's format in all three field studies. As such, these findings can be partly attributed to the previously described concern with differences in study behaviors that generally favor CBAs. Thus, these findings in field studies can also be characterized as mixed. In conclusion, the effect of assessment format on learning does not appear to depend on study setting.</p> <hd id="AN0184595461-17">Learning criteria</hd> <p>Learning can be assessed by testing for retention or transfer. When testing for retention, the objective is to determine to what extent factual information has been retained in long-term memory and can be retrieved, which is considered a basic conceptualization of learning (Brown, Roediger, and McDaniel [<reflink idref="bib8" id="ref118">8</reflink>]). Retention tests require individuals to recall facts, concepts, or ideas from readings or summarize this information. Tests of transfer not only require the retention of knowledge but also the ability to apply retained knowledge in a new context. Thus, in addition to what is required for tests of retention, assessments of transfer require drawing connections between concepts, making inferences from readings, and applying facts, concepts, or ideas from readings to solve novel problems.</p> <p>Seven studies assessed learning by only testing for retention (Agarwal et al. [<reflink idref="bib1" id="ref119">1</reflink>] [two studies]; Ebersbach [<reflink idref="bib16" id="ref120">16</reflink>]; Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref121">17</reflink>]; Moore and Jensen [<reflink idref="bib28" id="ref122">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref123">30</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref124">36</reflink>]). For example, both studies in Agarwal et al. ([<reflink idref="bib1" id="ref125">1</reflink>]) assessed retention with open-ended questions that required the recall of facts and concepts from various prose passages that participants had studied (e.g. 'Why did the Voyager have instruments that would measure ultraviolet and infrared light?' [<reflink idref="bib864" id="ref126">864</reflink>]). The results in the seven studies that only assessed retention were mixed as three found a CBA advantage, two found an OBA advantage, and two did not find a significant difference (see Table 1, column 8).</p> <p>The remaining four studies assessed both retention and transfer (Agarwal and Roediger [<reflink idref="bib2" id="ref127">2</reflink>], study 2; Arnold et al. [<reflink idref="bib3" id="ref128">3</reflink>] [two studies]; Roelle and Berthold [<reflink idref="bib34" id="ref129">34</reflink>]). For example, both studies in Arnold et al. ([<reflink idref="bib3" id="ref130">3</reflink>]) assessed transfer by open-ended questions that required making connections between facts in various passages and drawing complex inferences. The following is an example of a question that was used to assess transfer:</p> <p>You want to show a friend solar activity in the sky, but you do not have access to a telescope at the moment. Which of these solar activities (sunspots, faculae, eruptive prominences, solar flares) would you be most likely to be able to see? Please give two reasons to explain your answer. (<reflink idref="bib233" id="ref131">233</reflink>)</p> <p>In the four studies that assessed both retention and transfer, one study reported a CBA advantage, whereas the other three studies did not find a significant difference. Further, the results in each of these four studies were the same regardless of whether learning was assessed by a retention test or a transfer test. For example, Agarwal and Roediger ([<reflink idref="bib2" id="ref132">2</reflink>]) found that assessment format did not affect learning with either retention or transfer as the criterion. Thus, this overall pattern of results was also equivocal; therefore, learning criteria do not seem to moderate the effect of assessment format on learning.</p> <hd id="AN0184595461-18">Item overlap</hd> <p>The distinction between retention and transfer can also be conceptualized as the degree to which questions differ between an assessment in the acquisition process (i.e. during instruction) and a learning assessment post-acquisition. Given that transfer requires the application of retained knowledge in a new context, the manipulation of context can sometimes occur by using questions to assess learning (i.e. the final test) that differ from the questions used in the initial test. Thus, another variable examined as a potential moderator was item overlap, which refers to the extent that the assessment questions during acquisition overlap with the questions in the learning assessment. To simplify the classification of studies, a binary categorization was used with studies being coded as either having item overlap (when at least 50% of the items were the same) or not having item overlap.</p> <p>In five of the studies, there was item overlap (Agarwal et al. [<reflink idref="bib1" id="ref133">1</reflink>] [two studies]; Agarwal and Roediger [<reflink idref="bib2" id="ref134">2</reflink>], study 2; Ebersbach [<reflink idref="bib16" id="ref135">16</reflink>]; Roelle and Berthold [<reflink idref="bib34" id="ref136">34</reflink>]). The results in these studies were mixed as two studies reported an OBA advantage, one study reported a CBA advantage, and two other studies did not find a significant difference (see Table 1, column 9). In the six studies without item overlap (Arnold et al. [<reflink idref="bib3" id="ref137">3</reflink>] [two studies]; Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref138">17</reflink>]; Moore and Jensen [<reflink idref="bib28" id="ref139">28</reflink>]; Nsor-Ambala [<reflink idref="bib30" id="ref140">30</reflink>]; Rummer, Schweppe, and Schwede [<reflink idref="bib36" id="ref141">36</reflink>]), three studies found a CBA advantage, whereas the other three studies did not find a significant difference. Although the supportive evidence is inconclusive as half of the studies reported a null effect, there is tentative evidence that CBAs have a learning advantage when items do not overlap. Therefore, the pattern that appears to emerge as item overlap decreases is initially mixed results become clearer as only CBAs are found to have an advantage. In summary, there is suggestive evidence that the effect of assessment format on learning appears to be moderated by item overlap such that there is a CBA advantage when there is no item overlap.</p> <hd id="AN0184595461-19">Discussion</hd> <p>Although the overall effect of assessment format on learning is mixed, a robust finding from the studies we reviewed is that a CBA slows down forgetting relative to an OBA. One explanation for this finding is that a CBA requires individuals to engage in more elaborative retrieval that strengthens memory traces of successfully retrieved information compared to an OBA (see Cho et al. [<reflink idref="bib11" id="ref142">11</reflink>]; Neely and Cho [<reflink idref="bib29" id="ref143">29</reflink>]; Roediger and Karpicke [<reflink idref="bib33" id="ref144">33</reflink>], for a similar explanation of the testing effect). As such, taking a CBA represents a desirably difficult learning condition (Bjork and Bjork [<reflink idref="bib6" id="ref145">6</reflink>]) that promotes long-term learning. To our knowledge, our paper is the first to highlight this robust finding in this literature. Indeed, this finding is especially important in the education setting as instructors sometimes administer cumulative exams, and students' performance in upper-level courses often builds on knowledge acquired from their lower-level courses. Thus, when assessing student learning, focusing solely on retention (and excluding a discussion on forgetting) represents only part of the story. Another robust finding is that students prefer taking an OBA over a CBA. This conclusion is congruent with that reported in other similar reviews (e.g. Durning et al. [<reflink idref="bib15" id="ref146">15</reflink>]).</p> <p>Examining the potential moderators of assessment format indicates that the effect appears to be dependent on two of the variables included in this review. There is tentative evidence that the effect of assessment format on learning seems to be moderated by final test knowledge such that there is a CBA edge when participants are aware that the format of the final test is closed book. This finding is consistent with previous research that has found expectations about a final test's format can affect learning (e.g. Agarwal and Roediger [<reflink idref="bib2" id="ref147">2</reflink>], study 2). The awareness of a closed-book final test can encourage individuals to engage in more effective learning strategies, and differences in time spent on study-related activities generally favor CBAs (Durning et al. [<reflink idref="bib15" id="ref148">15</reflink>]). There is also tentative evidence that the effect of assessment format seems to be moderated by item overlap as CBAs appear to have a learning edge when post-acquisition assessment questions (i.e. final test) differ from assessment questions during the acquisition process (i.e. initial test).</p> <hd id="AN0184595461-20">Recommendations for educators</hd> <p>In this section, we delineate situations where each type of assessment format would be most appropriate for educators. Instructors administering cumulative exams would best serve their students by administering CBAs, as they unequivocally slow down forgetting relative to OBAs. However, an instructor who teaches a class in which many students report high test anxiety, which can be assessed using self-report measures (e.g. Schneider and Preckel [<reflink idref="bib37" id="ref149">37</reflink>]), might consider administering OBAs as they will alleviate students' test anxiety. As discussed earlier, students with test anxiety will likely have artificially low test scores due to their anxiety impairing knowledge retrieval at the time of a test. Thus, their test score would not reflect their actual knowledge of the material. Indeed, implementing OBAs can be one method an instructor can use to promote universal design in their class without compromising student learning, a concern that some instructors may have with implementing universal design principles. Nonetheless, for educators who remain highly reluctant to administer OBAs, a compromise between a CBA and an OBA is to use a hybrid approach. An example of a hybrid approach is a cheat-sheet exam, in which instructors allow students to prepare a limited set of handwritten notes (e.g. one page) that they can use on a CBA. There is limited evidence that while students still prefer taking an OBA to a cheat-sheet exam, the latter is still preferred over a CBA (Gharib, Phillips, and Mathew [<reflink idref="bib17" id="ref150">17</reflink>]). However, these data and learning data between these three types of tests are scarce.</p> <p>Because students tend to use less efficacious study strategies and engage in less optimal study habits when expecting an OBA, a best practice for instructors who will use an OBA is to discuss with their students that they should continue to use effective study and self-regulated learning strategies, as these variables are highly predictive of and conducive to learning (Credé and Kuncel [<reflink idref="bib13" id="ref151">13</reflink>]).</p> <hd id="AN0184595461-21">Future directions</hd> <p>In this section, we identify outstanding questions that future researchers should explore that would be of high interest to educators deciding between a CBA or an OBA. The first is to further investigate which assessment format students believe they learn more from, and equally important, the basis for their belief. Although a few studies collected students' predicted performance on a final test (e.g. Nsor-Ambala [<reflink idref="bib30" id="ref152">30</reflink>]), no study has asked participants to elaborate on why they might learn more from a CBA. This question is important given ample evidence of students' misconceptions about effective learning strategies (Dunlosky et al. [<reflink idref="bib14" id="ref153">14</reflink>]). Related to the first point is how students use available resources during an OBA, which was not assessed in any study we reviewed. One approach is that participants will answer questions from memory (akin to testing) and then check their answers with the available resources (feedback). Alternatively, students can immediately seek answers without attempting to answer each question first (akin to restudying, which is less effective than testing). This procedural difference in students' usage of resources in an OBA can provide more insight into assessment format effects and explain some of the null effects in this literature. Third, because students usually employ less effective self-regulated learning strategies when expecting an OBA, it would be important to control for these differences in future studies. Failure to account for these differences will prohibit researchers from drawing strong conclusions about the most effective assessment type, as assessment format would be confounded with self-regulated learning strategies. Fourth, although preliminary evidence indicates the effect of assessment format on learning is moderated by final test knowledge and item overlap, future research should test these potential moderators. Finally, although nearly all memory researchers agree that a 2-day minimum retention interval, which characterizes all the studies in this review, represents long-term memory, for most educators, long-term memory is likely defined in terms of months, not single-digit days. Since the retention interval in most reviewed studies was one week or less, future research is warranted to examine whether the observed lower forgetting rates of CBAs will generalize to longer retention intervals and thereby provide evidence of this assessment format's efficacy for subsequent courses.</p> <p>We hope this review provides useful recommendations for educators considering the merits of a CBA versus an OBA in their classroom. Given the increased flexibility of instructional methods in the past two years, owing to COVID-19, we anticipate that educators might also begin to show increased flexibility in the administration of in-class assessments. In addition, we hope that our findings provide fruitful avenues for future researchers.</p> <hd id="AN0184595461-22">Disclosure statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <ref id="AN0184595461-23"> <title> Notes </title> <blist> <bibl id="bib1" idref="ref19" type="bt">1</bibl> <bibtext> In the classroom setting, the duration of the acquisition process is the length of a term. In a laboratory study, the duration of the acquisition process is the length of time given to participants to study content that they are expected to learn.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref30" type="bt">2</bibl> <bibtext> If a study measured learning multiple times (e.g. an immediate and a delayed test), the latter assessment (i.e. the longer retention interval) was used to quantify learning because it is a better reflection of enduring knowledge.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref33" type="bt">3</bibl> <bibtext> We could not compute tests of statistical significance to compare each set of means because of inadequate information provided in the studies.</bibtext> </blist> </ref> <ref id="AN0184595461-24"> <title> References </title> <blist> <bibtext> Agarwal, Pooja, Jeffrey Karpicke, Sean Kang, Henry Roediger III, and Kathleen McDermott. 2008. Examining the testing effect with open- and closed-book tests. Applied Cognitive Psychology 22, no. 7 : 861 – 76. doi: 10.1002/acp.1391.</bibtext> </blist> <blist> <bibtext> Agarwal, Pooja, and Henry Roediger III. 2011. Expectancy of an open-book test decreases performance on a delayed closed-book test. Memory 19, no. 8 : 836 – 52. doi: 10.1080/09658211.2011.613840.</bibtext> </blist> <blist> <bibtext> Arnold, Kathleen, Emmaline Drew Eliseev, Alexandria Stone, Mark McDaniel, and Elizabeth Marsh. 2021. Two routes to the same place: Learning from quick closed-book essays versus open-book essays. Journal of Cognitive Psychology 33, no. 3 : 229 – 46. doi: 10.1080/20445911.2021.1903011.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref49" type="bt">4</bibl> <bibtext> Arnold, Kathleen, and Kathleen McDermott. 2013. Test-potentiated learning: Distinguishing between direct and indirect effects of tests. Journal of Experimental Psychology: Learning, Memory, and Cognition 39, no. 3 : 940 – 5. doi: 10.1037/a0029199.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref10" type="bt">5</bibl> <bibtext> Bengtsson, Lars. 2019. Take-home exams in higher education: A systematic review. Education Sciences 9, no. 4 : 267. doi: 10.3390/educsci9040267.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref22" type="bt">6</bibl> <bibtext> Bjork, Elizabeth, and Robert Bjork. 2011. Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In Psychology and the real world: Essays illustrating fundamental contributions to society, ed. Morton Gernsbacher, Richard Pew, Leaetta Hough, and James Pomerantz, 56 – 64. New York : Worth Publishers.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref29" type="bt">7</bibl> <bibtext> Block, Robert. 2012. A discussion of the effect of open-book and closed-book exams on student achievement in an introductory statistics course. PRIMUS 22, no. 3 : 228 – 38. doi: 10.1080/10511970.2011.565402.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref118" type="bt">8</bibl> <bibtext> Brown, Peter, Henry Roediger III, and Mark McDaniel. 2014. Make it stick: The science of successful learning. Cambridge : Belknap Press of Harvard University Press.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref5" type="bt">9</bibl> <bibtext> Cahill-Ripley, Amanda. 2015. Innovative methods of assessment in law: The value of open-book exams as a catalyst for improving teaching and learning in the law school. The Law Teacher 49, no. 2 : 206 – 18. doi: 10.1080/03069400.2015.1016724.</bibtext> </blist> <blist> <bibtext> Cho, Kit, and James Neely. 2017. The roles of encoding strategies and retrieval practice in test-expectancy effects. Memory 25, no. 5 : 626 – 35. doi: 10.1080/09658211.2016.1202983.</bibtext> </blist> <blist> <bibtext> Cho, Kit, James Neely, Stephanie Crocco, and Deana Vitrano. 2017. Testing enhances both encoding and retrieval for both tested and untested items. Quarterly Journal of Experimental Psychology 70, no. 7 : 1211 – 35. doi: 10.1080/17470218.2016.1175485.</bibtext> </blist> <blist> <bibtext> Clark, Courtney, and Robert Bjork. 2014. When and why introducing difficulties and errors can enhance instruction. In Applying the science of learning in education: Infusing psychological science into the curriculum, edited by Victor Benassi, Catherine Overson, and Christopher Hakala, 20 – 30. Washington, DC : Society for the Teaching of Psychology.</bibtext> </blist> <blist> <bibtext> Credé, Marcus, and Nathan Kuncel. 2008. Study habits, skills, and attitudes: The third pillar supporting collegiate academic performance. Perspectives on Psychological Science: A Journal of the Association for Psychological Science 3, no. 6 : 425 – 53. doi: 10.1111/j.1745-6924.2008.00089.x.</bibtext> </blist> <blist> <bibtext> Dunlosky, John, Katherine Rawson, Elizabeth Marsh, Mitchell Nathan, and Daniel Willingham. 2013. Improving students' learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest: A Journal of the American Psychological Society 14, no. 1 : 4 – 58. doi: 10.1177/1529100612453266.</bibtext> </blist> <blist> <bibtext> Durning, Steven, Ting Dong, Temple Ratcliffe, Lambert Schuwirth, Anthony Artino Jr, John Boulet, and Kevin Eva. 2016. Comparing open-book and closed-book examinations: A systematic review. Academic Medicine: Journal of the Association of American Medical Colleges 91, no. 4 : 583 – 99. doi: 10.1097/ACM.0000000000000977.</bibtext> </blist> <blist> <bibtext> Ebersbach, Mirjam. 2020. Access to the learning material enhances learning by means of generating questions: Comparing open- and closed-book conditions. Trends in Neuroscience and Education 19, no. June : 100130. doi: 10.1016/j.tine.2020.100130.</bibtext> </blist> <blist> <bibtext> Gharib, Afshin, William Phillips, and Noelle Mathew. 2012. Cheat sheet or open-book? A comparison of the effects of exam types on performance, retention, and anxiety. Psychology Research 2, no. 8 : 469 – 78. doi: 10.17265/2159-5542/2012.08.004.</bibtext> </blist> <blist> <bibtext> Harari, Michael, Heather Parola, Christopher Hartwell, and Amy Riegelman. 2020. Literature searches in systematic reviews and meta-analyses: A review, evaluation, and recommendations. Journal of Vocational Behavior 118 : 103377. doi: 10.1016/j.jvb.2020.103377.</bibtext> </blist> <blist> <bibtext> Hattie, John. 2015. The applicability of visible learning to higher education. Scholarship of Teaching and Learning in Psychology 1, no. 1 : 79 – 91. doi: 10.1037/stl0000021.</bibtext> </blist> <blist> <bibtext> Johanns, Beth, Amber Dinkens, and Jill Moore. 2017. A systematic review comparing open-book and closed-book examinations: Evaluating effects on development of critical thinking skills. Nurse Education in Practice 27, no. November : 89 – 94. doi: 10.1016/j.nepr.2017.08.018.</bibtext> </blist> <blist> <bibtext> Kang, Sean, Kathleen McDermott, and Henry Roediger III. 2007. Test format and corrective feedback modify the effect of testing on long-term retention. The European Journal of Cognitive Psychology 19, no. 4–5 : 528 – 58. doi: 10.1080/09541440601056620.</bibtext> </blist> <blist> <bibtext> Krasne, Sally, Paul Wimmers, Anju Relan, and Thomas Drake. 2006. Differential effects of two types of formative assessment in predicting performance of first-year medical students. Advances in Health Sciences Education: Theory and Practice 11, no. 2 : 155 – 71. doi: 10.1007/s10459-005-5290-9.</bibtext> </blist> <blist> <bibtext> Larwin, Karen, Jennifer Gorman, and David Larwin. 2013. Assessing the impact of testing aids on post-secondary student performance: A meta-analytic investigation. Educational Psychology Review 25, no. 3 : 429 – 43. doi: 10.1007/s10648-013-9227-1.</bibtext> </blist> <blist> <bibtext> Mathieson, Georgia, Roxanne Sutthakorn, and Olivia Thomas. 2020. Could the future of medical school examinations Be open-book - a medical student's perspective? Medical Education Online 25, no. 1 : 1787308. doi: 10.1080/10872981.2020.1787308.</bibtext> </blist> <blist> <bibtext> McDaniel, Mark, and Andrew Butler. 2011. A contextual framework for understanding when difficulties are desirable. In Successful remembering and successful forgetting: A festschrift in honor of Robert A. Bjork, edited by Aaron Benjamin, 175 – 99. New York : Psychology Press.</bibtext> </blist> <blist> <bibtext> Mitchell, Gregory. 2012. Revisiting truth or triviality: The external validity of research in the psychological laboratory. Perspectives on Psychological Science: A Journal of the Association for Psychological Science 7, no. 2 : 109 – 17. doi: 10.1177/1745691611432343.</bibtext> </blist> <blist> <bibtext> Momsen ‚ Jennifer, Tammy Long, Sara Wyse, and Diane Ebert-May. 2010. Just the facts? Introductory undergraduate biology courses focus on low-level cognitive skills. CBE—Life Sciences Education 9, no. 4 : 435 – 40. doi: 10.1187/cbe.10-01-0001.</bibtext> </blist> <blist> <bibtext> Moore, Randy, and Philip Jensen. 2007. Do open-book exams impede long-term learning in introductory biology course? Journal of College Science Teaching 36 : 46 – 9. https://my.nsta.org/resource/8452.</bibtext> </blist> <blist> <bibtext> Neely, James, and Kit Cho. 2015. Testing and retrieval practice effects: Assessing the contributions of encoding and retrieval mechanisms. In Remembering: Attributions, processes, and control in human memory: Essays in honor of Larry Jacoby, ed. Stephen Lindsay, Colleen Kelley, Andrew Yonelinas, and Henry Roediger III, 29 – 45. New York : Psychology Press.</bibtext> </blist> <blist> <bibtext> Nsor-Ambala, Randolph. 2020. Impact of exam type on exam scores, anxiety, and knowledge retention in a cost and management accounting course. Accounting Education 29, no. 1 : 32 – 56. doi: 10.1080/09639284.2019.1683871.</bibtext> </blist> <blist> <bibtext> Rao, Kavita, Min Wook Ok, and Brian Bryant. 2014. A review of research on universal design educational models. Remedial and Special Education 35, no. 3 : 153 – 66. doi: 10.1177/0741932513518980.</bibtext> </blist> <blist> <bibtext> Rivers, Michelle, and John Dunlosky. 2021. Are test-expectancy effects better explained by changes in encoding strategies or differential test experience? Journal of Experimental Psychology. Learning, Memory, and Cognition 47, no. 2 : 195 – 207. doi: 10.1037/xlm0000949.</bibtext> </blist> <blist> <bibtext> Roediger, Henry, and Jeffrey Karpicke. 2006. Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science 17, no. 3 : 249 – 55. doi: 10.1111/j.1467-9280.2006.01693.x.</bibtext> </blist> <blist> <bibtext> Roelle, Julian, and Kirsten Berthold. 2017. Effects of incorporating retrieval into learning tasks: The complexity of the tasks matters. Learning and Instruction 49, no. June : 142 – 56. doi: 10.1016/j.learninstruc.2017.01.008.</bibtext> </blist> <blist> <bibtext> Rowland, Christopher 2014. The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin 140, no. 6 : 1432 – 63. doi: 10.1037/a0037559.</bibtext> </blist> <blist> <bibtext> Rummer, Ralf, Judith Schweppe, and Annett Schwede. 2019. Open-book versus closed-book tests in university classes: A field experiment. Frontiers in Psychology 10, no. March. doi: 10.3389/fpsyg.2019.00463.</bibtext> </blist> <blist> <bibtext> Schneider, Michael, and Franzis Preckel. 2017. Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin 143, no. 6 : 565 – 600. doi: 10.1037/bul0000098.</bibtext> </blist> <blist> <bibtext> Schwinger, Malte, Linda Wirthwein, Gunnar Lemmer, and Ricarda Steinmayr. 2014. Academic self-handicapping and achievement: A meta-analysis. Journal of Educational Psychology 106, no. 3 : 744 – 61. doi: 10.1037/a0035832.</bibtext> </blist> <blist> <bibtext> Simkin, Mark, and William Kuechler. 2005. Multiple-choice tests and student understanding: What is the connection? Decision Sciences Journal of Innovative Education 3, no. 1 : 73 – 98. doi: 10.1111/j.1540-4609.2005.00053.x.</bibtext> </blist> <blist> <bibtext> Soderstrom, Nicholas, and Robert Bjork. 2015. Learning versus performance: An integrative review. Perspectives on Psychological Science: A Journal of the Association for Psychological Science 10, no. 2 : 176 – 99. doi: 10.1177/1745691615569000.</bibtext> </blist> <blist> <bibtext> Theophilides, Christos, and Mary Koutselini. 2000. Study behavior in the closed-book and the open-book examination: A comparative analysis. Educational Research and Evaluation: An International Journal on Theory and Practice 6, no. 4 : 379 – 93. doi: 10.1076/edre.6.4.379.6932.</bibtext> </blist> <blist> <bibtext> Vidya, Ganji. 2019. A study to evaluate the effectiveness of open book tests over close book tests and perception of medical students in physiology. International Journal of Physiology 7, no. 2 : 30 – 4. doi: 10.5958/2320-608X.2019.00038.6.</bibtext> </blist> <blist> <bibtext> White, Charles. 2007. Levels of understanding—A guide to the teaching and assessment of knowledge. Journal of Education for Business 82, no. 3 : 159 – 63. doi: 10.3200/JOEB.82.3.159-163.</bibtext> </blist> </ref> <aug> <p>By Vahe Permzadian and Kit W. Cho</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib19" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib24" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib31" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib20" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib27" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib39" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib43" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib15" firstref="ref11"></nolink> <nolink nlid="nl9" bibid="bib23" firstref="ref13"></nolink> <nolink nlid="nl10" bibid="bib37" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib40" firstref="ref20"></nolink> <nolink nlid="nl12" bibid="bib176" firstref="ref21"></nolink> <nolink nlid="nl13" bibid="bib12" firstref="ref24"></nolink> <nolink nlid="nl14" bibid="bib25" firstref="ref26"></nolink> <nolink nlid="nl15" bibid="bib18" firstref="ref27"></nolink> <nolink nlid="nl16" bibid="bib22" firstref="ref31"></nolink> <nolink nlid="nl17" bibid="bib42" firstref="ref32"></nolink> <nolink nlid="nl18" bibid="bib16" firstref="ref37"></nolink> <nolink nlid="nl19" bibid="bib17" firstref="ref38"></nolink> <nolink nlid="nl20" bibid="bib28" firstref="ref39"></nolink> <nolink nlid="nl21" bibid="bib30" firstref="ref40"></nolink> <nolink nlid="nl22" bibid="bib34" firstref="ref41"></nolink> <nolink nlid="nl23" bibid="bib36" firstref="ref42"></nolink> <nolink nlid="nl24" bibid="bib35" firstref="ref46"></nolink> <nolink nlid="nl25" bibid="bib21" firstref="ref48"></nolink> <nolink nlid="nl26" bibid="bib38" firstref="ref59"></nolink> <nolink nlid="nl27" bibid="bib41" firstref="ref78"></nolink> <nolink nlid="nl28" bibid="bib10" firstref="ref92"></nolink> <nolink nlid="nl29" bibid="bib32" firstref="ref93"></nolink> <nolink nlid="nl30" bibid="bib26" firstref="ref105"></nolink> <nolink nlid="nl31" bibid="bib864" firstref="ref126"></nolink> <nolink nlid="nl32" bibid="bib233" firstref="ref131"></nolink> <nolink nlid="nl33" bibid="bib11" firstref="ref142"></nolink> <nolink nlid="nl34" bibid="bib29" firstref="ref143"></nolink> <nolink nlid="nl35" bibid="bib33" firstref="ref144"></nolink> <nolink nlid="nl36" bibid="bib13" firstref="ref151"></nolink> <nolink nlid="nl37" bibid="bib14" firstref="ref153"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1468349
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Does the Format of an Assessment (Closed Book or Open Book) Affect Learning? A Systematic Review of the Literature
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Vahe+Permzadian%22">Vahe Permzadian</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1278-9056">0000-0003-1278-9056</externalLink>)<br /><searchLink fieldCode="AR" term="%22Kit+W%2E+Cho%22">Kit W. Cho</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0194-9851">0000-0003-0194-9851</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Teaching+in+Higher+Education%22"><i>Teaching in Higher Education</i></searchLink>. 2025 30(4):880-897.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 18
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Information Analyses
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Tests%22">Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Format%22">Test Format</searchLink><br /><searchLink fieldCode="DE" term="%22Long+Term+Memory%22">Long Term Memory</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Results%22">Test Results</searchLink><br /><searchLink fieldCode="DE" term="%22Retention+%28Psychology%29%22">Retention (Psychology)</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Selection%22">Test Selection</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/13562517.2023.2289128
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1356-2517<br />1470-1294
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: When administering an in-class exam, a common decision that confronts every instructor is whether the exam format should be closed book or open book. The present review synthesizes research examining the effect of administering closed-book or open-book assessments on long-term learning. Although the overall effect of assessment format on learning was mixed, two robust findings were that closed-book assessments led to a lower rate of forgetting (i.e. the percentage of change in information participants remembered on an initial test compared to that of the final test) relative to open-book assessments and students preferred to take open-book assessments. We also discuss possible moderators (e.g. item overlap) of the effect. Recommendations are provided to instructors who seek guidance regarding the most appropriate assessment format to adopt for their course. We conclude by outlining avenues for future research.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1468349
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1468349
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/13562517.2023.2289128
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 18
        StartPage: 880
    Subjects:
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Tests
        Type: general
      – SubjectFull: Test Format
        Type: general
      – SubjectFull: Long Term Memory
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Test Results
        Type: general
      – SubjectFull: Retention (Psychology)
        Type: general
      – SubjectFull: Test Selection
        Type: general
    Titles:
      – TitleFull: Does the Format of an Assessment (Closed Book or Open Book) Affect Learning? A Systematic Review of the Literature
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Vahe Permzadian
      – PersonEntity:
          Name:
            NameFull: Kit W. Cho
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1356-2517
            – Type: issn-electronic
              Value: 1470-1294
          Numbering:
            – Type: volume
              Value: 30
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Teaching in Higher Education
              Type: main
ResultId 1