A Systematic Literature Review of Student Evaluation of Peer Exemplars and Implications for Design, Technology, and Engineering Learning
Saved in:
| Title: | A Systematic Literature Review of Student Evaluation of Peer Exemplars and Implications for Design, Technology, and Engineering Learning |
|---|---|
| Language: | English |
| Authors: | Scott Thorne (ORCID |
| Source: | International Journal of Technology and Design Education. 2024 34(4):1441-1462. |
| Availability: | Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/ |
| Peer Reviewed: | Y |
| Page Count: | 22 |
| Publication Date: | 2024 |
| Document Type: | Journal Articles Information Analyses |
| Education Level: | Secondary Education Higher Education Postsecondary Education |
| Descriptors: | Design, Engineering Education, Learning Motivation, Self Efficacy, Barriers, Learning Processes, Technology Education, Research Reports, Secondary School Students, College Students, Correlation, Outcomes of Education, Peer Evaluation, Misconceptions, Student Attitudes |
| DOI: | 10.1007/s10798-023-09874-2 |
| ISSN: | 0957-7572 1573-1804 |
| Abstract: | In design courses, reviewing how others have solved design problems or completed projects is common practice and often encouraged by educators. Using student work as examples can provide context for assessment criteria and help students approach new design problems. While studies have explored the use of exemplars in various disciplines, little research has focused on which exemplars to use (e.g., high-quality, low-quality) in design, technology, and engineering fields. To address this gap, researchers conducted a literature review of 33 articles on exemplar use in secondary and post-secondary education. The analysis revealed nine themes related to exemplar use and their impact on student learning, including (1) Clarity of instruction, (2) Learner focus, (3) Motivation for learning, (4) Student reflection on learning, (5) Building student self-efficacy, (6) Identifying instructional challenges, (7) Providing contrasting cases, (8) The relationship between exemplar quality and student work quality, and (9) Raising the bar for learning outcomes. Findings suggest that simply providing an exemplar is not enough and that the selection of an exemplar can have positive or negative impacts on student motivation, understanding, and application. Carefully selecting exemplars and engaging in dialogue with students can help them identify expectations, recognize quality work, and identify potential misconceptions. These findings have implications for those involved in design, technology, and engineering education. Educators can use these findings to guide their selection of exemplars and engage students in meaningful dialogue to aid their learning. Researchers can also use these findings to further investigate the use of exemplars in these fields. |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| Accession Number: | EJ1429986 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFY1oEr-dAUNJ9-Er67y1jxAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDEvtJ7WcE_gcH4MlsQIBEICBmwjRg7f3xX3cMYboV3Z42NSxdOB585OYS3PKCUg_WL92r-oyjVhJImd61Pn3G9fGah8Qb1bgBeIBHU3015hnna7Plhp_qNVChB6ieG-3VapDUGkySfXqRdcaNXfDHrx7HjJdEzlIyxy0gUR8O7GRQzUEZgdqWq7Yh9tA60j-OznN9x1tpLGYfXt46f2VXpSxzXyWU1ylXbeCwwgY Text: Availability: 1 Value: <anid>AN0178333943;ogv01sep.24;2024Jul11.05:05;v2.2.500</anid> <title id="AN0178333943-1">A systematic literature review of student evaluation of peer exemplars and implications for design, Technology, and Engineering Learning </title> <p>In design courses, reviewing how others have solved design problems or completed projects is common practice and often encouraged by educators. Using student work as examples can provide context for assessment criteria and help students approach new design problems. While studies have explored the use of exemplars in various disciplines, little research has focused on which exemplars to use (e.g., high-quality, low-quality) in design, technology, and engineering fields. To address this gap, researchers conducted a literature review of 33 articles on exemplar use in secondary and post-secondary education. The analysis revealed nine themes related to exemplar use and their impact on student learning, including (<reflink idref="bib1" id="ref1">1</reflink>) Clarity of instruction, (<reflink idref="bib2" id="ref2">2</reflink>) Learner focus, (<reflink idref="bib3" id="ref3">3</reflink>) Motivation for learning, (<reflink idref="bib4" id="ref4">4</reflink>) Student reflection on learning, (<reflink idref="bib5" id="ref5">5</reflink>) Building student self-efficacy, (<reflink idref="bib6" id="ref6">6</reflink>) Identifying instructional challenges, (<reflink idref="bib7" id="ref7">7</reflink>) Providing contrasting cases, (<reflink idref="bib8" id="ref8">8</reflink>) The relationship between exemplar quality and student work quality, and (<reflink idref="bib9" id="ref9">9</reflink>) Raising the bar for learning outcomes. Findings suggest that simply providing an exemplar is not enough and that the selection of an exemplar can have positive or negative impacts on student motivation, understanding, and application. Carefully selecting exemplars and engaging in dialogue with students can help them identify expectations, recognize quality work, and identify potential misconceptions. These findings have implications for those involved in design, technology, and engineering education. Educators can use these findings to guide their selection of exemplars and engage students in meaningful dialogue to aid their learning. Researchers can also use these findings to further investigate the use of exemplars in these fields.</p> <p>Keywords: Student evaluation of exemplars; Selection of exemplars; Design-based learning; Student work quality; Building student self-efficacy; Motivation for learning</p> <p>Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</p> <hd id="AN0178333943-2">Introduction</hd> <p>One important factor for student success, across all levels of education, is the ability to critically evaluate their work's quality in alignment with their instructor's expectations (Amulya, [<reflink idref="bib2" id="ref10">2</reflink>]; Anderson &amp; Freiberg, [<reflink idref="bib4" id="ref11">4</reflink>]; McMillan &amp; Hearn, [<reflink idref="bib32" id="ref12">32</reflink>]). In an effort to aid students to be independent self-regulating learners, rubrics and criteria are often provided as part of design, technology, and engineering courses. These tools can communicate expectations with students and allow students to evaluate their own work, including open-ended design tasks, against a set of standards (Andrade, [<reflink idref="bib5" id="ref13">5</reflink>]). The type of rubric employed (analytic or holistic) is determined by the scope, purpose, and value attributed by researchers or practitioners in its context of application.</p> <p>However, research has shown that relying solely on rubrics is inadequate for guiding student learning as "verbal descriptions are always to some degree vague or fuzzy. A fuzzy standard cannot, therefore, be defined into existence" (Sadler, [<reflink idref="bib39" id="ref14">39</reflink>], p. 202). For this reason, Sadler ([<reflink idref="bib39" id="ref15">39</reflink>]), recommended to choose and present students with samples or peer exemplars of relevant coursework that align with the desired criteria in context, this is because the assessment process involving exemplars and rubrics can be intertwined to mutually reinforce each other. While literature has established that selecting exemplars that highlight important criteria is beneficial, little definition exists regarding how the quality of the exemplars impact learning.</p> <p>This review encompasses diverse studies on the use of peer exemplars across various fields including business, design, and science (Bouwer et al., [<reflink idref="bib9" id="ref16">9</reflink>]; Carless &amp; Chan, [<reflink idref="bib11" id="ref17">11</reflink>]; Knight et al., [<reflink idref="bib30" id="ref18">30</reflink>]; Kulkarni et al., [<reflink idref="bib31" id="ref19">31</reflink>]) to synthesize how exemplar use impacts learning. Participants in these studies are often presented with full text documents in which they evaluate the exemplar against a rubric to determine the criteria (clarity), define what makes it 'good' (quality), and identify misconceptions when attempting a similar task (calibration). This systematic literature review relies on secondary data sources to develop an informed stance about how the attributes of exemplar selection influence learning. This is done to advance both design-based learning and the learning of design.</p> <hd id="AN0178333943-3">Research purpose</hd> <p>The purpose of this systematized review was to search, identify, and synthesize themes from existing literature on the role of the evaluation of peer exemplars—samples of previous similar work—in the learning experience. This was approached by identifying relevant literature in the field, selective linking to expand the pool of relevant literature, and identifying themes from these studies for a systematized search. The systematized review was guided by the following questions:</p> <p></p> <ulist> <item> What are the potential influences of reviewing peer exemplars on student learning?</item> <p></p> <item> How does the selection of peer exemplars influence the learning experience?</item> </ulist> <p>Given the intention to apply findings from this research into design, technology, and engineering education, initial searches focused on publications in the related educational journals or conference proceedings. Specific methods for conducting and refining such a search are described below. Following the search protocol descriptions, themes identified from existing research emerged to address the above questions.</p> <hd id="AN0178333943-4">Method</hd> <p>Strategies used to refine a set of search terms specific to the above research questions were patterned from recommendations in conducting a systematic literature review by Borrego et al. ([<reflink idref="bib8" id="ref20">8</reflink>]). The use of peer exemplars and evaluation in education has been explored in multiple studies and disciplines, however, the intentional selection of peer exemplars, or which exemplars to show students, remains unanswered and unexplored. Therefore, as suggested by Petticrew and Roberts ([<reflink idref="bib36" id="ref21">36</reflink>]), a systematic literature review was chosen to evaluate existing research in a "transparent, methodical, and reproducible" manner (Borrego et al., [<reflink idref="bib8" id="ref22">8</reflink>], p. 46).</p> <p>Initial investigations to establish search criteria consisted of 23 articles selected based on teaching methods that rely on students interacting with and evaluating examples such as gallery walks, math thinking, worked examples, and visual thinking strategies. However, these did not provide data to answer the research questions. The primary focus of each was instead the student performance after embedding said strategy into the curriculum than the strategy itself. One article that not only began to address the research questions, but cited several relevant studies was "Why and how educators use exemplars" (Hawe et al., [<reflink idref="bib22" id="ref23">22</reflink>]). Through selective linking, four more high quality articles were identified and analyzed for key themes to start a systematic search (Carless et al., [<reflink idref="bib12" id="ref24">12</reflink>]; Carless &amp; Chan, [<reflink idref="bib11" id="ref25">11</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref26">21</reflink>]; Newlyn, [<reflink idref="bib35" id="ref27">35</reflink>]). A synthesis of these key articles as well as the previous 23 articles that focused on instructional strategies related to learning through the evaluation of peer exemplars resulted in the establishment of keywords for a targeted search (see Appendix 1).</p> <p>Journals related to design, education, technology, technology education, engineering, and engineering education were identified for their relevance on how to identify effective peer exemplars. For this reason, the databases EBSCOhost (ERIC), Scopus, and Elsevier (Engineering Village) were chosen as they index field specific conference proceedings including the American Society for Engineering Education (ASEE) and IEEE Frontiers in Education (FIE). Initial numbers of results from these databases were quite high. EBSCOhost (ERIC) returned 910, Scopus returned the most at 451,504, and Elsevier (Engineering Village) returned 14,422 articles. These results were narrowed by several factors including full text availability, peer reviewed, only articles from the past 10 years (2012–2022), and articles in relevant subject areas (i.e., engineering and math). This brought total search results down to 259 articles in EBSCOhost (ERIC), 382 in Scopus, and 231 in Elsevier (Engineering Village).</p> <p>The following criteria were established to prioritize studies with an empirical nature that explored the use of exemplars and their influence on student learning:</p> <p></p> <ulist> <item> The methods used in each study included an evaluation of samples, examples, or exemplars which may have been current student work, previous student work or from sources external to the classroom (e.g., professional work) where the central concept is consciously looking such that students are engaged in review or evaluation of the artifact to consider its perceived merits, scope and limitations.</item> <p></p> <item> The focus of the article was on supporting student learning.</item> <p></p> <item> Research was in an academic setting where teaching methods were at the forefront of the study.</item> <p></p> <item> Not included: Articles on machine learning, training algorithms, deep learning for artificial intelligence, measures of student response time or where exemplars were used to assess student learning.</item> </ulist> <p>Search results including the title, author, and abstract from each database were exported into individual spreadsheets in an Excel workbook. After eliminating duplicates, the number of unique articles was reduced to 872. Next, a face validity check was performed on the article titles, further narrowing the selection down to 158 articles. These 158 articles were further refined by assessing their content validity, which involved an analysis of both the abstracts and methods sections. This rigorous process resulted in seven articles for an in-depth analysis, namely those authored by Dixon et al. ([<reflink idref="bib16" id="ref28">16</reflink>]), Gonczi et al. ([<reflink idref="bib18" id="ref29">18</reflink>]), Hawe and Dixon ([<reflink idref="bib20" id="ref30">20</reflink>]), Headley and Pittson ([<reflink idref="bib23" id="ref31">23</reflink>]), Kean ([<reflink idref="bib28" id="ref32">28</reflink>]), Kulkarni et al. ([<reflink idref="bib31" id="ref33">31</reflink>]), and Rawson and Dunlosky ([<reflink idref="bib38" id="ref34">38</reflink>]).The same inclusion and exclusion criteria used during the face validity assessment of article titles were consistently applied to maintain methodological rigor, relevance to exemplars, and contextual appropriateness.</p> <p>To ensure the reliability of inclusion and exclusion decisions, frequently cited literature that was not included in the initial search results were reexamined. It was noted that while previously identified authors from the five articles focused on exemplars were represented in the dataset (Hawe, Carless, Chan, Newlyn), none of the articles themselves were present. Further, frequently cited authors, such as Carless, had no articles represented in the search results. At an attempt to be more inclusive, the original five articles on peer exemplars were made into a word frequency cloud using NVivo software (see Fig. 1). A new keyword search was developed using words unique to these articles, and passed again into EBSCOhost (ERIC), Scopus, and Elsevier (Engineering Village) (see Appendix 2). This additional step of checking for frequently cited publications that were not present in the first search query may help improve the reliability of the study.</p> <p>Graph: Fig. 1 Exemplar word cloud</p> <p>This limited list returned far fewer results through each of the databases, including 199 from EBSCOhost (ERIC), 134 in Scopus, and only 54 in Elsevier (Engineering Village). Using similar criteria as with the first search, these were further narrowed down by limiting to only journal articles and conference papers, full text availability, and articles in the past 10 years (2012–2022). This further reduced the number of results to 45 in EBSCOhost (ERIC), 14 in Scopus, and 3 in Elsevier (Engineering Village).</p> <p>Search results from each database were exported, including the title, author, and abstract, into individual spreadsheets in an Excel workbook. Once combined, with duplicates removed, the total number of unique articles was 59. Again, the same inclusion and exclusion criteria were utilized on each title for face validity, reducing the total number down to 36 articles. Each of the remaining articles were further checked for content validity by assessing the abstract and methods section, reducing the total number to 24 articles. These 24 articles focused on exemplars were combined with the seven articles from the initial search strategy and the five from the initial exemplar articles, and, after removing duplicates, was reduced to 33 articles for final synthesis (see Figs. 2 and 3).</p> <p>Graph: Fig. 2 Flowchart of search results by selection criteria</p> <p>Graph: Fig. 3 Number of articles in final selection by year from 2012 – 2022. Note Refer to the inclusion and exclusion criteria for the selection of these articles</p> <hd id="AN0178333943-5">Data analysis</hd> <p>After finalizing a list of articles, each study was coded in NVivo 12 using open-coding procedures (Merriam &amp; Tisdell, [<reflink idref="bib33" id="ref35">33</reflink>]). Although typically associated with qualitative research, this method was adapted for the systematic review as it offers a structured process for identifying any valuable data segments and aids in the development of categories and themes. Initially, a comprehensive reading of each article was conducted to establish familiarity. During this initial reading, codes were generated based on recurring keywords, phrases, and themes within the article, while detailed notes were recorded in a linked memo. Codes from each additional study were developed in much the same way using the constant-comparison technique to identify commonalities and deviations from initial codes (Glaser, [<reflink idref="bib17" id="ref36">17</reflink>]). As coding progressed and discernable patterns began to emerge, axial coding was used to reorganize and establish a hierarchy to create overarching themes (Charmaz, [<reflink idref="bib13" id="ref37">13</reflink>]). Following the identification of themes and a thorough understanding of the codes, each article underwent a secondary reading to enhance data familiarity and confirm alignment with the overarching themes.</p> <hd id="AN0178333943-6">Findings</hd> <p>The research questions focused on the influence of reviewing peer exemplars on student learning, and how the selection of exemplars influences the student learning experience. During the thorough examination of documents within NVivo through open coding techniques (Saldaña, [<reflink idref="bib43" id="ref38">43</reflink>]), overarching themes emerged from the existing research. Rather than strictly aligning these themes with specific questions, it became evident that they were intricately interconnected. As a result, we present the following comprehensive nine themes, each of which will be reviewed in detail:</p> <p></p> <ulist> <item> Clarity of instruction.</item> <p></p> <item> Learner focus.</item> <p></p> <item> Motivation for learning.</item> <p></p> <item> Student reflection on learning.</item> <p></p> <item> Building student self-efficacy.</item> <p></p> <item> Identifying instructional challenges.</item> <p></p> <item> Providing contrasting cases.</item> <p></p> <item> The relation between exemplar quality and student work quality.</item> <p></p> <item> Raising the bar for learning outcomes.</item> </ulist> <p>In the following sections, each theme will be described in the context of students evaluating peer exemplars as a learning experience. Citations and excerpts from the final articles selected for review, including qualitative comments from participants in their studies, are included below to contextualize and provide credibility to the findings.</p> <hd id="AN0178333943-7">Theme 1: clarity of instruction</hd> <p>Expanding and defining project criteria emerged as a major theme for the role of peer exemplar evaluation in enhancing the learning experience. This approach enables teachers to provide clarity and helps students recognize the quality or lack of quality of their work more effectively than other mechanisms (Sadler, [<reflink idref="bib41" id="ref39">41</reflink>]). Establishing clarity early in the process allows for both teachers and students to be in alignment on assignment expectations (Chong, [<reflink idref="bib15" id="ref40">15</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref41">19</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref42">20</reflink>]; Headley &amp; Pittson, [<reflink idref="bib23" id="ref43">23</reflink>]; Hendry et al., [<reflink idref="bib26" id="ref44">26</reflink>]; Yucel et al., [<reflink idref="bib50" id="ref45">50</reflink>]) as assessment criteria was sometimes be viewed as too dense or difficult to comprehend (Bell et al., [<reflink idref="bib7" id="ref46">7</reflink>]; Carless et al., [<reflink idref="bib12" id="ref47">12</reflink>]). These points were bolstered by quotes from students such as, "<emph>I would not have known what to do if we had not done this. It gave me an accurate expectation</emph>" (Yucel et al., [<reflink idref="bib50" id="ref48">50</reflink>], p. 976). A student from another study had a similar sentiment, "<emph>By you giving us those examples, we can determine what you deem to be the best answer, so we can kind of structure our own letter to match that, or improve on that</emph>" (Hendry et al., [<reflink idref="bib26" id="ref49">26</reflink>], p. 153).</p> <p>Establishing clarity for expectations at the start of a project can allow for students to establish forethought as they begin to plan for short and long-term goals (Zimmerman, [<reflink idref="bib51" id="ref50">51</reflink>]), especially as they are able to better visualize requirements and expectations (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref51">1</reflink>]; Bell et al., [<reflink idref="bib7" id="ref52">7</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref53">19</reflink>]; Tam, [<reflink idref="bib46" id="ref54">46</reflink>]). Providing transparency through peer exemplars was received very positively by students (Carless et al., [<reflink idref="bib12" id="ref55">12</reflink>]; Chong, [<reflink idref="bib15" id="ref56">15</reflink>]; Headley &amp; Pittson, [<reflink idref="bib23" id="ref57">23</reflink>]; Yucel et al., [<reflink idref="bib50" id="ref58">50</reflink>]). One student even described how the clarity in expectations helped bring understanding to grade distributions, making for a more enjoyable experience even when underperforming:<emph>Since this process I find that I am more aware now of what's expected of me in assessments and the marking criteria and how markers approach them is now more clear. I'm also no longer disappointed with a mark as I have some idea of what I'll receive.</emph> (Kearney &amp; Perkins, [<reflink idref="bib29" id="ref59">29</reflink>], p. 5)</p> <p>Clarity for learning, a major theme across multiple studies, may also be at the center of each of the themes across all study contexts. Having clarity in a project allows for focus, provides the opportunity for motivation, and enables reflection. Without fully understanding expectations, students are not likely to develop self-efficacy, identify challenges, or know where to start if attempting to raise the bar. It is the umbrella term that is in the background of each of the following themes.</p> <hd id="AN0178333943-8">Theme 2: learner focus</hd> <p>When attempting a project, students are often lost on where and how to start, even with a well written project description and detailed rubrics (Grainger et al., [<reflink idref="bib19" id="ref60">19</reflink>]). In fact, too detailed of rubrics can confuse students as they tend to present information in abstract terms (Bouwer et al., [<reflink idref="bib9" id="ref61">9</reflink>]; Sadler, [<reflink idref="bib40" id="ref62">40</reflink>], [<reflink idref="bib41" id="ref63">41</reflink>]) and the more standards and criteria are broken down, "the harder it is to make the bits work together as a whole" (Sadler, [<reflink idref="bib42" id="ref64">42</reflink>], p. 389). Hendry and Tomitsch ([<reflink idref="bib25" id="ref65">25</reflink>]) stated that when left to their own devices, students often ignore rubrics and criteria until they have nearly completed their task. One student elaborates "<emph>you don't even read the criteria of what you're making. You... go out, make it and think, 'Oh my God, this is the best ever' [but miss an important criterion]. It happens.</emph>" (Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref66">25</reflink>], p. 342). Evaluation of peer exemplars shifts the focus back to what is being assessed, encouraging students to check their assumptions against an agreed upon standard (Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref67">25</reflink>]; Kearney &amp; Perkins, [<reflink idref="bib29" id="ref68">29</reflink>]; Knight et al., [<reflink idref="bib30" id="ref69">30</reflink>]).</p> <p>Peer exemplars help to address this by showing what the criteria look like in a final product, allowing students to take a step back and see the bigger picture (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref70">1</reflink>]). In one study, researchers noted the shift in attention that when students evaluated documents side by side, explaining "Compared to the criteria condition, students in the comparative judgment condition [holistically evaluating two exemplars side-by-side] focused relatively more on aspects that were related to the content and structure of the text, and less so on aspects that were related to grammar and vocabulary." (Bouwer et al., [<reflink idref="bib9" id="ref71">9</reflink>]). Using peer exemplars can help broaden students' focus and enable them to differentiate between various levels of achievement in areas that are not covered by rubrics or task guidelines due to spatial constraints, such as formatting and layout. (Grainger et al., [<reflink idref="bib19" id="ref72">19</reflink>]).</p> <p>Students too found it useful to see what the teacher might focus on when evaluating an assignment (Hendry &amp; Jukic, [<reflink idref="bib24" id="ref73">24</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref74">25</reflink>]; Kean, [<reflink idref="bib28" id="ref75">28</reflink>]), as one stated, "<emph>It is always good to see how a teacher will evaluate your work. Some teachers have different styles and look for different things. Listening to the teacher [talk about peer exemplars] made me understand what to focus on.</emph>" (Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref76">25</reflink>], p. 341). After evaluation of peer exemplars, students not only started to recognize patterns of expectations (Kean, [<reflink idref="bib28" id="ref77">28</reflink>]; Yucel et al., [<reflink idref="bib50" id="ref78">50</reflink>]), they were also more aware of the current state of their work and found themselves more receptive to feedback (Kearney &amp; Perkins, [<reflink idref="bib29" id="ref79">29</reflink>]; Tam, [<reflink idref="bib46" id="ref80">46</reflink>]).</p> <hd id="AN0178333943-9">Theme 3: motivation for learning</hd> <p>The evaluation of authentic peer exemplars, those created by students, can be a source of motivation for starting a project (Grainger et al., [<reflink idref="bib19" id="ref81">19</reflink>]; Headley &amp; Pittson, [<reflink idref="bib23" id="ref82">23</reflink>]) as it shows students not only what it might look like, but that it can be done (Dixon et al., [<reflink idref="bib16" id="ref83">16</reflink>]; Hendry &amp; Jukic, [<reflink idref="bib24" id="ref84">24</reflink>]). One postgraduate student shared "<emph>Going through it with [the teacher] at the end and seeing what [the teacher] gave marks for did make you think, 'I actually – I could do that, I could do that"</emph> (Hendry &amp; Jukic, [<reflink idref="bib24" id="ref85">24</reflink>], p. 9). Other students noted that the process of making evaluations of peer exemplars was rather exciting and highly engaging (Kearney &amp; Perkins, [<reflink idref="bib29" id="ref86">29</reflink>]), with some looking forward to seeing if they were in alignment with their teacher:<emph>The first stage [marking] is making you aware of your own thoughts and what's going on in your own head and how you're judging things and then, when [the teacher explains], there's this big reveal kind of thing. You wanted to know, the suspense was killing you.</emph> (Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref87">25</reflink>], p. 342)</p> <p>While peer exemplars can help to motivate students, changing the quality of exemplars too far in <emph>either</emph> direction (I.e., much higher or lower quality) can act as a demotivating factor. Sharing work that is of very high quality, especially when only showing high quality work, can cause students to doubt their capabilities (Hawe &amp; Dixon, [<reflink idref="bib20" id="ref88">20</reflink>]) and undermine confidence (Dixon et al., [<reflink idref="bib16" id="ref89">16</reflink>]), but surprisingly, sharing low quality examples can cause similar reactions, especially with authentic peer exemplars. In one study, students felt anxious seeing that past students had performed poorly (Hendry &amp; Jukic, [<reflink idref="bib24" id="ref90">24</reflink>]), while in a separate study, students described examples of students who had failed as "unhelpful, demotivating, and even distracting" (Grainger et al., [<reflink idref="bib19" id="ref91">19</reflink>], p. 5). Recommendations to combat this include sharing a range of peer exemplars, either all of higher quality, or a wider range that goes no lower than 'passing' (Bouwer et al., [<reflink idref="bib9" id="ref92">9</reflink>]; Dixon et al., [<reflink idref="bib16" id="ref93">16</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref94">21</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref95">20</reflink>]; Hendry et al., [<reflink idref="bib26" id="ref96">26</reflink>]). When applied correctly, however, peer exemplars of the appropriate level of complexity can motivate and inspire students "to 'match or beat' the quality and standard" (Bell et al., [<reflink idref="bib7" id="ref97">7</reflink>], p. 772).</p> <hd id="AN0178333943-10">Theme 4: student reflection on learning</hd> <p>While not frequently cited, the act of metacognition when interpreting tacit knowledge is essential to the student learning experience. Evaluation helps to make this tacit knowledge explicit, as the process helps trigger a reflective process as considering the kinds of feedback they might provide others aids in transfer to self-assessment (Bell et al., [<reflink idref="bib7" id="ref98">7</reflink>]; Tam, [<reflink idref="bib46" id="ref99">46</reflink>]; Yucel et al., [<reflink idref="bib50" id="ref100">50</reflink>]).</p> <p>To help foster this, it is strongly encouraged, by nearly every study in this review focusing on the use of evaluating peer exemplars as an instructional method, that teachers model their thinking and engage students in dialogue after assessment (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref101">1</reflink>]; Bell et al., [<reflink idref="bib7" id="ref102">7</reflink>]; Bouwer et al., [<reflink idref="bib9" id="ref103">9</reflink>]; Carless et al., [<reflink idref="bib12" id="ref104">12</reflink>]; Carless &amp; Boud, [<reflink idref="bib10" id="ref105">10</reflink>]; Carless &amp; Chan, [<reflink idref="bib11" id="ref106">11</reflink>]; Chong, [<reflink idref="bib14" id="ref107">14</reflink>], [<reflink idref="bib15" id="ref108">15</reflink>]; Dixon et al., [<reflink idref="bib16" id="ref109">16</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref110">19</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref111">21</reflink>], [<reflink idref="bib22" id="ref112">22</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref113">20</reflink>]; Hendry et al., [<reflink idref="bib26" id="ref114">26</reflink>]; Hendry &amp; Jukic, [<reflink idref="bib24" id="ref115">24</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref116">25</reflink>]; Johnston et al., [<reflink idref="bib27" id="ref117">27</reflink>]; Kean, [<reflink idref="bib28" id="ref118">28</reflink>]; Tam, [<reflink idref="bib46" id="ref119">46</reflink>]; To &amp; Liu, [<reflink idref="bib48" id="ref120">48</reflink>]). One of the key reasons for this, is that students often under- or overestimate the work of others, as one student describes "<emph>We were marking it really low ... Or really high. We were thinking, 'It's a 7, or it's a 6', and [they'd] go, 'It's a 4', or 'It's a 3'. We were like, 'Wow!'</emph>" (Hendry et al., [<reflink idref="bib26" id="ref121">26</reflink>], p. 155). Other studies found that evaluating peer exemplars made students more attentive, asking questions about what makes a good answer. An undergraduate student elaborated:<emph>It made you pay attention more, so – then because you're curious too – yeah, because you can't assess how [the teacher's] marking unless you work out how you would mark it.... So you need to do that, otherwise it wouldn't sink in – I don't think it would sink in as much, what [the teacher] would say.</emph> (Hendry &amp; Jukic, [<reflink idref="bib24" id="ref122">24</reflink>], p. 7)</p> <p>Reflection when evaluating can also be enhanced through the use of rubrics or explicitly stated criteria either immediately before or during the marking session. One student reflected on his assumptions and knowledge about topics such as ethics when marking an exemplar:<emph>The marking criteria was useful as it provided avenues for reflection. For me it highlighted the elements of ethical perspectives and reasoning that should have drawn from the case. The marking criteria made me critically reflect on whether individual assumptions and knowledge about ethics were as strong and consistent as I previously thought.</emph> (Bell et al., [<reflink idref="bib7" id="ref123">7</reflink>], p. 777)</p> <p>Encouraging reflection during the process of evaluation can help students to internalize tacit knowledge, calibrate their understanding, and develop their own understanding of expectations.</p> <hd id="AN0178333943-11">Theme 5: building student self-efficacy</hd> <p>Student self-efficacy is a significant predictor of success and achievement for students (Zimmerman &amp; Schunk, [<reflink idref="bib52" id="ref124">52</reflink>]), and can be fostered through the evaluation of peer exemplars. The act of analyzing and evaluating peer exemplars has been shown to reduce some of the anxiety associated with the uncertainty of assessment standards and requirements (Carless &amp; Boud, [<reflink idref="bib10" id="ref125">10</reflink>]; Carless &amp; Chan, [<reflink idref="bib11" id="ref126">11</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref127">21</reflink>], [<reflink idref="bib22" id="ref128">22</reflink>]; Yucel et al., [<reflink idref="bib50" id="ref129">50</reflink>]). In fact, by evaluating multiple peer exemplars, students internalize a wider range of approaches to the assessment (Chong, [<reflink idref="bib14" id="ref130">14</reflink>]), which may help address concerns for stifling creativity and design fixation.</p> <p>Students engaging in exemplar evaluation, across multiple studies, indicated feeling more confident to work independently on challenging assignments (Bell et al., [<reflink idref="bib7" id="ref131">7</reflink>]; Dixon et al., [<reflink idref="bib16" id="ref132">16</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref133">21</reflink>], [<reflink idref="bib22" id="ref134">22</reflink>]; Headley &amp; Pittson, [<reflink idref="bib23" id="ref135">23</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref136">25</reflink>]). In a focus group with other professors who utilize peer exemplars at Harper Adams University, one professor shared that the evaluation of exemplars "<emph>gives students the confidence to work out how to do the assignment themselves without constant recourse to tutor to ask whether it is right"</emph> (Headley &amp; Pittson, [<reflink idref="bib23" id="ref137">23</reflink>], p. 23). This is reflected in Zimmerman's ([<reflink idref="bib51" id="ref138">51</reflink>]) theory of self-regulated learning, frequently cited in the literature around synthesis, which focuses on self-regulating behavior as a core component toward goal-completion.</p> <p>Researchers further describe the observation of self-efficacy in the actions of their students after evaluating peer exemplars as taking ownership in their work and actively monitoring and regulating their progress in concluding remarks:<emph>They were setting goals using the exemplars as benchmarks or targets; they were devising strategies to ensure they avoided plagiarism; they were managing resources (i.e., the various exemplars provided) by determining what exactly they were choosing to focus on, whether it was a FAILED exemplar or a quality exemplar or academic literacies.</emph> (Grainger et al., [<reflink idref="bib19" id="ref139">19</reflink>], p. 8)</p> <p>This, again, closely aligns with the establishment of appropriate actions, thoughts, feelings, and behaviors central to self-regulated learning (Zimmerman, [<reflink idref="bib51" id="ref140">51</reflink>]). Additionally, it seems that by seeing authentic peer exemplars, even of B and C level work, students who might doubt their abilities are inspired that they too can succeed (Dixon et al., [<reflink idref="bib16" id="ref141">16</reflink>]; Hawe et al., [<reflink idref="bib22" id="ref142">22</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref143">20</reflink>]).</p> <hd id="AN0178333943-12">Theme 6: identifying instructional challenges</hd> <p>Peer exemplars are carefully chosen samples that are representative of student work, and while they are used to clarify expectations, they may also be used to better visualize different dimensions of quality (Carless &amp; Chan, [<reflink idref="bib11" id="ref144">11</reflink>]). While often associated with high quality work, it is common to provide student samples that highlight common mistakes and pitfalls that are made in assessments throughout the semester (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref145">1</reflink>]; Dixon et al., [<reflink idref="bib16" id="ref146">16</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref147">19</reflink>]; Hawe et al., [<reflink idref="bib22" id="ref148">22</reflink>]; Headley &amp; Pittson, [<reflink idref="bib23" id="ref149">23</reflink>]; Kean, [<reflink idref="bib28" id="ref150">28</reflink>]; Newlyn, [<reflink idref="bib35" id="ref151">35</reflink>]; Rawson &amp; Dunlosky, [<reflink idref="bib38" id="ref152">38</reflink>]; Tam, [<reflink idref="bib46" id="ref153">46</reflink>]; Yucel et al., [<reflink idref="bib50" id="ref154">50</reflink>]).</p> <p>In general, researchers felt that this was an opportunity for students to learn from the mistakes of others and while some take a passive role, providing an exemplar and letting students ask questions or come to their own conclusions (Bell et al., [<reflink idref="bib7" id="ref155">7</reflink>]; Kean, [<reflink idref="bib28" id="ref156">28</reflink>]; Kearney &amp; Perkins, [<reflink idref="bib29" id="ref157">29</reflink>]; Newlyn, [<reflink idref="bib35" id="ref158">35</reflink>]; Rawson &amp; Dunlosky, [<reflink idref="bib38" id="ref159">38</reflink>]), other researchers advocate for dialogue to help explain <emph>why</emph> students may score poorly (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref160">1</reflink>]; Dixon et al., [<reflink idref="bib16" id="ref161">16</reflink>]; Hawe et al., [<reflink idref="bib22" id="ref162">22</reflink>]; Tam, [<reflink idref="bib46" id="ref163">46</reflink>]). This strategy of dialogue has gained a following in math education. Understanding and walking students through the thought process behind mistakes to learn from them have inspired pedagogical strategies, such as "My Favorite No" (Teaching Channel, [<reflink idref="bib47" id="ref164">47</reflink>]), websites dedicated to cataloging wrong answers and how to address them (mathmistakes.org), book chapters (Seeley, [<reflink idref="bib45" id="ref165">45</reflink>]), and entire books (Anderson, [<reflink idref="bib3" id="ref166">3</reflink>]).</p> <p>Other researchers methodologies utilized the experience of having students take a more active role by either providing a rating for the assessment (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref167">1</reflink>]; Headley &amp; Pittson, [<reflink idref="bib23" id="ref168">23</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref169">25</reflink>]), or even correcting the mistakes they see in peer exemplars, offering suggestions on how it might be improved (Dixon et al., [<reflink idref="bib16" id="ref170">16</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref171">21</reflink>], [<reflink idref="bib22" id="ref172">22</reflink>]; Tam, [<reflink idref="bib46" id="ref173">46</reflink>]; Yucel et al., [<reflink idref="bib50" id="ref174">50</reflink>]).</p> <p>Not all authors in the articles reviewed are in agreement with the practice of evaluating low quality peer exemplars, as some researchers state that it may draw too much focus on what not to do, rather than highlighting best practices. Hendry and Tomitsch ([<reflink idref="bib25" id="ref175">25</reflink>]) explain their reasoning for only using 'pass', 'credit', and 'high distinction' peer exemplars with students, stating:<emph>We believe that providing students with a 'fail' exemplar might be counterproductive because it could focus students on what not to do rather than what to strive for. It might also challenge students' positive outlook and belief in their ability to be successful in their design project.</emph> (Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref176">25</reflink>], p. 344)</p> <p>This statement is supported by previous work by Hendry and Tomitsch, finding that in courses where students were presented with low quality peer exemplars and the teacher emphasized errors and low quality, the students "felt less confident to attempt their assignment" (Hendry et al., [<reflink idref="bib26" id="ref177">26</reflink>], p. 157). Other researchers acknowledge that the use of 'fail' level peer exemplars may be counterproductive, as they do not show what is needed to achieve (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref178">1</reflink>]), and may even demotivate students (Grainger et al., [<reflink idref="bib19" id="ref179">19</reflink>]). Interestingly, when low quality samples were absent from studies, students indicated that they wanted them included (Bell et al., [<reflink idref="bib7" id="ref180">7</reflink>]), with one student explaining how they "<emph>Find it useful, of how people have approached it and how that's worked, so like not just shining examples but where people have gone wrong perhaps and how to avoid that"</emph> (Kean, [<reflink idref="bib28" id="ref181">28</reflink>], p. 92). In summary, providing too low quality of peer exemplars is counterproductive but leaving them out entirely leaves students uncertain of how to fully interpret assignment criteria.</p> <hd id="AN0178333943-13">Theme 7: providing contrasting cases</hd> <p>While Schwartz's ([<reflink idref="bib44" id="ref182">44</reflink>]) did not meet our inclusion criteria, it offers valuable insights into how students can benefit from reviewing exemplars of varying complexity. Contrasting cases guide students to discern similarities and differences in exemplar qualities, promoting deeper understanding of learning objectives and preparing them for future knowledge transfer. Although direct comparisons of drastically different peer exemplars were limited in the studies cited (Bouwer et al., [<reflink idref="bib9" id="ref183">9</reflink>]; Chong, [<reflink idref="bib15" id="ref184">15</reflink>]; Hendry &amp; Jukic, [<reflink idref="bib24" id="ref185">24</reflink>]), many studies presented a range of student work to more thoroughly define rubric criteria. To illustrate this range of approaches, we have included a table (Table 1) summarizing the methods of comparison employed across studies. Referencing the quality of peer exemplars evaluated when reviewing conclusions of various researchers helps to contextualize the student experience. It is worth noting that while other studies included in this systematic review may have also specified the use of different quality peer exemplars, only those that explicitly tied student performance to contrasting quality are identified in the table and the subsequent discussion.</p> <p>Table 1 Comparing qualities of peer exemplars</p> <p> <ephtml> &lt;table frame="hsides" rules="groups"&gt;&lt;thead&gt;&lt;tr&gt;&lt;th align="left"&gt;&lt;p&gt;Quality provided&lt;/p&gt;&lt;/th&gt;&lt;th align="left"&gt;&lt;p&gt;References&lt;/p&gt;&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;Credit and above&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;Bell et al. (&lt;xref ref-type="bibr" rid="bibr7"&gt;2013&lt;/xref&gt;), Hendry &amp; Tomitsch, (&lt;xref ref-type="bibr" rid="bibr25"&gt;2014&lt;/xref&gt;), Knight et al. (&lt;xref ref-type="bibr" rid="bibr30"&gt;2019&lt;/xref&gt;), Newlyn, (&lt;xref ref-type="bibr" rid="bibr35"&gt;2013&lt;/xref&gt;)&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;Average and high quality&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;Bouwer et al. (&lt;xref ref-type="bibr" rid="bibr9"&gt;2018&lt;/xref&gt;)&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;High, average, and low&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;Chong, (&lt;xref ref-type="bibr" rid="bibr14"&gt;2019&lt;/xref&gt;), Grainger et al., (&lt;xref ref-type="bibr" rid="bibr19"&gt;2018&lt;/xref&gt;), Hendry et al. (&lt;xref ref-type="bibr" rid="bibr26"&gt;2012&lt;/xref&gt;), Kean, (&lt;xref ref-type="bibr" rid="bibr28"&gt;2012&lt;/xref&gt;), Newlyn, (&lt;xref ref-type="bibr" rid="bibr35"&gt;2013&lt;/xref&gt;)&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;High and low&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;Chong, (&lt;xref ref-type="bibr" rid="bibr15"&gt;2021&lt;/xref&gt;), Hendry &amp; Jukic, (&lt;xref ref-type="bibr" rid="bibr24"&gt;2014&lt;/xref&gt;), Yucel et al. (&lt;xref ref-type="bibr" rid="bibr50"&gt;2014&lt;/xref&gt;)&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;Unspecified range&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;Hawe &amp; Dixon, (&lt;xref ref-type="bibr" rid="bibr20"&gt;2017&lt;/xref&gt;), Newlyn, (&lt;xref ref-type="bibr" rid="bibr35"&gt;2013&lt;/xref&gt;)&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>In "Providing exemplars in the learning environment: the case for and against" Newlyn ([<reflink idref="bib35" id="ref186">35</reflink>]) cites several studies with varying methodological approaches</p> <p>When presented with peer exemplars of contrasting quality, students begin to differentiate between average quality and high-quality work, and expectations of the teacher of what constitutes high quality work (Bouwer et al., [<reflink idref="bib9" id="ref187">9</reflink>]; Chong, [<reflink idref="bib14" id="ref188">14</reflink>], [<reflink idref="bib15" id="ref189">15</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref190">19</reflink>]; Hendry &amp; Jukic, [<reflink idref="bib24" id="ref191">24</reflink>]; Kean, [<reflink idref="bib28" id="ref192">28</reflink>]). A common trend from these studies is that they all provide either two comparisons (average against high quality or high against low quality), or a range of three peer exemplars (high, average, and low). Studies listed as providing qualities of peer exemplars that are "credit and above" typically describe the quality of their exemplars as being closer together such as "pass, distinction, and high distinction" (Bell et al., [<reflink idref="bib7" id="ref193">7</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref194">25</reflink>]; Newlyn, [<reflink idref="bib35" id="ref195">35</reflink>]) or evaluations that are even closer together such as "pass, credit, distinction, and high distinction" (Knight et al., [<reflink idref="bib30" id="ref196">30</reflink>]). While it may be that the distinctions students make when exposed to a variety of quality levels was not the interest of researchers conducting the study, and thus not reported, it could also be that as items are closer on the spectrum of quality, it becomes more difficult to distinguish one from another, especially when evaluated against a criterion.</p> <p>The use of contrasting cases did not always focus on quality of tasks, at times a variety of approaches were shared as well (Bell et al., [<reflink idref="bib7" id="ref197">7</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref198">19</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref199">25</reflink>]). This too is important, as a common criticism of using peer exemplars is the suppression of creativity (Headley &amp; Pittson, [<reflink idref="bib23" id="ref200">23</reflink>]; Hendry &amp; Jukic, [<reflink idref="bib24" id="ref201">24</reflink>]; Newlyn, [<reflink idref="bib35" id="ref202">35</reflink>]). However, when presented with a variety of approaches, students expressed that seeing peer exemplars may have helped them to be <emph>more</emph> creative (Hawe et al., [<reflink idref="bib22" id="ref203">22</reflink>]; Tam, [<reflink idref="bib46" id="ref204">46</reflink>]). One student shared how the evaluation of peer exemplars enhanced their design experience:<emph>Because we saw such a range of exemplars, like comparing the one where they had the cages, to the one where they did it in violet colours and moods and stuff. They were two completely different ideas. That doesn't say to me, I'm going to pick one and copy it. It says, there are so many ideas, what's mine?</emph> (Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref205">25</reflink>], p. 343)</p> <p>Another interest of researchers was that students not only recognized what makes something 'good', but that their own performance on assignments improved (Bouwer et al., [<reflink idref="bib9" id="ref206">9</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref207">20</reflink>]; Knight et al., [<reflink idref="bib30" id="ref208">30</reflink>]; Newlyn, [<reflink idref="bib35" id="ref209">35</reflink>]). One student explained their perceived benefit in an interview:<emph>You can compare, because he (the teacher) gave us two passages. So, while you are comparing the two passages, you can consider why this one gets higher mark, or somehow, to learn the ... how to say ... to learn from this comparison, and to improve your writing skills.</emph> (Chong, [<reflink idref="bib15" id="ref210">15</reflink>], p. 21)</p> <p>While the Hawe and Dixon study ([<reflink idref="bib20" id="ref211">20</reflink>]) did not specify the range of peer exemplars provided to students, the common thread for other studies indicated an increase in performance was the lack of a low-quality example. Further, Yucel ([<reflink idref="bib50" id="ref212">50</reflink>]) found that students who participated in their program achieved significantly lower marks than those who did not. While there are other contributing factors to these findings including student buy-in, lack of teacher-led discussion, and dissatisfaction with peer review, contributions from other studies indicate that too low of peer exemplars can have a negative impact on student understanding, motivation, and application (Aitken &amp; Thompson, [<reflink idref="bib1" id="ref213">1</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref214">19</reflink>]; Hendry et al., [<reflink idref="bib26" id="ref215">26</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref216">25</reflink>]).</p> <hd id="AN0178333943-14">Theme 8: the relationship between exemplar quality and student work quality</hd> <p>When considering theories of how peer exemplars might influence student work, one theoretical underpinning was the concept of tacit knowledge acquisition or tacit knowing (Carless &amp; Boud, [<reflink idref="bib10" id="ref217">10</reflink>]; Carless &amp; Chan, [<reflink idref="bib11" id="ref218">11</reflink>]; Chong, [<reflink idref="bib14" id="ref219">14</reflink>]). That is to say that not all that we know can be verbalized, and there is much that is understood or implied without being stated, relying on the experience of observation, imitation, and practice as it is otherwise difficult to share one's thinking process (Polanyi, [<reflink idref="bib37" id="ref220">37</reflink>]).</p> <p>Another set of theories frequently used were Vygotsky's sociocultural theory (SCT) and zone of proximal development (ZPD) (Carless et al., [<reflink idref="bib12" id="ref221">12</reflink>]; Chong, [<reflink idref="bib14" id="ref222">14</reflink>], [<reflink idref="bib15" id="ref223">15</reflink>]; Tam, [<reflink idref="bib46" id="ref224">46</reflink>]; To &amp; Liu, [<reflink idref="bib48" id="ref225">48</reflink>]). Sociocultural theory posits that learning takes place through social interactions with others, and is refined with the zone of proximal development, the space between what a learner can do independently and what they can do when collaborating with others or otherwise supported (Vygotsky, [<reflink idref="bib49" id="ref226">49</reflink>]). Accordingly, these studies emphasized a procedure where students would evaluate exemplars independently, with a peer, and through dialogue with their teacher.</p> <p>The socio-cognitive work of Zimmerman ([<reflink idref="bib51" id="ref227">51</reflink>];, [<reflink idref="bib52" id="ref228">52</reflink>]) was also used in multiple studies for the model of self-regulation (Grainger et al., [<reflink idref="bib19" id="ref229">19</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref230">21</reflink>], [<reflink idref="bib22" id="ref231">22</reflink>]). The model of self-regulation is a three-phase model consisting of forethought, performance, and self-reflection, with specific components to each phase, such as task analysis and self-motivational beliefs in the phase of forethought (Zimmerman, [<reflink idref="bib51" id="ref232">51</reflink>]). The act of evaluating the work of others was often linked to the task analysis, which leads to self-motivational beliefs as students realize that they are capable and progresses to performance where they can apply the insights, and finally to self-reflection to evaluate the degree of success in their attempt. Zimmerman's theory is built on work by Bandura which was also cited for his theory of self-efficacy in one study (Dixon et al., [<reflink idref="bib16" id="ref233">16</reflink>]), aligning with the first phase of forethought with regard to self-motivational beliefs. The theory of self-efficacy suggests that the motivations of a student are driven by the confidence they have to complete a task (Bandura, [<reflink idref="bib6" id="ref234">6</reflink>]).</p> <p>While only used in one study (Chong, [<reflink idref="bib15" id="ref235">15</reflink>]), Nelson's ([<reflink idref="bib34" id="ref236">34</reflink>]) conceptual framework of evaluative judgement was used to describe how students focus on various parts of work when making evaluations. Evaluative judgement is described in terms of hard, soft, and dynamic. In hard evaluative judgement, students focus on the mechanics of an exemplar, such as punctuation and grammatical decisions. Soft evaluative judgement tends to focus on word choice and patterns of speech. However, dynamic evaluative judgement is related to creativity and how ideas are organized (Nelson, [<reflink idref="bib34" id="ref237">34</reflink>]). It is a higher level of thinking, and one that students are scaffolded to through dialogue with peers and their teacher-led discussions.</p> <p>Multiple theories were used to justify how peer exemplars might transfer the knowledge of what makes an educational artifact 'good' to influence student performance, which are supported by evidence found in the research. Several studies have drawn correlations between using the evaluation of peer exemplars as learning strategy with improvements on assessments (Bell et al., [<reflink idref="bib7" id="ref238">7</reflink>]; Carless et al., [<reflink idref="bib12" id="ref239">12</reflink>]). One student voiced their experience in an interview, <emph>" When we use exemplar, we would automatically compare our own articles with the exemplar, thinking about the differences between our articles and the exemplar, then I know what I should improve on</emph>" (Chong, [<reflink idref="bib15" id="ref240">15</reflink>], p. 21). Other researchers expressed similar observations, noting how students would use peer exemplars as a benchmark to make adjustments to their own work (Hawe et al., [<reflink idref="bib21" id="ref241">21</reflink>]; Headley &amp; Pittson, [<reflink idref="bib23" id="ref242">23</reflink>]). Additionally, in a longitudinal study, it was found that students who evaluated peer exemplars outperformed their control group peers, but even a year after the initial study the participants were achieving significantly better results (Bell et al., [<reflink idref="bib7" id="ref243">7</reflink>]; Rawson &amp; Dunlosky, [<reflink idref="bib38" id="ref244">38</reflink>]). Part of the value of evaluating peer exemplars is reflecting on other approaches, especially if students are experiencing difficulties with not knowing where to start. One graduate student explained in an interview:<emph>It puts a structure in your head so when you see the question you're like, 'Okay.' You know how to break it down instead of before [when] you'd see the question but, 'Okay I've got all this knowledge in my head and I'm just going to blurt it out whereas in no logical order.</emph> (Hendry &amp; Jukic, [<reflink idref="bib24" id="ref245">24</reflink>], p. 6)</p> <p>Other research warns that while students will eventually apply what they have seen from peer exemplars, results for some may come more slowly than others (Kean, [<reflink idref="bib28" id="ref246">28</reflink>]). This study describes how students first pushed back when asked to evaluate their peers, and after the second week the majority of students "failed to show they could identify 'quality' in others' work or produce it in their own" (Kean, [<reflink idref="bib28" id="ref247">28</reflink>], p. 89). However, after repeated cycles and dialogic feedback students were able to identify and correctly apply criteria by the fifth week. This iterative process is consistent with other research on evaluation of peers and exemplars (Bell et al., [<reflink idref="bib7" id="ref248">7</reflink>]; Carless &amp; Boud, [<reflink idref="bib10" id="ref249">10</reflink>]; Chong, [<reflink idref="bib14" id="ref250">14</reflink>]; Knight et al., [<reflink idref="bib30" id="ref251">30</reflink>]).</p> <hd id="AN0178333943-15">Theme 9: raising the bar for learning outcomes</hd> <p>There seems to be some consensus in the advantages to engaging students by evaluating high quality work. In designing studies, excellent peer exemplars were occasionally chosen "on the grounds that students might have more to learn from high-quality work" (Carless &amp; Chan, [<reflink idref="bib11" id="ref252">11</reflink>], pp. 932–933). This may influence a student's own work as "students can sharpen appreciation of how quality is manifested, enhance their capacities to make sound academic judgments, and draw appropriate inferences for actions" (Carless &amp; Boud, [<reflink idref="bib10" id="ref253">10</reflink>], p. 1321). When selecting appropriate peer exemplars and students are actively engaging, there is a belief that students will push themselves to do better (Bell et al., [<reflink idref="bib7" id="ref254">7</reflink>]; Dixon et al., [<reflink idref="bib16" id="ref255">16</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref256">21</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref257">20</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref258">25</reflink>]). 'Appropriate' peer exemplars, are thought to be ones that are high, but not too far above a student's current capabilities, as one researcher explains:<emph>If the intent is to build robust efficacy beliefs for all students, use of a range of exemplars from across the grade range is recommended. We believe using those from the mid to low A range can encourage the more capable students who like to 'rise to the challenge'.</emph> (Dixon et al., [<reflink idref="bib16" id="ref259">16</reflink>], p. 465)</p> <p>The risk of presenting too high quality work is that students not only neglect to try to raise the bar, but they may also start to question their ability to successfully complete the task entirely (Dixon et al., [<reflink idref="bib16" id="ref260">16</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref261">21</reflink>], [<reflink idref="bib22" id="ref262">22</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref263">20</reflink>]). Dixon expands on this idea as they warn:<emph>Given the fragility of academic self-efficacy, the need for a delicate balance between building confidence and overwhelming students should be taken into account. Only selecting exemplars that exemplify the highest standards of achievement such as those illustrative of an A plus piece of work may be inspiring for some students, but for others such work may undermine confidence.</emph> (2020, p. 465)</p> <p>Student interviews from several studies support the notion that high quality peer exemplars motivate students to perform at a higher level. After evaluating samples from a variety of levels, one student stated that they felt "<emph>I can do better than that</emph>" and "<emph>felt I could achieve</emph>" (Hawe &amp; Dixon, [<reflink idref="bib20" id="ref264">20</reflink>], p. 1185). From an interview in a different study, other students shared similar sentiments "<emph>when seeing the quality of work [that got] higher marks, it encourages me to try and produce the same kind of work"</emph> and "[...] you kind of want to aim higher [...]" (Hawe et al., [<reflink idref="bib21" id="ref265">21</reflink>], p. 37). This extended past just the quality of work to the creativity of work as well as a student reflected on how peer exemplars pushed them to raise the bar in their design work, sharing "<emph>It made you want to work harder to... try and be more creative"</emph> (Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref266">25</reflink>], p. 342).</p> <hd id="AN0178333943-16">Discussions and recommendations for practice</hd> <p>According to literature presented in this review, there can be many benefits to using exemplars in design-focused coursework, including building self-efficacy, improving the quality of student work, and motivating students. Importantly, research supports that the careful selection and consistent use of exemplars contribute to the overall goal of providing clarity on instructor expectations beyond what is included in the task guidelines (Sadler, [<reflink idref="bib39" id="ref267">39</reflink>], [<reflink idref="bib40" id="ref268">40</reflink>], [<reflink idref="bib41" id="ref269">41</reflink>], [<reflink idref="bib42" id="ref270">42</reflink>]). Peer exemplars can provide an aid to students when differentiating between a variety of levels of achievement through tangible examples. Through the synthesis of themes from this study, the careful selection of exemplars can promote productive classroom dialogue (themes 4, 6, 8). It can help students identify expectations (themes 1, 2, 4, 6, 7) while also serving to identify potential misconceptions leading to improved student understanding (themes 1, 4, 7). Moreover, it aids in student motivation (themes 1, 3, 8, 9) and application (themes 3, 7, 8) in engineering and design education.</p> <p>When considering how the quality of peer exemplars influence student performance, literature from this review supports that in general, high-quality exemplars are most effective. This exposes students to effective problem-solving strategies and design solutions, while inspiring students and increasing confidence in their ability to succeed. Exemplars of too low of quality may confuse students and lead to misconceptions on expectations and how to achieve success. These low-quality examples may even bring about feelings of discouragement and low self-efficacy among students. Finally, the quality of exemplars may have an impact on dialogic discussions. Higher-quality exemplars provide an opportunity for students and teachers to have a clear standard with which to evaluate work, making it easier to provide constructive feedback and identify areas of improvement. This is in contrast to lower-quality exemplars, which may make discussions challenging beyond identifying areas of improvement.</p> <p>This collection of literature also addressed the role of exemplar quality on student decision making and self-efficacy in several ways. First, high-quality exemplars can provide students with a clear understanding of what constitutes a successful design solution, increasing confidence in their ability to produce similar outcomes. Second, seeing examples of well-designed solutions can inspire students to think more creatively and take risks in their own work. On the other hand, low-quality exemplars may lead to lower expectations of what is possible, negatively affecting confidence in their abilities and motivation to engage in the design process.</p> <p>When identifying high-quality exemplars for peer evaluation, literature in this review cautioned against selecting exemplars too far above the students' current level of achievement. Using peer exemplars that are of extremely high-quality may create unrealistic expectations and a sense of intimidation for students. This may leave students feeling overwhelmed as they may not be able to reach the level of quality presented, leading to reduced self-efficacy and motivation, negatively impacting their performance in the related coursework. Additionally, providing too high-quality exemplars may not provide enough variety for students to differentiate levels of achievement for acceptable work. This could lead to frustration and potential disengagement from the course.</p> <p>Beyond the scope of initial research questions, articles in this literature review overwhelmingly supported dialogic feedback following the evaluation of peer exemplars to address student misconceptions. Teachers should facilitate discussions around exemplars to highlight strengths, areas of improvement, and opportunities for transfer to help students apply what they have seen to their work. This can help scaffold student learning, especially as it pertains to evaluation criteria and how that criterion aligns with learning objectives for the course. By doing so, teachers can help students develop a more accurate understanding of the quality of their own work and improve their decision making and self-efficacy in the course. Overall, peer exemplars can significantly impact student decision making and self-efficacy in an engineering and design course, highlighting the importance of carefully selecting and using exemplars as a pedagogical tool.</p> <hd id="AN0178333943-17">Research recommendations</hd> <p>Authors cited in this study focused their research on the fields of foreign language, teacher education, law, first-year college experience courses, and other areas outside of design or engineering curriculum. Accordingly, for those wishing to expand this systematic literature review, it is recommended to expand search terms and include other education research databases.</p> <p>Additionally, results of the systematized literature review revealed both a scarcity of research on the purposeful selection of exemplars, and a limited focus on artifacts emphasizing writing in short answer and essay format, presented as a final product. Additional research around the influence of evaluation on student learning, and the purposeful selection of sketches, images, short statements, and video as exemplars for evaluation may build on current findings. This holds numerous implications for STEM education, as it assists students in visualizing abstract design problems, recognizing diverse approaches to problem-solving, facilitating comparisons among potential solutions, and synthesizing information to innovate new designs.</p> <p>Another recommended area of future research is if the selection of exemplars is as important as how they are used in the classroom. Common to nearly all articles identified for this study is the emphasis on the role of the teacher and dialogic strategies to frame the evaluation session before starting and to debrief with a partner and in large groups after. Often, students arrived at erroneous conclusions or fail to reach consensus on the quality of an exemplar when discussing with a peer or small group, necessitating a debriefing session. While the evaluation of exemplars remained the primary focus of the articles, in seven of the nine themes there was a repeated emphasis on how much the role of the teacher influences such an activity (themes 1, 2, 3, 4, 6, 7, 8).</p> <hd id="AN0178333943-18">Conclusion</hd> <p>The purpose of this research was to investigate foundational literature supporting the selection of peer exemplars to support student achievement in design, technology, and engineering related fields. The systematized search narrowed initial search results from 467,223 to 33 articles focused on secondary and post-secondary evaluative experiences over the past 10 years. It was found that research on the effectiveness of peer exemplars is primarily focused on written student samples for large projects, and the purposeful selection of peer exemplars is limited to two to five samples for a given class. Research on peer exemplars reflects a gap in studies focused on the variety of sample quality in peer exemplars for evaluation, especially in STEM-related fields.</p> <p>In analyzing the influence of peer exemplars on the impact on student's learning, nine themes were identified to address the research questions, specifically, (<reflink idref="bib1" id="ref271">1</reflink>) How the quality of exemplars influences student performance, (<reflink idref="bib2" id="ref272">2</reflink>) How comparing drastically different examples influences decision making, and (<reflink idref="bib3" id="ref273">3</reflink>) The effect of providing students with lower quality work. In addressing student performance, it was found that quality exemplars assist students with clarity and understanding of the task and what makes something 'good', motivate students in their own tasks, and facilitate knowledge transfer to enhance performance.</p> <p>A variety of ways that instructors and teachers use peer exemplars in the classroom were discovered such as to showcase a wide range of student work and using drastically different examples. It was found that by providing a spectrum of samples, students being able to differentiate between average and high-quality work (Bouwer et al., [<reflink idref="bib9" id="ref274">9</reflink>]; Chong, [<reflink idref="bib14" id="ref275">14</reflink>], [<reflink idref="bib15" id="ref276">15</reflink>]; Grainger et al., [<reflink idref="bib19" id="ref277">19</reflink>]; Hendry &amp; Jukic, [<reflink idref="bib24" id="ref278">24</reflink>]; Kean, [<reflink idref="bib28" id="ref279">28</reflink>]). Tangential to this, studies found that multiple peer exemplars counteract claims that providing students with examples hinders creativity. Findings indicate that when multiple peer exemplars are used it instead acts to inspire students into more divergent ideas when problem solving (Hawe et al., [<reflink idref="bib22" id="ref280">22</reflink>]; Hendry &amp; Tomitsch, [<reflink idref="bib25" id="ref281">25</reflink>]; Tam, [<reflink idref="bib46" id="ref282">46</reflink>]).</p> <p>Providing lower quality examples likewise helped students to identify challenges and misconceptions by learning from the mistakes of others. However, counterintuitive to teacher expectations, effects of using peer exemplars that are too high or too low of quality were shown to demotivate students and have a negative impact on understanding the assessment (Dixon et al., [<reflink idref="bib16" id="ref283">16</reflink>]; Hawe et al., [<reflink idref="bib21" id="ref284">21</reflink>], [<reflink idref="bib22" id="ref285">22</reflink>]; Hawe &amp; Dixon, [<reflink idref="bib20" id="ref286">20</reflink>]). As many of the studies cited contained qualitative aspects, multiple quotes from student reflections on the use of peer exemplars were introduced to strengthen the conclusions drawn from this paper and the claims made by researchers.</p> <hd id="AN0178333943-19">Limitations</hd> <p>This systematic review has some methodological limitations that need to be acknowledged. Firstly, as the search and coding were conducted by the first author, there is potential for bias in the search and selection process. While efforts were made to conduct a comprehensive search using multiple databases and search terms, the inclusion and exclusion criteria were set by a single author. This may have led to the exclusion of relevant studies or the inclusion of studies that do not meet the criteria.</p> <p>An additional limitation revolves around the possibility of overlooking certain studies. Despite the efforts to conduct a thorough search, it is possible that pertinent studies might have been missed, particularly those that were not present in the searched databases, accidentally screened out, or not published in English. This may have resulted in a skewed sample of studies.</p> <p>The act of synthesizing existing literature into thematic patterns inherently carries limitations. It is possible that the nuanced details of how exemplars are applied in various scenarios, as well as the true impact of student learning, may not have been fully captured. To mitigate this, several steps were taken. A diverse sample selection was employed, broadening the spectrum of examples and their patterns of use. An in-depth analysis was conducted to identify recurring patterns or trends related to the use of examples in different contexts. Quotations and citations were used whenever possible to provide contextual examples to add credibility and depth to the analysis.</p> <p>Finally, the generalizability of the findings may be limited due to the specific inclusion and exclusion criteria used in this study. The themes identified may not be representative of all studies on the topic, and the findings may not be applicable to other contexts or populations.</p> <hd id="AN0178333943-20">Declarations</hd> <p></p> <hd id="AN0178333943-21">Conflict of interest</hd> <p>On behalf of all authors, the corresponding author states that there is no conflict of interest.</p> <hd id="AN0178333943-22">Competing Interests</hd> <p>The authors have no competing interests to declare that are relevant to the content of this article.</p> <hd id="AN0178333943-23">Appendix 1</hd> <p>Initial Search Query (Conducted 7/20/2022).</p> <p></p> <p> <ephtml> &lt;table frame="hsides" rules="groups"&gt;&lt;thead&gt;&lt;tr&gt;&lt;th align="left"&gt;&lt;p&gt;#&lt;/p&gt;&lt;/th&gt;&lt;th align="left"&gt;&lt;p&gt;Query&lt;/p&gt;&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S1&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"exemplars" OR "samples" OR "worked examples" OR "examples" OR "model answer" OR "Gallery walk" OR "Design critique" OR "Peer critique" OR "Formative peer assessment" OR "Math thinking" OR "Learning by Evaluation" OR "LbE" OR "Peer learning"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S2&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"self-efficacy" OR "self efficacy" OR "self-confidence" OR "self confidence" OR "confidence" OR "efficacy" OR "concept of quality" OR "evaluative knowledge"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S3&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"secondary school" OR "high school" OR "secondary education" OR "postsecondary education" OR "college" OR "university"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S4&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"selection" OR "quality" OR "choice"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <hd id="AN0178333943-24">Appendix 2</hd> <p>Additional Search Query (Conducted 7/22/2022).</p> <p></p> <p> <ephtml> &lt;table frame="hsides" rules="groups"&gt;&lt;thead&gt;&lt;tr&gt;&lt;th align="left"&gt;&lt;p&gt;#&lt;/p&gt;&lt;/th&gt;&lt;th align="left"&gt;&lt;p&gt;Query&lt;/p&gt;&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S1&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"assessment" OR "marking" OR "criteria" OR "standards"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S2&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"discussion" OR "task" OR "understanding"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S3&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"exemplars"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td align="left"&gt;&lt;p&gt;S4&lt;/p&gt;&lt;/td&gt;&lt;td align="left"&gt;&lt;p&gt;"education"&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <hd id="AN0178333943-25">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0178333943-26"> <title> References </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Aitken A, Thompson DG. Using software to engage design students in academic writing. International Journal of Technology and Design Education. 2018; 28; 3: 885-898. 10.1007/s10798-017-9413-4</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Amulya J. What is reflective practice?. 2004; Center for Reflective Community Practice at the Massachusetts Institute of Technology</bibtext> </blist> <blist> <bibl id="bib3" idref="ref3" type="bt">3</bibl> <bibtext> Anderson N. What's right about wrong answers: Learning from math mistakes, grades 4–5. 2017; Stenhouse Publishers</bibtext> </blist> <blist> <bibl id="bib4" idref="ref4" type="bt">4</bibl> <bibtext> Anderson JB, Freiberg HJ. Using self-assessment as a reflective tool to enhance the student teaching experience. Teacher Education Quarterly. 1995; 22; 1: 77-91</bibtext> </blist> <blist> <bibl id="bib5" idref="ref5" type="bt">5</bibl> <bibtext> Andrade HG. Using rubrics to promote thinking and learning. Educational Leadership. 2000; 57; 5: 13-19</bibtext> </blist> <blist> <bibl id="bib6" idref="ref6" type="bt">6</bibl> <bibtext> Bandura A. Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review. 1977; 84; 2: 191-215. 10.1037/0033-295X.84.2.191</bibtext> </blist> <blist> <bibl id="bib7" idref="ref7" type="bt">7</bibl> <bibtext> Bell A, Mladenovic R, Price M. Students' perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars. Assessment &amp; Evaluation in Higher Education. 2013; 38; 7: 769-788. 10.1080/02602938.2012.714738</bibtext> </blist> <blist> <bibl id="bib8" idref="ref8" type="bt">8</bibl> <bibtext> Borrego M, Foster MJ, Froyd JE. Systematic literature reviews in engineering education and other developing Interdisciplinary fields. Journal of Engineering Education. 2014; 103; 1: 45-76. 10.1002/jee.20038</bibtext> </blist> <blist> <bibl id="bib9" idref="ref9" type="bt">9</bibl> <bibtext> Bouwer, R, Lesterhuis, M, Bonne, P, &amp; De Maeyer, S. (2018). Applying criteria to examples or learning by comparison: Effects on students' evaluative judgment and performance in writing. In Frontiers in education (Vol. 3, p. 86). Frontiers Media SA. https://doi.org/10.3389/feduc.2018.00086</bibtext> </blist> <blist> <bibtext> Carless D, Boud D. The development of student feedback literacy: Enabling uptake of feedback. Assessment &amp; Evaluation in Higher Education. 2018; 43; 8: 1315-1325. 10.1080/02602938.2018.1463354</bibtext> </blist> <blist> <bibtext> Carless D, Chan KKH. Managing dialogic use of exemplars. Assessment &amp; Evaluation in Higher Education. 2017; 42; 6: 930-941. 10.1080/02602938.2016.1211246</bibtext> </blist> <blist> <bibtext> Carless, D, Chan, K. K. H, To, J, Lo, M, &amp; Barrett, E. (2018). Developing students' capacities for evaluative judgement through analysing exemplars. Developing Evaluative Judgement in Higher Education: Assessment for Knowing and Producing Quality Work, 108–116.</bibtext> </blist> <blist> <bibtext> Charmaz K. Constructing grounded theory: A practical guide through qualitative analysis. 2006; SAGE</bibtext> </blist> <blist> <bibtext> Chong SW. The use of exemplars in English writing classrooms: From theory to practice. Assessment &amp; Evaluation in Higher Education. 2019; 44; 5: 748-763. 10.1080/02602938.2018.1535051</bibtext> </blist> <blist> <bibtext> Chong SW. University students' perceptions towards using exemplars dialogically to develop evaluative judgement: The case of a high-stakes language test. Asian-Pacific Journal of Second and Foreign Language Education. 2021; 6; 1: 12. 10.1186/s40862-021-00115-4</bibtext> </blist> <blist> <bibtext> Dixon H, Hawe E, Hamilton R. The case for using exemplars to develop academic self-efficacy. Assessment &amp; Evaluation in Higher Education. 2020; 45; 3: 460-471. 10.1080/02602938.2019.1666084</bibtext> </blist> <blist> <bibtext> Glaser BG. The constant comparative method of qualitative analysis. Social Problems. 1965; 12; 4: 436-445. 10.2307/798843</bibtext> </blist> <blist> <bibtext> Gonczi, A. L, Bergman, B. G, Huntoon, J, Allen, R, McIntyre, B, Turner, S, Davis, J, &amp; Handler, R. (2017). Decision matrices: Tools to enhance middle school engineering instruction. Science Activities, 54(1), 8–17. https://doi.org/10.1080/00368121.2016.1264922</bibtext> </blist> <blist> <bibtext> Grainger PR, Heck D, Carey MD. Are assessment exemplars perceived to support self-regulated learning in teacher education?. Frontiers in Education. 2018; 3: 60. 10.2307/798843</bibtext> </blist> <blist> <bibtext> Hawe E, Dixon H. Assessment for learning: A catalyst for student self-regulation. Assessment &amp; Evaluation in Higher Education. 2017; 42; 8: 1181-1192. 10.1080/02602938.2016.1236360</bibtext> </blist> <blist> <bibtext> Hawe E, Lightfoot U, Dixon H. First-year students working with exemplars: Promoting self-efficacy, self-monitoring and self-regulation. Journal of Further and Higher Education. 2019; 43; 1: 30-44. 10.1080/0309877X.2017.1349894</bibtext> </blist> <blist> <bibtext> Hawe, E, Dixon, H, &amp; Hamilton, R. (2021). Why and how educators use exemplars. Journal of University Teaching and Learning Practice. https://doi.org/10.53761/1.18.3.10</bibtext> </blist> <blist> <bibtext> Headley J, Pittson H. Bottom up! Establishing a University Wide community of practice for exemplars and exploration of its affect in enhancing the quality of student assessment. Practitioner Research in Higher Education. 2020; 13; 1: 18-26</bibtext> </blist> <blist> <bibtext> Hendry GD, Jukic K. Learning about the quality of work that teachers expect: Students' perceptions of exemplar marking versus teacher explanation. Journal of University Teaching and Learning Practice. 2014; 11; 2: 5. 10.53761/1.11.2.5</bibtext> </blist> <blist> <bibtext> Hendry GD, Tomitsch M. Implementing an exemplar-based approach in an interaction design subject: Enhancing students' awareness of the need to be creative. International Journal of Technology and Design Education. 2014; 24; 3: 337-348. 10.1007/s10798-013-9256-6</bibtext> </blist> <blist> <bibtext> Hendry GD, Armstrong S, Bromberger N. Implementing standards-based assessment effectively: Incorporating discussion of exemplars into classroom teaching. Assessment &amp; Evaluation in Higher Education. 2012; 37; 2: 149-161. 10.1080/02602938.2010.515014</bibtext> </blist> <blist> <bibtext> Johnston E, Burleigh C, Rasmusson X, Turner P, Valentine D, Bailey L. Multimedia open educational resource materials for teaching-online diversity and leadership: Aligning bloom's taxonomy and studio habits of mind. Higher Learning Research Communications. 2021; 11; 2: 40-67. 10.18870/hlrc.v11i2.1234</bibtext> </blist> <blist> <bibtext> Kean J. Show AND tell: Using peer assessment and exemplars to help students understand quality in assessment. Practitioner Research in Higher Education. 2012; 6; 2: 2</bibtext> </blist> <blist> <bibtext> Kearney SP, Perkins T. Engaging students through assessment: The Success and limitations of the ASPAL (authentic self and peer assessment for learning) Model. Journal of University Teaching and Learning Practice. 2014; 11; 3: 2. 10.53761/1.11.3.2</bibtext> </blist> <blist> <bibtext> Knight S, Leigh A, Davila YC, Martin LJ, Krix DW. Calibrating assessment literacy through benchmarking tasks. Assessment &amp; Evaluation in Higher Education. 2019; 44; 8: 1121-1132. 10.1080/02602938.2019.1570483</bibtext> </blist> <blist> <bibtext> Kulkarni, C. E, Socher, R, Bernstein, M. S, &amp; Klemmer, S. R. (2014). Scaling short-answer grading by combining peer assessment with algorithmic scoring. In Proceedings of the First ACM conference on learning @ scale conference (pp. 99–108). https://doi.org/10.1145/2556325.2566238</bibtext> </blist> <blist> <bibtext> McMillan JH, Hearn J. Student self-assessment: The key to stronger student motivation and higher achievement. Educational Horizons. 2008; 87; 1: 40-49</bibtext> </blist> <blist> <bibtext> Merriam SB, Tisdell EJ. Qualitative research. 20154; Wiley</bibtext> </blist> <blist> <bibtext> Nelson R. Barriers to the cultivation of evaluative judgement: A critical and historical perspective. Developing evaluative judgement in Higher Education. 2018; Routledge. 10.4324/9781315109251-6</bibtext> </blist> <blist> <bibtext> Newlyn D. Providing exemplars in the learning environment: The case for and against. Universal Journal of Educational Research. 2013; 1; 1: 26-32. 10.13189/ujer.2013.010104</bibtext> </blist> <blist> <bibtext> Petticrew M, Roberts H. Systematic reviews in the social sciences: A practical guide. 2008; Wiley</bibtext> </blist> <blist> <bibtext> Polanyi, M. (1997). The Tacit Dimension. In Knowledge in Organizations. Routledge.</bibtext> </blist> <blist> <bibtext> Rawson KA, Dunlosky J. How effective is example generation for learning declarative concepts?. Educational Psychology Review. 2016; 28; 3: 649-672. 10.1007/s10648-016-9377-zeric</bibtext> </blist> <blist> <bibtext> Sadler DR. Specifying and promulgating achievement standards. Oxford Review of Education. 1987; 13; 2: 191-209. 10.1080/0305498870130207</bibtext> </blist> <blist> <bibtext> Sadler DR. Formative assessment and the design of instructional systems. Instructional Science. 1989; 18; 2: 119-144. 10.1007/BF00117714</bibtext> </blist> <blist> <bibtext> Sadler, D. R. (2002). Ah!... so that's 'quality'. In Assessment. Routledge.</bibtext> </blist> <blist> <bibtext> Sadler DR. Perils in the meticulous specification of goals and assessment criteria. Assessment in Education: Principles Policy &amp; Practice. 2007; 14; 3: 387-392. 10.1080/09695940701592097</bibtext> </blist> <blist> <bibtext> Saldaña J. The Coding Manual for Qualitative Researchers. 20132; SAGE Publications Inc</bibtext> </blist> <blist> <bibtext> Schwartz, D. (2018). Designing contrasting cases for inductive learning. Human Sciences and Technologies Advanced Research Institiute. https://hstar.stanford.edu/research/designing-contrasting-cases-for-inductive-learning/</bibtext> </blist> <blist> <bibtext> Seeley, C. L. (2014). Chapter 8: Oops! The value of mistakes, failures, and wrong turns. In Smarter than we think: More messages about Math, Teaching, and learning in the 21st Century: A Resource for teachers, leaders, Policy makers, and families. Math Solutions.</bibtext> </blist> <blist> <bibtext> Tam ACF. Undergraduate students' perceptions of and responses to exemplar-based dialogic feedback. Assessment &amp; Evaluation in Higher Education. 2021; 46; 2: 269-285. 10.1080/02602938.2020.1772957</bibtext> </blist> <blist> <bibtext> Teaching Channel. (2014). My Favorite No (Uncut). Teaching Channel. https://learn.teachingchannel.com/video/my-favorite-no-complete-lesson</bibtext> </blist> <blist> <bibtext> To J, Liu Y. Using peer and teacher-student exemplar dialogues to unpack assessment standards: Challenges and possibilities. Assessment &amp; Evaluation in Higher Education. 2018; 43; 3: 449-460. 10.1080/02602938.2017.1356907</bibtext> </blist> <blist> <bibtext> Vygotsky LS. The collected works of L.S. Vygotsky: The fundamentals of defectology. 1987; Springer Science and Business Media</bibtext> </blist> <blist> <bibtext> Yucel R, Bird FL, Young J, Blanksby T. The road to self-assessment: Exemplar marking before peer review develops first-year students' capacity to judge the quality of a scientific report. Assessment &amp; Evaluation in Higher Education. 2014; 39; 8: 971-986. 10.1080/02602938.2014.880400</bibtext> </blist> <blist> <bibtext> Zimmerman BJBoekaerts M, Pintrich PR, Zeidner M. Chapter 2 - Attaining self-regulation: A social cognitive perspective. Handbook of Self-Regulation. 2000; Academic Press: 13-39. 10.1016/B978-012109890-2/50031-7</bibtext> </blist> <blist> <bibtext> Zimmerman BJ, Schunk DH. Motivation: An essential dimension of self-regulated learning. Motivation and self-regulated learning. 2007; Routledge</bibtext> </blist> </ref> <aug> <p>By Scott Thorne; Nathan Mentzer; Scott Bartholomew and Greg J. Strimel</p> <p>Reported by Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib32" firstref="ref12"></nolink> <nolink nlid="nl2" bibid="bib39" firstref="ref14"></nolink> <nolink nlid="nl3" bibid="bib11" firstref="ref17"></nolink> <nolink nlid="nl4" bibid="bib30" firstref="ref18"></nolink> <nolink nlid="nl5" bibid="bib31" firstref="ref19"></nolink> <nolink nlid="nl6" bibid="bib36" firstref="ref21"></nolink> <nolink nlid="nl7" bibid="bib22" firstref="ref23"></nolink> <nolink nlid="nl8" bibid="bib12" firstref="ref24"></nolink> <nolink nlid="nl9" bibid="bib21" firstref="ref26"></nolink> <nolink nlid="nl10" bibid="bib35" firstref="ref27"></nolink> <nolink nlid="nl11" bibid="bib16" firstref="ref28"></nolink> <nolink nlid="nl12" bibid="bib18" firstref="ref29"></nolink> <nolink nlid="nl13" bibid="bib20" firstref="ref30"></nolink> <nolink nlid="nl14" bibid="bib23" firstref="ref31"></nolink> <nolink nlid="nl15" bibid="bib28" firstref="ref32"></nolink> <nolink nlid="nl16" bibid="bib38" firstref="ref34"></nolink> <nolink nlid="nl17" bibid="bib33" firstref="ref35"></nolink> <nolink nlid="nl18" bibid="bib17" firstref="ref36"></nolink> <nolink nlid="nl19" bibid="bib13" firstref="ref37"></nolink> <nolink nlid="nl20" bibid="bib43" firstref="ref38"></nolink> <nolink nlid="nl21" bibid="bib41" firstref="ref39"></nolink> <nolink nlid="nl22" bibid="bib15" firstref="ref40"></nolink> <nolink nlid="nl23" bibid="bib19" firstref="ref41"></nolink> <nolink nlid="nl24" bibid="bib26" firstref="ref44"></nolink> <nolink nlid="nl25" bibid="bib50" firstref="ref45"></nolink> <nolink nlid="nl26" bibid="bib51" firstref="ref50"></nolink> <nolink nlid="nl27" bibid="bib46" firstref="ref54"></nolink> <nolink nlid="nl28" bibid="bib29" firstref="ref59"></nolink> <nolink nlid="nl29" bibid="bib40" firstref="ref62"></nolink> <nolink nlid="nl30" bibid="bib42" firstref="ref64"></nolink> <nolink nlid="nl31" bibid="bib25" firstref="ref65"></nolink> <nolink nlid="nl32" bibid="bib24" firstref="ref73"></nolink> <nolink nlid="nl33" bibid="bib10" firstref="ref105"></nolink> <nolink nlid="nl34" bibid="bib14" firstref="ref107"></nolink> <nolink nlid="nl35" bibid="bib27" firstref="ref117"></nolink> <nolink nlid="nl36" bibid="bib48" firstref="ref120"></nolink> <nolink nlid="nl37" bibid="bib52" firstref="ref124"></nolink> <nolink nlid="nl38" bibid="bib47" firstref="ref164"></nolink> <nolink nlid="nl39" bibid="bib45" firstref="ref165"></nolink> <nolink nlid="nl40" bibid="bib44" firstref="ref182"></nolink> <nolink nlid="nl41" bibid="bib37" firstref="ref220"></nolink> <nolink nlid="nl42" bibid="bib49" firstref="ref226"></nolink> <nolink nlid="nl43" bibid="bib34" firstref="ref236"></nolink> |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1429986 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: A Systematic Literature Review of Student Evaluation of Peer Exemplars and Implications for Design, Technology, and Engineering Learning – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Scott+Thorne%22">Scott Thorne</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-3742-8903">0000-0002-3742-8903</externalLink>)<br /><searchLink fieldCode="AR" term="%22Nathan+Mentzer%22">Nathan Mentzer</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-9788-7901">0000-0001-9788-7901</externalLink>)<br /><searchLink fieldCode="AR" term="%22Scott+Bartholomew%22">Scott Bartholomew</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-1680-2433">0000-0002-1680-2433</externalLink>)<br /><searchLink fieldCode="AR" term="%22Greg+J%2E+Strimel%22">Greg J. Strimel</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-4847-4526">0000-0002-4847-4526</externalLink>) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Technology+and+Design+Education%22"><i>International Journal of Technology and Design Education</i></searchLink>. 2024 34(4):1441-1462. – Name: Avail Label: Availability Group: Avail Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/ – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 22 – Name: DatePubCY Label: Publication Date Group: Date Data: 2024 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Information Analyses – Name: Audience Label: Education Level Group: Audnce Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink> – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Design%22">Design</searchLink><br /><searchLink fieldCode="DE" term="%22Engineering+Education%22">Engineering Education</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Motivation%22">Learning Motivation</searchLink><br /><searchLink fieldCode="DE" term="%22Self+Efficacy%22">Self Efficacy</searchLink><br /><searchLink fieldCode="DE" term="%22Barriers%22">Barriers</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Processes%22">Learning Processes</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Education%22">Technology Education</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Reports%22">Research Reports</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Correlation%22">Correlation</searchLink><br /><searchLink fieldCode="DE" term="%22Outcomes+of+Education%22">Outcomes of Education</searchLink><br /><searchLink fieldCode="DE" term="%22Peer+Evaluation%22">Peer Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Misconceptions%22">Misconceptions</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Attitudes%22">Student Attitudes</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1007/s10798-023-09874-2 – Name: ISSN Label: ISSN Group: ISSN Data: 0957-7572<br />1573-1804 – Name: Abstract Label: Abstract Group: Ab Data: In design courses, reviewing how others have solved design problems or completed projects is common practice and often encouraged by educators. Using student work as examples can provide context for assessment criteria and help students approach new design problems. While studies have explored the use of exemplars in various disciplines, little research has focused on which exemplars to use (e.g., high-quality, low-quality) in design, technology, and engineering fields. To address this gap, researchers conducted a literature review of 33 articles on exemplar use in secondary and post-secondary education. The analysis revealed nine themes related to exemplar use and their impact on student learning, including (1) Clarity of instruction, (2) Learner focus, (3) Motivation for learning, (4) Student reflection on learning, (5) Building student self-efficacy, (6) Identifying instructional challenges, (7) Providing contrasting cases, (8) The relationship between exemplar quality and student work quality, and (9) Raising the bar for learning outcomes. Findings suggest that simply providing an exemplar is not enough and that the selection of an exemplar can have positive or negative impacts on student motivation, understanding, and application. Carefully selecting exemplars and engaging in dialogue with students can help them identify expectations, recognize quality work, and identify potential misconceptions. These findings have implications for those involved in design, technology, and engineering education. Educators can use these findings to guide their selection of exemplars and engage students in meaningful dialogue to aid their learning. Researchers can also use these findings to further investigate the use of exemplars in these fields. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2024 – Name: AN Label: Accession Number Group: ID Data: EJ1429986 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1429986 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s10798-023-09874-2 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 22 StartPage: 1441 Subjects: – SubjectFull: Design Type: general – SubjectFull: Engineering Education Type: general – SubjectFull: Learning Motivation Type: general – SubjectFull: Self Efficacy Type: general – SubjectFull: Barriers Type: general – SubjectFull: Learning Processes Type: general – SubjectFull: Technology Education Type: general – SubjectFull: Research Reports Type: general – SubjectFull: Secondary School Students Type: general – SubjectFull: College Students Type: general – SubjectFull: Correlation Type: general – SubjectFull: Outcomes of Education Type: general – SubjectFull: Peer Evaluation Type: general – SubjectFull: Misconceptions Type: general – SubjectFull: Student Attitudes Type: general Titles: – TitleFull: A Systematic Literature Review of Student Evaluation of Peer Exemplars and Implications for Design, Technology, and Engineering Learning Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Scott Thorne – PersonEntity: Name: NameFull: Nathan Mentzer – PersonEntity: Name: NameFull: Scott Bartholomew – PersonEntity: Name: NameFull: Greg J. Strimel IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 09 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 0957-7572 – Type: issn-electronic Value: 1573-1804 Numbering: – Type: volume Value: 34 – Type: issue Value: 4 Titles: – TitleFull: International Journal of Technology and Design Education Type: main |
| ResultId | 1 |
