View in EDS HTML Full Text PDF Full Text

Learning by Evaluating (LbE): Promoting Meaningful Reasoning in the Context of Engineering Design Thinking Using Adaptive Comparative Judgment (ACJ)

Saved in:
Bibliographic Details
Title: Learning by Evaluating (LbE): Promoting Meaningful Reasoning in the Context of Engineering Design Thinking Using Adaptive Comparative Judgment (ACJ)
Language: English
Authors: Nathan Mentzer (ORCID 0000-0001-9788-7901), Wonki Lee (ORCID 0000-0002-0007-1023), Andrew Jackson (ORCID 0000-0003-2882-3052), Scott Bartholomew (ORCID 0000-0002-1680-2433)
Source: International Journal of Technology and Design Education. 2024 34(3):1145-1169.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 25
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Education Level: High Schools
Secondary Education
Descriptors: Evaluative Thinking, Design, Engineering Education, Evaluation Methods, Decision Making, High School Students, Abstract Reasoning, Content Analysis, Natural Language Processing, Thinking Skills, Word Frequency, Self Concept, Needs Assessment, Concept Formation
DOI: 10.1007/s10798-023-09853-7
ISSN: 0957-7572
1573-1804
Abstract: Adaptive comparative judgment (ACJ) has been widely used to evaluate classroom artifacts with reliability and validity. In the ACJ experience we examined, students were provided a pair of images related to backpack design. For each pair, students were required to select which image could help them ideate better. Then, they were prompted to provide a justification for their decision. Data were from 15 high school students taking engineering design courses. The current study investigated how students' reasoning differed based on selection. Researchers analyzed the comments in two ways: (1) computer-aided quantitative content analysis and (2) qualitative content analysis. In the first analysis, we performed sentiment analysis and word frequency analysis using natural language processing. Based on the findings, we explored how the design thinking process was embedded in student reasoning, and if the reasoning varied depending on the claim. Results from sentiment analysis showed that students tend to reveal more strong positive sentiment with short comments when providing reasoning for the selected design. In contrast, when providing reasoning for those items not chosen, results showed a weaker negative sentiment with more detailed reasons. Findings from word frequency analysis showed that students valued the function of design as well as the user perspective, specifically, convenience. Additionally, students took aesthetic features of each design into consideration when identifying the best of the two pairs. Within the engineering design thinking context, we found students empathize by identifying themselves as users, define user's needs, and ideate products from provided examples.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1426801
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwH4KHFZus1EnlDC-FAKNa0BAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJcsn2TdLghLvhhDUgIBEICBm7T75VQoFAfUsk0jd6V_Zcu8V126DFPtNEwLBsoPcW4cuVj13KZV4YRIgQkcZGISIgNseE8EKReT_tB_evTb3iQRlmkIwTURX-ppj6go1H2-ENr2XK02IdBckMFm-ER-gyin9fPq1xbDpne9fz2GYcK2w4PcabAHEuzJZcXN7JgQ5fDmh1KzWesp7oL_oPH2iB5NQea40L6OHGVR
Text:
  Availability: 1
  Value: <anid>AN0177623651;ogv01jul.24;2024Jun05.04:45;v2.2.500</anid> <title id="AN0177623651-1">Learning by Evaluating (LbE): promoting meaningful reasoning in the context of engineering design thinking using Adaptive Comparative Judgment (ACJ) </title> <p>Adaptive comparative judgment (ACJ) has been widely used to evaluate classroom artifacts with reliability and validity. In the ACJ experience we examined, students were provided a pair of images related to backpack design. For each pair, students were required to select which image could help them ideate better. Then, they were prompted to provide a justification for their decision. Data were from 15 high school students taking engineering design courses. The current study investigated how students' reasoning differed based on selection. Researchers analyzed the comments in two ways: (<reflink idref="bib1" id="ref1">1</reflink>) computer-aided quantitative content analysis and (<reflink idref="bib2" id="ref2">2</reflink>) qualitative content analysis. In the first analysis, we performed sentiment analysis and word frequency analysis using natural language processing. Based on the findings, we explored how the design thinking process was embedded in student reasoning, and if the reasoning varied depending on the claim. Results from sentiment analysis showed that students tend to reveal more strong positive sentiment with short comments when providing reasoning for the selected design. In contrast, when providing reasoning for those items not chosen, results showed a weaker negative sentiment with more detailed reasons. Findings from word frequency analysis showed that students valued the function of design as well as the user perspective, specifically, convenience. Additionally, students took aesthetic features of each design into consideration when identifying the best of the two pairs. Within the engineering design thinking context, we found students empathize by identifying themselves as users, define user's needs, and ideate products from provided examples.</p> <p>Keywords: Adaptive Comparative Judgment (ACJ); Learning by Evaluating (LbE); Claim, Evidence, Reasoning (CER) model; Design thinking; Technology education</p> <p>Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</p> <hd id="AN0177623651-2">Introduction</hd> <p>Recent studies in STEM education highlighted an emphasis on peer critique and feedback, as both formative assessment and learning strategy. Adaptive comparative judgment (ACJ; Pollitt & Whitehouse, [<reflink idref="bib46" id="ref3">46</reflink>]; Pollitt, [<reflink idref="bib47" id="ref4">47</reflink>], [<reflink idref="bib48" id="ref5">48</reflink>]) was developed based on the law of comparative judgment (Thurstone, [<reflink idref="bib61" id="ref6">61</reflink>]), which allows judges to compare items which are then ranked based on the results. ACJ is a modification on traditional Comparative Judgment (CJ) which exploits the power of adaptivity to expedite the comparison process (Pollitt, [<reflink idref="bib48" id="ref7">48</reflink>]). Other work has also demonstrated that ACJ can be used as a formative assessment, which challenges the evaluator to discern qualitative differences between artifacts (Pollitt, [<reflink idref="bib48" id="ref8">48</reflink>]) during the learning process. In the context of ACJ, judges are provided pairs of work, then, they make decisions about which one is better. From multiple comparisons, a measurement scale provides relative quality of the artifacts. Additionally, ACJ also can be incorporated in educational contexts as a learning mechanism. Along with the written justification for the decisions, studies found that ACJ can be implemented as a meaningful assessment and feedback tool, in which students can improve learning and achievement (Bartholomew & Jones, [<reflink idref="bib5" id="ref9">5</reflink>]; Bartholomew et al., [<reflink idref="bib9" id="ref10">9</reflink>]). In this case, our research team has named the priming process <emph>Learning by Evaluating</emph> (LbE) which stimulates and promotes meaningful learning for students (Bartholomew, [<reflink idref="bib3" id="ref11">3</reflink>]; Bartholomew & Yauney, [<reflink idref="bib4" id="ref12">4</reflink>]; Bartholomew et al., [<reflink idref="bib6" id="ref13">6</reflink>]). Beginning student efforts through LbE builds foundational understanding and has been shown to produce meaningful learning for students (Bartholomew, [<reflink idref="bib3" id="ref14">3</reflink>]; Bartholomew & Yauney, [<reflink idref="bib4" id="ref15">4</reflink>]; Bartholomew et al., [<reflink idref="bib6" id="ref16">6</reflink>]).</p> <p>For learning purposes, LbE has shown promise in facilitating student learning and growth (Bartholomew et al., [<reflink idref="bib7" id="ref17">7</reflink>], [<reflink idref="bib8" id="ref18">8</reflink>], [<reflink idref="bib9" id="ref19">9</reflink>]; Canty et al., [<reflink idref="bib12" id="ref20">12</reflink>]; Sluijsmans et al., [<reflink idref="bib56" id="ref21">56</reflink>]; Strimel et al., [<reflink idref="bib59" id="ref22">59</reflink>]). Additional work has shown that students' decision-making during the peer assessment process promotes critical thinking (Jackson et al., [<reflink idref="bib32" id="ref23">32</reflink>]; Sluijsmans et al., [<reflink idref="bib56" id="ref24">56</reflink>], [<reflink idref="bib55" id="ref25">55</reflink>]) and an analytical approach to design (Nicol et al., [<reflink idref="bib42" id="ref26">42</reflink>]). In multiple acts of making judgment, students think about not only the provided example but also their own work. Thus, LbE is beneficial in that it promotes active engagement in critical thinking, applying criteria, reflecting, and learning transfer derived from this process (Nicol et al., [<reflink idref="bib42" id="ref27">42</reflink>]). Especially, regular opportunities for students to participate in making feedback significantly enhance learning outcomes (Sadler, [<reflink idref="bib51" id="ref28">51</reflink>]). Additionally, Cowan ([<reflink idref="bib14" id="ref29">14</reflink>]) asserted the importance of systemic, objective, and sound criteria-based decision and reasoning for making effective evaluative judgment. Therefore, the learning benefits of LbE may be increased as students engage in providing rationales for their claims.[<reflink idref="bib1" id="ref30">1</reflink>]</p> <p>These findings around LbE are well aligned with the <emph>Claim, Evidence and Reasoning</emph> (CER) model developed by McNeill et al., ([<reflink idref="bib36" id="ref31">36</reflink>]) who position CER as a scientific argument model. A <emph>claim</emph> provides an answer to the question or problem. <emph>Evidence</emph> is data that supports the claim (e.g., measurements, observation, predetermined criteria), and <emph>reasoning</emph> explains why and how the evidence supports the claim using principles. In the LbE context, students engage in making a claim when they are tasked with selecting a better example from a pair of options. Student explanations are a rationale for their decision which may include evidence and their reasoning with criteria. The current study investigated the extent to which this reasoning differed by selecting and not selecting one of the pair of items displayed in each judgment, and how the reasoning differed in terms of word sentiment, usage, word frequency. Further, we compared the contents of the reasoning qualitatively in terms of design thinking. More specifically, we explored how students' reasoning incorporated the learned principles of the design thinking process and if there was any difference based on their claim (i.e., reasoning for the selected option vs. not selected option).</p> <hd id="AN0177623651-3">Previous studies</hd> <p>Three major strands of studies are introduced in this section. First, the origin and the notion of ACJ will be introduced. Then, the incorporation of ACJ and its learning benefits will be presented. Under this strand, LbE—which designates a specific application of ACJ and learning benefits coming from taking part in ACJ—is also discussed. Finally, we elaborate on the CER model and its implication in the STEM education context.</p> <hd id="AN0177623651-4">Adaptive Comparative Judgment (ACJ)</hd> <p>Adaptive comparative judgment was originally designed as an approach to assessment and evaluation. It was established on the "Law of Comparative Judgment" (Thurstone, [<reflink idref="bib61" id="ref32">61</reflink>]) by Louis Thurstone who sought to measure psychological values with discriminable differences. He presented a series of articles raising a concern with the methods of measurements of psychological perceptions such as mind, social attitudes, intelligence, and values (Thurstone, [<reflink idref="bib61" id="ref33">61</reflink>], [<reflink idref="bib62" id="ref34">62</reflink>], [<reflink idref="bib63" id="ref35">63</reflink>], [<reflink idref="bib64" id="ref36">64</reflink>]). Through this work, he yielded a meaningful finding of the "Law of Comparative Judgment" which can be used to measure the physical intensities, as well as quality of psychological values, through simultaneous and successive contrasts (Thurstone, [<reflink idref="bib61" id="ref37">61</reflink>]).</p> <p>Later, (Pollitt, [<reflink idref="bib48" id="ref38">48</reflink>]) outlined the possibility of adaptive comparative judgment (ACJ) which is applicable to various educational assessments and could provide high reliability and validity compared to the conventional assessment (i.e., operational marking) by graders (Mentzer et al., [<reflink idref="bib39" id="ref39">39</reflink>]; Pollitt, [<reflink idref="bib48" id="ref40">48</reflink>]). Pollitt incorporated an adaptive attribute to the comparative judgment by pairing similarly-ranked items in the process of judging (as opposed to random comparisons) to maximize the information gained from each comparison. Also, ACJ leverages a holistic statement which guides the judges to make a professional judgment (Pollitt, [<reflink idref="bib48" id="ref41">48</reflink>]) as to which item—of the pair displayed—demonstrates the highest quality. Judges make a comparison based on the provided holistic statement with overarching reasoning (e.g., "Which artifact demonstrates a better ideation process?").</p> <hd id="AN0177623651-5">Learning by evaluating (LbE) in STEM education</hd> <p>Several recent studies have incorporated adaptive comparative judgment (ACJ) into STEM education setting, especially in project-based design thinking processes (Bartholomew et al., [<reflink idref="bib7" id="ref42">7</reflink>], [<reflink idref="bib8" id="ref43">8</reflink>], [<reflink idref="bib9" id="ref44">9</reflink>], [<reflink idref="bib6" id="ref45">6</reflink>]; Dewit et al., [<reflink idref="bib18" id="ref46">18</reflink>]; Strimel et al., [<reflink idref="bib59" id="ref47">59</reflink>]). Bartholomew et al., ([<reflink idref="bib7" id="ref48">7</reflink>], [<reflink idref="bib8" id="ref49">8</reflink>]) examined ACJ to evaluate the middle school students' learning through an open-ended problem assigned in a technology and engineering education course. They found ACJ to be an effective assessment with reliability, validity, feasibility, and practicality. Results from assessment using ACJ also correspond to traditional marking results in project-based learning (Bartholomew et al., [<reflink idref="bib7" id="ref50">7</reflink>], [<reflink idref="bib8" id="ref51">8</reflink>]) but this innovation presents other opportunities. Not only was ACJ an effective measure, but, when students participate in ACJ, they can experience learning without necessarily increasing teacher workload (Nicol et al., [<reflink idref="bib42" id="ref52">42</reflink>]).</p> <p>Engaging students in the ACJ process has numerous intellectual merits (Collins et al., [<reflink idref="bib13" id="ref53">13</reflink>]). Critiquing, evaluating, and providing reasoning can foster critical thinking (Hokanson, [<reflink idref="bib30" id="ref54">30</reflink>]; McNeill & Krajcik, [<reflink idref="bib37" id="ref55">37</reflink>]). Other studies have asserted that student reasoning in the process of peer feedback has benefits such as a better understanding of learning goals (Stefani, [<reflink idref="bib58" id="ref56">58</reflink>]), more advanced understanding of assignments (Nicol et al., [<reflink idref="bib42" id="ref57">42</reflink>]), and higher achievements (Gielen & De Wever, [<reflink idref="bib26" id="ref58">26</reflink>]). Furthermore, the ACJ process gives an intuitive method to involve students in the assessment process as judges. Students reported ACJ as a fun and enjoyable task and students are keen to discuss their judgments (Jones & Wheadon, [<reflink idref="bib33" id="ref59">33</reflink>]). Later, Bartholomew et al. ([<reflink idref="bib9" id="ref60">9</reflink>]) implemented peer-evaluated ACJ in an open-ended design setting. In this study, students participated as judges to make assessments for their peer's design projects. A few studies delve into student-based ACJ and its educational benefits. Strimel et al. ([<reflink idref="bib59" id="ref61">59</reflink>]) explored students' ACJ process on their peer work and results showed that students reflected on their own design during the ACJ and thus gain future ideas to implement into their designs. Dewit et al. ([<reflink idref="bib18" id="ref62">18</reflink>]) incorporated ACJ in the higher educational context, a product-service-system design project for first year master's students, and found that these students could incorporate analytical thinking and metacognition, which improved their learning experiences. Collectively, the findings illuminated possibilities of using ACJ as an effective means for both teaching and learning.</p> <p>Extending findings from Bartholomew et al. ([<reflink idref="bib9" id="ref63">9</reflink>]), Bartholomew et al. ([<reflink idref="bib6" id="ref64">6</reflink>]) introduced a new term to describe the use of ACJ as a primer for learning: <emph>Learning by Evaluating</emph> (LbE). LbE engages students as judges using the ACJ platform, but facilitates design thinking insights at the start of a project, rather than waiting until students have progressed significantly further with their work. Moreover, instead of evaluating peer work, the examples used in LbE are selected by the instructor to target specific learning objectives or project elements. Working in this paradigm, we hypothesize that students will be oriented to the project and that the comparisons made will further their critical thinking and decision making when approaching open-ended problems (Jackson et al., [<reflink idref="bib32" id="ref65">32</reflink>]).</p> <hd id="AN0177623651-6">CER (Claim-Evidence-Reasoning) framework</hd> <p>The process of making and justifying decisions, also called argumentation, is central in engineering (Wilson-Lopez et al., [<reflink idref="bib66" id="ref66">66</reflink>]) and design (Daly et al., [<reflink idref="bib16" id="ref67">16</reflink>]). The claims, evidence, and reasoning (CER) framework, which emerged from research and instruction in science education, describes three central components of scientific thinking and argumentation that are applicable in these knowledge-construction settings (McNeill & Krajcik, [<reflink idref="bib37" id="ref68">37</reflink>]). This framework has been used in instruction, to support students' decision-making and explanation processes. By using the CER framework to craft an argument, students gain a deeper understanding of the learning content, because their explanations focus on bringing in their scientific background knowledge (Fielding-Wells, [<reflink idref="bib24" id="ref69">24</reflink>]). Even in complex inquiry-based problems, Gillies ([<reflink idref="bib27" id="ref70">27</reflink>]) supported this claim by asserting that students can use reasoning based on disciplinary knowledge. Context also plays a central role in CER framework, in defining the question, design, or problem within which the claims are situated (NGSS Standards, [<reflink idref="bib57" id="ref71">57</reflink>]).</p> <p>Based on the CER model (Slavit et al., [<reflink idref="bib54" id="ref72">54</reflink>]), we have chosen to concentrate on the role of claim-making and reasoning that occurs in engineering design thinking activities. In the design thinking context, students learn while designing, and can begin to make and explain knowledge-driven decisions (Crismond & Adams, [<reflink idref="bib15" id="ref73">15</reflink>]). They learn by doing (e.g., iteration, drawing, and troubleshooting), and use this knowledge from experiments to make a better, evolving design decision (Perkins et al., [<reflink idref="bib43" id="ref74">43</reflink>]). Yet, in engineering design thinking argumentation (see Fig. 1), the claims are tentative because the goal is to create and enact the most appropriate model for the contextualized problem (Slavit et al., [<reflink idref="bib54" id="ref75">54</reflink>]). Viability based on experiment and user-response are two major criteria that drive the claim-making process. Still, each claim, the statement or a conclusion to a problem, should be supported by justification based on data and explanation as in the CER framework and scientific argumentation (McNeill & Krajcik, [<reflink idref="bib37" id="ref76">37</reflink>]). Reasoning logically connects between claim and evidence, by expounding how those relationships can be established (McNeill & Krajcik, [<reflink idref="bib37" id="ref77">37</reflink>]; Toulmin, [<reflink idref="bib65" id="ref78">65</reflink>]; Van Eemeren et al., [<reflink idref="bib22" id="ref79">22</reflink>]).</p> <p>Graph: Fig. 1 Ways of thinking in engineering: using reasoning to make claims (Slavit et al., [<reflink idref="bib54" id="ref80">54</reflink>])</p> <p>In the current study, we determined to delve into students' reasoning in engineering design settings, when incorporating LbE. The learning objectives and artifacts used for the experiences were predetermined and meant to support students' design work, particularly in terms of enhancing their ideation. As seen in Fig. 2, by selecting option A or option B, they make an implicit claim that "Option A is better" or "Option B is better." Then, they are further prompted to answer a question: "Why you have chosen A (or B)?" While providing comments about their judgment decision, students need to provide reasoning for their claim. These claims and explanations provide the basis for our investigation which centered on student reasoning and content.</p> <p>Graph: Fig. 2 CER Model implication in the LbE process</p> <hd id="AN0177623651-7">Methods</hd> <p></p> <hd id="AN0177623651-8">Research questions</hd> <p>The current study aimed to examine students' reasoning from LbE in the context of design thinking. Specifically, our focus was on the reasoning provided in the comments section of LbE done by students working on a backpack design challenge. Considering that students provide reasoning for either the 'selected' option (e.g., "I like option A because it looks fancier.") or 'not selected' option (e.g., choosing option A because "Option B cannot carry a thing."), our intent was to determine if the content of the reasoning differed, and how it differed. In this empirical analysis, the following research questions were used to compare these different reasonings based on selection (i.e., selected and not selected).</p> <p></p> <ulist> <item> Is there a statistically meaningful difference between the scores of sentiment analysis (e.g., positive, negative, neutral, and compound scores) based on reasoning target?</item> <p></p> <item> How are the contents of reasoning situated in the design thinking context? In other words, what are the design thinking components students incorporated in their reasoning?</item> </ulist> <hd id="AN0177623651-9">Research context</hd> <p>This study, part of a larger project, was embedded in a 9th grade Engineering design course with five units during the school year. Students learned design experiences in the early units of lessons, and then worked to apply what they learned later in the unit. Researchers partnered with a large, urban district near Atlanta, Georgia, United States. The district serves a total of 93,703 students from a very diverse population (DeKalb County School District, [<reflink idref="bib21" id="ref81">21</reflink>]). Students and their families speak over 185 + languages and represent over 155 + nations. The majority of the students are underrepresented minorities (59% Black, 20% Hispanic), and 48.2% of students are eligible for free and reduced lunch (DeKalb County School District, [<reflink idref="bib21" id="ref82">21</reflink>]).</p> <p>Teachers in this study received professional development training multiple times as part of their <emph>Engineering by Design</emph> certification (International Technology and Engineering Educators Association, 2018) for the course, as well as training in LbE implementation in the classroom to ensure fidelity of use in our larger project. Teachers then introduced the design projects, led classroom discussions among the students, facilitated our LbE research, promoted skill development in the design process, and tracked student progression. With the guidance of teachers, students learned basic concepts of a design thinking process and its components. Further, they were motivated to participate in hands-on tasks through project-based learning of design thinking, and could address meaningful, real-world problems fostering critical thinking.</p> <hd id="AN0177623651-10">Participants</hd> <p>Prior to the data collection, consent and assent forms were distributed to students, teachers, and parents/guardians, as required by the researchers' University Institutional Review Board and the school district's research coordinators. This study focused where teachers were delivering a similar challenge to their introductory Engineering design classes—to complete a backpack design. All 140 students enrolled in the course across three schools were invited to participate in the study. Out of three schools, two teachers consented (66.67% return rate). Out of 140 students, 32 students completed assent forms (22.86% return rate) and 37 parents consented (26.43%). Thus, only 15 students were fully consented by school, teacher, and parents. Students were in their 9<sups>th</sups> grade engaged in the Engineering design course during Fall 2021 and Spring 2022.</p> <hd id="AN0177623651-11">Task: Backpack comparison</hd> <p>The goal of the current task was to promote the students' ideation process as part of design thinking. While identifying which one of the pair to choose, students were required to select which example helped or inspired them to ideate better. Our holistic statement for the current task was:</p> <p>What makes a good backpack? How could each item inspire you to solve the problem. Which item best assists you with your design and why?</p> <p>As an example of a comparison, in Fig. 3, students can think about which option is better and why it is better for their ideation of a new product design.</p> <p>Graph: Fig. 3 Screenshot of Backpack comparison session. "Left (A): ©WONG SZE FEI/ Adobe Stock #99,606,465", "Right (B): ©cegli/ Adobe Stock #27,278,935"</p> <p>Researchers uploaded a total of 39 different design artifacts from different sources related to the ideation of effective backpack design. To help in student brainstorming, images included various items such as actual backpacks, or other images related to backpacks that might carry things or hold things. During the activity, teachers introduced LbE with three steps: a student orientation, the LbE comparisons, and a post-session debrief. For instance, teachers provided a detailed and actionable problem statement to address the current problem of designing 'better' and 'inspiring' backpacks to users. Students then viewed pairs of images. In each case, they were required to select which they believed to be the better option (claim) to help them consider ways to improve their own backpack designs. Right after their decision, students were required to justify their choice of one item over another. Students either could provide their reasoning for 'selecting' the better image of the pair or describe why they didn't select the other option. After the completion of the judgments, teachers transitioned to a group discussion about which design artifact was better at inspiring ideation and why.</p> <p>Each student made an average of 10 comparisons with a minimum of 3 and a maximum of 20 judgments. After removing responses which do not fit the analysis criteria (e.g., not providing comments, writing something unrelated, including abusive/inappropriate language), a total of 145 comments from the judgments were collected.</p> <hd id="AN0177623651-12">Research design</hd> <p></p> <hd id="AN0177623651-13">Content analysis</hd> <p>According to Neuendorf ([<reflink idref="bib41" id="ref83">41</reflink>]), content analysis is a systematic and objective analysis of the characteristics of a message in context which can include both "human-coded analysis and computer-aided text analysis" (p. 19). Content analysis helps interpret meaning from the content of text data, and allows researchers to infer about the "states or properties of the sources of the analyzed data" (Krippendorff, [<reflink idref="bib34" id="ref84">34</reflink>], p. 20). It postulates concepts, attitudes, beliefs, intentions, emotions, and cognitive processes manifested in text; thus, it sets those attributes as a natural target for successful content analysis (Krippendorff, [<reflink idref="bib34" id="ref85">34</reflink>]). Both qualitative and quantitative approaches may be used in content analysis. In quantitative analysis, text is decontextualized to show patterns with reliability and validity. As compared to quantitative inquiry, qualitative inquiry aims to understand a phenomenon, rather than generalize the phenomenon (Forman & Damschroder, [<reflink idref="bib25" id="ref86">25</reflink>]), including thematic description.</p> <p>Since the first implementation in the field of content analysis, computer-aided content analysis (Sebeok & Zeps, [<reflink idref="bib53" id="ref87">53</reflink>]) is gaining its popularity, especially in the current big-data and machine-learning environment. It has proved high reliability, validity, and efficiency in different research settings (Su et al., [<reflink idref="bib60" id="ref88">60</reflink>]). However, computer-aided content analysis has key limitations, such as not always tapping the true meaning behind the relationships between the words (Krippendorff, [<reflink idref="bib34" id="ref89">34</reflink>]; Matthes & Kohring, [<reflink idref="bib35" id="ref90">35</reflink>]; Su et al., [<reflink idref="bib60" id="ref91">60</reflink>]). In response to the limitations noted above, we adopted a hybrid approach by combining computation and human-based methods to take advantage of researchers' insights in the analysis of text datasets (Su et al., [<reflink idref="bib60" id="ref92">60</reflink>]). As seen in Fig. 4, based on the text-mining approach, researchers utilized an additional interpretation to recognize underlying meanings embedded in the result of computer-aided content analysis.</p> <p>Graph: Fig. 4 Content analysis with a hybrid approach</p> <p>Researchers agreed upon utilizing students' reasoning in the comment section of each judgment to analyze their reasoning when making a claim in an engineering design thinking context. To explore differences of reasoning between 'selected' and 'not selected' options, we performed sentiment analysis and word frequency analysis. First, we labeled the data based on the justification of students. Then, we performed sentiment analysis to see if the perceived sentiment is different for the selected or the not selected option. A <emph>t</emph> test was used to identify if there was a statistically significant difference on sentiment between reasoning targets. Followed by this, we performed content analysis with word frequency. It revealed how word count was different between selected and not selected decision reasonings. Finally, researchers explored the students' thought process by identifying specific characteristics and then comparing those responses to an engineering design mindset. For the analysis, we used Python 3.9.7 within Visual Studio Code (version 1.71). Libraries used will be stated in each subsection.</p> <hd id="AN0177623651-14">Data labeling</hd> <p>Because student claims and justifications were made holistically, not on either item in the comparison, it was necessary to review and label whether the justification was based on the selected or not selected option. After the initial labeling from one of the research team members who led this analysis, each student justification was manually reviewed and coded by the research team based on our interpretation of the target of the comment. As seen in the examples in Table 1, when students provided reasoning about the selected option, we coded them as a 0. In contrast, when it was about not selected option, we coded them into 1.</p> <p>Table 1 Example codes for reasoning target, based on claim</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Comments</p></th><th align="left"><p>Selected*</p></th><th align="left"><p>Code</p></th></tr></thead><tbody><tr><td align="left"><p>Helps me show that straps could be involved in my design</p></td><td align="left"><p>Y</p></td><td align="left"><p>0</p></td></tr><tr><td align="left"><p>No one would buy the other one</p></td><td align="left"><p>N</p></td><td align="left"><p>1</p></td></tr><tr><td align="left"><p>This looks very impractical, but somewhat stylish</p></td><td align="left"><p>Y</p></td><td align="left"><p>0</p></td></tr><tr><td align="left"><p>The other one is hideous, uncomfortable, and heavy</p></td><td align="left"><p>N</p></td><td align="left"><p>1</p></td></tr><tr><td align="left"><p>It shows that the strap is adjustable so you can change to your likings</p></td><td align="left"><p>Y</p></td><td align="left"><p>0</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Selected*</bold> means if it is a reasoning about the selected option</p> <p>Three of the comments included reasonings related to both options in the comparison. Previous studies found that including mixed reviews in classification creates a problem when scoring words (Akhtyamova et al., [<reflink idref="bib1" id="ref93">1</reflink>]; Dave et al., [<reflink idref="bib17" id="ref94">17</reflink>]; Dey & Haque, [<reflink idref="bib19" id="ref95">19</reflink>], [<reflink idref="bib20" id="ref96">20</reflink>]). We decided to separate these comments out due to the small number of comments with mixed reasoning (N = 3) or about the not selected option (N = 16). Hence, we separated mixed comments (see Table 2) into two comments and coded them in the same way as before, based on whether the comment referred to the item selected or not selected (see Table 3). Therefore, the total number of comments slightly increased (N = 148).</p> <p>Table 2 Example codes for comments when participants provided mixed reasoning</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Comments</p></th><th align="left"><p>Selected*</p></th><th align="left"><p>Code</p></th></tr></thead><tbody><tr><td align="left"><p>Backpacks are way more comfortable and better in ALL ways than a purse; A backpack will lay on your shoulders with straps while a purse will irritate the skin on the inside of your elbow. It's just true</p></td><td align="left"><p>N/A</p></td><td align="left"><p>N/A</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Selected*</bold> means if it is a reasoning about the selected option</p> <p>Table 3 Example mixed coded comments after separating them into two reasonings</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Comments</p></th><th align="left"><p>Selected*</p></th><th align="left"><p>Code</p></th></tr></thead><tbody><tr><td align="left"><p>Backpacks are way more comfortable and better in ALL ways than a purse; A backpack will lay on your shoulders with straps</p></td><td align="left"><p>Y</p></td><td align="left"><p>0</p></td></tr><tr><td align="left"><p>A purse will irritate the skin on the inside of your elbow. It's just true</p></td><td align="left"><p>N</p></td><td align="left"><p>1</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Selected*</bold> means if it is a reasoning about the selected option</p> <p>Overall, students provided more reasoning for the selected option (<emph>N</emph> = 129) than the not selected options (<emph>N</emph> = 19). Our next step involved application of computer-aided content analysis via natural language processing.</p> <hd id="AN0177623651-15">Data preprocessing using natural language processing</hd> <p>After finishing the labeling, researchers processed the natural language into the vectorized text. Natural language processing incorporates computational techniques to learn, understand, and process human language (Hirschberg & Manning, [<reflink idref="bib29" id="ref97">29</reflink>]; Nadkarni et al., [<reflink idref="bib40" id="ref98">40</reflink>]). We removed punctuation, stemmed, lemmatized, and converted the text to the features into the model below (see Table 4). Specifically, we used lemmatization and count vectorizer function from NLTK packages (Bird et al., [<reflink idref="bib10" id="ref99">10</reflink>]). Using lemmatization, we processed words into their normalized form, for example to simplify 'straps' to 'strap' (Plisson et al., [<reflink idref="bib45" id="ref100">45</reflink>]). Then, we removed stop words—common words like 'I', 'me', 'was'—to find the context or the true meaning of decision criteria. Finally, count vectorizer was incorporated to quantify the words, by taking the frequency of words into account for the prediction. After initial preprocessing, researchers decided to add six more stop words to remove words repeated without significant meaning in the current research context (i.e., 'option', 'b').[<reflink idref="bib2" id="ref101">2</reflink>] After preprocessing, reasonings about the selected option had a lower character count (<emph>M</emph> = 54.40, <emph>SD</emph> = 33.25) and word count (<emph>M </emph>= 10.22, <emph>SD </emph>= 6.30) compared to the character count and word count for reasonings about the not selected option (<emph>M</emph> = 71.37, <emph>SD</emph> = 57.85; <emph>M</emph> = 13.84, <emph>SD</emph> = 10.66, respectively).</p> <p>Table 4 Example preprocessing of students' reasoning into normalized form</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Original text</p></th><th align="left"><p>→ </p></th><th align="left"><p>Remove punctuation</p></th><th align="left"><p> → </p></th><th align="left"><p>Remove stop words</p></th><th align="left"><p> → </p></th><th align="left"><p>Lemmatized text</p></th></tr></thead><tbody><tr><td align="left"><p>It looks like a more comfortable backpack</p></td><td align="left"><p> → </p></td><td align="left"><p>It looks like a more comfortable backpack</p></td><td align="left"><p> → </p></td><td align="left"><p>looks like comfortable backpack</p></td><td align="left"><p> → </p></td><td align="left"><p>look like comfortable backpack</p></td></tr><tr><td align="left"><p>It looks like it has more pockets and space, with the same comfort level</p></td><td align="left"><p> → </p></td><td align="left"><p>It looks like it has more pockets and spaces with the same comfort level</p></td><td align="left"><p> → </p></td><td align="left"><p>looks like pockets space comfort level</p></td><td align="left"><p> → </p></td><td align="left"><p>look like pocket space comfort level</p></td></tr><tr><td align="left"><p>Bags don't close!</p></td><td align="left"><p> → </p></td><td align="left"><p>Bags dont close</p></td><td align="left"><p> → </p></td><td align="left"><p>bags dont close</p></td><td align="left"><p> → </p></td><td align="left"><p>bag dont close</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0177623651-16">Sentiment analysis</hd> <p>Sentiment analysis was initiated to gauge opinions and identify sentiments (Mejova, [<reflink idref="bib38" id="ref102">38</reflink>]; Prabowo & Thelwall, [<reflink idref="bib49" id="ref103">49</reflink>]; Zhang et al., [<reflink idref="bib67" id="ref104">67</reflink>]). It often provides a polarity, dichotomized into two: positive and negative (Mejova, [<reflink idref="bib38" id="ref105">38</reflink>]). In this study, we used NLTK's Vader analysis, which classifies the texts into different sentiments as a range (Bird et al., [<reflink idref="bib10" id="ref106">10</reflink>]). Sentiment intensity analyzer was implemented, which yields scores of positive, negative, neutral, and compound results. The compound score is the normalized sum of all positive, negative, and neutral scores. Most extreme negative text (e.g., 'I hate A.') is scored as -1, and most extreme positive text (e.g., 'it contains all of the things you need for school work food etc.') is scored as + 1. Thus, the closer the compound score is to + 1, the more the reasoning is considered positive; conversely, the closer the compound score is to –1, the more the reasoning is considered negative. In Table 5, #Ex1—#Ex3 shows how the sentiment score is assigned to students' reasoning.</p> <p>Table 5 Exemplary results of sentiment analysis</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>#</p></th><th align="left"><p>Text</p></th><th align="left"><p>Code</p></th><th align="left"><p>Neg</p></th><th align="left"><p>Neu</p></th><th align="left"><p>Pos</p></th><th align="left"><p>Com</p></th></tr></thead><tbody><tr><td align="left"><p>Ex1</p></td><td align="left"><p>Man carrying everything on his back has a lot of <bold>pain</bold></p></td><td align="left"><p>1</p></td><td char="." align="char"><p>0.45</p></td><td char="." align="char"><p>0.55</p></td><td char="." align="char"><p>0.00</p></td><td align="left"><p>−0.51</p></td></tr><tr><td align="left"><p>Ex2</p></td><td align="left"><p>I think it is efficient, less <bold>painful</bold></p></td><td align="left"><p>0</p></td><td char="." align="char"><p>0.35</p></td><td char="." align="char"><p>0.27</p></td><td char="." align="char"><p>0.38</p></td><td align="left"><p>0.05</p></td></tr><tr><td align="left"><p>Ex3</p></td><td align="left"><p>Purse <bold>irritates</bold> skin inside elbow (...)</p></td><td align="left"><p>1</p></td><td char="." align="char"><p>0.29</p></td><td char="." align="char"><p>0.42</p></td><td char="." align="char"><p>0.30</p></td><td align="left"><p>0.00</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Neg</bold>: Negative sentiment score, <bold>Neu</bold>: Neutral sentiment score, <bold>Pos</bold>: Positive sentiment score, <bold>Com</bold>: Compound sentiment score</p> <p>Following sentiment analysis, we performed <emph>t</emph> tests to compare the difference of sentiments between two reasoning targets. Each sentiment score—positive, neutral, negative, and the compound result that represents an overall sentiment for the reasoning—was compared by group for the reasoning target: selected or not selected.</p> <hd id="AN0177623651-17">Word Frequency</hd> <p>Word frequency was also incorporated as part of the content analysis. Word frequency in a document provides insights about the patterns of word used and the content of a text (Baayen, [<reflink idref="bib2" id="ref107">2</reflink>]). After studying word frequency, we decided to present the top 10 most frequently used words.</p> <hd id="AN0177623651-18">Results</hd> <p>In this section, findings from computer-aided content analysis (i.e., sentiment analysis, word frequency) are presented. Then, researchers introduce conclusions from human-based content analysis, which elaborated on the findings from computer-aided content analysis.</p> <hd id="AN0177623651-19">Students' perceived sentiment in reasoning based on claim</hd> <p>As seen in Table 6, students used words with more positivity (<emph>M</emph> = 0.26, <emph>SD</emph> = 0.29) when providing reasoning for the selected option, while they used more negative words (<emph>M</emph> = 0.10, <emph>SD</emph> = 0.28) when providing their reasoning for not selecting an option. In terms of neutral sentiment score, we found reasoning for selected options showed lower neutral sentiment scores (<emph>M</emph> = 0.73, <emph>SD</emph> = 0.29) compared to reasoning related to the not selected option (<emph>M</emph> = 0.78, <emph>SD</emph> = 0.30). The differences in each reasoning sentiment (in isolation) also explains the higher compound score of the reasoning about the selected group (<emph>M</emph> = 0.22, <emph>SD</emph> = 0.25) in comparison to the reasoning about the not selected group (<emph>M</emph> = 0.00, <emph>SD</emph> = 0.28). These sentiment scores showed that students had a tendency to express strong positivity for the selected options, while providing less strong negativity toward the option which was not selected (see Fig. 5).</p> <p>Table 6 Scores from Sentiment Analysis: Negative, Neutral, Positive and Compound</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Neg</p><p><italic>M(SD)</italic></p></th><th align="left"><p>Neu</p><p><italic>M(SD)</italic></p></th><th align="left"><p>Pos</p><p><italic>M(SD)</italic></p></th><th align="left"><p>Com</p><p><italic>M(SD)</italic></p></th></tr></thead><tbody><tr><td align="left"><p>Selected (N = 129)</p></td><td align="left"><p>0.01 (0.15)</p></td><td align="left"><p>0.73 (0.29)</p></td><td align="left"><p>0.26 (0.29)</p></td><td align="left"><p>0.22 (0.25)</p></td></tr><tr><td align="left"><p>Not Selected (N = 19)</p></td><td align="left"><p>0.10 (0.28)</p></td><td align="left"><p>0.78 (0.30)</p></td><td align="left"><p>0.07 (0.14)</p></td><td align="left"><p>0.00 (0.28)</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Neg</bold>: Negative sentiment score, <bold>Neu</bold>: Neutral sentiment score, <bold>Pos</bold>: Positive sentiment score, <bold>Com</bold>: Compound sentiment score</p> <p>Graph: Fig. 5 Compound score distribution of two reasonings: Selected and not selected</p> <p>The results from the <emph>t</emph> test indicated that differences between the scores were statistically significant for the positive sentiment score, <emph>t</emph>(<reflink idref="bib146" id="ref108">146</reflink>) = 5.87, <emph>p</emph> < 0.01. There was no statistically significant difference found in negative sentiment score and neutral sentiment score using the threshold level of <emph>p</emph> < 0.01. In other words, whether student comments related to the selected option or the not selected option, students used similar levels of neutral and negative words in their reasoning. However, students tended to use strong positive words when providing reasoning for selected options.</p> <p>In Table 7, five comments with the highest negative sentiment score are presented (Neg1–Neg5). Because not all the words in the comment included negative sentiment, we could not find 1.0 point negative statement from the students' reasoning. Even though students provided reasoning for the selected option, the negative sentiment score might be high due to the strong negative words (e.g., no, doesn't). Neg3 stands out because it resulted in a high negative score in spite of its reading positively as a whole comment; this was due to the negative word usage (e.g., strain, doesn't want to). But it also has a similar neutral score and positive score. We will discuss this misinterpretation of context later in this section.</p> <p>Table 7 Comments with high negative sentiment score</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>#</p></th><th align="left"><p>Neg</p></th><th align="left"><p>Neu</p></th><th align="left"><p>Pos</p></th><th align="left"><p>Com</p></th><th align="left"><p>Text</p></th><th align="left"><p>Selected*</p></th></tr></thead><tbody><tr><td align="left"><p>Neg1</p></td><td char="." align="char"><p>0.61</p></td><td char="." align="char"><p>0.39</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>− 0.48</p></td><td align="left"><p>Poor horse wheels</p></td><td align="left"><p>N</p></td></tr><tr><td align="left"><p>Neg2</p></td><td char="." align="char"><p>0.45</p></td><td char="." align="char"><p>0.55</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>− 0.51</p></td><td align="left"><p>The man carrying everything on his back looks to be in a lot of pain</p></td><td align="left"><p>N</p></td></tr><tr><td align="left"><p>Neg3</p></td><td char="." align="char"><p>0.37</p></td><td char="." align="char"><p>0.32</p></td><td char="." align="char"><p>0.21</p></td><td char="." align="char"><p>0.35</p></td><td align="left"><p>Using a Lama to carry you belongings is a good idea because it puts no strain on yourself and who doesn't want to walk with a lama</p></td><td align="left"><p>Y</p></td></tr><tr><td align="left"><p>Neg4</p></td><td char="." align="char"><p>0.24</p></td><td char="." align="char"><p>0.76</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>− 0.44</p></td><td align="left"><p>Carrying everything box make arms tired especially carrying every day</p></td><td align="left"><p>N</p></td></tr><tr><td align="left"><p>Neg5</p></td><td char="." align="char"><p>0.23</p></td><td char="." align="char"><p>0.45</p></td><td char="." align="char"><p>0.33</p></td><td char="." align="char"><p>0.22</p></td><td align="left"><p>I feel like the left one is better because what if you don't have kids. Then the backpack has no value</p></td><td align="left"><p>N</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Selected*</bold> means if it is a reasoning about selected option. <bold>Neg</bold>: Negative sentiment score, <bold>Neu</bold>: Neutral sentiment score, <bold>Pos</bold>: Positive sentiment score, <bold>Com</bold>: Compound sentiment score</p> <p>Comments Pos1-Pos5 in Table 8 are five examples of comments which scored with the highest positive sentiment. Pos1 and Pos2 are relatively short reasoning with positive sentiment, so they both scored 1.00 in positive sentiment. Pos5 also showed shorter reasoning with positive words. Considering that Pos 1, 2, and Pos5 are reasoning from different students. The provided examples uphold the idea that students tend to provide less reasoning when selecting an option.</p> <p>Table 8 Comments with high positive sentiment score</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>#</p></th><th align="left"><p>Neg</p></th><th align="left"><p>Neu</p></th><th align="left"><p>Pos</p></th><th align="left"><p>Com</p></th><th align="left"><p>Text</p></th><th align="left"><p>Selected*</p></th></tr></thead><tbody><tr><td align="left"><p>Pos1</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>1.00</p></td><td char="." align="char"><p>0.42</p></td><td align="left"><p>It looks more efficient</p></td><td align="left"><p>Y</p></td></tr><tr><td align="left"><p>Pos2</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>1.00</p></td><td char="." align="char"><p>0.34</p></td><td align="left"><p>Looks more secure</p></td><td align="left"><p>Y</p></td></tr><tr><td align="left"><p>Pos3</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>0.12</p></td><td char="." align="char"><p>0.88</p></td><td char="." align="char"><p>0.93</p></td><td align="left"><p>Helps improve confidence and positivity to have a new backpack with a special message on it</p></td><td align="left"><p>Y</p></td></tr><tr><td align="left"><p>Pos4</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>0.18</p></td><td char="." align="char"><p>0.82</p></td><td char="." align="char"><p>0.58</p></td><td align="left"><p>That looks cool and that looks cool to the baby</p></td><td align="left"><p>Y</p></td></tr><tr><td align="left"><p>Pos5</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>0.26</p></td><td char="." align="char"><p>0.74</p></td><td char="." align="char"><p>0.69</p></td><td align="left"><p>Better material easier to carry</p></td><td align="left"><p>Y</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Selected*</bold> means if it is a reasoning about selected option. <bold>Neg</bold>: Negative sentiment score, <bold>Neu</bold>: Neutral sentiment score, <bold>Pos</bold>: Positive sentiment score, <bold>Com</bold>: Compound sentiment score</p> <p>From the score distribution of positive sentiment and negative sentiment (see Fig. 6), we also found evidence that positive sentiment scores were more widely distributed than negative sentiment. Most of the negative sentiment was found in the reasoning related to the options not selected, while most positive sentiment was found in the reasoning for selected options. The high peak for negative sentiment found in this graph is likely due to the larger number of statements which reasoned for 'selecting an option' over 'not selecting an option.' However, the distribution of the two sentiments indicates that positive sentiment clearly has a larger tail compared to negative sentiment. Specifically, the negative sentiment score had a maximum value around 0.6 while the positive sentiment score had several values higher than 0.6. Taken together, these show that students typically used words with stronger positive sentiments when providing reasoning for the option they claimed was better.</p> <p>Graph: Fig. 6 Sentiment analysis score distribution of positive and negative sentiment</p> <p>As mentioned earlier, it was also notable that some of the analysis performed by the computer did not capture the context appropriately. Although human researchers can infer students' intention underlying the text, sentiment analysis only discerns word-by-word (or partially syntactic) sentiment in the meaning (see Table 5). For instance, in Table 9, the first comment (Contextfail1) has a low compound score and high negative score due to words such as 'cartoony' and 'problem,' that would generally be interpreted as negatives. Another example, Contextfail2 showed 1.00 neutral sentiment score, but reading the comment actually reveals positive reasoning for the image because it looks 'clearer' and 'more simple.' As a result of preprocessing and limitations of sentiment analysis, some of the comments were not scored appropriately, nor did they reflect student intention.</p> <p>Table 9 Examples of sentiment failed to catch the sentiment and underlying context</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>#</p></th><th align="left"><p>Neg</p></th><th align="left"><p>Neu</p></th><th align="left"><p>Pos</p></th><th align="left"><p>Com</p></th><th align="left"><p>Text</p></th><th align="left"><p>Selected*</p></th></tr></thead><tbody><tr><td align="left"><p>Contextfail1</p></td><td char="." align="char"><p>0.25</p></td><td char="." align="char"><p>0.75</p></td><td char="." align="char"><p>0.00</p></td><td align="left"><p>−0.44</p></td><td align="left"><p>This image is also more appealing with the cartoony style of presentation, and showing a relatable problem</p></td><td align="left"><p>Y</p></td></tr><tr><td align="left"><p>Contextfail2</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>1.00</p></td><td char="." align="char"><p>0.00</p></td><td align="left"><p>0.00</p></td><td align="left"><p>The image seems clearer and more simple. It also shows an additional use for the backpack</p></td><td align="left"><p>Y</p></td></tr></tbody></table> </ephtml> </p> <p> <bold>Selected*</bold> means if it is a reasoning about selected option. <bold>Neg</bold>: Negative sentiment score, <bold>Neu</bold>: Neutral sentiment score, <bold>Pos</bold>: Positive sentiment score, <bold>Com</bold>: Compound sentiment score</p> <hd id="AN0177623651-20">Frequently used words in the reasoning process of backpack comparison</hd> <p>The most frequently used words in the students' reasoning are depicted below in Table 10, and we describe three insights about students' reasoning. Based on review of the comments, and modified stop words described earlier, we excluded some of the words that were used in relation to the LbE experience instead of the content of the options being shown (e.g., show, picture, image, option).</p> <p>Table 10 Frequently used words in reasoning, based on claim</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" colspan="3"><p>Reasoning of selected option</p></th><th align="left" colspan="3"><p>Reasoning of not selected option</p></th></tr><tr><th align="left"><p>Selected (S)#</p></th><th align="left"><p>Word</p></th><th align="left"><p>Count</p></th><th align="left"><p>Not Selected (NS) #</p></th><th align="left"><p>Word</p></th><th align="left"><p>Count</p></th></tr></thead><tbody><tr><td align="left"><p>S1</p></td><td align="left"><p>Carry</p></td><td align="left"><p>27</p></td><td align="left"><p>NS1</p></td><td align="left"><p>Carry</p></td><td align="left"><p>6</p></td></tr><tr><td align="left"><p>S2</p></td><td align="left"><p>Help</p></td><td align="left"><p>15</p></td><td align="left"><p>NS2</p></td><td align="left"><p>Heavy</p></td><td align="left"><p>5</p></td></tr><tr><td align="left"><p>S3</p></td><td align="left"><p>Design</p></td><td align="left"><p>15</p></td><td align="left"><p>NS3</p></td><td align="left"><p>Book</p></td><td align="left"><p>5</p></td></tr><tr><td align="left"><p>S4</p></td><td align="left"><p>Item</p></td><td align="left"><p>14</p></td><td align="left"><p>NS4</p></td><td align="left"><p>Multiple</p></td><td align="left"><p>3</p></td></tr><tr><td align="left"><p>S5</p></td><td align="left"><p>Thing</p></td><td align="left"><p>12</p></td><td align="left"><p>NS5</p></td><td align="left"><p>Everything</p></td><td align="left"><p>3</p></td></tr><tr><td align="left"><p>S6</p></td><td align="left"><p>Well</p></td><td align="left"><p>12</p></td><td align="left"><p>NS6</p></td><td align="left"><p>Stuff</p></td><td align="left"><p>2</p></td></tr><tr><td align="left"><p>S7</p></td><td align="left"><p>Easy</p></td><td align="left"><p>11</p></td><td align="left"><p>NS7</p></td><td align="left"><p>Hard</p></td><td align="left"><p>2</p></td></tr><tr><td align="left"><p>S8</p></td><td align="left"><p>Many</p></td><td align="left"><p>9</p></td><td align="left"><p>NS8</p></td><td align="left"><p>Item</p></td><td align="left"><p>2</p></td></tr><tr><td align="left"><p>S9</p></td><td align="left"><p>Space</p></td><td align="left"><p>7</p></td><td align="left"><p>NS9</p></td><td align="left"><p>Number</p></td><td align="left"><p>2</p></td></tr><tr><td align="left"><p>S10</p></td><td align="left"><p>Hold</p></td><td align="left"><p>7</p></td><td align="left"><p>NS10</p></td><td align="left"><p>Hold</p></td><td align="left"><p>2</p></td></tr></tbody></table> </ephtml> </p> <p>First, most comments made could be interpreted in relation to the design task and original value function of a backpack—to carry (S1, NS1) or to hold (S10, NS10) something—regardless of the claim. Other commonly used words, for instance, item (S3, NS8), (every)thing (S5, NS5), stuff (NS6), and more specifically, book (NS3), can be interpreted in light of what is carried. Expanding the function of 'carrying' something, students also considered how many (S8)/multiple (NS4) items the backpack can carry, which also related to the idea of "space" (S9); these words suggest that students considered spaces or number of items the backpack could hold as an important criterion for their decisions.</p> <p>Second, usage or user perspective was considered an important criterion for reasonings, both to select an option or not to select an option. In the reasoning of selected options, how easy (S7) or well (S6) the backpack could carry stuff was often mentioned. As well as thinking about the backpack itself, students took user perspectives into account. In contrast, the reasoning for not selected options showed that students commented on the backpack used words like 'heavy (NS2)' and/or 'hard (to do something) (NS7)', showing additional instances of student concern for the possible user experiences of those with the backpack.</p> <p>Third, when providing reasonings for the selected options, students also regarded the design (S3) of the backpack as an important justification. Design may represent a broad term that includes overall reactions to the options and details such as aesthetic features of the backpack. Put simply, students rejected options that explicitly failed to fulfill the very basic functions of a backpack (e.g., fail to carry stuff, cannot hold multiple item). Yet, the analysis showed that if both of the backpack options could meet the criteria of carrying multiple items, students seemed more inclined to make a decision based on the design features present or absent in each of the displayed backpacks.</p> <hd id="AN0177623651-21">Content analysis of comments in the context of engineering design thinking</hd> <p>For further elaboration and inference, our research team performed qualitative content analysis (Hsieh & Shannon, [<reflink idref="bib31" id="ref109">31</reflink>]) based on the previous computer-aided content analysis. As mentioned earlier, computer-aided content analysis may not always capture context or rhetorical nuance of the comments (Neuendorf, [<reflink idref="bib41" id="ref110">41</reflink>]; Su et al., [<reflink idref="bib60" id="ref111">60</reflink>]). Previous sections also demonstrated this problem (Table 9). Considering that students' comments were situated in the engineering design course, researchers qualitatively analyzed the comments through this lens of design thinking. This stage provided an interpretation of the underlying context of the reasoning and when students made a claim about selecting or not selecting backpacks.</p> <hd id="AN0177623651-22">Empathizing by identifying themselves as users</hd> <p>When taking a look at the word frequency, we found students treated themselves as the end-user of the items displayed (e.g., me, my, I). When taking a closer look, we found that some of the students (N = 5) tended to specifically articulate their opinions from this user perspective. Examples from different students are as below.<emph>"I do not like the clear backpack because you can see everything inside someone's bag, including valuable items such as phones."</emph><emph>"I like backpacks that you can put alot of stuff in."</emph><emph>"I have issues with my backpack being too heavy and not being able to carry a lot of stuff."</emph><emph>"I prefer using bags as opposed to carrying stuff in boxes."</emph><emph>"The cameras backpack is a very good idea because it has snug padding to keep all your gear safe (i need to get myself one of those)."</emph></p> <p>In design thinking, a designer's ability to empathize with the users is critical to developing an understanding of the problem. Our analysis showed that students could easily imagine themselves in the user environment demonstrating empathy toward users.</p> <hd id="AN0177623651-23">Defining user's needs and problems</hd> <p>As seen in word frequency analysis, students considered users' needs and problems when making a decision. In most cases, students considered four main areas concerning backpacks: the convenience of use (e.g., more secure, enough spaces), characteristics and features (e.g., charge your laptop, carry and move around), specific occasion (everyday use), and fulfilling specific user needs (e.g., people with no kids, carry a baby, walking a lot). Here are representative examples from the student comments:<emph>"I think that it is easier to carry a baby than a giant roll of materials."</emph><emph>"Rolling backpacks are very useful if you are walking a lot."</emph><emph>"I feel like the left one is better because what if you dont have kids. Than the backpack has no value."</emph><emph>"If you are more of a physical person this would be better."</emph><emph>"Helps improve a lightweight backpack for everyday use."</emph><emph>"A small backpack with a limited carrying capacity but is useful for carrying around art supplies."</emph><emph>"I have chosen A because your arms can get tired of holding your baby so you can hold him using your backpack and you also have space to put your things... cause its a backpack."</emph></p> <p>It is meaningful that students made connections to real-world, authentic problems as well as to users with different and specific needs. When students presume themselves as a user, they might specifically mention their own needs. Similarly, when they took possible users (e.g., as parents, physical person, someone carrying art supplies) into consideration, they tend to elaborate on what each specific user is in need. These types of comments reiterate students' empathy while solving problems as designers.</p> <hd id="AN0177623651-24">Aesthetic features or ideation to students' own design products</hd> <p>The word 'design' was only used for the reasoning about the selected option. When looking at the comments including this term, students used 'design' for two purposes: to refer to aesthetics or in reference to their own design process. First, in many reasonings the word design was a criterion regarding an option's aesthetically pleasing qualities. It was notable, however, that there were only two students regularly bringing this aesthetic rationale into their decision-making.<emph>"Much better design"</emph><emph>"Looking better design"</emph><emph>"Nicely designed and is more appealing to buyers"</emph><emph>"Has an organized design"</emph></p> <p>Secondly, students adopted the word 'design' in relation to a plan for their own design model. When provided images encompassing inspirational sketches, actual images of backpack, and up-to-date features, students utilized the images for ideation and commented on how it could inspire them to design a new backpack:<emph>"Helps show that my design should be able to hold heavy items without breaking."</emph><emph>"Mine should have many pockets and be organized."</emph><emph>"This image is more humorous and grabs the viewer's attention better."</emph><emph>"Helps me show that straps could be involved in my design."</emph></p> <p>However, it is also true that students compared and discussed the quality of the options, rather than ideating from the provided options. Many of the student reasonings simply judged the function, design, and quality of backpack. In other words, in spite of a holistic prompt and suggestion to base their choice on inspiration for iteration and though the comments provided a clear reason for selecting one option over another, it may not have lead them to new inspiration for their own designs.<emph>"It can hold more things"</emph><emph>"I have chosen B because the backpack looks nice"</emph><emph>"A multi-purpose bag is a good idea"</emph><emph>"It can hold larger items"</emph></p> <p>For instance, the comment 'It can hold more things' clearly showed why the student preferred the selected option. However, students reasoning here may not extend to their own ideation when designing (i.e., realizing that they also need to design a bag which can hold more things). Similarly, 'I have chosen B because the backpack looks nice' implies the student valued aesthetic features of the backpack. Even though they articulated the benefits of the selected option over the not selected one, we cannot conclude whether students considered these aspects when they designed their backpack. Hence, we cannot conclude how the judgment process impacted the design brainstorming of students.</p> <hd id="AN0177623651-25">Learning transfer</hd> <p>With respect to the LbE experience preparing students for their upcoming problem solving, it is also notable that some of the reasoning showed signs of failed learning transfer. Teachers included backpack designs and other images that were intended to stretch the creativity of the students' designs; however, many students could not make a connection between the images—for example, a possum carrying her family (Fig. 7) —and what would be helpful when designing a backpack. Since it was hard to understand the intention of the image being there, three students who were provided images of possum reported confusion in their reasoning. Several other reasonings showed students' failure in understanding the intention of pictures or in their ability to abstract the images to the design process.<emph>"Why is this relevant – hamster"</emph><emph>"The other is a literal animal"</emph></p> <p>Graph: Fig. 7 Picture of possum used to inspire students "©Evelyn / Adobe Stock #282,517,042"</p> <p>As a result, when presented with these images, the students selected the other option without much evidence supporting the claim. In general, these examples of concrete thinking imply how students' reasoning can be affected by the design and orientation of LbE, including the holistic statement and examples presented. When integrating LbE, this means that ultimately such students were not prepared to recognize opportunities for transfer of learning.</p> <hd id="AN0177623651-26">Discussion</hd> <p>This study incorporated content analysis through computer-aided methods (i.e., sentiment analysis and word frequency) and human-based methods altogether to investigate students' reasoning used in LbE. Findings from sentiment analysis indicated that students used words with more positive sentiment when providing reasoning for the option they selected. However, they used words with more negative sentiment when providing reasoning for the option they did not select. In the student reasonings for selected options, students expressed more biased, positive sentiment with less neutral expression and used shorter reasoning. This may be due to the lack of critical thinking or reflection or may result from failing to articulate their thinking in comments in spite of in-depth reasoning. Thus, further educational intervention is required to facilitate logical, detailed, design-thinking-based reasoning, which eventually can be provided in comments.</p> <p>Results from word frequency analysis showed how students perceive the importance of the design features they need to work on. Original functions of backpack, user convenience, and aesthetic design features were considered as crucial. We found aesthetic features were considered more often when selecting an option, and less considered when not selecting an option. Qualitative analysis of these comments found a few of students just briefly used the term 'design' to describe their intuitive impression or preference toward the option (e.g., a better looking design). Though these comments imply that students cared about aesthetic features when selecting a design, the reasoning itself is still vague.</p> <p>However, our qualitative analysis indicated that not all students are taking all features into consideration when making a judgment. This is in line with past research that suggests beginners tend to reason with a single explanation (Sandoval & Millwood, [<reflink idref="bib52" id="ref112">52</reflink>]). In other words, some of the students used just one or two criteria in their reasoning. For instance, there was one student who only cared about the function of backpack design and failed to incorporate other features making a more efficient backpack. Hence, further intervention such as activities sharing criteria with others can be considered when implementing the LbE into an educational setting. This finding also reinforces the importance of a class-wide discussion following LbE comparison making, as an opportunity to synthesize this information about what criteria matter in this design context and help students to recognize and deal with interconnectivity and trade-offs as an important part of design decision making (Crismond & Adams, [<reflink idref="bib15" id="ref113">15</reflink>]; Goel & Pirolli, [<reflink idref="bib28" id="ref114">28</reflink>]).</p> <p>Finally, content analysis from the researchers found students tend to (<reflink idref="bib1" id="ref115">1</reflink>) identify themselves as users, (<reflink idref="bib2" id="ref116">2</reflink>) consider user perspective and context, (<reflink idref="bib3" id="ref117">3</reflink>) use design as an aesthetic criterion or to ideate their design. When students identified themselves as users, they could emphasize better to users but also had a limitation of only reflecting their own perspective instead of considering broader target users. Furthermore, without the broader perspective, students considered limited user contexts. When requested to use options to ideate a new backpack design, students could incorporate design features successfully through their reasoning. However, there were a few signals of the failure of learning transfer, as well. Effective and explicit introduction of the holistic statement (preparation for the LbE comparisons) may help prevent confusion in students, which leads to the failure of learning transfer. Additional activities helping students share their reasoning process with peers or the entire classroom (debrief on the LbE comparisons) may help students think more broadly and critically.</p> <hd id="AN0177623651-27">Suggestion for teacher educators</hd> <p>Exploring the contents of students' reasoning in LbE provided insights into how teachers and STEM design educators can use ACJ as a tool to inform meaningful learning. First, teachers should introduce the LbE with detail and be explicit about the learning intentions. Teachers should present the goal of LbE (e.g., select an option better design, which inspires your own design), contents of judgment (e.g., backpack), with meaningful criteria of judging backpack design (e.g., function, user perspective, creative, aesthetic features) before LbE. This should be well-presented and summarized in a holistic statement the students will see in every judgment. After this, providing modeling of LbE can be helpful. For instance, actually modeling a couple of judgments of the sample work (e.g., previous students' artifact) is a viable and effective option to help students see design reasoning in action.</p> <p>Without explicit and specific guidance, students may simply select options that are visually pleasing (e.g., looks better, has a cool design) or based on the preference without any reasoning (e.g., It seems more appealing, it looks more efficient), or make poor selections when they fail to understand (e.g., what is this?). Students tendency for these types of selections were supported by evidence of short, positive reasoning without specific evidence in the sentiment analysis. Studies (McNeill & Krajcik, [<reflink idref="bib37" id="ref118">37</reflink>]; Ping et al., [<reflink idref="bib44" id="ref119">44</reflink>]; Slavit et al., [<reflink idref="bib54" id="ref120">54</reflink>]) view reasoning as an essential process of claims based on evidence, and highlight the importance of explicit teaching. This is especially the case when disciplinary content and practices become more explicit, and the claims and reasoning becomes more centered on a specific content area with epistemological shifts (Slavit et al., [<reflink idref="bib54" id="ref121">54</reflink>]). Thus, when specific and explicit context provided to the students with more details, student-thinking in an articulated manner is expected to be exhibited.</p> <p>Second, post-LbE activities such as teacher-led or peer-led discussion based on actual judgments can be helpful for leveraging understanding. While some of the students could articulate their reasoning in a substantially diverse way, with a good understanding within design thinking context, other students fell short. When given an opportunity to follow up on comparisons, students can think about how to use preferred features they have seen in the ideation process, while preventing inclusion of the less preferred features. Also, students have a chance to clarify understanding of the options with peer discussion. For instance, as presented earlier, some of the students could not successfully understand what the possum picture has to offer backpack design (Fig. 7). They could not imagine the transfer applications and may have assumed it was an attempt at testing engagement. Not only can a discussion facilitate expanding the characteristics and features of good design, but it can also help students articulate why they think it looks better. In a more public setting (e.g., small group, entire classroom discussion) which requires interaction and participation, students need to provide explanation with more details, which eventually promotes articulation of their knowledge (Collins et al., [<reflink idref="bib13" id="ref122">13</reflink>]).</p> <p>Third, post-LbE discussion may help students think from others' perspective, including users. In some cases, students have contrasting opinions toward the same option (e.g., a simplistic backpack that can carry a good amount of beachwear vs. I do not like the clear backpack because you can see everything inside someone's bag, including valuable items such as phones). Discussion on these subjects help students foster a design mindset with openness to diverse opinion (Brenner et al., [<reflink idref="bib11" id="ref123">11</reflink>]; Ekman & Ekman, [<reflink idref="bib23" id="ref124">23</reflink>]). Though students in this study empathized with users, they may be biased if they stick to their own perspective and identify only themselves as users. In this sense, discussion after LbE can help balance empathy and open mindset toward a broad user group. Rauth et al., ([<reflink idref="bib50" id="ref125">50</reflink>]) underscored the importance of the involvement of all perspectives when designing. They see each team and individual developing their own process as they work on a problem and adapting and adding to the solution as the essence of design thinking. Being mindful of the opinion of others can foster creative thinking among students, allowing them 'thinking out of the box' from one single individual's perspective.</p> <hd id="AN0177623651-28">Limitation</hd> <p>Several limitations in this study should be acknowledged. Sample size of the current study is small compared to other studies using machine learning and natural language processing. Though qualitative content analysis compensates for the sample size, an imbalanced dataset with small samples may not be generalizable to all contexts. Some aspects of the LbE session were inconsistent in their presentation to students. There were some technical glitches students faced while using the software, thus they could not see options clearly. Additionally, size of the images was not consistent throughout the judgment process. Therefore, students may be inclined to select larger images with high resolution unless the resolution was too high, and the image was slow to load. These challenges may have affected students' judgment or explanations in ways that are not associated with the content of the options presented.</p> <hd id="AN0177623651-29">Declarations</hd> <p></p> <hd id="AN0177623651-30">Conflict of interest</hd> <p>The authors declare that they have no conflict of interest. The authors have no relevant financial or non-financial interests to disclose.</p> <hd id="AN0177623651-31">Ethical approval</hd> <p>The current study was supported by funding from National Science Foundation (NSF), Division of Research on Learning (DRL), Project no. 2101235. Entire research process in the current study involving human participants were in compliance with the ethical standards (i.e., IRB) of the institutions Purdue University (IRB-2021–543). Brigham Young University (R0303035), and University of Georgia (PROJECT 00004032). and school district RRB (file 2021–015). Informed consent was obtained from all individual participants in the study. All the software (RM Compare, Python, Python libraries) has an appropriate permission for the use.</p> <hd id="AN0177623651-32">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0177623651-33"> <title> References </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Akhtyamova, L, Alexandrov, M, Cardiff, J, & Koshulko, O. (2018). Opinion mining on small and noisy samples of health-related texts. Conference on Computer Science and Information Technologies, 379–390.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Baayen, R. H. (2001). Word frequency distributions (Vol. 18). Springer Science & Business Media.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref11" type="bt">3</bibl> <bibtext> Bartholomew, S. R. (2021). Investigating the impacts of differentiated stimulus materials in a learning by evaluating activity. Mississippi Valley Technology Teacher Education Conference.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref12" type="bt">4</bibl> <bibtext> Bartholomew, S. R, & Yauney, J. (2022). The impact of differentiated stimulus materials in learning by evaluating. Pupils' Attitudes Towards Technology Annual Conference.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref9" type="bt">5</bibl> <bibtext> Bartholomew SR, Jones MD. A systematized review of research with adaptive comparative judgment (ACJ) in higher education. International Journal of Technology and Design Education. 2022; 32; 1: 1159-1190. 10.1007/s10798-020-09642-6</bibtext> </blist> <blist> <bibl id="bib6" idref="ref13" type="bt">6</bibl> <bibtext> Bartholomew SR, Mentzer N, Jones M, Sherman D, Baniya S. Learning by evaluating (LbE) through adaptive comparative judgment. International Journal of Technology and Design Education. 2020; 32; 2: 1-15</bibtext> </blist> <blist> <bibl id="bib7" idref="ref17" type="bt">7</bibl> <bibtext> Bartholomew SR, Nadelson LS, Goodridge WH, Reeve EM. Adaptive comparative judgment as a tool for assessing open-ended design problems and model eliciting activities. Educational Assessment. 2018; 23; 2: 85-101. 10.1080/10627197.2018.1444986</bibtext> </blist> <blist> <bibl id="bib8" idref="ref18" type="bt">8</bibl> <bibtext> Bartholomew SR, Strimel GJ, Jackson A. A comparison of traditional and adaptive comparative judgment assessment techniques for freshmen engineering design projects. International Journal of Engineering Education. 2018; 34; 1: 20-33</bibtext> </blist> <blist> <bibl id="bib9" idref="ref10" type="bt">9</bibl> <bibtext> Bartholomew SR, Strimel GJ, Yoshikawa E. Using adaptive comparative judgment for student formative feedback and learning during a middle school design project. International Journal of Technology and Design Education. 2019; 29; 2: 363-385. 10.1007/s10798-018-9442-7</bibtext> </blist> <blist> <bibtext> Bird, S, Klein, E, & Loper, E. (2009). Natural Language Processing with Python. O'Reilly Media, Inc.</bibtext> </blist> <blist> <bibtext> Brenner W, Uebernickel F, Abrell TBrenner W, Uebernickel F. Design thinking as mindset, process, and toolbox. Design thinking for innovation. 2016; Springer: 3-21. 10.1007/978-3-319-26100-3_1</bibtext> </blist> <blist> <bibtext> Canty, D, Seery, N, Hartell, E, & Doyle, A. (2017). Integrating peer assessment in technology education through adaptive comparative judgment. PATT34 Technology & Engineering Education–Fostering the Creativity of Youth Around The Globe, Millersville University, Pennsylvania, USA, 10–14.</bibtext> </blist> <blist> <bibtext> Collins, A, Brown, J. S, Newman, S. E, & Resnick, R. (2018). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In Knowing, learning, and instruction (pp. 453–494). Routledge.</bibtext> </blist> <blist> <bibtext> Cowan J. Developing the ability for making evaluative judgements. Teaching in Higher Education. 2010; 15; 3: 323-334. 10.1080/13562510903560036</bibtext> </blist> <blist> <bibtext> Crismond DP, Adams RS. The informed design teaching & learning matrix. Journal of Engineering Education-Washington. 2012; 101; 4: 738-747. 10.1002/j.2168-9830.2012.tb01127.x</bibtext> </blist> <blist> <bibtext> Daly SR, Adams RS, Bodner GM. What does it mean to design? A qualitative investigation of design professionals' experiences. Journal of Engineering Education. 2012; 101; 2: 187-219. 10.1002/j.2168-9830.2012.tb00048.x</bibtext> </blist> <blist> <bibtext> Dave, K, Lawrence, S, & Pennock, D. M. (2003). Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. Proceedings of the 12th International Conference on World Wide Web, 519–528.</bibtext> </blist> <blist> <bibtext> Dewit I, Rohaert S, Corradi D. How can comparative judgement become an effective means toward providing clear formative feedback to students to improve their learning process during their product-service-system design project?. Design and Technology Education. 2021; 26; 3: 276-293</bibtext> </blist> <blist> <bibtext> Dey, L, & Haque, S. M. (2008). Opinion mining from noisy text data. Proceedings of the Second Workshop on Analytics for Noisy Unstructured Text Data, 83–90.</bibtext> </blist> <blist> <bibtext> Dey, L, & Haque, S. M. (2009). Studying the effects of noisy text on text mining applications. Proceedings of The Third Workshop on Analytics for Noisy Unstructured Text Data, 107–114.</bibtext> </blist> <blist> <bibtext> DeKalb County School District. (2021). DeKalb County School District Demographics (2021–2022 School Year). https://<ulink href="http://www.dekalbschoolsga.org/about/">www.dekalbschoolsga.org/about/</ulink></bibtext> </blist> <blist> <bibtext> Van Eemeren, F. H, Grootendorst, R, & Kruiger, T. (2019). Handbook of argumentation theory. In Handbook of Argumentation Theory. De Gruyter Mouton.</bibtext> </blist> <blist> <bibtext> Ekman, S, & Ekman, A. (2009). Designing an entrepreneurial mindset in engineering and management. DS 58–9: Proceedings of ICED 09, the 17th International Conference on Engineering Design, Vol. 9, Human Behavior in Design, Palo Alto, CA, USA, 24.-27.08. 2009, 179–190.</bibtext> </blist> <blist> <bibtext> Fielding-Wells, J. (2016). Mathematics Is Just 1+ 1= 2, What Is There to Argue About?: Developing a Framework for Argument-Based Mathematical Inquiry. Mathematics Education Research Group of Australasia.</bibtext> </blist> <blist> <bibtext> Forman, J, & Damschroder, L. (2007). Qualitative content analysis. In L. Jacoby & L. A. Siminoff (Eds.), Empirical Methods for Bioethics: A Primer (Vol. 11, pp. 39–62). Emerald Group Publishing Limited. https://doi.org/10.1016/S1479-3709(07)11003-7</bibtext> </blist> <blist> <bibtext> Gielen M, De Wever B. Structuring peer assessment: Comparing the impact of the degree of structure on peer feedback content. Computers in Human Behavior. 2015; 52: 315-325. 10.1016/j.chb.2015.06.019</bibtext> </blist> <blist> <bibtext> Gillies RM. Inquiry-based science education. 2020; CRC Press</bibtext> </blist> <blist> <bibtext> Goel V, Pirolli P. The structure of design problem spaces. Cognitive Science. 1992; 16; 3: 395-429. 10.1207/s15516709cog1603_3</bibtext> </blist> <blist> <bibtext> Hirschberg J, Manning CD. Advances in natural language processing. Science. 2015; 349; 6245: 261-266. 10.1126/science.aaa8685</bibtext> </blist> <blist> <bibtext> Hokanson BMoller L, Huett J. The design critique as a model for distributed learning. The next generation of distance education. 2012; Springer: 71-83. 10.1007/978-1-4614-1785-9_5</bibtext> </blist> <blist> <bibtext> Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qualitative Health Research. 2005; 15; 9: 1277-1288. 10.1177/1049732305276687</bibtext> </blist> <blist> <bibtext> Jackson, A, Bartholomew, S, & Mentzer, N. (2022, November 17). Priming the design process: Activating and characterizing students' critical thinking in design [Paper presentation]. 108th Mississippi Valley Technology Teacher Education Conference and the 60th Southeastern Technology Education Conference, Nashville, TN.</bibtext> </blist> <blist> <bibtext> Jones I, Wheadon C. Peer assessment using comparative and absolute judgement. Studies in Educational Evaluation. 2015; 47: 93-101. 10.1016/j.stueduc.2015.09.004</bibtext> </blist> <blist> <bibtext> Krippendorff, K. (2018). Content analysis: An introduction to its methodology. Sage publications.</bibtext> </blist> <blist> <bibtext> Matthes J, Kohring M. The content analysis of media frames: Toward improving reliability and validity. Journal of Communication. 2008; 58; 2: 258-279. 10.1111/j.1460-2466.2008.00384.x</bibtext> </blist> <blist> <bibtext> McNeill, K. L, Katsh-Singer, R, & Pelletier, P. (2015). Assessing Science Practices: Moving your class along a continuum. Science Scope, 39(4), 21–28. https://<ulink href="http://www.proquest.com/scholarly-journals/assessing-science-practices-moving-your-class/docview/1735273036/se-2?accountid=13360">www.proquest.com/scholarly-journals/assessing-science-practices-moving-your-class/docview/1735273036/se-2?accountid=13360</ulink></bibtext> </blist> <blist> <bibtext> McNeill KL, Krajcik JLuft J, Bell R, Gess-Newsome J. Inquiry and scientific explanations: Helping students use evidence and reasoning. Science as inquiry in the secondary setting. 2008; National Science Teaching Association: 121-134</bibtext> </blist> <blist> <bibtext> Mejova, Y. (2009). Sentiment analysis: An overview.</bibtext> </blist> <blist> <bibtext> Mentzer N, Lee W, Bartholomew SR. Examining the validity of adaptive comparative judgment for peer evaluation in a design thinking course. Frontiers in Education. 2021; 6: 1-15. 10.3389/feduc.2021.772832</bibtext> </blist> <blist> <bibtext> Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: An introduction. Journal of the American Medical Informatics Association. 2011; 18; 5: 544-551. 10.1136/amiajnl-2011-000464</bibtext> </blist> <blist> <bibtext> Neuendorf, K. A. (2017). The content analysis guidebook. In The content analysis guidebook (2nd edition.). SAGE Publications, Inc.</bibtext> </blist> <blist> <bibtext> Nicol D, Thomson A, Breslin C. Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education. 2014; 39; 1: 102-122. 10.1080/02602938.2013.795518</bibtext> </blist> <blist> <bibtext> Perkins DN, Crismond D, Simmons R, Unger C. Inside understanding. 1995; Oxford University Press</bibtext> </blist> <blist> <bibtext> Ping ILL, Halim L, Osman K. Explicit teaching of scientific argumentation as an approach in developing argumentation skills, science process skills and biology understanding. Journal of Baltic Science Education. 2020; 19; 2: 276-288. 10.33225/jbse/20.19.276</bibtext> </blist> <blist> <bibtext> Plisson J, Lavrac N, Mladenic D. A rule based approach to word lemmatization. Proceedings of IS. 2004; 3: 83-86</bibtext> </blist> <blist> <bibtext> Pollitt, A, & Whitehouse, C. (2012). Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment. Center for Education Research and Policy.</bibtext> </blist> <blist> <bibtext> Pollitt A. Comparative judgement for assessment. International Journal of Technology and Design Education. 2012; 22; 2: 157-170. 10.1007/s10798-011-9189-x</bibtext> </blist> <blist> <bibtext> Pollitt A. The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice. 2012; 19; 3: 281-300</bibtext> </blist> <blist> <bibtext> Prabowo R, Thelwall M. Sentiment analysis: A combined approach. Journal of Informetrics. 2009; 3; 2: 143-157. 10.1016/j.joi.2009.01.003</bibtext> </blist> <blist> <bibtext> Rauth, I, Köppen, E, Jobst, B, & Meinel, C. (2010). Design thinking: An educational model towards creative confidence. DS 66–2: Proceedings of the 1st International Conference on Design Creativity (ICDC 2010).</bibtext> </blist> <blist> <bibtext> Sadler DR. Beyond feedback: Developing student capability in complex appraisal. Assessment and Evaluation in Higher Education. 2010; 35; 5: 535-550. 10.1080/02602930903541015</bibtext> </blist> <blist> <bibtext> Sandoval WA, Millwood KA. The quality of students' use of evidence in written scientific explanations. Cognition and Instruction. 2005; 23; 1: 23-55. 10.1207/s1532690xci2301_2</bibtext> </blist> <blist> <bibtext> Sebeok TA, Zeps VJ. An analysis of structured content, with application of electronic computer research, in psycholinguistics. Language and Speech. 1958; 1; 3: 181-193. 10.1177/002383095800100304</bibtext> </blist> <blist> <bibtext> Slavit D, Grace E, Lesseig K. Student ways of thinking in STEM contexts: A focus on claim making and reasoning. School Science and Mathematics. 2021; 121; 8: 466-480. 10.1111/ssm.12501</bibtext> </blist> <blist> <bibtext> Sluijsmans, D, Dochy, F, & Moerkerke, G. (1998b). The use of self-, peer-and co-assessment in higher education.</bibtext> </blist> <blist> <bibtext> Sluijsmans D, Dochy F, Moerkerke G. Creating a learning environment by using self-, peer-and co-assessment. Learning Environments Research. 1998; 1; 3: 293-319. 10.1023/A:1009932704458</bibtext> </blist> <blist> <bibtext> NGSS Standards. (2013). Next generation science standards: For states, by states. The National Academies Press DC.</bibtext> </blist> <blist> <bibtext> Stefani LA. Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education. 1994; 19; 1: 69-75. 10.1080/03075079412331382153</bibtext> </blist> <blist> <bibtext> Strimel GJ, Bartholomew SR, Purzer S, Zhang L, Ruesch EY. Informing engineering design through adaptive comparative judgment. European Journal of Engineering Education. 2021; 46; 2: 227-246. 10.1080/03043797.2020.1718614</bibtext> </blist> <blist> <bibtext> Su LY-F, Cacciatore MA, Liang X, Brossard D, Scheufele DA, Xenos MA. Analyzing public sentiments online: Combining human- and computer-based content analysis. Information, Communication & Society. 2017; 20; 3: 406-427. 10.1080/1369118X.2016.1182197</bibtext> </blist> <blist> <bibtext> Thurstone LL. A law of comparative judgment. Psychological Review. 1927; 34; 4: 273-286. 10.1037/h0070288</bibtext> </blist> <blist> <bibtext> Thurstone LL. The measurement of social attitudes. The Journal of Abnormal and Social Psychology. 1931; 26; 3: 249-269. 10.1037/h0070363</bibtext> </blist> <blist> <bibtext> Thurstone LL. Factorial studies of intelligence. 1941; University of Chicago Press</bibtext> </blist> <blist> <bibtext> Thurstone LL. The measurement of values. Psychological Review. 1954; 61; 1: 47-58. 10.1037/h0060035</bibtext> </blist> <blist> <bibtext> Toulmin SE. The uses of argument. 2003; Cambridge University Press. 10.1017/CBO9780511840005</bibtext> </blist> <blist> <bibtext> Wilson-Lopez A, Strong AR, Hartman CM, Garlick J, Washburn KH, Minichiello A, Weingart S, Acosta-Feliz J. A systematic review of argumentation related to the engineering-designed world. Journal of Engineering Education. 2020; 109; 2: 281-306. 10.1002/jee.20318</bibtext> </blist> <blist> <bibtext> Zhang L, Wang S, Liu B. Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. 2018; 8; 4</bibtext> </blist> </ref> <ref id="AN0177623651-34"> <title> Footnotes </title> <blist> <bibtext> In the present study, students did not write feedback for the purpose of providing it to their peers. Instead, they evaluated example artifacts where reasoning provided in the comment functioned as a primer task for students. Hence, we will use the term 'comment' to designate the output of reasoning. Thus, eliciting feedback will concentrate on activating students as owners of their own learning, helping students articulating what success looks like, and critically reflecting on their own learning.</bibtext> </blist> <blist> <bibtext> 'A' in the 'Option A' was not included in the new stop words because 'a/A' is already included in the stop words due to the article 'a'.</bibtext> </blist> </ref> <aug> <p>By Nathan Mentzer; Wonki Lee; Andrew Jackson and Scott Bartholomew</p> <p>Reported by Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib46" firstref="ref3"></nolink> <nolink nlid="nl2" bibid="bib47" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib48" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib61" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib12" firstref="ref20"></nolink> <nolink nlid="nl6" bibid="bib56" firstref="ref21"></nolink> <nolink nlid="nl7" bibid="bib59" firstref="ref22"></nolink> <nolink nlid="nl8" bibid="bib32" firstref="ref23"></nolink> <nolink nlid="nl9" bibid="bib55" firstref="ref25"></nolink> <nolink nlid="nl10" bibid="bib42" firstref="ref26"></nolink> <nolink nlid="nl11" bibid="bib51" firstref="ref28"></nolink> <nolink nlid="nl12" bibid="bib14" firstref="ref29"></nolink> <nolink nlid="nl13" bibid="bib36" firstref="ref31"></nolink> <nolink nlid="nl14" bibid="bib62" firstref="ref34"></nolink> <nolink nlid="nl15" bibid="bib63" firstref="ref35"></nolink> <nolink nlid="nl16" bibid="bib64" firstref="ref36"></nolink> <nolink nlid="nl17" bibid="bib39" firstref="ref39"></nolink> <nolink nlid="nl18" bibid="bib18" firstref="ref46"></nolink> <nolink nlid="nl19" bibid="bib13" firstref="ref53"></nolink> <nolink nlid="nl20" bibid="bib30" firstref="ref54"></nolink> <nolink nlid="nl21" bibid="bib37" firstref="ref55"></nolink> <nolink nlid="nl22" bibid="bib58" firstref="ref56"></nolink> <nolink nlid="nl23" bibid="bib26" firstref="ref58"></nolink> <nolink nlid="nl24" bibid="bib33" firstref="ref59"></nolink> <nolink nlid="nl25" bibid="bib66" firstref="ref66"></nolink> <nolink nlid="nl26" bibid="bib16" firstref="ref67"></nolink> <nolink nlid="nl27" bibid="bib24" firstref="ref69"></nolink> <nolink nlid="nl28" bibid="bib27" firstref="ref70"></nolink> <nolink nlid="nl29" bibid="bib57" firstref="ref71"></nolink> <nolink nlid="nl30" bibid="bib54" firstref="ref72"></nolink> <nolink nlid="nl31" bibid="bib15" firstref="ref73"></nolink> <nolink nlid="nl32" bibid="bib43" firstref="ref74"></nolink> <nolink nlid="nl33" bibid="bib65" firstref="ref78"></nolink> <nolink nlid="nl34" bibid="bib22" firstref="ref79"></nolink> <nolink nlid="nl35" bibid="bib21" firstref="ref81"></nolink> <nolink nlid="nl36" bibid="bib41" firstref="ref83"></nolink> <nolink nlid="nl37" bibid="bib34" firstref="ref84"></nolink> <nolink nlid="nl38" bibid="bib25" firstref="ref86"></nolink> <nolink nlid="nl39" bibid="bib53" firstref="ref87"></nolink> <nolink nlid="nl40" bibid="bib60" firstref="ref88"></nolink> <nolink nlid="nl41" bibid="bib35" firstref="ref90"></nolink> <nolink nlid="nl42" bibid="bib17" firstref="ref94"></nolink> <nolink nlid="nl43" bibid="bib19" firstref="ref95"></nolink> <nolink nlid="nl44" bibid="bib20" firstref="ref96"></nolink> <nolink nlid="nl45" bibid="bib29" firstref="ref97"></nolink> <nolink nlid="nl46" bibid="bib40" firstref="ref98"></nolink> <nolink nlid="nl47" bibid="bib10" firstref="ref99"></nolink> <nolink nlid="nl48" bibid="bib45" firstref="ref100"></nolink> <nolink nlid="nl49" bibid="bib38" firstref="ref102"></nolink> <nolink nlid="nl50" bibid="bib49" firstref="ref103"></nolink> <nolink nlid="nl51" bibid="bib67" firstref="ref104"></nolink> <nolink nlid="nl52" bibid="bib146" firstref="ref108"></nolink> <nolink nlid="nl53" bibid="bib31" firstref="ref109"></nolink> <nolink nlid="nl54" bibid="bib52" firstref="ref112"></nolink> <nolink nlid="nl55" bibid="bib28" firstref="ref114"></nolink> <nolink nlid="nl56" bibid="bib44" firstref="ref119"></nolink> <nolink nlid="nl57" bibid="bib11" firstref="ref123"></nolink> <nolink nlid="nl58" bibid="bib23" firstref="ref124"></nolink> <nolink nlid="nl59" bibid="bib50" firstref="ref125"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1426801
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Learning by Evaluating (LbE): Promoting Meaningful Reasoning in the Context of Engineering Design Thinking Using Adaptive Comparative Judgment (ACJ)
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Nathan+Mentzer%22">Nathan Mentzer</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-9788-7901">0000-0001-9788-7901</externalLink>)<br /><searchLink fieldCode="AR" term="%22Wonki+Lee%22">Wonki Lee</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-0007-1023">0000-0002-0007-1023</externalLink>)<br /><searchLink fieldCode="AR" term="%22Andrew+Jackson%22">Andrew Jackson</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-2882-3052">0000-0003-2882-3052</externalLink>)<br /><searchLink fieldCode="AR" term="%22Scott+Bartholomew%22">Scott Bartholomew</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-1680-2433">0000-0002-1680-2433</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Technology+and+Design+Education%22"><i>International Journal of Technology and Design Education</i></searchLink>. 2024 34(3):1145-1169.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 25
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Evaluative+Thinking%22">Evaluative Thinking</searchLink><br /><searchLink fieldCode="DE" term="%22Design%22">Design</searchLink><br /><searchLink fieldCode="DE" term="%22Engineering+Education%22">Engineering Education</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+Making%22">Decision Making</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Abstract+Reasoning%22">Abstract Reasoning</searchLink><br /><searchLink fieldCode="DE" term="%22Content+Analysis%22">Content Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Thinking+Skills%22">Thinking Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Word+Frequency%22">Word Frequency</searchLink><br /><searchLink fieldCode="DE" term="%22Self+Concept%22">Self Concept</searchLink><br /><searchLink fieldCode="DE" term="%22Needs+Assessment%22">Needs Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Concept+Formation%22">Concept Formation</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s10798-023-09853-7
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0957-7572<br />1573-1804
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Adaptive comparative judgment (ACJ) has been widely used to evaluate classroom artifacts with reliability and validity. In the ACJ experience we examined, students were provided a pair of images related to backpack design. For each pair, students were required to select which image could help them ideate better. Then, they were prompted to provide a justification for their decision. Data were from 15 high school students taking engineering design courses. The current study investigated how students' reasoning differed based on selection. Researchers analyzed the comments in two ways: (1) computer-aided quantitative content analysis and (2) qualitative content analysis. In the first analysis, we performed sentiment analysis and word frequency analysis using natural language processing. Based on the findings, we explored how the design thinking process was embedded in student reasoning, and if the reasoning varied depending on the claim. Results from sentiment analysis showed that students tend to reveal more strong positive sentiment with short comments when providing reasoning for the selected design. In contrast, when providing reasoning for those items not chosen, results showed a weaker negative sentiment with more detailed reasons. Findings from word frequency analysis showed that students valued the function of design as well as the user perspective, specifically, convenience. Additionally, students took aesthetic features of each design into consideration when identifying the best of the two pairs. Within the engineering design thinking context, we found students empathize by identifying themselves as users, define user's needs, and ideate products from provided examples.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1426801
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1426801
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10798-023-09853-7
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 25
        StartPage: 1145
    Subjects:
      – SubjectFull: Evaluative Thinking
        Type: general
      – SubjectFull: Design
        Type: general
      – SubjectFull: Engineering Education
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Decision Making
        Type: general
      – SubjectFull: High School Students
        Type: general
      – SubjectFull: Abstract Reasoning
        Type: general
      – SubjectFull: Content Analysis
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Thinking Skills
        Type: general
      – SubjectFull: Word Frequency
        Type: general
      – SubjectFull: Self Concept
        Type: general
      – SubjectFull: Needs Assessment
        Type: general
      – SubjectFull: Concept Formation
        Type: general
    Titles:
      – TitleFull: Learning by Evaluating (LbE): Promoting Meaningful Reasoning in the Context of Engineering Design Thinking Using Adaptive Comparative Judgment (ACJ)
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Nathan Mentzer
      – PersonEntity:
          Name:
            NameFull: Wonki Lee
      – PersonEntity:
          Name:
            NameFull: Andrew Jackson
      – PersonEntity:
          Name:
            NameFull: Scott Bartholomew
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0957-7572
            – Type: issn-electronic
              Value: 1573-1804
          Numbering:
            – Type: volume
              Value: 34
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: International Journal of Technology and Design Education
              Type: main
ResultId 1