View in EDS HTML Full Text PDF Full Text

What Distinguishes Students' Engineering Design Performance: Design Behaviors, Design Iterations, and Application of Science Concepts

Saved in:
Bibliographic Details
Title: What Distinguishes Students' Engineering Design Performance: Design Behaviors, Design Iterations, and Application of Science Concepts
Language: English
Authors: Hanxiang Du (ORCID 0000-0002-9081-0706), Gaoxia Zhu (ORCID 0000-0003-4589-0775), Wanli Xing (ORCID 0000-0002-1446-889X), Charles Xie
Source: Journal of Science Education and Technology. 2025 34(2):314-326.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 13
Publication Date: 2025
Sponsoring Agency: National Science Foundation (NSF)
Contract Number: 2105695
2131097
Document Type: Journal Articles
Reports - Research
Education Level: High Schools
Secondary Education
Descriptors: Engineering, Design, Scientific Concepts, Problem Solving, High School Students, Energy, Computer Assisted Design, Computer Software
DOI: 10.1007/s10956-024-10184-y
ISSN: 1059-0145
1573-1839
Abstract: Engineering design that requires mathematical analysis, scientific understanding, and technology is critical for preparing students for solving engineering problems. In simulated design environments, students are expected to learn about science and engineering through their design. However, there is a lack of understanding concerning linking science concepts with design problems to design artifacts. This study investigated how 99 high school students applied science concepts to solarize their school using a computer-aided engineering design software, aiming to explore the interaction between students' science concepts and engineering design behaviors. Students were assigned to three groups based on their design performance: the achieving group, proficient group, and emerging group. By mining log activities, we explored the interactions among students' application of science concepts, engineering design behaviors, design iterations, and their design performance. We found that the achieving group has a statistically higher number of design iterations than the other two performance groups. We also identified distinctive transition patterns in students' applying science concepts and exercising design behaviors among three groups. The implications of this study are then discussed.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1463742
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFwGLlK2x1mLRXBQdEQaRYnAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDLaPXPRMf57S4o27eAIBEICBm4jUmFlKEOpEZLlPhKZ7kjjCraYDGamLQa9jD3goOaNOphV3uN5F3EskQAw0YiIBlg1G8M5tNVeoH6a5gJAWyp74zcXq0SB6CaKSINeoDJJoICzkMdm80ZDz0YYIhOEI04_44ui52JxzTyhTUnZNRP8fCCHpCxhpKuP3sFKrANrJetHQlEKp_5VtY8q9o8hHZJdDKLtG2NsiztgV
Text:
  Availability: 1
  Value: <anid>AN0183971589;4n601apr.25;2025Mar26.05:26;v2.2.500</anid> <title id="AN0183971589-1">What Distinguishes Students' Engineering Design Performance: Design Behaviors, Design Iterations, and Application of Science Concepts </title> <p>Engineering design that requires mathematical analysis, scientific understanding, and technology is critical for preparing students for solving engineering problems. In simulated design environments, students are expected to learn about science and engineering through their design. However, there is a lack of understanding concerning linking science concepts with design problems to design artifacts. This study investigated how 99 high school students applied science concepts to solarize their school using a computer-aided engineering design software, aiming to explore the interaction between students' science concepts and engineering design behaviors. Students were assigned to three groups based on their design performance: the achieving group, proficient group, and emerging group. By mining log activities, we explored the interactions among students' application of science concepts, engineering design behaviors, design iterations, and their design performance. We found that the achieving group has a statistically higher number of design iterations than the other two performance groups. We also identified distinctive transition patterns in students' applying science concepts and exercising design behaviors among three groups. The implications of this study are then discussed.</p> <p>Keywords: Engineering design; Science concepts; Design iterations; Computer-aided engineering design; Sequential analysis</p> <p>Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</p> <hd id="AN0183971589-2">Introduction</hd> <p>The National Academy of Engineering (NAE) recognizes the STEM-educated workforce as the main driving force of worldwide economic growth (National Academy of Engineering, [<reflink idref="bib27" id="ref1">27</reflink>]). In STEM (i.e., Science, Technology, Engineering, and Mathematics) education, engineering education plays a "catalyst" role because mathematical analysis, scientific understanding, and technology are required to solve engineering problems, to engage in engineering design, and to produce engineering products. Furthermore, engineering research might lead to new scientific discoveries. Therefore, it is critical to integrate engineering learning in K–12 education to prepare students to "design a system, design and conduct experiments, function on multidisciplinary teams, and identify, formulate, and solve engineering problems" (Khalaf et al., [<reflink idref="bib19" id="ref2">19</reflink>], p.2; The Accreditation Board for Engineering and Technology, [<reflink idref="bib33" id="ref3">33</reflink>]). Engineering design, a systematic and intelligent process of devising systems, components, or processes through iterative generation, analysis, and evaluation to meet given needs under specific constraints, is fundamental to engineering education (Berland et al., [<reflink idref="bib7" id="ref4">7</reflink>]; Dym et al., [<reflink idref="bib12" id="ref5">12</reflink>]). In engineering design, students do not aim to and are not likely to achieve the goal in one attempt. Instead, engineering design is an iterative process during which students often find themselves back and forth between problem identification, design formulation and implementation, solution evaluation, and reformulation based on evaluations (Dym et al., [<reflink idref="bib12" id="ref6">12</reflink>]). Iterative design has also been proven integral to successful design (e.g., Zhang et al., [<reflink idref="bib38" id="ref7">38</reflink>]).</p> <p>Solving engineering problems involves conducting research on previous problems and solutions and identifying possible solutions based on design criteria and constraints (Arık & Topçu, [<reflink idref="bib2" id="ref8">2</reflink>]). Hence, acquiring, analyzing, and applying relevant science concepts and knowledge is critical to solving engineering problems. Previous studies (e.g., Ball et al., [<reflink idref="bib5" id="ref9">5</reflink>]; Lammi et al., [<reflink idref="bib21" id="ref10">21</reflink>]; Mentzer, [<reflink idref="bib25" id="ref11">25</reflink>]) identified a difference between experienced designers and students: students usually focus less on science and invest less effort in collecting background information and analyzing problems. In engineering education, educators and students also face the "design-science gap"—a challenge of bridging and integrating engineering and science concepts and mechanisms to solve problems or design products (Vattam & Kolodner, [<reflink idref="bib34" id="ref12">34</reflink>]). Understanding how students' engineering design behaviors interact with their science knowledge has the potential to shine a light on how to bridge the "design-science gap."</p> <p>Previous research on design iterations tends to adopt qualitative research methods or quantitative comparison of pre- and post-tests (e.g., Chao et al., [<reflink idref="bib8" id="ref13">8</reflink>]; Purzer et al., [<reflink idref="bib29" id="ref14">29</reflink>]) which limit the possibility of providing students and teachers timely feedback to support adaptive teaching and learning. To address these issues, this study derived students' design iterations and application of science concepts from their design behaviors (i.e., log data) recorded in a computer-aided design (CAD) platform (i.e., Energy3D) and examined whether and how these factors were related to students' design performance. This study was guided by the following research questions (RQs):</p> <p></p> <ulist> <item> Do students with different design performances differ in the numbers of design interactions?</item> <p></p> <item> Do different design performance groups differ in their application of science concepts and frequency during their design?</item> <p></p> <item> What are the dynamics between science concepts and design behaviors in design interactions of students with different design performances?</item> </ulist> <hd id="AN0183971589-3">Literature Review</hd> <p></p> <hd id="AN0183971589-4">Engineering Design Iterations</hd> <p>Engineering design process includes defining the problem, selecting among multiple possible solutions, modeling and analysis, and iteration (Berland et al., [<reflink idref="bib7" id="ref15">7</reflink>]). Iteration, a process of revising a solution or product through multiple design cycles, is inherent in complex engineering design (Hjalmarson & Cardella, [<reflink idref="bib17" id="ref16">17</reflink>]). Due to various social, economic, scientific, or technological constraints around a design task, ambiguity and uncertainty are inevitable (Hjalmarson & Cardella, [<reflink idref="bib17" id="ref17">17</reflink>]). The inherent ambiguity and uncertainty in design require designers to iteratively engage in design procedures such as exploring the problem and solution spaces, identifying key variables to work with, reworking to respond to emergent problems, and converging on a satisfying design that meets design requirements (Wynn et al., [<reflink idref="bib36" id="ref18">36</reflink>]). A design iteration usually consists of common steps such as design, test, and analyze (Hartmann et al., [<reflink idref="bib16" id="ref19">16</reflink>]). In a design iteration, designers usually start with design, then test whether their design works as intended, and analyze whether the design meets the given requirements and why or why not (Marks & Chase, [<reflink idref="bib23" id="ref20">23</reflink>]). These procedures provide opportunities for designers to collect feedback, recheck the design constraints and requirements, and reflect on areas that need to be improved, which inform a new design cycle (Yen et al., [<reflink idref="bib37" id="ref21">37</reflink>]). Multiple design cycles, including repetitive design, analysis, and refinement, enable learners to continuously improve their design (Arik & Topçu, [<reflink idref="bib2" id="ref22">2</reflink>]; Zhu et al., [<reflink idref="bib40" id="ref23">40</reflink>]). Furthermore, learners' understanding of relevant science knowledge may be enhanced as they filter information, synthesize, and create knowledge to improve their design solutions or products (Hjalmarson & Cardella, [<reflink idref="bib17" id="ref24">17</reflink>]).</p> <p>Given the importance of iteration to engineering design, it is critical to provide students with opportunities to engage in iterations before they enter the workplace, including in K–12 context (Moore et al., [<reflink idref="bib26" id="ref25">26</reflink>]), and to study how iterations influence their design performance. Iterations provide a lens for analyzing how design changes over time, and relevant empirical research has been conducted in the K–12 context. For instance, by studying students' engineering design behaviors recorded in a 3-dimensional (3D) CAD design platform, Li et al. ([<reflink idref="bib22" id="ref26">22</reflink>]) found that students' reformulation behaviors on the one hand, positively predict design performance, and on the other hand, mediate the relationship between evaluation behaviors and design performance across time, suggesting the importance of iterative evaluation and reformulation on students' design. Marks and Chase ([<reflink idref="bib23" id="ref27">23</reflink>]) found that participating in an iterative prototyping intervention helped young students develop their knowledge of iterative practices, adaptive reactions to failure, and design performance on novel challenges.</p> <p>However, it is challenging to implement iterative design in K–12 classrooms (Kolodner et al., [<reflink idref="bib20" id="ref28">20</reflink>]) for many reasons. First, when being asked to iterate, novice designers who are inclined to build a perfect design with one attempt may feel anxious or satisfied with their initial solution and may not be able to engage in productive iterations without facilitation (Andrews, [<reflink idref="bib1" id="ref29">1</reflink>]; Dow et al., [<reflink idref="bib11" id="ref30">11</reflink>]; English et al., [<reflink idref="bib15" id="ref31">15</reflink>]). Second, from the perspective of teachers, they may not feel comfortable or familiar with facilitating multiple design iterations (Kolodner et al., [<reflink idref="bib20" id="ref32">20</reflink>]). Finally, time limitation is a constant issue which limits the possibility for students to engage in more than one design iteration (Hmelo et al., [<reflink idref="bib18" id="ref33">18</reflink>]). In this study, we aimed to provide students with a design tool, Energy3D, and adequate design time to examine the extent to which students could engage in design iterations and how design interactions are related to their design performance and application of science concepts. We provided limited teacher support to minimize the potential influence of variations in teacher instructions.</p> <hd id="AN0183971589-5">Engineering Design and Science Knowledge</hd> <p>Previous research suggests the way of applying science knowledge in engineering design processes is a characteristic that differentiates novice and expert engineering designers. Expert designers can rapidly identify and apply abstract knowledge to develop solutions to engineering design problems, while novice engineers tend to strategically identify a particular prior problem whose solutions can be mapped systematically to the current problem (Ball et al., [<reflink idref="bib5" id="ref34">5</reflink>]). Experts spend more time scoping the problem and collecting information, which are important competencies for engineering students to develop (Atman et al., [<reflink idref="bib3" id="ref35">3</reflink>], [<reflink idref="bib4" id="ref36">4</reflink>]). In contrast, students invest less effort in collecting background information and analyzing problems, or even if they do, they focus more on contextualized information but less on science (Lammi et al., [<reflink idref="bib21" id="ref37">21</reflink>]; Mentzer, [<reflink idref="bib25" id="ref38">25</reflink>]). Reflection is critical for advanced professional designers, and reflective practitioners constantly engage in "reflective conversation(s)" with the situation, changing the situation and eventually, achieving successful solutions (Schön, [<reflink idref="bib31" id="ref39">31</reflink>]).</p> <p>What remains unclear is how students apply and develop science knowledge as they engage in design tasks (Hjalmarson & Cardella, [<reflink idref="bib17" id="ref40">17</reflink>]). Learners often face the "design-science gap" (Vattam & Kolodner, [<reflink idref="bib34" id="ref41">34</reflink>]), the challenge of connecting and bridging engineering and science concepts and mechanisms. The understanding gap of how students' engineering design behaviors interact with their domain knowledge needs to be addressed. This requires understanding the temporal design and learning process (Li et al., [<reflink idref="bib22" id="ref42">22</reflink>]; Reimann, [<reflink idref="bib30" id="ref43">30</reflink>]). Some attempts have been made to address this issue. For instance, English and King, ([<reflink idref="bib14" id="ref44">14</reflink>]) studied how sixth graders applied STEM disciplinary knowledge when designing a building that can withstand an earthquake by qualitatively coding the transcripts of whole class discussions and focus group interviews. Purzer et al. ([<reflink idref="bib29" id="ref45">29</reflink>]) studied how secondary school students' design behaviors were associated with scientific explanations by coding and visualizing students' design decisions demonstrated in video playbacks of students' design process and electronic notes taken during the design. Chao et al. ([<reflink idref="bib8" id="ref46">8</reflink>]) examined how ninth graders learned science concepts through designing energy-saving houses by analyzing students' responses to a science test before and after the design.</p> <p>However, these studies tended to adopt post-hoc analysis after the design processes using qualitative coding or quantitative comparison. Qualitative coding that involves tedious work is not easy to scale, and does not enable timely feedback to support adaptive teaching or learning. Quantitative comparison between pre- and post-tests does not enable the examination of the design process. To address these issues, this study aimed to study students' design iterations and application of science concepts during the design process and examine the extent to which these factors and dynamics are related to students' design performance.</p> <hd id="AN0183971589-6">Conceptual Model</hd> <p>This study represents an attempt to explore the relationships between design behaviors and the application of science concepts in design iterations over time of K–12 students with different design performances. The model shown in Fig. 1 guides the conceptualization and analyses of the current study. This model builds on the learning with designing model (Purzer et al., [<reflink idref="bib29" id="ref47">29</reflink>]) but highlights multiple design iterations. As the model demonstrates, starting with design criteria/constraints, students engage in various numbers of design iterations to move towards successful design prototypes or products that meet the requirements. In each design iteration, students are expected to apply science knowledge and execute design behaviors to make knowledge-based decisions. The number of design iterations depends on available time, students' abilities and commitment, teacher facilitation, and other relevant factors. This study will extend our understanding of how design behaviors, application of science concepts, and design iterations are related to design performance and build steps towards how to scaffold students' engineering design just in time from the perspectives of their design behaviors and application of science concepts.</p> <p>Graph: Fig. 1 A model of students' design behaviors and application of science concepts in engineering design iterations</p> <hd id="AN0183971589-7">Methods</hd> <p></p> <hd id="AN0183971589-8">Participants</hd> <p>A total number of 99 grade 9 students who registered for the Science of Energy course in a northeastern public school in the USA participated in this study. However, 15 of them did not have valid design performance due to their behaviors in the CAD environment and thus were excluded from the analysis, leaving us a final sample size of 84. According to the school enrollment data, 53.4% of students were identified as White, 26.6% as African American, 9.5% as Hispanic, 6.9% as Asian, 3.1% as Multi-Race, Non-Hispanic, 0.2% as Native American, and 0.2% as Native Hawaiian, Pacific Islander; concerning gender, 48.24% of students in the school were female. The CAD software used in this study is like what professionals use in industry. Participants had not been exposed to such authentic CAD projects or tools before.</p> <hd id="AN0183971589-9">Research Design</hd> <p>The course was delivered by three teachers in three different classes. Through the course, students learned how atmospheric air masses, human actions, and global winds interact with land masses and ocean currents to determine regional climates. Solar science and energy was one of the course topics, discussing how the Sun's energy flows through a variety of interacting systems and why solar energy is the driving force of daily weather change on our planet. The learning activities we designed were authentic in terms of context, real-life scenarios (e.g., business and bidding models), hands-on CAD tools, and a school-building model overlaid on Google Maps.</p> <p>Energy3D is a simulated CAD environment for engineering design, supporting engineering and science education. It supports the design, construction, and analysis of buildings that use solar energy. Specifically, it provides simulations such as <emph>Show Shadow</emph>, <emph>Show Heliodon</emph> (i.e., a device that simulates the angle at which sunbeams strike a building or landscape), <emph>MonthlySunshineHours</emph>, and <emph>Change Monthly Tilt Angles</emph>, which are relevant to science concepts such as seasonal change of the Sun's path and projection effect. These simulations animate the Sun to help students learn about the design environment. In Energy3D, students started with a practice task (Fig. 2) to modify an example building by exploring functions related to the learning task, such as changing the roof, wall, window, and color of the building; viewing the Sun's path and the shadow of this building; adding solar panels to the roof; and analyzing annual electricity production of solar panels. They could also change the location of the house, add trees, and change the landscape.</p> <p>Graph: Fig. 2 The interface of Energy3D with an example building</p> <p>In this study, students were tasked with designing cost-effective solutions to transform their school building into a power generator (see Fig. 3) that (<reflink idref="bib1" id="ref48">1</reflink>) generates more than 400,000 kWh of electricity per year, (<reflink idref="bib2" id="ref49">2</reflink>) has a budget less than $800, 000, and (<reflink idref="bib3" id="ref50">3</reflink>) achieves a payback period of less than ten years. To achieve these requirements, students needed to choose from three different types of solar panels with various costs, sizes, and efficiency. It is worth mentioning that the school was not solarized when the study took place. When students launched the program, they were provided with a model of their school building (see the model screenshot in Fig. 3) which could inform their design project.</p> <p>Graph: Fig. 3 Solarize your school</p> <p>Regarding the procedures, the activities lasted for 7 days within 2 weeks. To study individual design behaviors and application of science concepts, we intentionally limited teachers' facilitation and students' collaboration. On the first day, the research team introduced this study. They helped students set up an account and test-run the program. Students started with a practice task (see Fig. 2) to explore various functions of Energy3D, for instance, how to change the roof, wall, window, and color of the building; view the Sun's path and the shadow of the building; add solar panels to the roof; and analyze daily and annual electricity production of solar panels. On the second and third day, students learned about science concepts such as the Sun's path, seasonal change of solar angles, projection effect, the effect of weather, and solar radiation in detail. Students were given opportunities to explore related functions in the CAD software. On the fourth day, students were provided with an opportunity to apply what they had learned to solve a real-world problem: find a position for a solar panel around their school building so that it generates the most electricity throughout the year. Worksheets, see an example in Table 1, were provided to guide students' exploration and learning. Over the last 3 days, students were asked to apply what they had learned from previous lessons to solarize their school building. All the data analyzed in this study was from the final task, solarizing their school.</p> <p>Table 1 A sample of the worksheet: daily change of solar angles</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" colspan="3"><p>Boston, MA; June 22</p></th></tr></thead><tbody><tr><td align="left"><p>Time</p></td><td align="left"><p>Zenith (°)</p></td><td align="left"><p>Azimuth (°)</p></td></tr><tr><td align="left"><p>6:00 AM</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>9:00 AM</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>12:00 PM</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>3:00 PM</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>6:00 PM</p></td><td align="left" /><td align="left" /></tr><tr><td align="left" colspan="3"><p>Can the Zenith angle be 0 on June 22 in Boston? If yes, when is it? If not, why?</p></td></tr><tr><td align="left" colspan="3"><p>Which direction does the Sun rise in the summer, exactly east, northeast, or southeast?</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0183971589-10">Data Collection</hd> <p></p> <hd id="AN0183971589-11">Design Behaviors</hd> <p>Referring to the data schema of Energy3D, two researchers familiar with the system discussed the similarities and roles of design behaviors, classifying them into five major categories: observing surroundings, building, editing, setting a value, and file, as shown in Table 2. Observing surroundings describes students viewing the structure of their design from different perspectives (e.g., general view, spin view, top view, and show axes) and displaying heliodon, showing windows, animating sun, and showing annotation. Building involves students constructing their school by adding and removing parts and elements such as walls, windows, and trees. Editing is about students modifying and refining their design, for example, changing the sizes or properties of floor, wall, window, and door. Setting a value consists of changing settings (e.g., date, time) to fine-tune the school parameters and compare the results. Filing includes behaviors of redo, undo, and save.</p> <p>Table 2 Sample design behaviors</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Categories</p></th><th align="left"><p>Design behaviors</p></th></tr></thead><tbody><tr><td align="left"><p>Observe surroundings</p></td><td align="left"><p>Camera, spin view, top view, zoom, show annotation</p></td></tr><tr><td align="left"><p>Building</p></td><td align="left"><p>Add human/wall/window/tree, remove wall/rack/tree/window</p></td></tr><tr><td align="left"><p>Editing</p></td><td align="left"><p>Edit human/window/wall/tree/rack/foundation</p></td></tr><tr><td align="left"><p>Setting a value</p></td><td align="left"><p>Change label of foundation, set solar panel model for all racks, set size for all racks</p></td></tr><tr><td align="left"><p>File</p></td><td align="left"><p>Save, undo, redo</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0183971589-12">Design Iterations</hd> <p>Students' design iterations were segmented by the measurement of annual energy (Zhang et al., [<reflink idref="bib38" id="ref51">38</reflink>]). We extracted students' log data and examined the frequency of their analysis behavior (i.e., <emph>PvAnnualAnalysis</emph>). When students selected this analysis function, the CAD tool would calculate and display the annual energy output of the current design. Students were encouraged to display the outputs after each change to compare whether the result improved compared to the previous version, as well as to make decisions based on the comparison rather than guessing. Two metrics were used in this study to measure students' design iterations: the number of design iterations and the length of each design iteration. A design iteration typically consists of all the activities between the two nearest uses of the <emph>PvAnnualAnalysis</emph> function, or from the beginning to the first use of <emph>PvAnnualAnalysis</emph>. In other words, the total number of design iterations by a student equals the total number of <emph>PvAnnualAnalysis</emph> uses. The length of a design iteration was defined as the total number of log activities within the iteration, including <emph>PvAnnualAnalysis</emph>. Design iteration length can be as short as 1, meaning the student selected the analyze function twice in a row and did nothing in between.</p> <hd id="AN0183971589-13">Science Concepts</hd> <p>The learning task was designed for students to explore how the Sun moves in the sky as the Earth orbits the Sun and rotates around its own axis. Specifically, students investigated how the Sun's path changes from season to season, how the length of the day varies, how the Sun's position relative to a surface affects the intensity of sunlight that shines on it, and why the intensity depends on the time of the day and the weather. Based on the learning materials, we identified four types of science-concepts-related items: seasonal change, projection effect, the effect of weather about insolation, and solar radiation pathways, as shown in Table 3. It is worth mentioning that we identified these log activities based on the CAD tool's functions. For instance, when students select <emph>Animate Sun</emph>, the tool will show how the Sun's path changes within a specific time window. But the animation will be different if students choose the function at other time windows.</p> <p>Table 3 Sample activities for each type of science concept</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Science concepts</p></th><th align="left"><p>Sample behaviors</p></th></tr></thead><tbody><tr><td align="left"><p>Seasonal change</p></td><td align="left"><p>Change monthly tilt angles, change monthly tilt angles for all racks, change time and date, show heliodon, animate sun</p></td></tr><tr><td align="left"><p>Projection effect</p></td><td align="left"><p>Temperature effects change for selected solar panel, sun beam, rotate building, adjust thermostat, horizontal/vertical single-axis tracker</p></td></tr><tr><td align="left"><p>Effect of weather about insolation</p></td><td align="left"><p>Change latitude, change city, monthly sunshine hours</p></td></tr><tr><td align="left"><p>Solar radiation pathways</p></td><td align="left"><p>Set texture for all walls on selected foundation, set texture for all roofs on selected foundation, land color change, add mirror</p></td></tr></tbody></table> </ephtml> </p> <p>To measure how students applied science concepts in their design iterations, three metrics were calculated: the percentage of science-concepts-related behaviors for each student (<emph>SC%</emph>), the percentage of design iterations involving science concepts for each student (<emph>SCDI%</emph>), and the average number of science concept types involved in each student's design iterations (<emph>SCType</emph>). For <emph>SCType</emph>, we first calculated the types of science concepts (a possible value is {0, 1, 2, 3, 4}) involved in each design iteration and then averaged the number of all design iterations for each student.</p> <hd id="AN0183971589-14">Design Performance</hd> <p>Students' design performance was retrieved through their log data, indicated by the highest annual energy generated by their designed solarized school. When students selected the analysis option (i.e., <emph>PvAnnualAnalysis</emph>), they were able to review the outputs to check whether their design met all the given design criteria and constraints. When students made changes, they were encouraged to compare the outputs after each change and make decisions on their next change based on the analysis results. A total number of 15 students whose design performance is unknown were excluded from this study: two of them did not leave any valid data, while the rest never analyzed their design. For the remaining 84 students, we identified three groups. The first group consisted of 28 students whose design met the requirements, and we labeled this group as the achieving group. To study the interplays of design behaviors, design iterations, and application of science concepts in scrutiny, we separated the 56 students whose design did not meet the requirements into another two groups by their medium output (i.e., 242,313.97): one group consisted of 28 students whose design output was higher than the medium, while another group included 28 students whose design output was lower than the medium output. We labeled these two groups as the proficient group and emerging group, respectively.</p> <hd id="AN0183971589-15">Data Analysis</hd> <p>To address our first research question about group differences in design iterations, we examined the total number of design iterations for each student in the three groups and conducted a one-way ANOVA test. To answer the second research question about group differences in the application of science concepts, we investigated students' design iteration length, the frequency of science concept use, and the percentage of design iterations with science concepts in the three groups. We fitted an ordinary least squares (OLS) regression model by considering performance as the dependent variable and the rest as independent variables. To address the third question about the dynamics between science concepts and design behaviors, we applied the Markov chain to examine the sequential transition between the application of science concepts and design behaviors among three groups and entropy analysis to study the distribution of design behaviors and application of science concepts. Next, we elaborate on Markov chain and entropy analysis.</p> <hd id="AN0183971589-16">Markov Chain</hd> <p>Markov chain can be used to capture the sequential relationships between students' design behaviors. It is a mathematical model that describes the transition from one state to another based on certain probabilistic rules. In other words, in a Markov chain model, the probability of each event to occur depends only on the previous state. No matter how the process comes to the current state, the possible next states are fixed. Markov-chain–based techniques have been used in educational research to make sequential recommendations on college course selection, to predict students' learning behaviors, to assess students' learning progress in intelligent tutoring systems, and to predict students' early dropout in the online learning environment (Beal & Cohen, [<reflink idref="bib6" id="ref52">6</reflink>]; Polyzou & Karypis, [<reflink idref="bib28" id="ref53">28</reflink>]). Mathematically, for states <emph>X</emph><subs><emph>1</emph></subs>, <emph>X</emph><subs><emph>2</emph></subs>, ..., <emph>X</emph><subs><emph>n</emph></subs>, the probability (<emph>Pr</emph>) of moving to the next state is:</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mi>r</mi><mfenced close=")" open="("><msub><mtext>X</mtext><mrow><mtext>n</mtext><mo>+</mo><mn>1</mn></mrow></msub><mo>=</mo><mi>x</mi><msub><mrow><mo stretchy="false">|</mo><mtext>X</mtext></mrow><mn>1</mn></msub><msub><mrow><mo>=</mo><msub><mi>x</mi><mn>1</mn></msub><mo>,</mo><mspace width="0.333333em" /><mtext>X</mtext></mrow><mn>2</mn></msub><mo>=</mo><msub><mi>x</mi><mn>2</mn></msub><mo>,</mo><mo>⋯</mo><mo>,</mo><msub><mtext>X</mtext><mtext>n</mtext></msub><mo>=</mo><msub><mi>x</mi><mtext>n</mtext></msub></mfenced><mo>=</mo><mi>P</mi><mi>r</mi><mfenced close=")" open="("><msub><mtext>X</mtext><mrow><mtext>n</mtext><mo>+</mo><mn>1</mn></mrow></msub><mrow><mo>=</mo><mi>x</mi><mo stretchy="false">|</mo></mrow><msub><mtext>X</mtext><mtext>n</mtext></msub><mo>=</mo><msub><mi>x</mi><mtext>n</mtext></msub></mfenced></mrow></math> </ephtml> </p> <p>Graph</p> <p>In this study, each log data point is considered a state. Based on the design behaviors and science concepts, there are nine states in total: namely five categories of design behaviors (i.e., observing surroundings, building, editing, setting a value, and file) and four types of science concepts (i.e., seasonal change, projection effect, the effect of weather about insolation, and solar radiation pathways). Markov chain is used to examine the transition patterns of the nine states of the three groups.</p> <hd id="AN0183971589-17">Entropy Analysis</hd> <p>Shannon's entropy analysis, which is also known as Shannon's entropy index, has been widely used in machine learning models such as decision trees. It has also been used in social science studies to measure the distribution of students' collaboration and discourse features (Matei et al., [<reflink idref="bib24" id="ref54">24</reflink>]; Zheng et al., [<reflink idref="bib39" id="ref55">39</reflink>]). In the context of information theory, entropy measures the "amount of information" or disorder (Shannon, [<reflink idref="bib32" id="ref56">32</reflink>]). In this study, entropy analysis can be used to calculate the distribution of students' design behaviors and their application of science concepts. Mathematically, for given events <emph>X</emph><subs><emph>1</emph></subs>, <emph>X</emph><subs><emph>2</emph></subs>,..., <emph>X</emph><subs><emph>n</emph></subs>, the entropy:</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>E</mi><mfenced close=")" open="("><mi>X</mi></mfenced><mo>=</mo><mo>-</mo><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msub><mi>p</mi><mi>i</mi></msub><mi>l</mi><mi>o</mi><mi>g</mi><mfenced close=")" open="("><msub><mi>p</mi><mi>i</mi></msub></mfenced></mrow></math> </ephtml> </p> <p>Graph</p> <p>In a data set with two classes, the entropy lies between 0 and 1. If all data in the data set has the same value, the entropy would be either 0 or 1. When the total number of events (i.e., <emph>n</emph>) increases, the range of entropy also changes. A larger entropy value indicates a higher level of disorder or uncertainty in the class distribution. In this study, there are nine different states in total, the entropy value may range from 0 to 3.17. The entropy value is used to examine the differences in students' design activities among the three groups.</p> <hd id="AN0183971589-18">Results</hd> <p></p> <hd id="AN0183971589-19">RQ1. Do Students with Different Design Performance Differ in Numbers of Design Interactions?</hd> <p>We retrieved students' log data and identified their application of science concepts and design behaviors. Figure 4 visualizes the distribution of science concepts and design behaviors of the three groups. For all three groups, the frequency of <emph>File</emph> and <emph>SoalRa</emph> is lower than 100, making them hard to be seen in the bar chart. Except for <emph>File</emph> and <emph>IsolWe</emph>, the achieving group had the highest frequencies of all other codes, followed by the proficient group, while the emerging group had the lowest frequencies of other science concepts and design behaviors. <emph>ProjEf</emph> was the most applied science concept in all three groups, while <emph>Observe</emph> was the most commonly used design behavior among the three groups.</p> <p>Graph: Fig. 4 Distribution of science concepts application and design behaviors (SeasCh, seasonal change; ProjEf, projection effect; IsolWe, the effect of weather about insolation; and SolaRa, solar radiation pathways)</p> <p>Descriptive statistics on the number of design iterations of the three groups are shown in Table 4. The minimum number of design iteration for a student is 1, while the maximum is 106; both are in the achieving group. The proficient and emerging groups shared similar ranges of the number of iteration and mean values. The one-way ANOVA showed that there was a statistically significant difference in the number of design iteration (<emph>F</emph> (<reflink idref="bib2" id="ref57">2</reflink>, 82) = 4.728, <emph>p</emph> = 0.01) across the three groups. The achieving group had the highest number of design iteration, followed by the emerging group, and last, the proficient group. To summarize, on average, students in the achieving group had significantly more design iterations than students in the other two groups.</p> <p>Table 4 Descriptive statistics on the number of design iteration for three groups</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Group</p></th><th align="left"><p>Range</p></th><th align="left"><p>Mean</p></th><th align="left"><p><italic>Std</italic></p></th></tr></thead><tbody><tr><td align="left"><p>Achieving (<italic>N</italic> = 28)</p></td><td align="left"><p>[1, 106]</p></td><td char="." align="char"><p>33.50</p></td><td char="." align="char"><p>21.47</p></td></tr><tr><td align="left"><p>Proficient (<italic>N</italic> = 28)</p></td><td align="left"><p>[5, 61]</p></td><td char="." align="char"><p>21.00</p></td><td char="." align="char"><p>13.84</p></td></tr><tr><td align="left"><p>Emerging (<italic>N</italic> = 28)</p></td><td align="left"><p>[2, 59]</p></td><td char="." align="char"><p>20.18</p></td><td char="." align="char"><p>18.37</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0183971589-20">RQ.2 Do Different Design Performance Groups Differ in Their Application of Science Concepts a...</hd> <p>To answer the second research question concerning the application of science concepts of the three groups, we investigated the following variables: <emph>DIL</emph> (i.e., design iteration length), <emph>SC%</emph> (i.e., the percentage of science-concepts-related behaviors for each student), <emph>SCDI%</emph> (i.e., the percentage of design iterations with science concepts for each student), and <emph>SCType</emph> (i.e., the types of science concepts involved in each student's design), and applied these variables to fit an OLS regression model using performance as the independent variable.</p> <p>Descriptive statistics of students' design iteration length are shown in Table 5, the longest design iteration has 2579 log activities, which was produced by the student who had only one design iteration. In other words, one student in the achieving group had only one extremely long design iteration consisting of 2579 log records. Interestingly, the proficient group has longer design iterations than the other two groups. On average, each design iteration in the proficient group has approximately 50% more log activities than the other two groups. The results of one-way ANOVA showed that there was a statistically significant difference in design iteration length (<emph>F</emph> (<reflink idref="bib2" id="ref58">2</reflink>, 97) = 3.461, <emph>p</emph> = 0.03) among the three groups. Interestingly, the achieving group has the shortest design iteration length on average and used analysis tools more often than the other two groups as suggested by their number of design iterations. Considering the design iterations of the achieving group were shorter, the students in the group seemed to understand the importance of analysis and design iterations and engage in intensive design iterations. Meanwhile, there was no statistically significant difference in students' <emph>SC%</emph> (<emph>F</emph> (<reflink idref="bib2" id="ref59">2</reflink>, 97) = 0.180, <emph>p</emph> = 0.84), <emph>SCDI%</emph> (<emph>F</emph> (<reflink idref="bib2" id="ref60">2</reflink>, 97) = 1.499, <emph>p</emph> = 0.23), or <emph>SCType</emph> (<emph>F</emph> (<reflink idref="bib2" id="ref61">2</reflink>, 97) = 0.037, <emph>p</emph> = 0.963) among the three groups.</p> <p>Table 5 Descriptive statistics of design iteration length for three groups</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Group</p></th><th align="left"><p>Range</p></th><th align="left"><p>Mean</p></th><th align="left"><p><italic>Std</italic></p></th><th align="left"><p>Total number of design iterations</p></th></tr></thead><tbody><tr><td align="left"><p>Achieving (<italic>N</italic> = 28)</p></td><td align="left"><p>[1, 2579]</p></td><td char="." align="char"><p>66.07</p></td><td char="." align="char"><p>194.90</p></td><td align="left"><p>938</p></td></tr><tr><td align="left"><p>Proficient (<italic>N</italic> = 28)</p></td><td align="left"><p>[1, 1783]</p></td><td char="." align="char"><p>94.27</p></td><td char="." align="char"><p>224.54</p></td><td align="left"><p>565</p></td></tr><tr><td align="left"><p>Emerging (<italic>N</italic> = 28)</p></td><td align="left"><p>[1, 2389]</p></td><td char="." align="char"><p>69.79</p></td><td char="." align="char"><p>222.08</p></td><td align="left"><p>588</p></td></tr></tbody></table> </ephtml> </p> <p>We fitted a robust OLS regression model using students' performance as the independent variable and other variables (i.e., the number of design iterations, <emph>SC%</emph>, <emph>SCDI%</emph>, and <emph>SCType</emph>) as dependent variables. The variable <emph>DIL</emph> was removed from the final model due to multicollinearity. The results are shown in Table 6. The overall regression was statistically significant (<emph>R</emph><sups><emph>2</emph></sups> = 0.119, <emph>F</emph> (<reflink idref="bib4" id="ref62">4</reflink>, 79) = 3,373, <emph>p</emph> = 0.013). Among the four variables, the number of design iterations significantly predicted students' design performance.</p> <p>Table 6 Coefficients for regressing students' design performance</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Explainable variables</p></th><th align="left"><p>Estimates</p></th><th align="left"><p><italic>SE</italic></p></th><th align="left"><p><italic>p</italic>-value</p></th></tr></thead><tbody><tr><td align="left"><p>Number of design iterations</p></td><td char="." align="char"><p>0.012</p></td><td char="." align="char"><p>0.005</p></td><td char="." align="char"><p>0.015</p></td></tr><tr><td align="left"><p><italic>SC%</italic></p></td><td char="." align="char"><p>0.214</p></td><td char="." align="char"><p>0.650</p></td><td char="." align="char"><p>0.743</p></td></tr><tr><td align="left"><p><italic>SCDI%</italic></p></td><td char="." align="char"><p>1.169</p></td><td char="." align="char"><p>0.782</p></td><td char="." align="char"><p>0.139</p></td></tr><tr><td align="left"><p><italic>SCType</italic></p></td><td char="." align="char"><p>−0.299</p></td><td char="." align="char"><p>0.477</p></td><td char="." align="char"><p>0.533</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0183971589-21">RQ 3. What Are the Dynamics Between Science Concepts and Design Behaviors in Design Interacti...</hd> <p>To address the third question, we applied Markov chain and entropy analysis to study the transition between the use of science concepts and design behaviors, as well as the distribution of these activities among the three groups. Since the application of the analysis tool was used to segment design iterations, analysis was excluded from the Markov chain and entropy analysis. The entropy values for the achieving, proficient, and emerging groups are 1.34, 1.35, and 1.31, respectively. To our surprise, the distribution of nine codes among the three groups is similar.</p> <p>Then, we applied Markov chain to calculate the state transition between science concepts and design behaviors. We set out to understand how students' application of science concepts interacts with their designs, hence, excluded observing surroundings from Markov chain analysis because observation does not change design artifacts nor imply science concept application. The results are shown in Fig. 5. Yellow nodes are design behaviors, while blue nodes are science concepts. For easy reference purposes, we only visualized the transition probability greater than 10%.</p> <p>Graph: Fig. 5 (a), (b), and (c) Log activities transition of the three groups. Transition is displayed only if its probability is greater than 0.1 (SR, solar radiation pathways; SV, set a value; PE, projection effect; IW, effect of weather about insolation; SC, seasonal change)</p> <p>Overall, the three groups shared most of the same frequent transitions. The self-loops of all states in all three groups suggested frequent repetitive activities, which is anticipated since engineering design is an iterative process. Within each design iteration, individuals tend to take multiple behaviors to complete their intended design. For instance, individuals may need to adjust the positions and angles of a rack multiple times before conducting an analysis. In relation to the use of science concepts, all three groups had a frequent transition from the effect of IW (i.e., weather about insolation) to SC (seasonal change). Students of all three groups seemed to associate these two concepts (i.e., seasonal changes of the sun's path, the effect of weather and isolation) more often in comparison with the other two (i.e., solar radiation pathways and projection effect).</p> <p>The three groups differed in terms of how science concepts related to others. In the achieving group, all science concepts were related to other science concepts or design behaviors, while a few science concepts were isolated in the proficient and emerging groups. The results indicated that there were more explicit and salient transition patterns between science concepts and design behaviors and between different science concepts in the achieving group. However, for the proficient group, the use of seasonal change and the effect of weather on insolation were isolated from the other design behaviors and science concepts. Meanwhile, the use of the projection effect has no frequent connection with others for the emerging group. We believe that the achieving group was able to manipulate various science concepts in their design and frequently transit between science concepts and design behaviors. The other two groups, however, seemed to have an impasse in applying different types of science concepts.</p> <hd id="AN0183971589-22">Discussion</hd> <p>This study set out to explore the relationships among design iterations, application of science concepts, and design behaviors of students with different design performances in a CAD engineering design learning environment, Energy3D. We retrieved students' log data and identified three different groups based on their performance: the achieving, proficient, and emerging groups. Out of the log, we calculated multiple variables such as the number of design iterations, the length of each design iteration, the percentage of science-concepts-related behaviors for each student, the percentage of design iterations with science concepts for each student, and the types of science concepts used by each student. We conducted a one-way ANOVA test and fitted an OLS model to examine the relationship between these variables and students' design performance. Finally, we applied Markov chain and entropy analysis to explore how students transit between science concepts and design behaviors. A few interesting findings are worth further discussion.</p> <p>First, there is a statistically significant difference in the number of design iterations and design iteration length among the three design performance groups, but no significant differences in science-concept–related variables were found among the three groups. These findings on the one hand, confirm the importance of design iteration emphasized in previous studies (e.g., Arık & Topçu, [<reflink idref="bib2" id="ref63">2</reflink>]; Marks & Chase, [<reflink idref="bib23" id="ref64">23</reflink>]) and, on the other hand, extend previous studies by exploring the possibility of using the number of design iterations to predict design performance when developing prediction models of students' engineering design. This finding is critical especially in contexts when it is impossible for teachers to provide timely feedback to a big class of students (Zheng et al., [<reflink idref="bib39" id="ref65">39</reflink>]).</p> <p>What is worth mentioning is that more log activities do not necessarily indicate longer design iterations, and longer design iterations may not lead to better design performance. Although the achieving group has 16% more log activities than the proficient group, the mean length of their design iteration is shorter than that of the proficient group (66.07 versus 94.27). These results indicate that instead of supporting students in designing without clear ideas or goals—relying on trial and error, which can lead to extended processes and not necessarily better design performance—it is critical to help them understand and reflect on how well they are designing and what steps to take next (Edwards, [<reflink idref="bib13" id="ref66">13</reflink>]) during the design process.</p> <p>Second, the three groups had similar distributions in their application of science concepts and design behaviors, as well as some transition patterns between science concepts and design behaviors. Students of all three groups frequently used two science concepts sequentially, namely seasonal change of the Sun's path and the effect of weather on isolation, but differed in the use of projection effect and solar radiation pathways. We speculate that seasonal changes in the Sun's path and the effect of weather on isolation were commonly used when students, for instance, decided how to place the direction and angle of solar panels to maximize energy generation considering the four seasons and weather. However, the projection effect and solar radiation, which concern daily change of solar angles, the effect of air mass, and how land surface affects the solar radiation pathways, were more complex to understand and use. The use of the projection effect requires students to think about how to adjust solar panels daily, while the use of solar radiation pathways requires understanding how wall textures affect the solar radiation pathways and the generated power. These two concepts are less directly related to the installation of solar panels and more focused on improving their efficiency. These findings align with previous research which suggests the challenges and importance of integrating science concepts and engineering processes in engineering education (Dorie et al., [<reflink idref="bib10" id="ref67">10</reflink>]; English & King, [<reflink idref="bib14" id="ref68">14</reflink>]; Wendell et al., [<reflink idref="bib35" id="ref69">35</reflink>]). Addressing these issues requires teachers' knowledge and effort to design problem situations in which students can learn from and about problems, apply, reflect on, and reshape their prior knowledge and experiences (Crismond & Adams, [<reflink idref="bib9" id="ref70">9</reflink>]) with the support of technical tools when needed (e.g., digital note booking tool in Wendell et al. ([<reflink idref="bib35" id="ref71">35</reflink>]), simulated environment for engineering design in Chao et al., ([<reflink idref="bib8" id="ref72">8</reflink>])).</p> <p>We found that science-concept–relevant variables (i.e., the percentage of science-concepts-related behaviors for each student, the percentage of design iterations with science concepts for each student, the types of science concepts involved in each student's design) did not significantly predict design performance, but the three groups displayed different transition patterns between science concepts and design behaviors. The results indicate that compared with how frequently students used science concepts or whether they used all science concepts, how they used the concepts and in what ways and sequences matter more. Although being novice designers, the achieving group could manipulate different types of science concepts in their design, whereas the other two groups lacked the overall understanding of the connection between science concepts. This result extends previous research on the difference between expert and novice designers in engineering design (e.g., Ball et al., [<reflink idref="bib5" id="ref73">5</reflink>]; Lammi et al., [<reflink idref="bib21" id="ref74">21</reflink>]; Mentzer, [<reflink idref="bib25" id="ref75">25</reflink>]) by studying the application of science concepts derived from design log data. This result also suggests that when designing timely feedback for students during the design process, educators and researchers shall consider both individual temporal dimensions (e.g., design iteration) and between group differences (e.g., what behavior patterns do more achieving students display in the same design task).</p> <p>This study has both theoretical and practical implications. Theoretically, we extended qualitative analysis of design iterations by segmenting students' design iterations based on their analysis behavior and researched how design iteration and science-concept–relevant variables are related to design performance. Practically, the findings on the relations between design iterations and science concepts and the sequence of science concepts among different design performance groups provide implications regarding how to design learning analytics to scaffold students' engineering design by providing just-in-time feedback (e.g., reminding students when they engage in lengthy design without iterations, fail to apply some science concepts, or do not employ the concepts in connected ways). These results can also inform educators to design instructions to facilitate students to better integrate science and engineering knowledge during design iterations.</p> <p>There are several limitations. First, in our design, teachers' support and students' collaboration were intentionally limited so that we could study individual design behaviors and application of science concepts in Energy3D. Examining the impact of instructors' instruction or peer collaboration was beyond the scope of this work. However, it should be noted that teachers' instructional support and peer collaboration are critical in supporting students' engineering design in schools, which are also important future research directions. Second, our findings are based on the specific version of Energy3D used in 2019 and the log data of 84 grade 9 students recruited in a northeastern public school in the USA. Updates or new features of Energy3D, especially changes to the interface, may alter users' behaviors, leading to somewhat different learning pathways. A different group of students with different prior science knowledge and learning motivation, commitment, and engagement could also produce different results. Generalizing the findings to other versions of the tool or other learning environments or different samples needs to be cautious. Third, our analysis focuses on the differences among different performance groups and may overlook the dynamics in each student's design process. A qualitative study of the sequence of design may bring new perspectives to our understanding of design iterations.</p> <hd id="AN0183971589-23">Funding</hd> <p>This project is supported by the National Science Foundation (NSF) under grant numbers #2105695, #2131097, and #2301164. Any opinions, findings, and conclusions or recommendations expressed in this material, however, are those of the authors and do not necessarily reflect the views of NSF.</p> <hd id="AN0183971589-24">Data Availability</hd> <p>The datasets generated during and/or analyzed during the current study are not publicly available due to the participants' consent but are available from the corresponding author on reasonable request.</p> <hd id="AN0183971589-25">Declarations</hd> <p></p> <hd id="AN0183971589-26">Conflict of Interest</hd> <p>The authors declare no competing interests.</p> <hd id="AN0183971589-27">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0183971589-28"> <title> References </title> <blist> <bibl id="bib1" idref="ref29" type="bt">1</bibl> <bibtext> Andrews, C. (2016). Failure and idea evolution in an elementary engineering workshop (fundamental). 2016 ASEE Annual Conference & Exposition. https://peer. asee. org/26895</bibtext> </blist> <blist> <bibl id="bib2" idref="ref8" type="bt">2</bibl> <bibtext> Arık M, Topçu MS. Implementation of engineering design process in the K-12 science classrooms: Trends and issues. Research in Science Education. 2020. 10.1007/s11165-019-09912-x</bibtext> </blist> <blist> <bibl id="bib3" idref="ref35" type="bt">3</bibl> <bibtext> Atman CJ, Chimka JR, Bursic KM, Nachtmann HL. A comparison of freshman and senior engineering design processes. Design Studies. 1999; 20; 2: 131-152. 10.1016/S0142-694X(98)00031-3</bibtext> </blist> <blist> <bibl id="bib4" idref="ref36" type="bt">4</bibl> <bibtext> Atman CJ, Adams RS, Cardella ME, Turns J, Mosborg S, Saleem J. Engineering design processes: A comparison of students and expert practitioners. Journal of Engineering Education. 2007; 96; 4: 359-379. 10.1002/j.2168-9830.2007.tb00945.x</bibtext> </blist> <blist> <bibl id="bib5" idref="ref9" type="bt">5</bibl> <bibtext> Ball LJ, Ormerod TC, Morley NJ. Spontaneous analogising in engineering design: A comparative analysis of experts and novices. Design Studies. 2004; 25; 5: 495-508. 10.1016/j.destud.2004.05.004</bibtext> </blist> <blist> <bibl id="bib6" idref="ref52" type="bt">6</bibl> <bibtext> Beal, C. R, & Cohen, P. R. (2008). Temporal data mining for educational applications. In PRICAI 2008: Trends in artificial intelligence: 10th Pacific Rim International Conference on Artificial Intelligence, Hanoi, Vietnam, December 15-19, 2008. Proceedings 10 (pp. 66–77). Springer Berlin Heidelberg.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref4" type="bt">7</bibl> <bibtext> Berland L, Steingut R, Ko P. High school student perceptions of the utility of the engineering design process: Creating opportunities to engage in engineering practices and apply math and science content. Journal of Science Education and Technology. 2014; 23; 6: 705-720. 10.1007/s10956-014-9498-4</bibtext> </blist> <blist> <bibl id="bib8" idref="ref13" type="bt">8</bibl> <bibtext> Chao J, Xie C, Nourian S, Chen G, Bailey S, Goldstein MH, Purzer S, Adams RS, Tutwiler MS. Bridging the design-science gap with tools: Science learning and design behaviors in a simulated environment for engineering design. Journal of Research in Science Teaching. 2017; 54; 8: 1049-1096</bibtext> </blist> <blist> <bibl id="bib9" idref="ref70" type="bt">9</bibl> <bibtext> Crismond DP, Adams RS. The informed design teaching and learning matrix. Journal of Engineering Education. 2012; 101; 4: 738-797. 10.1002/j.2168-9830.2012.tb01127.x</bibtext> </blist> <blist> <bibtext> Dorie, B. L, Cardella, M, & Svarovsky, G. N. (2014). Capturing the design thinking of young children interacting with a parent. School of Engineering Education Graduate Student Series, Purdue University. Paper 52. <ulink href="http://docs.lib.purdue.edu/enegs/52">http://docs.lib.purdue.edu/enegs/52</ulink></bibtext> </blist> <blist> <bibtext> Dow, S. P, Heddleston, K, & Klemmer, S. R. (2009). The efficacy of prototyping under time constraints. In Proceedings of the seventh ACM conference on Creativity and cognition (pp. 165–174). Association for Computing Machinery. https://doi.org/10.1145/1640233.1640260</bibtext> </blist> <blist> <bibtext> Dym CL, Agogino AM, Eris O, Frey DD, Leifer LJ. Engineering design thinking, teaching, and learning. Journal of Engineering Education. 2005; 94; 1: 103-120. 10.1002/j.2168-9830.2005.tb00832.x</bibtext> </blist> <blist> <bibtext> Edwards, S. H. (2004). Using software testing to move students from trial-and-error to reflection-in-action. In Proceedings of the 35th SIGCSE technical symposium on Computer science education (pp. 26–30). Association for Computing Machinery. https://doi.org/10.1145/971300.971312</bibtext> </blist> <blist> <bibtext> English LD, King D. STEM integration in sixth grade: Desligning and constructing paper bridges. International Journal of Science and Mathematics Education. 2018; 17: 863-884. 10.1007/s10763-018-9912-0</bibtext> </blist> <blist> <bibtext> English L, Hudson P, Dawes L. Engineering-based problem solving in the middle school: Design and construction with simple machines. Journal of Pre-College Engineering Education Research. 2013; 3; 2: 1-13</bibtext> </blist> <blist> <bibtext> Hartmann, B, Klemmer, S. R, & Bernstein, M. (2006). Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of the 19th annual ACM symposium on User interface software and technology (pp. 299–308). Association for Computing Machinery. https://doi.org/10.1145/1166253.1166300</bibtext> </blist> <blist> <bibtext> Hjalmarson, M. A, & Cardella, M. (2020). Uncertainty and iteration in design tasks for engineering students. Foundations for the future in mathematics Education (pp. 409–429). Routledge</bibtext> </blist> <blist> <bibtext> Hmelo CE, Holton DL, Kolodner JL. Designing to learn about complex systems. Journal of the Learning Sciences. 2000; 9; 3: 247-298. 10.1207/S15327809JLS0903_2</bibtext> </blist> <blist> <bibtext> Khalaf K, Balawi S, Hitt GW, Radaideh A. Engineering design education: When, what, and how. Advances in Engineering Education. 2013; 3; 3: n3</bibtext> </blist> <blist> <bibtext> Kolodner JL, Camp PJ, Crismond D, Fasse B, Gray J, Holbrook J, Puntambekar S, Ryan M. Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design(tm) into practice. Journal of the Learning Sciences. 2003; 12; 4: 495-547. 10.1207/S15327809JLS1204_2</bibtext> </blist> <blist> <bibtext> Lammi, M. D, Denson, C, Wells, J, & Paretti, M. (2014). Initial findings of high school pre-engineering and non-engineering students' design cognition. In 2014 IEEE Frontiers Proceedings (pp. 1–4). https://doi.org/10.1109/FIE.2014.7044313</bibtext> </blist> <blist> <bibtext> Li S, Du H, Xing W, Zheng J, Chen G, Xie C. Examining temporal dynamics of self-regulated learning behaviors in STEM learning: A network approach. Computers & Education. 2020; 158: 103987</bibtext> </blist> <blist> <bibtext> Marks J, Chase CC. Impact of a prototyping intervention on middle school students' iterative practices and reactions to failure. Journal of Engineering Education. 2019; 108; 4: 547-573. 10.1002/jee.20294</bibtext> </blist> <blist> <bibtext> Matei, S. A, Bruno, R, & Morris, P. L. (2015). Visible effort: Visualizing and measuring group structuration through social entropy. In Transparency in Social Media: Tools, Methods and Algorithms for Mediating Online Interactions (pp. 109–123).</bibtext> </blist> <blist> <bibtext> Mentzer N. High school student information access and engineering design performance. Journal of Pre-College Engineering Education (J-PEER). 2014; 4; 1: 4</bibtext> </blist> <blist> <bibtext> Moore, T. J, Glancy, A. W, Tank, K. M, Kersten, J. A, Smith, K. A, & Stohlmann, M. S. (2014). A framework for quality K-12 engineering education: Research and development. Journal of Pre-College Engineering Education Research (J-PEER), 4(1), 2. https://doi.org/10.7771/2157-9288.1069</bibtext> </blist> <blist> <bibtext> National Academy of Engineering. (2019). Engineering the Future: 2019 Annual Report. NAE. https://<ulink href="http://www.nae.edu/237786/NAE-Annual-Report-2019">www.nae.edu/237786/NAE-Annual-Report-2019</ulink></bibtext> </blist> <blist> <bibtext> Polyzou A, Karypis G. Feature extraction for next-term prediction of poor student performance. IEEE Transactions on Learning Technologies. 2019; 12; 2: 237-248</bibtext> </blist> <blist> <bibtext> Purzer Ş, Goldstein MH, Adams RS, Xie C, Nourian S. An exploratory study of informed engineering design behaviors associated with scientific explanations. International Journal of STEM Education. 2015; 2: 1-12</bibtext> </blist> <blist> <bibtext> Reimann P. Time is precious: Variable-and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning. 2009; 4: 239-257</bibtext> </blist> <blist> <bibtext> Schön, D. A. (2017). The reflective practitioner: How professionals think in action. Routledge. https://doi.org/10.4324/9781315237473</bibtext> </blist> <blist> <bibtext> Shannon CE. A mathematical theory of communication. The Bell System Technical Journal. 1948; 27; 3: 379-423</bibtext> </blist> <blist> <bibtext> The Accreditation Board for Engineering and Technology (1998). "Engineering criteria 2000", 3rd edition, https://user.eng.umd.edu/~zhang/414_97/abet.html</bibtext> </blist> <blist> <bibtext> Vattam SS, Kolodner JL. On foundations of technological support for addressing challenges facing design-based science learning. Pragmatics & Cognition. 2008; 16; 2: 406-437. 10.1075/pc.16.2.08vat</bibtext> </blist> <blist> <bibtext> Wendell KB, Andrews CJ, Paugh P. Supporting knowledge construction in elementary engineering design. Science Education. 2019; 103; 4: 952-978</bibtext> </blist> <blist> <bibtext> Wynn, D. C, Eckert, C. M, & Clarkson, P. J. (2007). Modelling iteration in engineering design. In DS 42: Proceedings of ICED 2007, the 16th International Conference on Engineering Design, Paris, France (pp. 693–694).</bibtext> </blist> <blist> <bibtext> Yen, Y. C. G, Dow, S. P, Gerber, E, & Bailey, B. P. (2017). Listen to others, listen to yourself: Combining feedback review and reflection to improve iterative design. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (pp. 158–170). Association for Computing Machinery. https://doi.org/10.1145/3059454.3059468</bibtext> </blist> <blist> <bibtext> Zhang HZ, Xie C, Nourian S. Are their designs iterative or fixated? Investigating design patterns from student digital footprints in computer-aided design software. International Journal of Technology and Design Education. 2018; 28: 819-841</bibtext> </blist> <blist> <bibtext> Zheng J, Xing W, Zhu G, Chen G, Zhao H, Xie C. Profiling self-regulation behaviors in STEM learning of engineering design. Computers & Education. 2020; 143: 103669</bibtext> </blist> <blist> <bibtext> Zhu, G, Zeng, Y, Xing, W, Du, H, & Xie, C. (2021). Reciprocal relations between students' evaluation, reformulation behaviors, and engineering design performance over time. Journal of Science Education and Technology, 30, 595–607. https://doi.org/10.1007/s10956-021-09906-3</bibtext> </blist> </ref> <aug> <p>By Hanxiang Du; Gaoxia Zhu; Wanli Xing and Charles Xie</p> <p>Reported by Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib27" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib19" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib33" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib12" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib38" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib21" firstref="ref10"></nolink> <nolink nlid="nl7" bibid="bib25" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib34" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib29" firstref="ref14"></nolink> <nolink nlid="nl10" bibid="bib17" firstref="ref16"></nolink> <nolink nlid="nl11" bibid="bib36" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib16" firstref="ref19"></nolink> <nolink nlid="nl13" bibid="bib23" firstref="ref20"></nolink> <nolink nlid="nl14" bibid="bib37" firstref="ref21"></nolink> <nolink nlid="nl15" bibid="bib40" firstref="ref23"></nolink> <nolink nlid="nl16" bibid="bib26" firstref="ref25"></nolink> <nolink nlid="nl17" bibid="bib22" firstref="ref26"></nolink> <nolink nlid="nl18" bibid="bib20" firstref="ref28"></nolink> <nolink nlid="nl19" bibid="bib11" firstref="ref30"></nolink> <nolink nlid="nl20" bibid="bib15" firstref="ref31"></nolink> <nolink nlid="nl21" bibid="bib18" firstref="ref33"></nolink> <nolink nlid="nl22" bibid="bib31" firstref="ref39"></nolink> <nolink nlid="nl23" bibid="bib30" firstref="ref43"></nolink> <nolink nlid="nl24" bibid="bib14" firstref="ref44"></nolink> <nolink nlid="nl25" bibid="bib28" firstref="ref53"></nolink> <nolink nlid="nl26" bibid="bib24" firstref="ref54"></nolink> <nolink nlid="nl27" bibid="bib39" firstref="ref55"></nolink> <nolink nlid="nl28" bibid="bib32" firstref="ref56"></nolink> <nolink nlid="nl29" bibid="bib13" firstref="ref66"></nolink> <nolink nlid="nl30" bibid="bib10" firstref="ref67"></nolink> <nolink nlid="nl31" bibid="bib35" firstref="ref69"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1463742
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: What Distinguishes Students' Engineering Design Performance: Design Behaviors, Design Iterations, and Application of Science Concepts
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Hanxiang+Du%22">Hanxiang Du</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-9081-0706">0000-0002-9081-0706</externalLink>)<br /><searchLink fieldCode="AR" term="%22Gaoxia+Zhu%22">Gaoxia Zhu</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-4589-0775">0000-0003-4589-0775</externalLink>)<br /><searchLink fieldCode="AR" term="%22Wanli+Xing%22">Wanli Xing</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-1446-889X">0000-0002-1446-889X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Charles+Xie%22">Charles Xie</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Science+Education+and+Technology%22"><i>Journal of Science Education and Technology</i></searchLink>. 2025 34(2):314-326.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 13
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 2105695<br />2131097
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Engineering%22">Engineering</searchLink><br /><searchLink fieldCode="DE" term="%22Design%22">Design</searchLink><br /><searchLink fieldCode="DE" term="%22Scientific+Concepts%22">Scientific Concepts</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+Solving%22">Problem Solving</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Energy%22">Energy</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Design%22">Computer Assisted Design</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Software%22">Computer Software</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s10956-024-10184-y
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1059-0145<br />1573-1839
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Engineering design that requires mathematical analysis, scientific understanding, and technology is critical for preparing students for solving engineering problems. In simulated design environments, students are expected to learn about science and engineering through their design. However, there is a lack of understanding concerning linking science concepts with design problems to design artifacts. This study investigated how 99 high school students applied science concepts to solarize their school using a computer-aided engineering design software, aiming to explore the interaction between students' science concepts and engineering design behaviors. Students were assigned to three groups based on their design performance: the achieving group, proficient group, and emerging group. By mining log activities, we explored the interactions among students' application of science concepts, engineering design behaviors, design iterations, and their design performance. We found that the achieving group has a statistically higher number of design iterations than the other two performance groups. We also identified distinctive transition patterns in students' applying science concepts and exercising design behaviors among three groups. The implications of this study are then discussed.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1463742
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1463742
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10956-024-10184-y
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 314
    Subjects:
      – SubjectFull: Engineering
        Type: general
      – SubjectFull: Design
        Type: general
      – SubjectFull: Scientific Concepts
        Type: general
      – SubjectFull: Problem Solving
        Type: general
      – SubjectFull: High School Students
        Type: general
      – SubjectFull: Energy
        Type: general
      – SubjectFull: Computer Assisted Design
        Type: general
      – SubjectFull: Computer Software
        Type: general
    Titles:
      – TitleFull: What Distinguishes Students' Engineering Design Performance: Design Behaviors, Design Iterations, and Application of Science Concepts
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hanxiang Du
      – PersonEntity:
          Name:
            NameFull: Gaoxia Zhu
      – PersonEntity:
          Name:
            NameFull: Wanli Xing
      – PersonEntity:
          Name:
            NameFull: Charles Xie
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1059-0145
            – Type: issn-electronic
              Value: 1573-1839
          Numbering:
            – Type: volume
              Value: 34
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Journal of Science Education and Technology
              Type: main
ResultId 1