View in EDS HTML Full Text PDF Full Text

Measuring Pedagogy and the Integration of Engineering Design in STEM Classrooms

Saved in:
Bibliographic Details
Title: Measuring Pedagogy and the Integration of Engineering Design in STEM Classrooms
Language: English
Authors: Williams, Tory (ORCID 0000-0002-3600-1174), Singer, Jonathan, Krikorian, Jacqueline, Rakes, Christopher, Ross, Julia
Source: Journal of Science Education and Technology. Jun 2019 28(3):179-194.
Availability: Springer. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 16
Publication Date: 2019
Sponsoring Agency: National Science Foundation (NSF)
Contract Number: DRL1418183
Document Type: Journal Articles
Reports - Research
Education Level: Secondary Education
Descriptors: Engineering Education, Design, STEM Education, Secondary School Science, Teacher Student Relationship, Scientific Concepts, Faculty Development, Educational Change, Science Education, Active Learning, Inquiry
DOI: 10.1007/s10956-018-9756-y
ISSN: 1059-0145
Abstract: The present study examined changes in high school biology and technology education pedagogy during the first year of a three-year professional development (PD) program using the INSPIRES educative curriculum. The Next Generation Science Standards (NGSS) calls for the integration of science and engineering through inquiry-based pedagogy that shifts the burden of thinking from the teacher to the student. This call is especially challenging for teachers untrained in inquiry teaching and engineering or science concepts. The INSPIRES educative curriculum materials and PD provided a mechanism for teachers to transform their teaching to meet the NGSS challenges. This study followed a longitudinal triangulation mixed methods design. Selected lessons were video recorded, scored on the Reformed Teaching Observation Protocol (RTOP) rubric, and examined for qualitative trends. Year 1 results indicated that teachers had begun to transform their teaching and pointed to particular lessons within the INSPIRES curriculum that most facilitated the reform. Instructional practices of participants improved significantly as a result of the INSPIRES PD program and also aligned with previous, similar studies. These findings provide insights for rethinking the structure of professional development, particularly in the integrated use of an educative curriculum aligned with intended professional development goals.
Abstractor: As Provided
Entry Date: 2019
Accession Number: EJ1212690
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGStI9NpFnObkKMytKmjG4XAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDBQo1RZOgrEAlSyHkwIBEICBm7LagwWlEJ5JZ4aqKYWEMOiTEYagi1hX609NIeOsOQs4qHWk_D-wTwWHQ26-trm2f5L4gpJSFMfyMzehkQgPXLPgCkRcekTELlZ14-0s2cZYvyEaY3RRyULBwXfXDp1PJI3l-0tEtBHJoZBBiyLApFtC6QOhxb-9J5ZF6lUm2764QW-FnzNDPqpZCid5XLyVYq-XH220kzqDvz9v
Text:
  Availability: 1
  Value: <anid>AN0135909703;4n601jun.19;2019Apr18.05:04;v2.2.500</anid> <title id="AN0135909703-1">Measuring Pedagogy and the Integration of Engineering Design in STEM Classrooms </title> <p>The present study examined changes in high school biology and technology education pedagogy during the first year of a three-year professional development (PD) program using the INSPIRES educative curriculum. The Next Generation Science Standards (NGSS) calls for the integration of science and engineering through inquiry-based pedagogy that shifts the burden of thinking from the teacher to the student. This call is especially challenging for teachers untrained in inquiry teaching and engineering or science concepts. The INSPIRES educative curriculum materials and PD provided a mechanism for teachers to transform their teaching to meet the NGSS challenges. This study followed a longitudinal triangulation mixed methods design. Selected lessons were video recorded, scored on the Reformed Teaching Observation Protocol (RTOP) rubric, and examined for qualitative trends. Year 1 results indicated that teachers had begun to transform their teaching and pointed to particular lessons within the INSPIRES curriculum that most facilitated the reform. Instructional practices of participants improved significantly as a result of the INSPIRES PD program and also aligned with previous, similar studies. These findings provide insights for rethinking the structure of professional development, particularly in the integrated use of an educative curriculum aligned with intended professional development goals.</p> <p>Keywords: Educative curriculum; Engineering education; Mixed methods; Pedagogical reform; Professional development</p> <hd id="AN0135909703-2">Introduction</hd> <p>The publication of the <emph>Framework for K–12 Science Education: Practices, Crosscutting Concepts, and Core Ideas</emph> (National Research Council [<reflink idref="bib30" id="ref1">30</reflink>]) and the subsequent adoption of Next Generation Science Standards (NGSS) have led to a significant shift in instruction and student learning expectations in K–12 science classrooms (Cuban [<reflink idref="bib9" id="ref2">9</reflink>]; Roseman et al. [<reflink idref="bib39" id="ref3">39</reflink>]). In addition to the use of student performance expectations, the NGSS has multiple components that are significantly different from past reforms, including the incorporation of science and engineering "practices," "disciplinary core ideas," and "crosscutting concepts" (Next Generation Science Standards [<reflink idref="bib31" id="ref4">31</reflink>]). These changes to STEM teaching and learning will require both the need for new curricular materials, as well as support in reformed instructional practices (Richmond et al. [<reflink idref="bib38" id="ref5">38</reflink>]; Fishman et al. [<reflink idref="bib16" id="ref6">16</reflink>]; Ross et al. [<reflink idref="bib41" id="ref7">41</reflink>]; Singer et al. [<reflink idref="bib47" id="ref8">47</reflink>]). For example, inclusion of pedagogical practices, such as coaching student groups through an open-ended design challenge and probing students for science- or math-based rationale, supports success in addressing the NGSS. Teacher professional development (PD) is a critical strategy for supporting in-service educators in the use of new materials and the implementation of reform-based instructional practices (Reiser [<reflink idref="bib36" id="ref9">36</reflink>]). This shift presents significant challenges to teachers unfamiliar with engineering-based pedagogy and engineering or science concepts.</p> <p>The INcreasing Student Participation, Interest, and Recruitment in Engineering and Science (INSPIRES) curriculum is written for grades 9–12 and focuses on integrating all areas of STEM. These materials use a real-world engineering design challenge (building a functional hemodialysis system for an adolescent patient) and inquiry-based learning strategies (e.g., phenomena-first, artifact sharing, probing questions) to engage students, increase technological literacy, and develop key practices foundational for success in STEM disciplines. The curriculum was designed to be flexible, low cost, and approximately three weeks in length (Ross et al. [<reflink idref="bib41" id="ref10">41</reflink>]). The curriculum is well-aligned to the ideas and practices of engineering articulated in the <emph>Framework for K–12 Science Education: Practices, Crosscutting Concepts, and Core Ideas</emph> (National Research Council [<reflink idref="bib30" id="ref11">30</reflink>]). As a result, the INSPIRES curriculum targets all four NGSS Engineering Design performance expectations (HS-ETS1) and all eight Science and Engineering Practices (Next Generation Science Standards [<reflink idref="bib31" id="ref12">31</reflink>]). In addition, the INSPIRES curriculum has been constructed to include explicit, imbedded supports that highlight specific elements in the lesson plan that may impact student learning. The inclusion of these elements may support teachers "to learn about teaching within the curriculum materials, making them educative" (Schneider et al. [<reflink idref="bib44" id="ref13">44</reflink>]). The educative curriculum materials support teachers by including features that encourage reflection and promote connections among specific content, pedagogy and pedagogical-content knowledge (Ball and Cohen [<reflink idref="bib3" id="ref14">3</reflink>]; Schneider et al. [<reflink idref="bib44" id="ref15">44</reflink>]; Knaggs and Schneider [<reflink idref="bib17" id="ref16">17</reflink>]). These characteristics make INSPIRES unique compared to other currently available engineering-based curriculum materials (Ross et al. [<reflink idref="bib41" id="ref17">41</reflink>]). Within each INSPIRES lesson, the educative components appear in a column adjacent to particular sections that are potentially challenging for teachers or learners. Similar to the support described by Davis and Krajcik ([<reflink idref="bib11" id="ref18">11</reflink>]), the INSPIRES educative traits highlight strategies or information that is intended to address (among other things) student misconceptions, additional content knowledge for teachers, potential probing questions, or specific pedagogical strategies. For example, in INSPIRES lesson 7, <emph>Introduction to Dialysis</emph>, the lesson plan describes how the teacher can facilitate student experiments that explore the movement of "waste" products across a semi-permeable membrane. Here, the educative elements include (<reflink idref="bib1" id="ref19">1</reflink>) highlighting student misconceptions related to "equilibria," (<reflink idref="bib2" id="ref20">2</reflink>) teacher content knowledge regarding experimental variables that impact the "rate of diffusion" versus the amount of "mass transfer," and (<reflink idref="bib3" id="ref21">3</reflink>) a description of how the lesson moves from a macroscopic phenomenon to a particle-level simulation.</p> <p>The present study explored the benefits and limitations of infusing the INSPIRES educative curriculum materials within a professional development (PD) system. Such an enhancement of PD is posed as a mechanism for strengthening teacher pedagogical skills for integrating engineering practices in high school biology and technology education classrooms. The research questions were the following:</p> <p></p> <ulist> <item> Did teachers' classroom practice change as a function of INSPIRES-based professional development and curriculum enactment as measured by the Reformed Teaching Observation Protocol (RTOP)?</item> <p></p> <item> Did teacher pedagogical skill development differ for biology and technology education teachers?</item> </ulist> <hd id="AN0135909703-3">Conceptual Framework</hd> <p>The Professional Development: Research, Implementation, and Evaluation (PrimeD) framework (Saderholm et al. [<reflink idref="bib43" id="ref22">43</reflink>]) guided the PD throughout the study. Elements of the PrimeD framework were developed through a synthesis of PD theory from multiple sources, such as Darling-Hammond and McLaughlin ([<reflink idref="bib10" id="ref23">10</reflink>]), McAleer ([<reflink idref="bib27" id="ref24">27</reflink>]), Desimone ([<reflink idref="bib12" id="ref25">12</reflink>]), Loucks-Horsley et al. ([<reflink idref="bib22" id="ref26">22</reflink>]), and Sztajn ([<reflink idref="bib50" id="ref27">50</reflink>]). PrimeD divides PD into the following four phases: design and development, implementation, evaluation, and research. In the design and development phase, the PD providers met with district personnel and teachers to develop a common vision and design, including the establishment of goals, strategies, needs assessment, targets, and contextual factors (<emph>challenge space</emph>). The implementation phase consisted of cycles of whole and small group meetings and utilized classroom implementation activities. Whole group meetings occurred during summer workshops and periodically throughout the school year. Small group meetings occurred during the school year between whole group meetings. Classroom implementation activities were guided by Plan-Do-Study-Act (PDSA) cycles (Bryk et al. [<reflink idref="bib6" id="ref28">6</reflink>]). For each PDSA cycle, teachers implemented activities to address a particular challenge discussed during a whole or small group meeting. Teachers collected artifacts during classroom implementation to bring back to the whole and small group meetings. Feedback was provided throughout each phase of the program and findings initiated a revisiting of the challenge space prior to subsequent rounds of implementation. Research goals, design, data, threats to validity and reliability, and ongoing results were an integral component of the development and adjustment of the challenge space. However, even with effective PD programs, research has shown that teachers struggle to successfully integrate engineering design- and inquiry-based practices (Schneider et al. [<reflink idref="bib44" id="ref29">44</reflink>]).</p> <hd id="AN0135909703-4">Educative Curriculum</hd> <p>The integration of educative curriculum materials with PD has shown promise in small-scale studies (e.g., Rushton et al. [<reflink idref="bib42" id="ref30">42</reflink>]; Singer et al. [<reflink idref="bib46" id="ref31">46</reflink>]; Lotter et al. [<reflink idref="bib21" id="ref32">21</reflink>]). In a PD guided by an educative curriculum, the curriculum acts as a scaffold to illustrate pedagogical principles to be transferred to teaching practice. In this study, the classroom enactment of the educative materials (INSPIRES) was intended to be a critical component of the PD strategy. Thus, teachers were given the guided experience of grappling with the educative materials from both the student's and teacher's perspectives, followed by reflective discussions on the lessons' pedagogical design. These experiences provided opportunities for teachers to encounter the affordances and limitations of each activity from the student's perspective and then to discuss the rationale for how the activity was constructed and how it may be adapted (Remillard [<reflink idref="bib37" id="ref33">37</reflink>]). The curricular materials serve as a scaffold by providing the teachers concrete examples for how to translate abstract ideas into a tangible useful product. Employing such a strategy may promote significant change in the content knowledge and pedagogical practices of high school STEM teachers (Singer et al. [<reflink idref="bib46" id="ref34">46</reflink>]; Lotter et al. [<reflink idref="bib21" id="ref35">21</reflink>]). Arias et al. ([<reflink idref="bib2" id="ref36">2</reflink>]) found that teachers better supported students in qualifying predictions, forming evidence-based claims, documenting observations, and planning next steps when utilizing an educative curriculum for electric circuits; educative features included practice overviews, in-lesson "how and why supports," practice reminder boxes, rubrics, examples, and narratives. Teachers have reported that enacting the educative curriculum profoundly changed their attitudes and methods for teaching science (Pringle et al. [<reflink idref="bib34" id="ref37">34</reflink>]). With the proper educative features, these curricula are already thought to be appropriate for addressing challenges of the NGSS (Roseman et al. [<reflink idref="bib40" id="ref38">40</reflink>]). Additionally, there is a call to further shift teachers' perspective of educative materials from merely a source of student activities to a dynamic tool for supporting teachers' own pedagogical growth (McNeill et al. [<reflink idref="bib28" id="ref39">28</reflink>]).</p> <hd id="AN0135909703-5">INSPIRES Educative Curriculum and PD Program</hd> <p>The INSPIRES educative curriculum materials and accompanying teacher PD framework are intended to facilitate teacher adoption of design-based pedagogical practices necessary for integrating engineering and biology concepts and practices. The PD program began with a 5-day summer institute (SI) followed by a series of 2-h, monthly sessions sustained across the academic year. The year 1 SI focused on the following four key components: (<reflink idref="bib1" id="ref40">1</reflink>) the INSPIRES educative curriculum materials, (<reflink idref="bib2" id="ref41">2</reflink>) STEM practices, (<reflink idref="bib3" id="ref42">3</reflink>) pedagogical practices, and (<reflink idref="bib4" id="ref43">4</reflink>) reflective critiques. The INSPIRES hemodialysis materials were developed to model and scaffold the other three components. During the STEM practices segment of summer PD, specific activities from the pre-selected materials were used by the facilitators to illustrate key ideas or as "jumping off" points for deeper discussion. The key foci of the STEM practices component were on building content knowledge, an understanding of the engineering design process, and skills with the tools needed for the design challenge. Teacher teams participated in the curriculum as students and performed all design-, build-, and test-based engineering activities. The key focus of the pedagogical practices component was on building pedagogical content knowledge. Core elements of this component focused on modeling various pedagogical strategies, STEM practices, and curriculum materials. Example practices that were emphasized include phenomena-first, inquiry, and design-based learning (e.g., Predict, Observe, Explain; integration of an engineering design loop), collaboration (e.g., jigsaws, Think-Pair-Share), context (e.g., driving questions, KWL charts), technology integration (e.g., simulations, data collection), and sense-making and assessment (e.g., wait time, probing questions, prior knowledge). The reflective critiques component supported both STEM and pedagogical practices as well as classroom management issues. Following each lesson, the PD facilitators engaged teachers in discussions relating the lessons' content to its structure and strategies.</p> <hd id="AN0135909703-6">Method</hd> <p></p> <hd id="AN0135909703-7">The INSPIRES Curriculum</hd> <p>The INSPIRES curriculum was developed to integrate engineering design principles into high school science and technology classes. The present study used <emph>Engineering in Health Care: Hemodialysis</emph>, one of five modules that comprise the INSPIRES curriculum (Ross et al. [<reflink idref="bib41" id="ref44">41</reflink>]). In this module, students learn about kidney function, dialysis, diffusion of waste across membranes, and factors that influence mass transfer and diffusion rates. By the end of the module, students design, build, test, and revise an apparatus that mimics the function of a hemodialysis system. The module applies a project-based approach (Blumenfeld et al. [<reflink idref="bib4" id="ref45">4</reflink>]; Marx et al. [<reflink idref="bib26" id="ref46">26</reflink>]; Krajcik and Blumenfeld [<reflink idref="bib18" id="ref47">18</reflink>]; Willis [<reflink idref="bib51" id="ref48">51</reflink>]), in which the design challenge is introduced at the beginning of the module and is used throughout multiple lessons to drive the learning of important science and engineering concepts.</p> <hd id="AN0135909703-8">Participants</hd> <p>The present study was conducted in collaboration with a large mid-Atlantic public school system. With 174 schools, programs, and centers, nearly 9000 classroom teachers and over 105,000 students, this district is one of the largest school systems in the US. The district's 800,000+ residents live in suburban, rural, and urban neighborhoods comprising of cultures and backgrounds representative of the nation's diversity. Overall, 54.8% of the district's students represent racial and ethnic groups other than White, 48.9% are female, and 44.8% are eligible for free/reduced price meals.</p> <p>Twenty-seven biology and technology education teachers from eleven high schools participated in the study. These schools represent traditional and alternative schools that offer both biology and technology education courses and form a representative cross-section of the district. The group of teachers included both males (<emph>N</emph> = 16) and females (<emph>N</emph> = 11) who reported their race/ethnicity as Black or African American (22%) or White (78%) and whose classroom teaching experience ranged from 2 to 28 years (16% of teachers had 0–5, 47% had 6–10, 26% had 11–15, 11% had > 15 years of experience).</p> <hd id="AN0135909703-9">Data Sources</hd> <p>The data presented in this study represents those from the first year of a three-year, longitudinal research project. The data were obtained from scoring classroom videos at four time points. The first data point (Baseline lesson) was collected during the spring prior to the summer PD event in which the teachers were asked to provide their best attempt of incorporating NGSS Engineering Design Standards (HS-ETS1) into a lesson. This same prompt was utilized approximately 1 year later during the following semester to serve as a measure of potential growth during year 1 (Transfer lesson). Two additional lessons associated with the enactment of the INSPIRES educative materials were recorded during the intervening fall (Lessons 7 and 11).</p> <p>The INSPIRES Hemodialysis Lesson 7 is structured as a phenomena-first, science-rich, inquiry activity. It provides an opportunity for students to collect visual and quantitative evidence of "waste" removal from artificial blood by diffusion. The lesson's base activity involves dialysis tubing formed into a "bag" and filled with 20 ml of simulated blood. The dialysis bag is then placed in a beaker of water. By identifying and altering variables (e.g., porosity of the bag membrane, water temperature), the conditions affecting waste removal and, therefore, diffusion, can be identified and tested. This creates opportunities for students to work collaboratively in teams, identify experimental variables, form predictions, design protocols and procedures, and carry out experiments. In addition, the lesson is designed to allow student teams to share results with the whole class, analyze data, and reflect on outcomes. The strategy of sharing results is expected to deepen understanding of the critical scientific concepts and to inform design choices in the larger design challenge.</p> <p>The objective for INSPIRES Lesson 11 was for students to apply the knowledge and experiences from all previous INSPIRES lessons and use the design process to design, build, and test a hemodialysis system. Lesson 11 begins with a review of the design challenge, the various preceding activities, and the connections between activities that address the challenge. Teams are shown various supplies (e.g., tubes, membranes, pumps, bottles) and are prompted to plan their designs. Before construction can begin, the teacher probes teams for evidence-based rationale for their various design decisions. Research-based observations of Lesson 11 typically captured the design phase and sometimes the beginning of the build phase. Overall, Lesson 11 was crafted to lapse 2–3 class periods where students could continue building their systems, complete testing, and further revise their design.</p> <p>Collected classroom videos were scored using the RTOP observational instrument. The RTOP was developed by the Arizona Collaborative for Excellence in the Preparation of Teachers to capture current elements of pedagogical reform. The instrument was written based on constructivist theory and with national standards of math and science in mind. The RTOP is widely applied in STEM educational research as both a quantitative and qualitative tool (e.g., MacIsaac et al. [<reflink idref="bib24" id="ref49">24</reflink>]; Enderle et al. [<reflink idref="bib13" id="ref50">13</reflink>]; Amolins et al. [<reflink idref="bib1" id="ref51">1</reflink>]) by outlining characteristics of reform in a 25-item rubric on a 0–4 performance scale. The training manual defines level 0 as "not descriptive of the lesson" and level 4 as "very descriptive of the lesson," and prior psychometrics on the RTOP instrument revealed an "exceptionally high" estimate of reliability (Piburn and Sawada [<reflink idref="bib33" id="ref52">33</reflink>]).</p> <p>RTOP items are divided into the following five subcategories: Lesson Design, Propositional Knowledge, Procedural Knowledge, Classroom Culture, and Teacher–Student Relationships. Lesson Design items ask the extent to which class instruction incorporates prior knowledge, social construction of knowledge, the progression from concrete to abstract concepts, valuing multiple solutions or approaches, and flexibly in following students' ideas or needs. Items in the Propositional Knowledge subcategory ask whether significant STEM ideas are the focus, if explicit connections are made between STEM ideas and with real-world applications, and the extent of teacher comfort and expertise in the STEM content. Rating Procedural Knowledge items will indicate the extent of multiple means of representation and the opportunity for students to make predictions, think critically, reflect on learning, and engage in argumentation. Items representing Classroom Culture assess multiple means of expression, the facilitation of divergent thinking, the value of student discourse, and the classroom as a safe place to express individual ideas. Finally, Teacher–Student Relationship items evaluate the level of leadership and empowerment passed from teacher to students, intended use of wait time, and teacher facilitation of student understanding (Piburn and Sawada [<reflink idref="bib33" id="ref53">33</reflink>]).</p> <p>Prior to data collection, four coders were trained to identify the characteristics of each RTOP item and performance level. The coders developed and refined performance indicators within the RTOP rubric to bring validity to particular score levels and to enhance inter-rater reliability. Classroom video data were deidentified by replacing teacher names with random numeric codes. Subjectivity was further discouraged by frequent checks of inter-rater reliability; the four coders achieved high agreement despite their varied expertise within STEM fields or education. Twenty percent of the videos was coded by all four researchers with an additional 14% being double coded. Interclass correlation coefficients (<emph>K</emph>) that ranked in the range of 0.75–1.00 were considered excellent, and ranks between 0.60 and 0.74 were considered good (Cicchetti [<reflink idref="bib7" id="ref54">7</reflink>]). Interclass correlation coefficients for videos scored by all four coders were the following: Baseline lesson (<emph>K</emph> = 0.705), Lesson 7 (<emph>K</emph> = 0.826), Lesson 11 (<emph>K</emph> = 0.711), and Transfer lesson (<emph>K</emph> = 0.718). For all co-scored videos, discrepancies in item scores between raters were deliberated on until mutual consensus was reached. Classroom videos were given a performance level score (0–4) on all 25 items in the RTOP rubric. Summing scores within each subcategory and then averaging across all teachers yielded summary performance within subcategories. Summing scores of all 25 items and then averaging across teachers determined summary total RTOP performance.</p> <hd id="AN0135909703-10">Data Analyses</hd> <p>For statistical analysis, each teacher's video received a single score for each subcategory by averaging the scores for its five items. Overall trends were identified during the first year of the study by relating teacher instruction of the four lessons (Baseline lesson, Lesson 7, Lesson 11, and Transfer lesson). Additionally, a total average score was computed for all 25 RTOP items. Differences in total and subcategory averages across the four lessons were analyzed with a repeated measures analysis of variance (ANOVA) with one fixed factor to compare biology and technology education teachers.</p> <p>A subsample of teacher participants was selected for qualitative analysis. Raters further characterized typical practices that were generally representative of qualitative traits observed in Baseline and Transfer lessons. The systematic approach used in this characterization involved the selection of three biology and three technology education teachers whose Baseline RTOP scores were in the mean range for at least two out of three of the following subcategories: Procedural Knowledge, Classroom Culture, or Teacher–Student Relationships. Focus was placed on these subcategories as they represented areas of notable growth between Baseline and Transfer lessons for teachers overall. By selecting teachers whose assigned RTOP scores were around the means representative to all teachers, the raters aimed to capture the common traits of teaching practices at the different time points of the study. Further, each focal teacher represented a different high school in the district. This systematic approach was adapted from both domain analysis methods (Spradley [<reflink idref="bib49" id="ref55">49</reflink>]) and analytic coding techniques (Coffey and Atkinson [<reflink idref="bib8" id="ref56">8</reflink>]).</p> <p>Raters critically examined the RTOP scoring notes and lesson summaries for focal teachers' lessons across the four time points. For each lesson, the raters reached consensus on identifiable pedagogical traits. Themes were recognized across all six focal teachers' Baseline lessons which led to the development of a typical Baseline lesson qualitative description. The process was repeated respective to Lesson 7, Lesson 11, and the Transfer lessons.</p> <hd id="AN0135909703-11">Results</hd> <p></p> <hd id="AN0135909703-12">Quantitative Analysis</hd> <p>Total RTOP scores were averaged for all 25 items and for each subcategory (Table 1). At the Baseline, teachers scored an average of about half the possible points, indicating that they were not initially teaching with strong reform pedagogies. Lesson 7 scores were similar to Baseline scores. For Lesson 11, teachers scored approximately two thirds of the possible points. The Transfer lesson scores were slightly lower than those of Lesson 11.</p> <p>Mean total scores for RTOP overall and subcategories</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th rowspan="2"><p>RTOP categories</p></th><th><p>Baseline</p></th><th><p>Lesson 7</p></th><th><p>Lesson 11</p></th><th><p>Transfer</p></th></tr><tr><th colspan="4"><p>Mean total score (SD)</p></th></tr></thead><tbody><tr><td colspan="5"><p>All teachers (<italic>N</italic> = 27)</p></td></tr><tr><td><p> Overall<sup>a</sup></p></td><td><p>50.7 (12.0)</p></td><td><p>56.3 (11.5)</p></td><td><p>68.5 (10.4)</p></td><td><p>59.6 (11.1)</p></td></tr><tr><td><p> Lesson design<sup>b</sup></p></td><td><p>9.2 (3.8)</p></td><td><p>10.5 (3.1)</p></td><td><p>14.2 (2.4)</p></td><td><p>11.4 (3.9)</p></td></tr><tr><td><p> Propositional knowledge<sup>b</sup></p></td><td><p>13.9 (2.3)</p></td><td><p>14.4 (3.2)</p></td><td><p>14.4 (3.2)</p></td><td><p>14.1 (2.4</p></td></tr><tr><td><p> Procedural knowledge<sup>b</sup></p></td><td><p>9.0 (2.9)</p></td><td><p>10.9 (1.9)</p></td><td><p>13.8 (1.8)</p></td><td><p>11.0 (2.6)</p></td></tr><tr><td><p> Classroom culture<sup>b</sup></p></td><td><p>8.6 (2.7)</p></td><td><p>9.8 (2.1)</p></td><td><p>12.7 (2.4)</p></td><td><p>11.3 (2.4)</p></td></tr><tr><td><p> Teacher–student relationships<sup>b</sup></p></td><td><p>10.0 (2.9)</p></td><td><p>10.8 (2.7)</p></td><td><p>13.4 (2.1)</p></td><td><p>11.9 (2.4)</p></td></tr><tr><td colspan="5"><p>Biology teachers (<italic>N</italic> = 14)</p></td></tr><tr><td><p> Overall<sup>a</sup></p></td><td><p>50.4 (10.6)</p></td><td><p>59.5 (8.5)</p></td><td><p>70.1 (7.2)</p></td><td><p>57.4 (10.6)</p></td></tr><tr><td><p> Lesson design<sup>b</sup></p></td><td><p>9.0 (3.3)</p></td><td><p>10.8 (2.5)</p></td><td><p>14.4 (1.8)</p></td><td><p>10.7 (3.8)</p></td></tr><tr><td><p> Propositional knowledge<sup>b</sup></p></td><td><p>14.4 (2.3)</p></td><td><p>15.4 (2.7)</p></td><td><p>14.8 (2.3)</p></td><td><p>14.1 (2.1)</p></td></tr><tr><td><p> Procedural knowledge<sup>b</sup></p></td><td><p>8.9 (2.7)</p></td><td><p>11.5 (1.9)</p></td><td><p>14.1 (1.3)</p></td><td><p>10.1 (3.0)</p></td></tr><tr><td><p> Classroom culture<sup>b</sup></p></td><td><p>8.4 (2.2)</p></td><td><p>10.2 (1.5)</p></td><td><p>12.8 (2.1)</p></td><td><p>10.9 (2.6)</p></td></tr><tr><td><p> Teacher–student relationships<sup>b</sup></p></td><td><p>9.8 (2.2)</p></td><td><p>11.6 (2.5)</p></td><td><p>14.1 (1.4)</p></td><td><p>11.6 (2.3)</p></td></tr><tr><td colspan="5"><p>Technology education teachers (<italic>N</italic> = 13)</p></td></tr><tr><td><p> Overall<sup>a</sup></p></td><td><p>51.0 (14.0)</p></td><td><p>52.5 (13.8)</p></td><td><p>66.6 (13.4)</p></td><td><p>62.1 (11.6)</p></td></tr><tr><td><p> Lesson design<sup>b</sup></p></td><td><p>9.5 (4.4)</p></td><td><p>10.1 (3.7)</p></td><td><p>13.9 (3.0)</p></td><td><p>12.1 (4.1)</p></td></tr><tr><td><p> Propositional knowledge<sup>b</sup></p></td><td><p>13.2 (2.4)</p></td><td><p>13.2 (3.8)</p></td><td><p>14.1 (4.1)</p></td><td><p>14.1 (2.7)</p></td></tr><tr><td><p> Procedural knowledge<sup>b</sup></p></td><td><p>9.2 (3.2)</p></td><td><p>10.2 (1.8)</p></td><td><p>13.5 (2.3)</p></td><td><p>11.9 (1.9)</p></td></tr><tr><td><p> Classroom culture<sup>b</sup></p></td><td><p>8.8 (3.2)</p></td><td><p>9.2 (2.6)</p></td><td><p>12.5 (2.8)</p></td><td><p>11.8 (2.2)</p></td></tr><tr><td><p> Teacher–student relationships<sup>b</sup></p></td><td><p>10.2 (3.6)</p></td><td><p>9.8 (2.6)</p></td><td><p>12.6 (2.5)</p></td><td><p>12.2 (2.6)</p></td></tr></tbody></table> </ephtml> </p> <p> <sups>a</sups>100 points possible <sups>b</sups>20 points possible</p> <p>The repeated measures ANOVA indicated significant differences across the four lessons for the overall RTOP as well as for all subcategories except for Propositional Knowledge (Table 2). No significant differences were found between the biology and technology education teachers (Table 2).</p> <p>Repeated measures ANOVA across four lessons</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th /><th><p>Comparisons among Baseline, Lesson 7, Lesson 11, and Transfer</p></th><th><p>Comparisons between biology and technology ed. teachers</p></th></tr><tr><th><p>RTOP categories</p></th><th><p><italic>F</italic>(2,50)</p></th><th><p><italic>F</italic>(2,50)</p></th></tr></thead><tbody><tr><td><p>Total</p></td><td><p>15.857<sup>***</sup></p></td><td><p>2.067</p></td></tr><tr><td><p>Lesson design</p></td><td><p>11.872<sup>***</sup></p></td><td><p>0.819</p></td></tr><tr><td><p>Propositional knowledge</p></td><td><p>0.596</p></td><td><p>1.342</p></td></tr><tr><td><p>Procedural knowledge</p></td><td><p>23.667<sup>***</sup></p></td><td><p>2.529</p></td></tr><tr><td><p>Classroom culture</p></td><td><p>17.347<sup>***</sup></p></td><td><p>0.766</p></td></tr><tr><td><p>Teacher–student relationships</p></td><td><p>12.113<sup>***</sup></p></td><td><p>2.447</p></td></tr></tbody></table> </ephtml> </p> <p> <sups>*</sups> <emph>p</emph> < 0.05 <sups>**</sups><emph>p</emph> < 0.01 <sups>***</sups><emph>p</emph> < 0.001</p> <p>Pairwise comparisons revealed that the Baseline and Lesson 7 scores were not significantly different except in the case of the Procedural Knowledge subcategory. Lesson 11, however, scored significantly higher than all other lessons for the overall RTOP for all teachers (Table 3). Further, Transfer lessons scored significantly higher than Baseline lessons for both the Classroom Culture and Teacher–Student Relationships subcategories.</p> <p>Pairwise comparisons for RTOP</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th /><th><p>Lesson 7—Baseline</p></th><th><p>Lesson 11—Baseline</p></th><th><p>Lesson 11—Lesson 7</p></th><th><p>Transfer—Baseline</p></th><th><p>Transfer—Lesson 7</p></th><th><p>Lesson 11—Transfer</p></th></tr><tr><th><p>RTOP categories</p></th><th colspan="6"><p>Mean difference (SE)</p></th></tr></thead><tbody><tr><td><p>Total</p></td><td><p>0.250 (0.123)</p></td><td><p>0.726 (0.098)<sup>***</sup></p></td><td><p>0.476 (0.075)<sup>***</sup></p></td><td><p>0.388 (0.118)<sup>*</sup></p></td><td><p>0.138 (0.118)</p></td><td><p>0.338 (0.105)<sup>*</sup></p></td></tr><tr><td><p>Lesson design</p></td><td><p>0.307 (0.192)</p></td><td><p>1.013 (0.166)<sup>***</sup></p></td><td><p>0.706 (0.113)<sup>***</sup></p></td><td><p>0.472 (0.195)</p></td><td><p>0.164 (0.202)</p></td><td><p>0.542 (0.163)<sup>*</sup></p></td></tr><tr><td><p>Propositional knowledge</p></td><td><p>0.157 (0.145)</p></td><td><p>0.156 (0.151)</p></td><td><p>− 0.001 (0.096)</p></td><td><p>0.070 (0.123)</p></td><td><p>− 0.086 (0.147)</p></td><td><p>0.086 (0.159)</p></td></tr><tr><td><p>Procedural knowledge</p></td><td><p>0.401 (0.124)<sup>*</sup></p></td><td><p>0.981 (0.102)<sup>***</sup></p></td><td><p>0.580 (0.078)<sup>***</sup></p></td><td><p>0.441 (0.156)</p></td><td><p>0.041 (0.113)</p></td><td><p>0.540 (0.116)<sup>**</sup></p></td></tr><tr><td><p>Classroom culture</p></td><td><p>0.217 (0.126)</p></td><td><p>0.805 (0.122)<sup>***</sup></p></td><td><p>0.588 (0.091)<sup>***</sup></p></td><td><p>0.557 (0.132)<sup>**</sup></p></td><td><p>0.340 (0.130)</p></td><td><p>0.248 (0.121)</p></td></tr><tr><td><p>Teacher–student relationships</p></td><td><p>0.161 (0.149)</p></td><td><p>0.674 (0.103)<sup>***</sup></p></td><td><p>0.513 (0.094)<sup>***</sup></p></td><td><p>0.400 (0.135)<sup>*</sup></p></td><td><p>0.239 (0121)</p></td><td><p>0.274 (0.104)</p></td></tr></tbody></table> </ephtml> </p> <p> <sups>*</sups> <emph>p</emph> < 0.05 <sups>**</sups><emph>p</emph> < 0.01 <sups>***</sups><emph>p</emph> < 0.001</p> <hd id="AN0135909703-13">Qualitative Analysis</hd> <p>Here, we evaluate qualitative themes that reflect shifts in teacher instruction across the four focal lessons. Evidence of qualitative trends fits into the following six themes: Guided vs. Open Strategy, Probing of Prior Knowledge, Making Predictions, Making Connections, Student Reflection, and Teacher Sharing (Tables 4–7). Guided vs. Open Strategy highlights traits that may characterize a lesson as either more prescribed or open-ended. Probing of Prior Knowledge characterizes the degree to which teachers facilitate students' application of prior knowledge to the current lesson. Making Predictions refers to elements of prediction formulation, justification, and verification that may occur throughout a STEM lesson. Making Connections highlights instances where teachers or students explicitly think about how past lessons inform the current lesson or how the current lesson may inform future lessons. Student Reflection captures elements of divergent and critical thinking and the strategies used to support these processes. Teacher Sharing refers to teacher comments that convey personal experiences, notably their struggles while working through the Hemodialysis curriculum as learners. Elements of Teacher Sharing were unique to Lesson 11 (Table 6).</p> <p>Qualitative trends among baseline lessons</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th><p>Theme</p></th><th><p>Baseline trends</p></th></tr></thead><tbody><tr><td><p>Guided vs. open strategy</p></td><td><p>More "Hands-on than Minds-on"</p><p>• Activities are preceded by teacher-centered introduction of key ideas</p><p>• Teacher provided variables and procedures</p><p>• Focus on consistent process (doing it correctly)</p><p>• Activities are used to confirm information presented in the lesson</p></td></tr><tr><td><p>Probing of prior knowledge</p></td><td><p>Traditional "Bell work"</p><p>• Review of prior concepts at start of lesson</p><p>• Completed as individuals</p><p>• Ascertained information does not alter instructional sequence</p></td></tr><tr><td><p>Making predictions</p></td><td><p>Prediction as "Formality"</p><p>• Teacher directs individuals to make predictions</p><p>• Predictions are typically made before the activity</p></td></tr><tr><td><p>Making connections</p></td><td><p>Connecting "Past to Present"</p><p>• Reminds students of introduced concepts from prior lessons</p><p>• Teacher provides real world examples</p><p>• Connections mostly "Past to Present"</p></td></tr><tr><td><p>Student reflection</p></td><td><p>Traditional "Exit Ticket"</p><p>• Individuals respond in writing to teacher prompt of student knowledge from the day's lesson</p><p>• Short, factual information from the day's lesson is the focus of the prompt</p></td></tr><tr><td><p>Teacher sharing</p></td><td><p>Not a hallmark of this lesson</p></td></tr></tbody></table> </ephtml> </p> <p>Qualitative trends of Lesson 7 enactment</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th><p>Theme</p></th><th><p>Lesson 7 trends</p></th></tr></thead><tbody><tr><td><p>Guided vs. open strategy</p></td><td><p>More "Hands-on than Minds-on"</p><p>• Activities preceded by extensive teacher-centered summary of key ideas/vocabulary</p><p>• Student ideas for the activity are solicited; use is limited</p><p>• Variables and procedures are provided by teacher</p><p>• Different groups investigate different variables</p><p>• Teacher discusses results with individual groups</p><p>• Teacher often does calculations of dependent variable for students</p></td></tr><tr><td><p>Probing of prior knowledge</p></td><td><p>Traditional "Bell work"</p><p>• Review of prior science concepts at start of lesson</p><p>• Structured as a warm-up (individual student work), followed by class discussion led by the teacher</p><p>• Student prior knowledge does not alter the instructional sequence</p></td></tr><tr><td><p>Making predictions</p></td><td><p>Prediction as "Confirmation"</p><p>• Predictions for activities shared within student groups</p><p>• Some teacher probing for information introduced earlier in the lesson as rationale</p><p>• Teacher often confirms prediction rationale before the activity</p><p>• Predictions and rationale discussed mostly prior to the activity</p></td></tr><tr><td><p>Making connections</p></td><td><p>"Incomplete" Connections</p><p>• Teacher reminds students of concepts from prior lessons</p><p>• Teacher provides real world examples</p><p>• Superficial connections are made to the engineering design process (e.g., "Where are we?")</p><p>• Connections are mostly "Past to Present"</p><p>• Frequent reference made to reviewing data during the next class</p></td></tr><tr><td><p>Student reflection</p></td><td><p>Journals used for "Documentation"</p><p>• Student notebooks used throughout the lesson for notes, predictions, experimental designs, data, results, and to summarize outcomes</p></td></tr><tr><td><p>Teacher sharing</p></td><td><p>Not a hallmark of this lesson</p></td></tr></tbody></table> </ephtml> </p> <p>Qualitative trends of Lesson 11 enactment</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th><p>Theme</p></th><th><p>Lesson 11 trends</p></th></tr></thead><tbody><tr><td><p>Guided vs. open strategy</p></td><td><p>Mostly open; "Student Autonomy"</p><p>• Open-ended group work</p><p>• Divergent thinking valued through student-determined designs and procedures</p><p>• Activity has multiple correct solutions</p><p>• Students encouraged to use additional materials brought from home</p></td></tr><tr><td><p>Probing of prior knowledge</p></td><td><p>Relevant "Science Concepts"</p><p>• Student-selected artifacts or use of KWL charts replaces traditional written drill</p><p>• Discussion of counter-current flow</p><p>• Revisiting the relationship between height and flow rate</p><p>• Cost emphasized over integration of science concepts</p></td></tr><tr><td><p>Making predictions</p></td><td><p>Student "Planning"</p><p>• Design sketching precedes building</p><p>• Teachers check designs/predictions before students "buy" materials</p><p>• Groups are expected to combine ideas from multiple designs, or use rationale to select a best design to build</p></td></tr><tr><td><p>Making connections</p></td><td><p>Connecting "Past to Present"</p><p>• Connecting to prior lessons ("Computer Simulation" and "Flow Rate" lessons)</p><p>• Reminding class of the current step within the engineering design process</p><p>• References to the multiple criteria and constraints of the design target</p></td></tr><tr><td><p>Student reflection</p></td><td><p>Journals used as a "Dynamic Resource"</p><p>• Notebooks are frequently used for note-taking, data recording, design sketching, and referencing notes from prior lessons to inform design decisions or provide rationale for design decisions</p></td></tr><tr><td><p>Teacher sharing</p></td><td><p>"Teachers Share" their own experiences of designing, building, and testing systems</p><p>• Shared photographs of multiple teacher-built systems</p><p>• Revealed that teacher systems did not meet all criteria and constraints</p><p>• Noted that teacher designs were successful without use of pumps</p></td></tr></tbody></table> </ephtml> </p> <p>Qualitative trends among Transfer lessons</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th><p>Theme</p></th><th><p>Transfer</p></th></tr></thead><tbody><tr><td><p>Guided vs. open strategy</p></td><td><p>"Increased Autonomy"</p><p>• Student groups pursue different approaches</p><p>• Planning documents utilized</p><p>• Shallow emphasis on rationale/adaptations</p></td></tr><tr><td><p>Probing of prior knowledge</p></td><td><p>"Increased Student Interest"</p><p>• Presentation of prior results</p><p>• Base experiment prior to student designing</p><p>• Increased student engagement</p></td></tr><tr><td><p>Making predictions</p></td><td><p>"Shallow Rationale"</p><p>• Sharing to teacher within groups</p><p>• Limited pressing for conceptual rationale</p></td></tr><tr><td><p>Making connections</p></td><td><p>"Improve Future Results"</p><p>• Data collected to improve performance or test hypothesis</p><p>• Mostly implicit connection to concepts discussed prior to investigations</p><p>• Increased use of "modeling"</p></td></tr><tr><td><p>Student reflection</p></td><td><p>"Teacher Prompted Closure"</p><p>• Limited time set aside at lesson conclusion</p><p>• Teacher probes and prompts students for key ideas</p><p>• Superficial response accepted</p></td></tr><tr><td><p>Teacher sharing</p></td><td><p>Not a hallmark of this lesson</p></td></tr></tbody></table> </ephtml> </p> <p>Prior to the first INSPIRES summer PD Institute, the teacher participants were asked to conduct a classroom lesson that addressed their best attempt of incorporating NGSS Engineering Design Standards (HSETS1). This event served as a baseline measure of teachers' initial understanding of integrating engineering design into their instruction. Baseline data revealed that teachers' lessons addressed a wide range of foci varying from classical biological topics, such as evolution and endangered species, to physical sciences, such as propeller designs, fluid flow rates, and simple machines, as well as specialized subjects like forensic science. Despite the large range of topics, multiple themes could be distilled (Table 4).</p> <p>One emergent Baseline theme was that instruction involved a central activity requiring the collection of data, yet, the activities were confirmational in nature and the introduction of concepts preceded the actual investigation (Table 4, <emph>Guided</emph> vs <emph>Open Strategy</emph>). Additionally, probing for student predictions was limited, and no connections were made between predictions and the corresponding results (Table 4, <emph>Making Predictions</emph>). Baseline lessons typically included connections to prior classroom activities, such as illustrating how the lesson was part of a larger challenge (Table 4, <emph>Making Connections</emph>). While most of the Baseline lessons attempted to make connections to other lessons, limited attempts were made to integrate student prior knowledge as a means to engage students or adapt the instruction (Table 4, <emph>Probing of Prior Knowledge</emph>). Most of the sampled teachers opened instruction with a traditional drill asking students to provide a definition of a key term related to the day's activity. Generally, student responses were relayed back to the teacher with an emphasis on presenting a correct response.</p> <p>The INSPIRES Hemodialysis Lesson 7 is structured as a phenomena-first inquiry activity. However, forthcoming qualitative analysis and discussion suggest that the provided, written plan for Lesson 7 was not closely followed by several teachers. Various qualitative traits characteristic of Lesson 7 enactment are listed in Table 5.</p> <p>In general, Lesson 7 instruction was guided and often teacher-directed (Table 5, <emph>Guided vs. Open Strategy</emph>). Commonly, teachers probed students' prior knowledge of relevant scientific concepts and vocabulary during Lesson 7. Student misconceptions were usually clarified by teachers, but they did not alter the instructional sequence of the lesson (Table 5, <emph>Probing Prior Knowledge</emph>). Teachers typically prompted students to identify possible variables for the experimental system and to make predictions on the effects of changing each variable (Table 5, <emph>Making Predictions</emph>). Making explicit connections to science concepts from a prior lesson was a common practice in enactments of Lesson 7, yet, connections to the engineering design process were sparse (Table 5, <emph>Making Connections</emph>). Student journals were frequently used as a tool to record notes, predictions, data, experimental design plans, and results. Use of notebooks for written reflection on rationale (such as <emph>explaining</emph> the results after experiment completion) was minimal or absent (Table 5, <emph>Student Reflection</emph>).</p> <p>The objective for INSPIRES Lesson 11 was for students to apply the knowledge and experiences they had acquired from all previous INSPIRES lessons and effectively employ a design process in order to design, build, and test a hemodialysis system. Common qualitative traits are evident across Lesson 11 teacher enactments (Table 6).</p> <p>During Lesson 11, teachers generally allowed student autonomy by encouraging the development of multiple designs and/or procedures. In addition to following the INSPIRES lesson plan, teachers typically granted students opportunities for divergent thinking by fostering open-ended group work (Table 6, <emph>Guided vs. Open Strategy</emph>). Many teachers facilitated explicit connections to both prior lessons and knowledge (Table 6, <emph>Probing Prior Knowledge</emph>) and established links to the engineering design loop or target (Table 6, <emph>Making Connections</emph>). Students frequently used engineering notebooks for sketching designs or referencing relevant prior knowledge (Table 6, <emph>Making Predictions</emph>, <emph>Student Reflection</emph>). Teachers also referenced their own prior experiences designing, building, and testing hemodialysis systems as they trained in the INSPIRES curriculum (Table 6, <emph>Teacher Sharing</emph>).</p> <p>For the final class observation of the present study, teachers were asked to select and share a lesson from their repertoire that best highlighted NGSS engineering design practices. Although lesson topics varied widely, the collective group of these lessons is referred to as Transfer lessons. In other words, we wanted to measure how effectively teachers transferred elements of reformed pedagogy, learned through the INSPIRES PD and educative curriculum, into their own original lessons. Table 7 lists common traits evident across teachers' Transfer lessons.</p> <p>During Transfer lessons, teachers generally allowed some level of student autonomy, demonstrated through students working within small groups and pursuing different approaches to a problem (Table 7, <emph>Guided vs. Open Strategy</emph>). Transfer lessons frequently incorporated strategies to elicit student prior knowledge of a STEM concept (Table 7, <emph>Probing of Prior Knowledge</emph>). Such strategies appeared to spark interest among students and encourage their full participation in the activity. Further, teachers pressed students for shallow levels of rationale, which could be construed as students making predictions about the outcomes of their activity (Table 7, <emph>Making Predictions</emph>). Commonly, students made connections between concepts or multiple activities, during Transfer lessons (Table 7, <emph>Making Connections</emph>). For example, students were engaged in data collection as a means to improve performance or test a hypothesis. Many of the Transfer lessons concluded with a teacher-prompted closure activity that limited the opportunity for student reflection (Table 7, <emph>Student Reflection</emph>). Often, the time reserved for a lesson's conclusion was short in duration, and the discussion was rushed or absent.</p> <hd id="AN0135909703-14">Discussion</hd> <p></p> <hd id="AN0135909703-15">Quantitative Findings</hd> <p>One striking trend revealed by the quantitative analysis is that enactment of Lesson 11 scored significantly higher than all other lessons overall. As part of the larger INSPIRES Hemodialysis curriculum, Lesson 11 was carefully crafted to incorporate explicit connections to both the engineering design process as well as the underpinning scientific and quantitative rationale. The lesson is further designed to shift the responsibility of learning from the teacher to the students, resulting in a student-centered, inquiry- and project-based experience for learning. Examples of such exemplar lesson traits include but are not limited to the following: small student groups working to communicate designs and procedures, the expectation for science and quantitative rationale to justify design decisions, teachers acting as listeners and facilitators, student groups reporting out and offering critique, teachers enforcing wait time and encouraging divergent thinking, and opportunities to explore phenomena related to real-world engineering challenges (Piburn and Sawada [<reflink idref="bib33" id="ref57">33</reflink>]). When Lesson 11 is taught as intended, the resulting RTOP analysis would indicate use of highly reformed pedagogy. Therefore, teachers that made a strong effort to facilitate the lesson as written were well prepared to attain high RTOP scores. Finding high levels of pedagogical reform on this engineering-focused lesson provides support for how quality engineering lessons offer ideal opportunities for student learning. Therefore, teachers equipped with the pedagogical skillset to accompany quality engineering lessons will be better prepared to address the challenges of the NGSS.</p> <p>Subcategorical RTOP performance revealed that Lesson 11 outscored Transfer lessons only in Lesson Design and Procedural Knowledge (Table 3). Alternatively, Lesson 11 outscored Baseline and Lesson 7 in all subcategories except Propositional Knowledge. We speculate that the engineering design structure of Lesson 11 allowed teachers to score significantly higher in the subcategories of Lesson Design and Procedural Knowledge, as aspects of design and procedure were made explicit within the lesson plans and are central to a quality engineering-focused lesson. Lesson 11 scores were not significantly higher than Transfer lesson scores in the subcategories of Classroom Culture and Teacher–Student Relationships (Table 3), which we attribute to the successful transfer of pedagogical skills, possibly as a result of teachers' participation in the educative curriculum-based PD. This hypothesis is further supported by the fact that teachers' Transfer lessons also scored significantly higher than teachers' Baseline lessons in the subcategories of Classroom Culture and Teacher–Student Relationships.</p> <p>Performance on Lesson 11 did not significantly exceed that of any other lesson in the subcategory of Propositional Knowledge (Table 3). This suggests that teachers had a solid foundation in the content related to their selected (Baseline and Transfer) or assigned (Hemodialysis Lessons 7 and 11) lessons. That is, teachers likely consciously shared lessons that were rich in the STEM content they were comfortable teaching, which resulted in high Baseline (and Transfer) Propositional Knowledge scores. Notably, there was not much room for pedagogical improvement within this subcategory. Similarly, both INSPIRES Lessons 7 and 11 were designed to be rich in STEM content and may yield comparably high scores in Propositional Knowledge when instructed as intended.</p> <p>Subcategorical and overall RTOP comparisons between Baseline lesson and Lesson 7 performance revealed no significant differences (Table 3). As part of the INSPIRES Hemodialysis curriculum, Lesson 7 is written to be rich in STEM content and also reformed in the suggested pedagogy of the STEM process. For example, teachers are encouraged to allow students to select their own independent variables, develop and justify their own predictions, design their own procedure, and share their findings with the class. Since RTOP scores did not indicate growth in pedagogical reform between the Baseline lesson and Lesson 7 enactment, we speculate that several teachers may have veered from the INSPIRES lesson plan. Forthcoming discussion of the qualitative findings explains the traits of Lesson 7 enactment that may have hindered pedagogical growth at this time point.</p> <p>The present study addresses whether growth in pedagogical reform is evident in teacher-selected and teacher-written lessons (i.e., the Transfer lessons). Indeed, significant growth occurred between the Baseline and Transfer lessons, in both overall and the subcategories of Classroom Culture and Teacher–Student Relationships. These subcategories assess the degree to which teachers act as patient facilitators while creating a classroom environment that invites student communication, divergent thinking, active participation, and other qualities of student-directed learning (Piburn and Sawada [<reflink idref="bib33" id="ref58">33</reflink>]). We speculate that growth in teachers' pedagogical reform was influenced by their participation in the INSPIRES PD institute and the subsequent enactment of the Hemodialysis curriculum, which incorporates several pedagogical skills valued on the RTOP scale. Future discussion of qualitative findings helps identify common pedagogical traits that explain growth in the areas of Classroom Culture and Teacher–Student Relationships. Over the course of this longitudinal research study, we will make comparisons between the teacher participants and a group of teachers in a control group, which will better enable us to draw causal conclusions about the effects of the combined PD and educative curriculum on pedagogical growth.</p> <p>Finally, the quantitative results indicate that comparisons between biology and technology education teachers' performance did not yield any significant differences. This finding was surprising, as we speculated that biology teachers may be stronger than technology education teachers in enactment of the science-rich Lesson 7. Likewise, we thought technology education teachers may be stronger than biology teachers in the enactment of engineering-rich Lesson 11. These assumptions may still be true, as the RTOP scale may not be the instrument that can best capture this content-specific difference. That is, the RTOP instrument measures levels of pedagogical reform in STEM fields but does not necessarily differentiate between specific STEM domains. The forthcoming exploration of qualitative trends reveals some indication that despite experience and strong content knowledge in science, biology teachers do not always teach biology lessons using reformed pedagogy and may not have followed the Lesson 7 plan as written; similarly, even with experience and a background in designing and building projects, technology education teachers do not always incorporate reformed pedagogy when teaching the engineering process.</p> <hd id="AN0135909703-16">Qualitative Findings</hd> <p>Qualitative analysis was explored to explain and enhance the quantitative findings of the RTOP instrument. As lessons progressed longitudinally, teachers provided more prescribed, guided parameters within Baseline lessons and Lesson 7 (Tables 4 and 5, <emph>Guided vs. Open Strategy</emph>) and then progressed to allowing open-ended and autonomous elements within Lesson 11 and Transfer lessons (Tables 6 and 7, <emph>Guided vs. Open Strategy</emph>). The nature of Lesson 11 (as written) supported the open-ended design of a hemodialysis system which likely allowed this lesson to score significantly higher than others on the RTOP scale. Although Lesson 7 was written to allow student autonomy, we found that both biology and technology education teachers often controlled the lesson by presenting vocabulary prior to the experiment, telling/assigning independent variables to student groups, providing explicit procedures, and doing mathematical computations for students. It is not surprising that even the biology teachers altered Lesson 7 in these ways, which are common practices in traditionally taught science lessons, and some level of prior pedagogical discontentment may be necessary to motivate teachers to adopt reformed methodology (e.g., Southerland et al. [<reflink idref="bib48" id="ref59">48</reflink>]; McNeill et al. [<reflink idref="bib28" id="ref60">28</reflink>]). Such reworking of the Lesson 7 plan may account, in part, for why the RTOP analysis did not reveal a significant difference between biology and technology education teacher performance (Table 2) and, more generally, why quantitative RTOP scores are relatively low for Lesson 7 enactments. As a longitudinal study with progressive PD and experience implementing the INSPIRES Hemodialysis curriculum, we predict that the reformed qualities of Lesson 7 enactment may improve in subsequent years and that increased RTOP measures would naturally follow. Research in the field of PD programs has demonstrated that increasingly difficult changes in practice (e.g., biology teachers infusing engineering practices and technology education teachers incorporating scientific rationale) require increased PD time. Further, teachers evolve their practices differently over time and therefore require flexible instructional support to continue their pedagogical and content knowledge growth (Luft and Hewson [<reflink idref="bib23" id="ref61">23</reflink>]). The INSPIRES PD Institute takes a learner-centered approach, where teachers are the learners and their needs guide the focal topics for continued PD sessions over the course of a three-year study. Although Transfer lessons were typically not as strongly reformed as Lesson 11 (Table 3), we have found that Transfer lessons still incorporate more aspects of autonomy than Baseline lessons, such as students guiding the procedure instead of the teacher, and more emphasis on student rationale rather than the teacher telling key ideas (Tables 4 and 7, <emph>Guided vs. Open Strategy</emph>). This suggests that (<reflink idref="bib1" id="ref62">1</reflink>) these qualitative elements may account for some of the significant growth in RTOP scores between Baseline and Transfer lessons and (<reflink idref="bib2" id="ref63">2</reflink>) teacher participation in the INSPIRES PD and educative curriculum enactment may influence their pedagogical growth.</p> <p>The absence of pedagogical growth between Baseline lessons and Lesson 7 (Table 3) may also be attributed to how teachers probed for prior student knowledge over the course of the four documented time points. There was a tendency to utilize traditional bell work (i.e., drills to review prior concepts and completed individually; elicited student knowledge does not change the focus or sequence of the day's lesson) in both Baseline lessons and Lesson 7 (Tables 4 and 5, <emph>Probing of Prior Knowledge</emph>). By Lesson 11, teachers more frequently utilized reformed methods of eliciting prior knowledge (e.g., student artifacts from previous lessons) that typically progressed into a whole class discussion of scientific concepts relevant to aiding students in the next steps of their design challenge (Table 6, <emph>Probing of Prior Knowledge</emph>). A more widespread use of artifact sharing was observed; this pedagogical technique was explicitly modeled and encouraged during all INSPIRES PD sessions. Proper artifact sharing challenges students to make connections between the STEM concept underlying their chosen artifact and the greater design challenge of the Hemodialysis unit (Blumenfeld et al. [<reflink idref="bib4" id="ref64">4</reflink>]; Singer et al. [<reflink idref="bib45" id="ref65">45</reflink>]; Krajcik [<reflink idref="bib19" id="ref66">19</reflink>]). Transfer lessons often avoided traditional bell work and generally engaged students (Table 7, <emph>Probing of Prior Knowledge</emph>), although employed strategies were not as reformed as Lesson 11 (i.e., student presentation of prior results in lieu of artifacts).</p> <p>The use of "prediction making" revealed qualitative differences in the areas of student sharing and student rationale. That is, Baseline lessons treated predictions as formalities in the scientific process while enactment of Lesson 7 posed predictions as a confirmational strategy, yet during both lessons teachers did not typically ask students to share their predictions or provide scientific rationale (Tables 4 and 5, <emph>Making Predictions</emph>). Alternatively, most teachers expected students to share their design ideas within groups and with the teacher during Lesson 11. Students were also expected to provide scientific or mathematic rationale for their design decisions (Table 6, <emph>Making Predictions</emph>). Since Lesson 11 is engineering-based, the authors treated "designs with rationale" as well-constructed predictions, as they demonstrate students' justified belief that their idea will succeed. Transfer lessons were typically more reformed in the area of students sharing their predictions with the teacher; yet, in general, the press for rationale was shallow (Table 7, <emph>Making Predictions</emph>). However, this gradual improvement in reformed pedagogy may help explain why the overall RTOP scores demonstrate growth from Baseline to Transfer lessons (Table 3).</p> <p>When considering "connection making" within lessons, there is some level of (<reflink idref="bib1" id="ref67">1</reflink>) "past-to-present" and (<reflink idref="bib2" id="ref68">2</reflink>) "real world" connection evident at all four time points (Tables 4–7, <emph>Making Connections</emph>). That is, teachers commonly revisited concepts, data, etc., from previous lessons and helped students apply that prior knowledge to the current lesson. Lesson 11 continued to stand out, however, in that teachers encouraged students to make more explicit connections between multiple STEM domains (e.g., connecting science concepts to engineering design decisions) and more frequently referenced the engineering design loop and design challenge requirements (criteria and constraints). Connections to the overall engineering design challenge during Lesson 7 were typically superficial.</p> <p>Qualitative findings suggest that the INSPIRES lessons were more conducive to student reflection than either the Baseline or Transfer lessons. Both Lesson 7 and Lesson 11 encouraged students to use a journal to record and reference scientific and engineering concepts. However, journals were typically used as documentation tools during the science-based Lesson 7 (Table 5, <emph>Student Reflection</emph>). Student reflection on how Lesson 7 could inform their approach of the engineering design challenge was limited. Notably, teachers often ran short on time during Lesson 7 and could not include all concluding elements of the lesson plan in a single 90-min period. This often played out in students not finishing their experiments, teachers stepping in to do mathematical computations for students, and teachers announcing that class-wide experimental findings would be discussed in a future class (qualitative data not shown). Although the alterations some teachers made to the INSPIRES Lesson 7 plan might influence the duration of the lesson (see discussion on <emph>Open vs</emph>. <emph>Guided Strategy</emph> in the previous texts), it is understandable that the absence of result sharing, interpretation, and application would confer a lower score on the RTOP scale. According to the literature, when teachers engage their students in an inquiry-based lesson, sometimes more focus is placed on completing the activity correctly than on taking the proper steps to assist students' understanding of the underlying STEM concepts (Blumenfeld et al. [<reflink idref="bib4" id="ref69">4</reflink>]; Singer et al. [<reflink idref="bib45" id="ref70">45</reflink>]). Reserving time to connect the activity to concepts during the introduction and conclusion of the lesson is an approach outlined in all lessons of the INSPIRES Hemodialysis unit. Commonly, teachers would alter the lesson plan by front-loading information (i.e., vocabulary review) before the inquiry-based lab activity of Lesson 7. Consequently, many teachers did not have time to complete the experiment and/or engage in a deep reflection at the conclusion of the period. Student reflection during Lesson 11 was enhanced as journal use became more dynamic. Engineering journals served as a forum for critical thinking in addition to documentation (Table 6, <emph>Student Reflection</emph>). Baseline and Transfer lessons yielded shallow student reflections centered around teacher-prompted recollection of facts at the end of the lesson (Tables 4 and 7, <emph>Student Reflection</emph>).</p> <p>One reason why Lesson 11 may be more reform-oriented than the other lessons is because the design-based lesson may have pushed teachers from their comfort zones and encouraged them to follow the lesson plan more closely. Evidence for this speculation is presented when teachers enact specific pedagogical strategies in Lesson 11, but not Lesson 7, although such strategies are outlined in both lesson plan guides. For example, artifacts are explicitly encouraged in the guides for both Lessons 7 and 11; we observed teachers enacting student artifact-sharing more in Lesson 11 than in Lesson 7. Similarly, both lesson plan guides encourage teachers to prompt students in sketching their experimental systems. Within our qualitative subsample, we found that only technology education teachers followed this strategy during Lesson 7, while both biology and technology education teachers prompted design sketches in Lesson 11. In the latter example, technology education teachers may have followed the Lesson 7 plan more closely than the biology teachers, perhaps because the non-science teachers require more support while enacting a science-based lesson. Then, perhaps all teachers sought extra support from the Lesson 11 guide when enacting a novel, engineering design-based lesson. Therefore, while there were no quantitative significant differences identified between technology education and biology teacher RTOP scores, the qualitative analysis suggests that technology education teachers may have been following the lesson plan more closely than biology teachers during Lesson 7. Anecdotal evidence, based on conversations with multiple biology teacher participants during the INSPIRES summer PD institute, revealed that several of these teachers had previously instructed lab-based lessons on the concept of diffusion. Although the underlying concept of diffusion and some of the materials (e.g., dialysis membrane) may be similar between the INSPIRES Hemodialysis Lesson 7 and a traditional high school biology diffusion lab, the overall structure and supportive pedagogy were likely very different. Often, traditional labs are conducted as confirmational activities where information is front-loaded, rather than opportunities to exercise students' ability to think critically. Although Lesson 7 is framed as an inquiry-based lesson, its structure may have been traditionalized if science teachers felt they had enacted similar diffusion labs before and therefore reverted to the traditional strategies they used to teach a typical diffusion lab lesson. That is, if teachers believe they are enacting something familiar or do not recognize the need for, or nuance in, the reform (i.e., conducting the lab in a different manner to highlight different practices), then there may be less motivation to adjust an existing schema of how-to-teach a seemingly familiar lesson (e.g., Southerland et al. [<reflink idref="bib48" id="ref71">48</reflink>]; McNeill et al. [<reflink idref="bib28" id="ref72">28</reflink>]).</p> <p>Lesson 11 was the only documented time point where teachers shared their personal experiences with students of grappling with the INSPIRES Hemodialysis unit (Table 6, <emph>Teacher Sharing</emph>). By conveying their personal struggles, teachers brought a humanizing component to their teaching and the lesson. Teachers and students could relate in their experience of a challenging open-ended problem. By relating to the students as they wrestled with the project, teacher–student bonds may have been established that in turn could influence students' persistence, as teacher–student relationships and teacher empathy have positive influences on student learning outcomes (e.g., Faber and Mazlish [<reflink idref="bib15" id="ref73">15</reflink>]; Jennings and Greenberg [<reflink idref="bib20" id="ref74">20</reflink>]). During the summer PD institute, many teachers voiced concerns over their students' fragility over failure and the INSPIRES unit presenting too great of a challenge for students' self-esteem. Previous research has shown that students of varied abilities are capable of success in open-ended design challenges similar to the INSPIRES Hemodialysis unit (Reeves and Ross [<reflink idref="bib35" id="ref75">35</reflink>]), although teachers often underestimate students' abilities to pursue and learn from these challenges (e.g., Bryan and Atwater [<reflink idref="bib5" id="ref76">5</reflink>]). Other research on the use of educative curricula has shown that teachers' approaches to teaching science are transformed (Pringle et al. [<reflink idref="bib34" id="ref77">34</reflink>]), and perhaps, the INSPIRES teachers are beginning to transform their methodology based on their experience working through the curriculum. Relating experiences of struggles and persistence to even small victories may have supported or maintained student confidence and participation for the duration of Lesson 11.</p> <p>One of the questions that the present study posed was whether a shift toward reformed pedagogy would be evident between Baseline and Transfer lessons. Indeed, quantitative analyses have revealed that such a shift has begun, especially in the areas of Classroom Culture and Teacher–Student Relationships (Table 3). Qualitative analyses further explain how teachers demonstrate growth in these specific areas (Tables 4 and 7). In particular, there is an increase in elicited student ideas, student engagement, communicating (shallow) rationale with teachers, student autonomy, and (implicit) connections to data or concepts of prior lessons. The RTOP subcategories of Classroom Culture and Teacher–Student Relationships assess the degree to which teachers act as patient facilitators while creating a classroom environment that invites student communication, divergent thinking, active participation, and other qualities of student-directed learning (Piburn and Sawada [<reflink idref="bib33" id="ref78">33</reflink>]). Therefore, the qualitative evidence that characterizes typical Baseline and Transfer lessons supports the significant quantitative gains observed in these domains. The present study documents teacher growth after one year of participation in a three-year longitudinal study. Continued participants will experience two subsequent summer INSPIRES PD institutes, spanned by multiple monthly PD sessions. Therefore, we predict that this extended PD model will support increased growth in pedagogical reform over the final two years of the study. Substantial and difficult change in practice and content knowledge requires an increased commitment to PD-based support (Luft and Hewson [<reflink idref="bib23" id="ref79">23</reflink>]).</p> <p>Overall pedagogical growth between Baseline and Transfer lessons may be further supported by increased incidence of argumentation. In Lesson 11 and Transfer lessons, teachers typically set higher standards of pressing students for providing STEM-based rationale. Previous research in argumentation within STEM classrooms has documented significant gains in both the frequency and quality of arguments between the first and second years of implementation (Erduran et al. [<reflink idref="bib14" id="ref80">14</reflink>]). Yet, in a separate study, Osborne et al. ([<reflink idref="bib32" id="ref81">32</reflink>]) found that teachers' participation in a argumentation-focused PD program that ran 3–6 h once a month for nine months influenced growth in the quality of students' arguments, albeit not significantly. It is thought that recurrent argumentation throughout the curriculum would better support significant growth in the skill, rather than argumentation occurring primarily during nine lessons taught over the nine-month period. In the INSPIRES unit, teachers are encouraged to incorporate argumentation in multiple lessons and are further supported in developing this skill throughout three consecutive, annual, week-long summer PD institutes spanned by multiple 2-h-long monthly PD sessions. Thus, there is great potential that argumentation will grow significantly by the end of the longitudinal study. McNeill et al. ([<reflink idref="bib28" id="ref82">28</reflink>]) supported a group of middle school science teachers in enacting an educative curriculum focused on improving argumentation; they found that while some teachers used instructional practices in line with argumentation, several others oversimplified the structured curriculum, which resulted in traditionally led lessons where students engaged in <emph>pseudoargumentation</emph>. Those teachers that best supported their students in developing argumentation discourse were those that (<reflink idref="bib1" id="ref83">1</reflink>) understood argumentation to be a cognitively enriching process, (<reflink idref="bib2" id="ref84">2</reflink>) actively reflected on the educative curriculum, and (<reflink idref="bib3" id="ref85">3</reflink>) exhibited discontent with their prior teaching methodology. Similarly, Marco-Bujosa et al. ([<reflink idref="bib25" id="ref86">25</reflink>]) found that teachers who openly engaged in their own learning, while enacting an educative curriculum, made larger learning gains in argumentation practices than those teachers that treated the educative curriculum primarily as a resource for student activities. Therefore, INSPIRES participants may benefit from ongoing PD opportunities to actively reflect on their growth in reformed pedagogy. At this time, argumentation witnessed in the INSPIRES classrooms somewhat resembles Osborne et al.'s ([<reflink idref="bib32" id="ref87">32</reflink>]) and McNeill et al.'s ([<reflink idref="bib28" id="ref88">28</reflink>]) findings, as much of the teacher press and student rationale observed during Transfer lessons was present yet shallow in quality, and discourse quickly ended following students' superficial contributions. Parallel work has utilized instruments to document teachers' self-reported engineering self-efficacy and areas of concern, longitudinally over the course of the three-year INSPIRES project, which may shed light on which teachers felt discontent with their practices at different stages of the study. Finally, McNeill and Knight ([<reflink idref="bib29" id="ref89">29</reflink>]) found that classroom argumentation was significantly enhanced following a PD program that included the following components: (<reflink idref="bib1" id="ref90">1</reflink>) analyzing evidence of prior classroom practice, (<reflink idref="bib2" id="ref91">2</reflink>) supporting teachers in infusing argumentation within lessons, (<reflink idref="bib3" id="ref92">3</reflink>) expecting teachers to share selected evidence of their classroom practice, and (<reflink idref="bib4" id="ref93">4</reflink>) encouraging teacher reflection on past practices to modify practices for the future. The INSPIRES PD institute also captures elements of these four themes as it includes the following: (<reflink idref="bib1" id="ref94">1</reflink>) documentation and analysis of baseline-level teacher practices (as described in the present study), (<reflink idref="bib2" id="ref95">2</reflink>) continued discussion and modeling of how teachers can press students for scientific and quantitative rationale for design decisions, (<reflink idref="bib3" id="ref96">3</reflink>) requesting that teachers prepare and share artifacts from their recent infusion of reformed pedagogical strategies, and (<reflink idref="bib4" id="ref97">4</reflink>) creating space for reflection and setting new goals during monthly PD sessions. Once again, the deliberate planning of the INSPIRES PD program alongside the careful structuring of the educative curriculum holds promise for substantial teacher growth and student learning.</p> <hd id="AN0135909703-17">Conclusion</hd> <p>Overall, we find that results addressing our first research question demonstrate that reformed pedagogy improved significantly during the first year of the study. Particularly, the instructional practices of the teachers improved significantly between enactment of the Baseline and Transfer lessons during the first year of the PD program. The findings are well aligned with previous studies when a similar PD model was utilized with middle school science teachers (Singer et al. [<reflink idref="bib46" id="ref98">46</reflink>]) and with high school technology education teachers (Singer et al. [<reflink idref="bib47" id="ref99">47</reflink>]). Both prior studies used a similar repeated measures design to analyze RTOP scores. Results from the present study were conducted with a much larger population of teachers and also demonstrated significant differences on more RTOP subcategories than prior studies. Unlike the present study, Singer et al. ([<reflink idref="bib47" id="ref100">47</reflink>]) found gains in Propositional Knowledge. Video coders noted that while teachers enacted the INSPIRES curriculum, the teachers often failed to connect the design challenge (building a hemodialysis machine) to the science concepts (e.g., diffusion). Student ideas were often solicited then discarded for the teachers' preconceived ideas of how the lesson should proceed. By Lesson 11, teachers began releasing control of the lesson direction to students and allowed them to design and build their own machines. Even with stronger emphasis on student ideas, connections to the underlying STEM practices were inconsistent. After-school PD meetings used a lesson-study model and fostered discussions about how to connect the science and engineering more strongly to Lesson 11 and how to lead other lessons more similarly to Lesson 11.</p> <p>Qualitative analysis demonstrated that Transfer lessons exhibited more reformed qualities (i.e., student autonomy, connections to prior knowledge, open-ended design-based activities) than Baseline lessons. Multiple themes emerged that were used to characterize each lesson: Baseline, Lesson 7, Lesson 11, Transfer (Tables 4–7).</p> <p>Regarding our second research question, we do not see a significant difference between biology and technology education teachers' pedagogical growth at this time. We recognize that this finding may change as this research project continues to unfold. The following two years of this longitudinal study are expected to yield further reform in pedagogical skills and the integration of engineering practices into STEM classrooms. Close observation of this pedagogical evolution has the potential to reveal differences between the biology and technology education teacher populations that may surface at later times. To date, these findings provide insights for rethinking the structure of professional development, particularly in the integrated use of an educative curriculum aligned with intended professional development goals.</p> <hd id="AN0135909703-18">Recommendations</hd> <p>Results from the present study will be compared against RTOP data and qualitative trends measured from teachers in a control group. The control group comprises biology and technology education teachers in the same district who did not participate in the INSPIRES PD or implement the INSPIRES curriculum. Additionally, while the RTOP rubric facilitated the present study of student-centered pedagogical change in STEM classroom environments, other observational tools exist that more specifically address changes in classroom engineering practices and principles. The teacher lessons evaluated here via the RTOP were simultaneously coded using a research instrument sensitive to explicit engineering lesson qualities. Next steps in research include the analysis and dissemination of forthcoming findings pertaining to engineering-specific changes and how they may align to the broader RTOP results. In general, we recommend that educative curricula be used as a vector for integrating elements of educational reform to address NGSS challenges, especially in engineering education. Professional development that supports teaches in implementing a strongly written engineering educative curriculum can allow the transfer of design-based pedagogy into teacher-developed curricula.</p> <hd id="AN0135909703-19">Funding Information</hd> <p>This project was funded by a Discovery Research K–12 National Science Foundation grant (DRL 1418183).</p> <hd id="AN0135909703-20">Acknowledgements</hd> <p>We wish to thank the teachers, administrators, and program staff of our partner school district for their time and efforts in this study. The following UMBC students assisted in the collection of classroom recordings: Abby Singer, Ahmed Al-Salihi, Goureesh Paranjpe, Garrett Bockmiller, Marcus Foster, and Ekaterina DiBenedetto.</p> <hd id="AN0135909703-21">Compliance with Ethical Standards</hd> <p></p> <hd id="AN0135909703-22">Conflict of Interest</hd> <p>The authors declare that they have no conflict of interest.</p> <hd id="AN0135909703-23">Ethical Approval</hd> <p>All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.</p> <hd id="AN0135909703-24">Informed Consent</hd> <p>Informed consent was obtained from all individual participants included in the study.</p> <ref id="AN0135909703-25"> <title> References </title> <blist> <bibl id="bib1" idref="ref19" type="bt">1</bibl> <bibtext> Amolins MW, Ezrailson CM, Pearce DA, Elliott AJ, Vitiello PF. Evaluating the effectiveness of a laboratory-based professional development program for science educators. Advances in Physiology Education. 2015; 39; 4: 341-351</bibtext> </blist> <blist> <bibl id="bib2" idref="ref20" type="bt">2</bibl> <bibtext> Arias AM, Davis EA, Marino JC, Kademian SM, Palincsar AS. Teachers' use of educative curriculum materials to engage students in science practices. International Journal of Science Education. 2016; 38; 9: 1504-1526</bibtext> </blist> <blist> <bibl id="bib3" idref="ref14" type="bt">3</bibl> <bibtext> Ball DL, Cohen DK. Reform by the book: what is—or might be—the role of curriculum materials in teacher learning and instructional reform?. Educational Research. 1996; 25: 6-8</bibtext> </blist> <blist> <bibl id="bib4" idref="ref43" type="bt">4</bibl> <bibtext> Blumenfeld PC, Soloway E, Marx RW, Krajcik JS, Guzdial M, Palincsar A. Motivating project-based learning: sustaining the doing, supporting the learning. Educational Psychologist. 1991; 26; 3–4: 369-398</bibtext> </blist> <blist> <bibl id="bib5" idref="ref76" type="bt">5</bibl> <bibtext> Bryan LA, Atwater MM. Teacher beliefs and cultural models: a challenge for science teacher preparation programs. Science Education. 2002; 86; 6: 821-839</bibtext> </blist> <blist> <bibl id="bib6" idref="ref28" type="bt">6</bibl> <bibtext> Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: building networked improvement communities in education. In Frontiers in sociology of education (pp. 127–162). Dordrecht: Springer.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref54" type="bt">7</bibl> <bibtext> Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluation normed and standardized assessment instruments in psychology. Psychology Assessment. 1994; 6; 4: 284-290</bibtext> </blist> <blist> <bibl id="bib8" idref="ref56" type="bt">8</bibl> <bibtext> Coffey A, Atkinson P. Making sense of qualitative data: complementary research strategies. 1996: Thousand Oaks; Sage Publications: 54-82</bibtext> </blist> <blist> <bibl id="bib9" idref="ref2" type="bt">9</bibl> <bibtext> Cuban L. Inside the black box of classroom practice: change without reform in American education. 2013: Cambridge; Harvard Education Press</bibtext> </blist> <blist> <bibtext> Darling-Hammond L, Mclaughlin MW. Policies that support professional development in an era of reform. Phi Delta Kappan. 1995; 76: 597-604</bibtext> </blist> <blist> <bibtext> Davis E, Krajcik J. Designing educative curriculum materials to promote teacher learning. Educational Researcher. 2005; 34; 3: 3-14</bibtext> </blist> <blist> <bibtext> Desimone LM. Improving impact studies of teachers' professional development: toward better conceptualizations and measures. Educational Research. 2009; 38; 3: 181-199</bibtext> </blist> <blist> <bibtext> Enderle P, Dentzau M, Roseler K, Southerland S, Granger E, Hughes R, Saka Y. Examining the influence of RETs on science teacher beliefs and practice. Science Education. 2014; 98; 6: 1077-1108</bibtext> </blist> <blist> <bibtext> Erduran S, Simon S, Osborne J. TAPping into argumentation: developments in the application of Toulmin's argument pattern for studying science discourse. Science Education. 2004; 88; 6: 915-933</bibtext> </blist> <blist> <bibtext> Faber A, Mazlish E. How to talk so kids can learn. 2008: New York; Simon and Schuster</bibtext> </blist> <blist> <bibtext> Fishman EJ, Borko H, Osborne J, Gomez F, Rafanelli S, Reigh E, Tseng A, Million S, Berson E. A practice-based professional development program to support scientific argumentation from evidence in the elementary classroom. Journal of Science Teacher Education. 2017; 28; 3: 222-249</bibtext> </blist> <blist> <bibtext> Knaggs C, Schneider R. Thinking like a scientist: using vee-maps to understand process and concepts in science. Research in Science Education. 2012; 42; 4: 609-632</bibtext> </blist> <blist> <bibtext> Krajcik JS, Blumenfeld PC. Project-based learning. 2006: 317-334</bibtext> </blist> <blist> <bibtext> Krajcik J. Project-based science: engaging students in three-dimensional learning. The Science Teacher. 2015; 82; 1: 25</bibtext> </blist> <blist> <bibtext> Jennings PA, Greenberg MT. The prosocial classroom: teacher social and emotional competence in relation to student and classroom outcomes. Review of Educational Research. 2009; 79; 1: 491-525</bibtext> </blist> <blist> <bibtext> Lotter C, Rushton G, Singer JE. Teacher enactment patterns: how can we help move all teachers to reform-based inquiry practice through professional development?. Journal of Science Teacher Education. 2013; 24: 1263-1291</bibtext> </blist> <blist> <bibtext> Loucks-Horsley S, Stiles E, Mundry S, Love N, Hewson P. Designing professional development for teachers of science and mathematics. 20103: Thousand Oaks; Corwin</bibtext> </blist> <blist> <bibtext> Luft JA, Hewson PW. Research on teacher professional development programs in science. Handbook of Research on Science Education. 2014; 2: 889-909</bibtext> </blist> <blist> <bibtext> MacIsaac, D., Sawada, D., & Falconer, K. (2001). Using the Reformed Teaching Observation Protocol (RTOP) as a catalyst for self-reflective change in secondary science teaching. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.</bibtext> </blist> <blist> <bibtext> Marco-Bujosa LM, McNeill KL, González-Howard M, Loper S. An exploration of teacher learning from an educative reform-oriented science curriculum: case studies of teacher curriculum use. Journal of Research in Science Teaching. 2017; 54; 2: 141-168</bibtext> </blist> <blist> <bibtext> Marx RW, Blumenfeld PC, Krajcik JS, Fishman B, Soloway E, Geier R, Tal RT. Inquiry-based science in the middle grades: assessment of learning in urban systemic reform. Journal of Research in Science Teaching. 2004; 41; 10: 1063-1080</bibtext> </blist> <blist> <bibtext> McAleer, S. D. (2008). Professional growth through mentoring: a study of experienced mathematics teachers participating in a content-based online mentoring and induction program. Dissertation Abstracts International-A, 69 (08).</bibtext> </blist> <blist> <bibtext> McNeill KL, González-Howard M, Katsh-Singer R, Loper S. Moving beyond pseudoargumentation: teachers' enactments of an educative science curriculum focused on argumentation. Science Education. 2017; 101; 3: 426-457</bibtext> </blist> <blist> <bibtext> McNeill KL, Knight AM. Teachers' pedagogical content knowledge of scientific argumentation: the impact of professional development on K–12 teachers. Science Education. 2013; 97; 6: 936-972</bibtext> </blist> <blist> <bibtext> . A framework for K-12 science education: practices, crosscutting concepts, and core ideas. 2012: Washington, DC; National Academies Press</bibtext> </blist> <blist> <bibtext> . Next Generation Science Standards. [Website]. 2013: Washington, DC; National Research Council, National Science Teachers Association, & American Association for the Advancement of Science</bibtext> </blist> <blist> <bibtext> Osborne J, Erduran S, Simon S. Enhancing the quality of argumentation in school science. Journal of Research in Science Teaching. 2004; 41; 10: 994-1020</bibtext> </blist> <blist> <bibtext> Piburn M, Sawada D. Reformed teaching observation protocol (RTOP): reference manual (ACEPT technical report IN00-3). 2000: Tempe; Arizona State University, Arizona Collaborative for Excellence in the Preparation of Teachers</bibtext> </blist> <blist> <bibtext> Pringle RM, Mesa J, Hayes L. Professional development for middle school science teachers: does an educative curriculum make a difference?. Journal of Science Teacher Education. 2017; 28; 1: 57-72</bibtext> </blist> <blist> <bibtext> Reeves R, Ross J. AC 2010–1952: a novel approach to professional development. American Society for Engineering Education, Conference Proceedings. 2010; 15: 1</bibtext> </blist> <blist> <bibtext> Reiser, B. J. (2014). Designing coherent storylines aligned with NGSS for the K-12 classroom. Paper presented at the Professional Development Institute of the National Science Education Leadership Association, Boston, MA. Retrieved from https://<ulink href="http://www.academia.edu/6884962/Designing%5fCoherent%5fStorylines%5fAligned%5fwith%5fNGSS%5ffor%5fthe%5fK-12%5fClassroom">www.academia.edu/6884962/Designing%5fCoherent%5fStorylines%5fAligned%5fwith%5fNGSS%5ffor%5fthe%5fK-12%5fClassroom</ulink>. Accessed 22 June 2017.</bibtext> </blist> <blist> <bibtext> Remillard JT. Can curriculum materials support teachers' learning? Two fourth-grade teachers' use of a new mathematics text. The Elementary School Journal. 2000; 100; 4: 331-350</bibtext> </blist> <blist> <bibtext> Richmond G, Parker JM, Kaldaras L. Supporting reform-oriented secondary science teaching through the use of a framework to analyze construction of scientific explanations. Journal of Science Teacher Education. 2016; 27; 5: 477-493</bibtext> </blist> <blist> <bibtext> Roseman, J. E., Fortus, D., Krajcik, J., & Reiser, B. J. (2015). Curriculum materials for Next Generation Science Standards: what the science education research community can do. In NARST Annual International Conference, Chicago, IL.</bibtext> </blist> <blist> <bibtext> Roseman JE, Herrmann-Abell CF, Koppal M. Designing for the Next Generation Science Standards: educative curriculum materials and measures of teacher knowledge. Journal of Science Teacher Education. 2017; 28; 1: 111-141</bibtext> </blist> <blist> <bibtext> Ross J, Bayles T, Singer J Sneider C. The inspires curriculum. The go-to guide for engineering curricula, grades 9–12. 2015: London; SAGE Publications Ltd.: 19-30</bibtext> </blist> <blist> <bibtext> Rushton G, Lotter C, Singer J. Chemistry teachers' emerging expertise in inquiry teaching: the effect of an authentic professional development model on beliefs and practice. Journal of Science Teacher Education. 2011; 22: 23-52</bibtext> </blist> <blist> <bibtext> Saderholm, J., Ronau, R. N., Rakes, C. R., Bush, S. B., & Mohr-Schroeder, M. (2016). The critical role of a well-articulated conceptual framework to guide professional development: an evaluation of a state-wide two-week program for mathematics and science teachers. Professional Development in Education. 10.1080/19415257.2016.1251485.</bibtext> </blist> <blist> <bibtext> Schneider RM, Krajcik J, Blumenfeld P. Enacting reform-based science materials: the range of teacher enactments in reform classrooms. Journal of Research in Science Teaching. 2005; 42; 3: 283-312</bibtext> </blist> <blist> <bibtext> Singer J, Marx RW, Krajcik J, Clay Chambers J. Constructing extended inquiry projects: curriculum materials for science education reform. Educational Psychologist. 2000; 35; 3: 165-178</bibtext> </blist> <blist> <bibtext> Singer J, Lotter C, Feller R, Gates. Exploring a model of situated professional development: impact on classroom practice. Journal of Science Teacher Education. 2011; 22; 3: 203-227</bibtext> </blist> <blist> <bibtext> Singer JE, Ross JM, Jackson-Lee Y. Professional Development for the Integration of Engineering in High School STEM Classrooms. Journal of Pre-College Engineering Education Research. 2016; 6; 1: 30-44</bibtext> </blist> <blist> <bibtext> Southerland SA, Nadelson L, Sowell S, Saka Y, Kahveci M, Granger EM. Measuring one aspect of teachers' affective states: development of the science teachers' pedagogical discontentment scale. School Science and Mathematics. 2012; 112; 8: 483-494</bibtext> </blist> <blist> <bibtext> Spradley JP. Participant observation. 1980: Orlando; Harcourt Brace Jovanovich College Publishers</bibtext> </blist> <blist> <bibtext> Sztajn P. Research commentary: standards for reporting mathematics professional development in research studies. Journal for Research in Mathematics Education. 2011; 42; 3: 220-236</bibtext> </blist> <blist> <bibtext> Willis, M. B. (2018). Examination of novice science teachers' use of project-based instructional strategies (Doctoral dissertation, University of Massachusetts Lowell).</bibtext> </blist> </ref> <aug> <p>By Tory Williams; Jonathan Singer; Jacqueline Krikorian; Christopher Rakes and Julia Ross</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib30" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib39" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib31" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib38" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib41" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib47" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib36" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib44" firstref="ref13"></nolink> <nolink nlid="nl10" bibid="bib17" firstref="ref16"></nolink> <nolink nlid="nl11" bibid="bib11" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib43" firstref="ref22"></nolink> <nolink nlid="nl13" bibid="bib10" firstref="ref23"></nolink> <nolink nlid="nl14" bibid="bib27" firstref="ref24"></nolink> <nolink nlid="nl15" bibid="bib12" firstref="ref25"></nolink> <nolink nlid="nl16" bibid="bib22" firstref="ref26"></nolink> <nolink nlid="nl17" bibid="bib50" firstref="ref27"></nolink> <nolink nlid="nl18" bibid="bib42" firstref="ref30"></nolink> <nolink nlid="nl19" bibid="bib46" firstref="ref31"></nolink> <nolink nlid="nl20" bibid="bib21" firstref="ref32"></nolink> <nolink nlid="nl21" bibid="bib37" firstref="ref33"></nolink> <nolink nlid="nl22" bibid="bib34" firstref="ref37"></nolink> <nolink nlid="nl23" bibid="bib40" firstref="ref38"></nolink> <nolink nlid="nl24" bibid="bib28" firstref="ref39"></nolink> <nolink nlid="nl25" bibid="bib26" firstref="ref46"></nolink> <nolink nlid="nl26" bibid="bib18" firstref="ref47"></nolink> <nolink nlid="nl27" bibid="bib51" firstref="ref48"></nolink> <nolink nlid="nl28" bibid="bib24" firstref="ref49"></nolink> <nolink nlid="nl29" bibid="bib13" firstref="ref50"></nolink> <nolink nlid="nl30" bibid="bib33" firstref="ref52"></nolink> <nolink nlid="nl31" bibid="bib49" firstref="ref55"></nolink> <nolink nlid="nl32" bibid="bib48" firstref="ref59"></nolink> <nolink nlid="nl33" bibid="bib23" firstref="ref61"></nolink> <nolink nlid="nl34" bibid="bib45" firstref="ref65"></nolink> <nolink nlid="nl35" bibid="bib19" firstref="ref66"></nolink> <nolink nlid="nl36" bibid="bib15" firstref="ref73"></nolink> <nolink nlid="nl37" bibid="bib20" firstref="ref74"></nolink> <nolink nlid="nl38" bibid="bib35" firstref="ref75"></nolink> <nolink nlid="nl39" bibid="bib14" firstref="ref80"></nolink> <nolink nlid="nl40" bibid="bib32" firstref="ref81"></nolink> <nolink nlid="nl41" bibid="bib25" firstref="ref86"></nolink> <nolink nlid="nl42" bibid="bib29" firstref="ref89"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1212690
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Measuring Pedagogy and the Integration of Engineering Design in STEM Classrooms
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Williams%2C+Tory%22">Williams, Tory</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-3600-1174">0000-0002-3600-1174</externalLink>)<br /><searchLink fieldCode="AR" term="%22Singer%2C+Jonathan%22">Singer, Jonathan</searchLink><br /><searchLink fieldCode="AR" term="%22Krikorian%2C+Jacqueline%22">Krikorian, Jacqueline</searchLink><br /><searchLink fieldCode="AR" term="%22Rakes%2C+Christopher%22">Rakes, Christopher</searchLink><br /><searchLink fieldCode="AR" term="%22Ross%2C+Julia%22">Ross, Julia</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Science+Education+and+Technology%22"><i>Journal of Science Education and Technology</i></searchLink>. Jun 2019 28(3):179-194.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 16
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2019
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: DRL1418183
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Engineering+Education%22">Engineering Education</searchLink><br /><searchLink fieldCode="DE" term="%22Design%22">Design</searchLink><br /><searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Science%22">Secondary School Science</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Student+Relationship%22">Teacher Student Relationship</searchLink><br /><searchLink fieldCode="DE" term="%22Scientific+Concepts%22">Scientific Concepts</searchLink><br /><searchLink fieldCode="DE" term="%22Faculty+Development%22">Faculty Development</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Change%22">Educational Change</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Education%22">Science Education</searchLink><br /><searchLink fieldCode="DE" term="%22Active+Learning%22">Active Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Inquiry%22">Inquiry</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s10956-018-9756-y
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1059-0145
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The present study examined changes in high school biology and technology education pedagogy during the first year of a three-year professional development (PD) program using the INSPIRES educative curriculum. The Next Generation Science Standards (NGSS) calls for the integration of science and engineering through inquiry-based pedagogy that shifts the burden of thinking from the teacher to the student. This call is especially challenging for teachers untrained in inquiry teaching and engineering or science concepts. The INSPIRES educative curriculum materials and PD provided a mechanism for teachers to transform their teaching to meet the NGSS challenges. This study followed a longitudinal triangulation mixed methods design. Selected lessons were video recorded, scored on the Reformed Teaching Observation Protocol (RTOP) rubric, and examined for qualitative trends. Year 1 results indicated that teachers had begun to transform their teaching and pointed to particular lessons within the INSPIRES curriculum that most facilitated the reform. Instructional practices of participants improved significantly as a result of the INSPIRES PD program and also aligned with previous, similar studies. These findings provide insights for rethinking the structure of professional development, particularly in the integrated use of an educative curriculum aligned with intended professional development goals.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2019
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1212690
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1212690
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10956-018-9756-y
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
        StartPage: 179
    Subjects:
      – SubjectFull: Engineering Education
        Type: general
      – SubjectFull: Design
        Type: general
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: Secondary School Science
        Type: general
      – SubjectFull: Teacher Student Relationship
        Type: general
      – SubjectFull: Scientific Concepts
        Type: general
      – SubjectFull: Faculty Development
        Type: general
      – SubjectFull: Educational Change
        Type: general
      – SubjectFull: Science Education
        Type: general
      – SubjectFull: Active Learning
        Type: general
      – SubjectFull: Inquiry
        Type: general
    Titles:
      – TitleFull: Measuring Pedagogy and the Integration of Engineering Design in STEM Classrooms
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Williams, Tory
      – PersonEntity:
          Name:
            NameFull: Singer, Jonathan
      – PersonEntity:
          Name:
            NameFull: Krikorian, Jacqueline
      – PersonEntity:
          Name:
            NameFull: Rakes, Christopher
      – PersonEntity:
          Name:
            NameFull: Ross, Julia
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Type: published
              Y: 2019
          Identifiers:
            – Type: issn-print
              Value: 1059-0145
          Numbering:
            – Type: volume
              Value: 28
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Journal of Science Education and Technology
              Type: main
ResultId 1