View in EDS HTML Full Text PDF Full Text

Measuring the Amorphous: Substantive and Methodological Outcomes from Concept Maps

Saved in:
Bibliographic Details
Title: Measuring the Amorphous: Substantive and Methodological Outcomes from Concept Maps
Language: English
Authors: Foley, Rider W. (ORCID 0000-0002-9362-8790), Ferguson, Sean M., Pollack, Catherine C. (ORCID 0000-0002-7434-5306)
Source: Journal of Engineering Education. Jan 2021 110(1):161-183.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 23
Publication Date: 2021
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Undergraduate Students, Engineering Education, Student Evaluation, Concept Mapping, Concept Formation, Problem Solving, Thinking Skills, Evaluation Methods, Accreditation (Institutions)
DOI: 10.1002/jee.20373
ISSN: 1069-4730
Abstract: Background: Accreditation organizations broadened program assessment criteria in ways that present challenges for the evaluation of learning outcomes. This is especially the case in courses where there are not narrowly defined questions or definitive solutions, such as engineering ethics. While protocols for learning assessment exist, there is limited research exploring sociotechnical learning outcomes in a manner that combines theoretical, empirical, and procedural aspects of assessment. Purpose/Hypothesis: This paper shares 3 years of research into the effectiveness of concept maps as an assessment tool for engineering students in courses that prioritize professional skills development and sociotechnical thinking. We show that concept maps can offer evidence of knowledge formation and learning outcomes associated with courses that introduce complex problems with multiple possible interpretations or viable solutions. Design/Method: A concept mapping activity was completed by 614 undergraduate engineering students at the start and end of three different courses to evaluate sociotechnical thinking. Student-level longitudinal changes were evaluated using paired t tests and simple linear regression. The concept mapping activity was improved iteratively in response to pilot tests and focus groups. Results: Undergraduate engineers demonstrated greater complexity in the representation of sociotechnical relationships as observed in the structure and content of the concept maps. The methodological results offer lessons about the administration and analysis of concept maps often excluded from conversations on course and curriculum level assessment. Conclusions: Concept maps offer insights into student learning outcomes, and they can be deployed and analyzed with minimal resources. However, assessments must be carefully designed to account for administrative and analytical challenges.
Abstractor: As Provided
Entry Date: 2021
Accession Number: EJ1286729
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFZzbSX8itNCuI8HnZ9-NQhAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDOK3OPKoASA04mzUZAIBEICBmg3Ypcum_pHqZ5oDZDQ0SrNUgF5Ums8J5hl7KFgheAeogfR3QFdcjwpYc-YyBS7N3Fgssoq27Z2AKUR5YNf_1LbkE2j7Zv9zXXFuGx3VSj1kr2lGSeSO2NZx0VIMYd6nvwdwkuZCfP1l9VkiNdvJu62DkU7jz-IsGpO7MI9AQqqwc7dd7Bg37S9dEI4jK9ZoEOtUzZpXyj56MvQ=
Text:
  Availability: 1
  Value: <anid>AN0148778237;6m401jan.21;2021Feb19.05:39;v2.2.500</anid> <title id="AN0148778237-1">Measuring the amorphous: Substantive and methodological outcomes from concept maps </title> <p>Background: Accreditation organizations broadened program assessment criteria in ways that present challenges for the evaluation of learning outcomes. This is especially the case in courses where there are not narrowly defined questions or definitive solutions, such as engineering ethics. While protocols for learning assessment exist, there is limited research exploring sociotechnical learning outcomes in a manner that combines theoretical, empirical, and procedural aspects of assessment. Purpose/Hypothesis: This paper shares 3 years of research into the effectiveness of concept maps as an assessment tool for engineering students in courses that prioritize professional skills development and sociotechnical thinking. We show that concept maps can offer evidence of knowledge formation and learning outcomes associated with courses that introduce complex problems with multiple possible interpretations or viable solutions. Design/Method: A concept mapping activity was completed by 614 undergraduate engineering students at the start and end of three different courses to evaluate sociotechnical thinking. Student‐level longitudinal changes were evaluated using paired t tests and simple linear regression. The concept mapping activity was improved iteratively in response to pilot tests and focus groups. Results: Undergraduate engineers demonstrated greater complexity in the representation of sociotechnical relationships as observed in the structure and content of the concept maps. The methodological results offer lessons about the administration and analysis of concept maps often excluded from conversations on course and curriculum level assessment. Conclusions: Concept maps offer insights into student learning outcomes, and they can be deployed and analyzed with minimal resources. However, assessments must be carefully designed to account for administrative and analytical challenges.</p> <p>Keywords: accreditation criteria; assessment tools; concept inventory; knowledge gain; knowledge retention</p> <hd id="AN0148778237-2">INTRODUCTION</hd> <p>The assessment of learning outcomes is a growing challenge for educators and school administrators in engineering programs throughout the world. Many international efforts build on ABET, an organization based in the United States (U.S.), which issued the Engineering Criteria 2000 (EC2000) to broaden the scope of learning outcomes. The EC2000 moved away from strict curricular mandates, instead favoring learning outcomes that offered greater freedom to educational programs to explore the creation of novel curriculum and courses. More recently, revisions of the ABET EC2000 were proposed and resulted in the "Student Outcomes 3a‐k." Taken together, EC2000 and Student Outcomes 3a‐k indicate that both technical and professional skills are important for the education of future engineers. As a result, U.S. engineering programs, specifically, and other engineering programs that follow ABET, need to perform self‐assessments to demonstrate compliance with these standards for courses that aim to deliver both technical and professional skills.</p> <p>Yet, challenges arise when educators attempt to assess curricula and courses that introduce complex problems with multiple possible interpretations and no correct answer or definitive solutions. This is particularly the case for courses that focus on the political, social, and cultural context of engineering, such as engineering ethics. Further complicating matters, ABET as an organization does not catalog best practices during the evaluation, which hinders the ability of faculty and administrators to coherently share lessons learned among institutions. Neither ABET nor engineering schools document and widely publish the necessary details that would support iterative improvement of assessment practices. While this allows engineering programs the autonomy to self‐assess in a manner of their choosing, there remains a paucity of knowledge‐sharing about best practices among institutions, thus undermining the comparability of assessment context and procedural challenges. This has generated some confusion about valid assessment strategies for professional skills, specifically in regards to learning outcomes addressing ethical reasoning, global cultural awareness, and documenting continuous improvement (Mayes & Bennett, 2005).</p> <p>Prior reviews of best practices pertaining to assessment often yields portrayals of one‐off case studies (Henri et al., 2017), while comparisons among engineering programs with identical evaluation techniques are rare; for example, Cech's (2014) four‐school comparison is treated as exemplary. Our research project is premised on the idea that engineering educators can benefit from sharing more than just the findings of the assessment (e.g., communication skills improve). Furthermore, there is a need to interrogate the use of context‐specific assessment tools and reflect on the factors that contribute to the success of an assessment tool. Henri et al. (2017) took a step in this direction with a literature review of assessment techniques. Their analysis offers seven assessment techniques for engineering content, five for professional skills, and seven for assessing the combination of content and professional skills.</p> <p>We contribute to this field by critically evaluating concept maps as an assessment tool. Concept mapping—also called mind mapping or cognitive mapping—has been used extensively as a teaching tool among Science, Technology, Engineering, and Mathematics (STEM) and non‐STEM communities. Concept maps have proven valuable as a flexible learning assessment methodology through augmenting labor‐intensive, contextually dependent free‐form essays as well as rigid surveys. Shavelson and Ruiz‐Primo (2005) make it clear that concept mapping is a complementary form of assessment to other strategies and not the only "correct way" to assess content and professional skills.</p> <p>Our research asks two fundamental questions. First, can concept maps be used to assess sociotechnical learning outcomes in courses that are centered around questions without an absolute, correct answer? Second, what methodological considerations are necessary to control for during the deployment and analysis of concept maps? These questions facilitate the presentation of the primary outcome (e.g., whether course learning outcomes were satisfied) and showcase an evidence‐based approach to evaluate the use of concept maps as an assessment tool. Our evaluation of concept mapping activities includes the use of physical artifacts as prompts, intercoder reliability, and administration of the activity. Collected data sheds light on the students' prior knowledge and the local context where concept mapping is deployed (cf. Novak & Cañas, 2008). The secondary goal is to interrogate effective assessment strategies intending to meet ABET requirements, particularly the challenges associated with assessing courses where the learning outcomes address professional skills and sociotechnical thinking.</p> <p>To investigate these questions, this article reviews 3 years of research on alternative approaches to deploying and analyzing concept maps in the Department of Engineering & Society at the University of Virginia (UVA). For decades, this department has been recognized for deconstructing technical and nontechnical boundaries while building professional skills within undergraduate engineers (National Academy of Engineering, 2016). While substantive undergraduate research projects serve as exemplars of sociotechnical thinking in the undergraduate experience, we sought to implement a flexible and repeatable assessment tool of student learning outcomes.</p> <p>By giving readers understanding of the context, timing, and other procedural elements of assessment, we provide critical self‐reflection that can be of great value to our peers who seek similar insights into student learning and assessment strategies. As Douglas and Purzer (2015) note in a guest editorial in the <emph>Journal of Engineering Education</emph>,</p> <p>discourse within the engineering education research community is lacking about what exactly constitutes high‐quality engineering education assessment and how to provide evidence of the meaning and relevance (that is, valid interpretation) of the resulting scores. To improve the quality of engineering education assessment, there needs to more dialogue regarding validity in the context of assessment instrument research and use. (p. 109)</p> <p>Taking that claim to heart, we do not assume that concept mapping is valid. Rather, we aim to offer multiple interpretations of where concept mapping may indicate change in student outcomes after taking courses designed to address social and global context of engineering, engineering ethics, and sociotechnical systems thinking.</p> <hd id="AN0148778237-3">BACKGROUND LITERATURE</hd> <p></p> <hd id="AN0148778237-4">Cultures of engineering: From disengagement to sociotechnical systems thinking</hd> <p>It is critical to document the success of sociotechnical thinking in engineering training. Downey and Lucena's (2005) historical work showed that the U.S. culture of engineering drew upon values of hard work, honesty, and respect. Engineering culture in the U.S. also privileged mathematics, discrete boundaries, and technological solutions that resulted in greater efficiency and higher productivity. However, Weinberg's (1967) essay cautioned against a reductionist approach to engineering where societal challenges, such as poverty and geo‐political struggles, were broken down into smaller pieces. When this occurred, engineers created narrowly bounded solutions for unwieldy, complex issues, often called "wicked problems" (Rittel & Webber, 1973). Engineering culture reinforces individual ethics and decision‐making that is often uncritical of culture, politics, and broader institutional context. As Herkert (2001) noted, since systemic failures arise more often from macro‐ethical challenges and group responsibility, most problems far exceed an individual's decision‐making capacity. He argued that professional engineering societies needed to address broader systemic effects that transcended individual decisions but are only loosely addressed in the codes of conduct shared among engineering professionals. Without concerted effort to foster reflexive learning and problem framing, systemic failures may continue to occur (Foley & Gibbs, 2019).</p> <p>Cech (2014) argued that the origin of some of these systemic problems stemmed from a "culture of disengagement" in engineering education that is upheld by three ideological pillars:</p> <p></p> <ulist> <item> <emph>Dualism</emph> —Technology and society are distinct, and technical knowledge and competencies have more value than social ones.</item> <p></p> <item> <emph>Apolitical</emph> —Engineering is a "pure" space free of political and cultural concerns.</item> <p></p> <item> <emph>Meritocracy</emph> —Engineers earn entry into professional engineering organizations in a manner that is unbiased and with fair systems of advancements.</item> </ulist> <p>In a similar vein, Slaton (2015) detailed how engineering education is dominated by the ideologies of technocracy (lacking attention to the social, cultural, and political dimensions of engineering processes) and meritocracies (relying solely on merit‐based eligibility and evaluations). Cech and Slaton, two leading scholars in liberal arts and engineering, share a central critique that engineering education has negative effects on both the individual engineers and on the broader society that relies on engineers' specialized skills and training.</p> <p>This is not to say that engineers have not considered the social dimensions of their work. As Callon (1987) observed, engineers often work to translate knowledge from quasi‐sociological research into design practices. Furthermore, engineers have long been heralded as "systems builders" that create complex sociotechnical systems (Carlson, 1991; Hughes, 1987). Engineers must attend to functionality in the mechanistic sense while also leveraging knowledge of the societal and environmental conditions in which the work will be performed in order for sociotechnical systems to work. Functionality from a sociotechnical perspective involves the enrollment of political, organizational, legal, labor, environmental, and material resources, to name a few critical skills in engineering.</p> <hd id="AN0148778237-5">Sociotechnical systems thinking: Courses and curricular interventions</hd> <p>In light of Cech, Slaton, and others' critiques, many educators and scholars such as York (2018) have advocated for more sociotechnical integration as a way to confront Cech's cultures of disengagement. To achieve integrated learning, engineering students are asked to consider cases of ethics, public policy, or alternative global contexts in lieu of uncontextualized problems with simple solutions. Courses like those studied here are designed to facilitate learning about broader contexts while accommodating epistemic and ontological diversity (Conley et al., 2017). Thus, liberal arts courses, including the curriculum under investigation in this paper, aim to teach engineers how to explore divergent problem‐solution options and understand the consequences of those alternatives. By framing engineering as a practice embedded within sociotechnical systems, narrow problem framings are replaced by open‐ended challenges that do not privilege technical expertise over other forms of expertise (Stirling, 2008). This section highlights several instances where sociotechnical, sustainable, and other holistic forms of learning were incorporated into engineering courses and assessed through several methods.</p> <p>Leydens et al. (2018) studied the integration of sociotechnical thinking and liberal education elements in engineering education at several universities. Their intent was to broaden the scope of technical courses (e.g., electrical magnetism in a first‐year course) to include more complex, realistic scenarios that introduced engineering failure as both a technical and social phenomenon. They explored a variety of assessment strategies and settled on a survey methodology that reveals sociotechnical thinking. In a similar vein, Shankar et al. (2017) presented a case study that integrated humanities into engineering education. Their course exposed engineering students to social and community issues such as police homicide rates against minorities in metropolitan cities. Their pedagogical intervention positioned students to engage by interacting with one another and learning about real‐world issues that were out of their comfort zones. Taken together, these two cases illustrate that engineers can acquire new knowledge about the broader societal context if they engage in interdisciplinary liberal arts education.</p> <p>David and Marshall (2017) claimed that students acquired knowledge when they had opportunities to explore open‐ended topics that the students themselves deem important. They employed project‐based learning techniques that created near‐term milestones and culminated in final project presentations. Students set goals, assessed their progress, and reflected on their performance. This provided more autonomy and guidance that motivated students to be more active in assessing their own learning. Students demonstrated knowledge acquisition by applying lessons from the course to their real‐world projects without expectation of a perfect answer being possible. Students were able to learn from their mistakes and develop initial strategies to problem solve. The course offered clear metrics of success in both technical and professional skills designated by ABET's EC2000 criteria. Yet, the authors critically reflected on the lack of clear assessment measures for engineering fundamentals and the challenge of scaling this approach to meet the demands of engineering courses with larger enrollments.</p> <p>Using different techniques to augment an existing course, Garlock et al. (2017) sought to teach literacy through an interactive course design in three phases of interaction: predict, experience, and reflect. The classes started with an online poll that gave students an opportunity to share their preconceived notions, heuristics, and knowledge about the topic of the day. This provided the instructor with an understanding of the students' prior knowledge. While students entered the classroom with preconceived notions, the instructor could facilitate learning in a manner that encouraged students to critically reflect on their own heuristics. Thus, well‐designed courses could augment existing knowledge, challenge heuristics, and develop connections between the course topic and the students' lived experiences. Similarly, Garlock et al. (2017) suggested knowledge displacement of heuristics was achieved through critical reflection and assessed with reflection essays written by the students.</p> <p>Jesiek et al. (2018) delineated discrete knowledge deriving from boundary spanning, objects, practice, and discourse. Jesiek and colleagues define knowledge boundary as "representing the pool of knowledge (occupational, contextual, etc.) relevant to the project [or artifact]" (Jesiek et al., 2018, p. 392). This framework is helpful in conceptualizing the knowledge forms and disruptions that we aim to assess, yet the authors only hint at how engineering education relates to building these forms of knowledge and the role of training and culture on boundary spanning. Henri et al. (2017) offered a set of assessment tools to evaluate boundary spanning competencies that covered mixed technical and professional knowledge. Their review demonstrated that it is difficult to assess and develop professional skills within the classroom environment. Their review also suggests that multiple techniques are needed to perform a comprehensive learning evaluation (Henri et al., 2017). Thus, our research explores the use of concept mapping to assess professional skills development and cognitive representations of boundary spanning when confronted with a physical artifact.</p> <hd id="AN0148778237-6">Research design and methodological approach</hd> <p>The best practice for educational evaluation requires that a student's development of skills, competencies, or knowledge be tracked longitudinally and include their performance prior to enrollment in an institution (Mertens, 2005). Ideally, engineering programs would issue an entrance exam or initial engineering challenge to capture data about a student's competencies in order to develop a comprehensive baseline against which all future learning outcomes could be assessed. Each instructor, in coordination with their program administrators, would then design pre‐, mid‐, and post‐tests to assess learning outcomes from each course. While extremely effective, Donna Mertens (2005) argued that such an approach is impractical given the constraints of resources and administrative time. To address these limitations, our research design contributes a pragmatic middle ground that is time‐effective, moves beyond ad hoc measures of learning outcomes, and does not stray from the best practices that call for longitudinal assessment across all four undergraduate years.</p> <p>To attend to the many questions of student knowledge acquisition and assessment, this research explores substantive and methodological aspects of a concept mapping exercise, including the prompt given, administration of the activity, map coding protocols, and forms of data analysis. At the same time, we share key substantive findings gleaned from 3 years of research and our reflections of the methodological opportunities and challenges of concept maps. This article builds on prior work which can be traced back to Novak's (1990) special issue on methodological designs in concept mapping. Significant scholarship has shown the effectiveness of using concept maps in the teaching and assessment of STEM courses where a correct answer can be replicated by the students. In contrast, this paper explores the use concept maps in interdisciplinary courses that cover subject matter for which there is no "perfect" answer, such as sustainability (Foley et al., 2017), science policy (Harsh et al., 2017), and the social context of engineering (Conley et al., 2017). Concept maps are understood to capture divergent thinking, complex systems, and knowledge acquisition. By deploying concept maps at the outset of a course and again at the end of the course, the tool can evaluate near‐term learning outcomes and, when deployed again after longer time intervals, knowledge retention. Specifically, this research design explores learning outcomes from courses that aim to support complex sociotechnical systems thinking. To reflexively interrogate the overarching research question and methods, we asked these subquestions about concept mapping as a form of assessment:</p> <p></p> <ulist> <item> Can concept maps transparently capture learning outcomes from courses associated with sociotechnical thinking and professional skill development for robust internal and external review?</item> <p></p> <item> Can this approach be modified to address multiple courses and programs without losing comparability?</item> <p></p> <item> Do concept maps show how students blend life experiences with technical and professional skills?</item> <p></p> <item> Can this approach be time and cost efficient and still comply with ABET's Criterion 4 of Continuous Improvement, which is a call for longitudinal evaluation?</item> </ulist> <hd id="AN0148778237-7">Case context: Courses and the curriculum</hd> <p>The UVA's School of Engineering and Applied Science (SEAS) curriculum was noted as an exemplary program in the field of engineering ethics by the National Academy of Engineering (2016). The SEAS curriculum requires that all undergraduate students pass four courses offered by the Science, Technology and Society (STS) program in SEAS's Department of Engineering and Society. First‐year students take Science, Technology and Contemporary Issues (STS 1500) as an introduction to engineering practice and its social dimensions. Students then select a subject‐specific elective (STS 2000/3000) to fulfill the second course requirement. These courses can feature diverse topics such as computer ethics, privacy, nanotechnology, science policy, the history of technology, and business practices of entrepreneurs. Students then take two paired courses in their final year (STS 4500/4600). The shared learning objective for all sections of STS 4500/4600 engages students with the challenge of framing and solving engineering problems in a manner that requires attention to social dimensions. Students are introduced to STS theories and methods as a means to prepare them for their STS research papers. Many students explore problems that can be defined as emergent, and thus there are multiple possible interpretations and future conditions.</p> <p>This research project explored the integration of humanities and social science into the engineering curriculum at UVA. We sampled from students enrolled in the first‐year STS course (STS 1500) during the 2016–2017 and 2017–2018 academic school year. We also sampled from students within the fourth‐year courses (STS 4500/4600) during the 2015–2016 and 2016–2017 academic year. Students volunteered their time during class and did not receive compensation.</p> <hd id="AN0148778237-8">Measuring the amorphous</hd> <p>This subsection describes the deployed baseline research design that assessed learning within the first‐ and fourth‐year courses. The procedure included a brief introduction to the mapping activity in the form of a two‐sided handout. On the front page, a short description of a concept map, a visual depiction of a concept map, and examples of linking phrases were listed after the title "STS 4500 – STS and Engineering Practice." On the reverse side, space was provided to create a concept map. Participants were shown a physical object and instructed to draw a concept map centered on that object—no further instructions were provided beyond the brief verbal and text cues. Students were given 15 min to complete the activity, while students who opted to not participate given the option to sit quietly. This process was conducted at the start of the semester on the first day of classes (STS 4500/4600) or within the first week of the discussion section (STS 1500); see Figure 1 for the courses and timeline. Since a primary goal of the protocol was to minimize both time and resource expenditures, the study avoids software‐based concept map programs that require training prior to data collection (Watson & Barrella, 2017). Instead, the protocol requires only a paper and pencil (or pen), which Muryanto (2006) argued is an acceptable format and affords greater flexibility in terms of time and resources.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6M4/01jan21/jee20373-fig-0001.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jee20373-fig-0001.jpg" title="1 Timeline and course of data collection. Triangles indicate the baseline concept map participation (T0) conducted at the beginning of the semester. Diamonds indicate the first posttest instance (T1). The dashed arrow shows that a second posttest (T2) was conducted in the 2015–2016 fourth‐year cohort" /> </p> <p></p> <p>All concept maps were anonymized with a number and labeled "A" (denoting a "pre" map) or "B" (denoting a "post" map) prior to coding. The concept maps were then photocopied, and the black‐and‐white copies were coded for the number of nodes, the number of connections, and structure complexity with blue or red pen. Exclusion criteria included blank maps, maps with three or fewer total nodes, maps that were not related to the prompt in any way, and maps that were illegible. There was no predefined correct map due to the interdisciplinary nature of the course material, methodological question of preexisting knowledge, and desire to welcome students' personal knowledge of the topic. During the analysis, the researchers counted the nodes by marking and counting each. Counts were then compiled into a database. The connections were marked with a cross‐out line ("X" or slash) and compiled into the same database. The researchers observed that many links were not labeled despite the instructions given by the instructors.</p> <p>The research team also reviewed the substantive codes used to classify the content (i.e., subject matter of the words expressed). Table 1 lists the codes used by the research team and their corresponding alignment with ABET Student Outcomes 3a‐k. The left‐hand column are the aggregate clusters of all codes related to materiality (M) and technoscientific (T), as well as the technical outcomes (TO), which are grouped into the code "MTTO." The other codes were clustered as "All Social" codes and shortened to "AllSoc" in the reported statistical analysis. The creation of this table was introduced by Ferguson and Foley (2017) and draws from Shallcross (2016) and Segalás et al. (2008). Researchers applied these codes by writing the abbreviated letter (e.g., SO for social outcome) onto the concept map (see Supplemental Material Map 106A).</p> <p>1 TABLEContent codes applied to concept map nodes</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Clusters</th><th align="left">Content codes</th><th align="left">Alignment with ABET student outcomes 3a–k</th></tr></thead><tbody valign="top"><tr><td>MTTO<xref ref-type="fn" rid="tfn1" /></td><td>Materiality (M)</td><td>3E: An ability to identify, formulate, and solve engineering problems</td></tr><tr><td /><td>Technoscience (T)</td><td>3E</td></tr><tr><td /><td>Technical Outcomes (TO)</td><td>3E</td></tr><tr><td>All Social (AllSoc)<xref ref-type="fn" rid="tfn2" /></td><td>Ethics/Values (E/V)</td><td>3F: An understanding of professional and ethical responsibility</td></tr><tr><td /><td>Nature, Environment, Ecology (N)</td><td>3H: The broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context</td></tr><tr><td /><td>Social Outcomes (SO)</td><td>3H</td></tr><tr><td /><td>Locality (L)</td><td>3H</td></tr><tr><td /><td>Economics (ECON)</td><td>3H</td></tr><tr><td /><td>Social Groups/Users/Non‐Users (SU)</td><td>3H</td></tr><tr><td /><td>STS Concepts (STS)</td><td>3H</td></tr><tr><td /><td>History, Future, or Temporality (H)</td><td>3J: A knowledge of contemporary issues</td></tr><tr><td /><td>Policy/Politics/Regulation (P)</td><td>3J</td></tr></tbody></table> </ephtml> </p> <p>1 a Additive combination of Materiality (M), Technoscience (T), and Technical Outcomes (TO) content codes.</p> <p>2 b Additive combination of Ethics/Values; Nature, Environment, Ecology; Social Outcomes; Locality; Economics; Social Groups/Users/Non‐Users; STS Concepts; History, Future, or Temporality; and Policy/Politics/Regulation.</p> <p>The codes and all other data were tabulated into a data table. Data were analyzed with the statistical software R (Version 3.5.0) with the RStudio graphical user interface (Version 1.1.453). The database comprised participant identification numbers, course, instructor, year, semester, object, node count (pre and post), connection count (pre and post), structure code (pre and post), and all subject codes (pre and post). The research team applied basic statistical methods using concept maps and related open assessment tools, including paired <emph>t</emph> tests, box plots, scatter plots, and trend lines.</p> <hd id="AN0148778237-10">Iterative assessment development with focus group</hd> <p>The administration and coding of the concept maps was created through an iterative design process. After a pilot test in 2016, the researchers led a focus group with 15 randomly selected fourth‐year students. The focus group included a basic debriefing on the course, and the students reflected upon and coded their own concept maps. The focus groups helped to evaluate the validity of the experimental design and coding procedures. Students were presented with copies of their concept maps from the beginning and end of the course. They were then asked to compare and contrast their maps and corresponding nuances of their participation, including key differences they noted between the pre and post concept maps, their level of motivation or disposition to the course over the semester, and whether specific experiences occurred during the mapping time period that might have influenced the development of the maps (e.g., illness or impending course‐specific assignment deadlines). They were also asked to use the coding scheme on their concept maps with only a brief description of the codes. The researchers then compared the codes provided by the students and those defined by the research team. Coding was largely congruous except for an emphasis on coding for Materiality (M) rather than for Technoscience (T) among the focus group participants compared to researchers. This ambiguity was resolved by combining the M and T codes during subsequent analysis (Table 1). The focus group participants were asked about instances where there was no label on the connecting line. Students stated that while they were confident in the connections, they were not always clear on how to describe the nature of the relationship. The researchers therefore decided to include maps where the links were not all labeled as indicative of sociotechnical relationships even if they did not fully articulate the essence of that relationship.</p> <hd id="AN0148778237-11">Hierarchy and structure of concept maps</hd> <p>Prior work on the hierarchy, structures, and forms of concept maps is quite robust. In regard to hierarchy, Kinchin et al. (2000) employed three analytical levels: no hierarchy (spoke and wheel), multilevel hierarchy in a linear fashion (chain‐link), and nets (or networks) that involve multilevels of hierarchy and linkages among the branches. Turns et al. (2000) interpreted cross‐linkages as feedback loops that bring dynamism to the concept maps. Thus, as McClure et al. (1999) argued, the hierarchy and form of concept maps is a secondary measure of map complexity that can be interpreted independently of nodes and connection counts as shown by Jablokow et al. (2013). The measures for hierarchy and map structure followed a scoring system developed from these sources. The scoring systems entail the following five levels (Figure 2), which are analyzed as interval variables:</p> <p></p> <ulist> <item> Shows a linear structure with no branching or a spoke‐and‐wheel structure with up to two nodes emanating from the center.</item> <p></p> <item> Resembles three possible simple forms with up to two nodes from center: (a) branching off central node, (b) one interaction between branches, and (c) interactions on the same linear path.</item> <p></p> <item> Represents linear structures or spoke‐and‐wheel extending beyond two nodes from center.</item> <p></p> <item> Builds on the three simple forms, but with more than two nodes from the center and 1 or 2 interactions between branches that may include feedback loops.</item> <p></p> <item> Displays the most complex network with multiple branches from the center, more than two nodes beyond the center, and multiple interactions that may include feedback loops.</item> </ulist> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6M4/01jan21/jee20373-fig-0002.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jee20373-fig-0002.jpg" title="2 Hierarchy structure used to analyze concept maps generated in STS 4500/4600 from 2015 to 2016" /> </p> <p></p> <hd id="AN0148778237-13">Methodological experiments</hd> <p>Over the course of 3 years and in two different courses, six small tests explored alternative strategies for the deployment and analysis of this assessment tool. The prompts, test intervals, and direct versus indirectadministration of the activity pertain to the deployment, while the alternative forms of analysis involved the maps structure, intercoder reliability of content codes, and the use of automated text recognition software.</p> <hd id="AN0148778237-14">Prompts: Objects in place of words</hd> <p>Participants were shown one of two artifacts on which to base their concept maps: a 2008 iPod Touch and a road bicycle (Supplemental Material Concept Maps 106A for an iPod example and 46A and 47A bicycle examples). The use of a physical artifact is novel but justifiable given the visual and three‐dimensional focus of engineering, the intent of the mapping activity to rapidly encourage concept development, and previous student experiences with the objects themselves. The iPod is a relatively small "black box" with very little outward facing complexity. It is small in size, computer‐oriented, and connects to the Internet. This artifact has been significantly transformed within the students' lived‐experience and is associated with Apple, a large and controversial company. The bike offers an open design with components readily visualized. Parts can be seen as mechanistically complex or boring depending on the interpretation of the participant. The bike is a more mundane object and easily relatable to different user types and markets. While the bicycle is not as likely to solicit a sense of innovation considering it has undergone a less dramatic change in the past decade relative to the evolution of iPods and iPhones, most of the students will have encountered the bicycle as a classic example of the sociotechnical relationships from their STS courses (cf. Pinch & Bijker, 1987).</p> <p>These two objects were thought to offer a broad range of opportunities for the participants to depict their conceptualization of technology and society from the lenses of engineering and their lived experience. All maps were coded as "iPod," "Bike," or "Other," (i.e., noncompliant objects) as categorical variables for this analysis. Box plots for each category of pre and post nodes were generated to better visualize the spread of the data and identify possible outliers (e.g., students who generated significantly more or fewer nodes compared to peers in the course). Comparison of the box plots explored differences in median scores for the pre and post maps based on the object shown to the participants.</p> <hd id="AN0148778237-15">Time intervals</hd> <p>The researchers explored alternative time intervals within the yearlong course sequence of STS 4500/4600, which is visually depicted in Figure 1. The researchers consistently used the first day of the course in late August for the pretest (T0 or "Time Zero"), but then varied the time interval for the posttest. For the 2015–2016 cohort, posttest data (T1 or "Time 1") was collected before the winter break in early December, and a second posttest (T2 or "Time 2") was performed at the end of the spring semester in May. For the 2016–2017 cohort, a single posttest was collected in the middle of the semester—specifically, the week after the spring break in March 2017. Time intervals were investigated in STS 4500/4600 cohorts because students took the course over an academic year (August to May), whereas STS 1500 only lasted for one semester (approximately 3 months). An earlier publication on this research project reports on the time intervals when the concept mapping activity was conducted mid‐semester, but the results indicated no statistical difference (Ferguson & Foley, 2017).</p> <hd id="AN0148778237-16">Direct versus indirect administration</hd> <p>We also allowed for the deployment of the concept map test by the graduate teaching assistants leading STS 1500. Undergraduate researchers used timers to record the amount of time the STS 1500 instructors allotted for students to complete the activity relative to the amount of time they were asked to provide students (i.e., 15 min). Outcomes (e.g., number of nodes and connections) between sections based on the amount of time allotted were then evaluated.</p> <hd id="AN0148778237-17">Intercoder reliability</hd> <p>Several approaches to coding the concept maps were tested using the categories in Table 1. Prior to finalizing the coding schema, the lead researchers and research assistants worked together to code the map separately and then compared the results until a consensus was reached. Secondarily, the focus group (introduced above in Section 3.2) served as a cross‐check between the participants' and researchers' interpretation of concepts. The researchers shared photocopies of the participant maps, and participants were instructed to assign the codes from Table 1 to their concept nodes. That iterative process yielded the final list of categories shown in Table 1 and was then subjected to intercoder reliability testing.</p> <p>To test intercoder reliability, 150 concept maps and the brief written description of content codes (Table 1) were given to an undergraduate research assistant (URA) with no training. This untrained URA was given simple instructions to apply one content code to each node on the concept maps. The second approach involved one faculty researcher (Foley) working with four URAs to apply content codes to the same 150 concept maps. The four URAs met with the faculty researcher (Foley) on five occasions for 2hr, and the URAs could ask a lead researcher questions for clarification at any time. For the third approach, the other faculty researcher (Ferguson) coded the same 150 concept maps. Statistical differences in intercoder reliability between the untrained undergraduate, four URAs trained and managed by Foley, and coding performed by Ferguson were assessed with Krippendorff's Alpha. This is a typical test to determine agreement among raters in content analysis cases (Hayes & Krippendorff, 2007). It is important to note that intercoder reliability was not used as an inclusion or exclusion criteria for the maps themselves, but rather an evaluative tool for the methodological manner in which the concept maps were coded. An initial fourth approach intended to use the automated text recognition software ABBYY FineReader (2016), but the results were insufficient to compare against the human coders.</p> <hd id="AN0148778237-18">Findings</hd> <p>This research addresses two central questions. First, how can sociotechnical learning outcomes be assessed over time when there is no absolute correct answer? Second, what methodological considerations are necessary to control for during deployment and analysis of student learning outcomes? The next section of the results addresses this first question and offers substantive results suggesting that concept maps can assess changes in how undergraduate engineers conceptualize artifacts after taking courses that address sociotechnical thinking, contemporary issues in engineering, and engineering ethics. In particular, the results indicate that students have a greater awareness of the broader social context and dynamic sociotechnical relationships after participating in these types of courses. In the second half of the findings, we share methodological results about the deployment and analysis of concept maps for assessment purposes, which offers lessons about the administration of concept maps. The methodological findings should give confidence to undergraduate engineering programs that want to use this instrument for evaluating students' performance in courses that introduce complex problems with multiple possible interpretations or viable solutions.</p> <p>Table 2 shows the demographic breakdown of the sample of students who participated in this study. Of the cohort of 614 students, 167 were fourth years at the time of the study (78 in 2015–2016 and 89 in 2016–2017). In each year, this represented approximately 13% of the overall number of fourth years in SEAS. In contrast, the 152 first‐year students sampled in 2016–2017 represented about 21% of the student body, while the 295 sampled in 2017–2018 were about 39% of all first‐year students in that year. Overall, the demographics of the sample differ slightly from the general student‐body population in a few ways. The sample is slightly over‐representative of female students as well as students who identified as White or Asian when compared to the student body. Furthermore, the sample includes more Biomedical Engineers (BME) and Systems and Information Engineers (SIE) than the general population, which results in fewer computer science majors than the general student‐body. There is greater gender parity in BME and SIE at UVA, and fewer Non‐White and Non‐Asian students opted in. Students in SEAS do not declare an engineering major until their second year, so the percentage of students in each major does not exist for students in the first‐year cohort. Thus, the results have not been analyzed for differences between gender and race.</p> <p>2 TABLEAcademic cohort demographic information</p> <p> <ephtml> <table><thead valign="bottom"><tr><th /><th align="left">2015–2016</th><th align="left">2016–2017</th><th align="left">2016–2017 (Fall/Spring)</th><th align="left">2017–2018 (Fall/Spring)</th></tr><tr><th align="left">Characteristic</th><th>STS 4500/4600 4th year (senior)</th><th>STS 4500/4600 4th year (senior)</th><th>STS 1500 1st year (freshman)</th><th>STS 1500 1st year (freshman)</th></tr></thead><tbody valign="top"><tr><td>Participants, n (%<xref ref-type="fn" rid="tfn5" />)</td><td>78 (12%)</td><td>89 (13%)</td><td>152 (21%)</td><td>295 (39%)</td></tr><tr><td>Male gender<xref ref-type="fn" rid="tfn6" /> (%)</td><td>58.0</td><td>60.0</td><td>65.1</td><td>61.5</td></tr><tr><td>Race<xref ref-type="fn" rid="tfn7" /> (%)</td><td /><td /><td /><td /></tr><tr><td>White</td><td>69.2</td><td>67.1</td><td>50.7</td><td>57.6</td></tr><tr><td>Asian</td><td>16.7</td><td>18.1</td><td>25.0</td><td>22.4</td></tr><tr><td>Non‐White/Non‐Asian<xref ref-type="fn" rid="tfn8" /></td><td>14.1</td><td>14.8</td><td>24.3</td><td>20.0</td></tr><tr><td>Major (%)</td><td /><td /><td /><td /></tr><tr><td>BME</td><td>23.1</td><td>32.0</td><td>—</td><td>—</td></tr><tr><td>CEE</td><td>5.1</td><td>4.0</td><td>—</td><td>—</td></tr><tr><td>CHE</td><td>4.8</td><td>10.0</td><td>—</td><td>—</td></tr><tr><td>CS</td><td>9.0</td><td>14.0</td><td>—</td><td>—</td></tr><tr><td>ECE</td><td>19.2</td><td>6.0</td><td>—</td><td>—</td></tr><tr><td>ES</td><td>2.6</td><td>8.0</td><td>—</td><td>—</td></tr><tr><td>MAE</td><td>17.9</td><td>6.0</td><td>—</td><td>—</td></tr><tr><td>SIE</td><td>19.2</td><td>20.0</td><td>—</td><td>—</td></tr></tbody></table> </ephtml> </p> <ulist> <item>3 <emph>Note:</emph> All 1st year (freshman) students are "undeclared" and have not been assigned a major.</item> <item>4 Abbreviations: BME, Biomedical Engineering; CEE, Civil and Environmental Engineering; CHE, Chemical Engineering; CS, Computer Science; ECE, Electrical and Computer Engineering; ES, Engineering Science; MAE, Mechanical and Aerospace Engineering; SIE, Systems and Information Engineering.</item> <item>5 a Denotes percent of population that the sample represents.</item> <item>6 b Gender denoted as male or female. Binary gender identity is increasingly a problematic notion, but the number reported are those on record with the academic affairs office.</item> <item>7 c Racial and ethnic identities are often more complex than the categories afforded to incoming students, but the number reported are those on record with the academic affairs office.</item> <item>8 d All students that identified as non‐White or non‐Asian are combined to avoid the individual identification of students in under‐represented ethnic and minority categories.</item> </ulist> <hd id="AN0148778237-19">TOWARD COMPLEX, SOCIOTECHNICAL REPRESENTATIONS</hd> <p>Student‐participants from the STS 4500/4600 courses in the academic years of 2015–2016 and 2016–2017 created pretest concept maps that contained a mean of 14.9 nodes (<emph>SD</emph> = 5.89 nodes), and their posttest concept maps contained a mean of 16.9 nodes (standard deviation = 9.49 nodes). Examples of the students' concept maps are included in the Supplemental Material. The difference in total nodes expressed in the students' pre and post concept maps demonstrate an increase in the students' knowledge and ability to relate more concepts to the central prompt, which was a physical artifact. In this way, our results confirm that concept maps capture learning outcomes associated with sociotechnical thinking and professional skill development for assessment purposes. The results suggest that students demonstrate a more constructivist understanding of artifacts after completing of STS 4500/4600 and more readily make connections between that object and the broader world. However, the results may also indicate that the students are just making connections to more technical aspects and might even be reducing the artifact into more discrete subparts and components, thus requiring a different strategy beyond numerical increases in nodes. The categorical coding of all the nodes offers evidence supporting the interpretation that students are constructing knowledge in a way that offers greater connectivity to non‐technical elements of the artifact.</p> <p>The pre and post concept maps were categorically coded (see Table 3), and those codes were clustered into two groups: MTTO and All Social (AllSoc). The mean pretest contained 14.9 nodes, with 8.03 coded as MTTO and 5.86 coded as AllSoc (Table 3). In contrast, the posttest yielded a mean score of 16.9 nodes per concept map, with 9.01 nodes coded as MTTO and 8.86 coded as AllSoc. There was a larger average percent increase in AllSoc nodes between pre‐ and posttests compared to MTTO nodes (51 and 12% increase, respectively). A paired <emph>t</emph> test suggested that there was a significant difference between pre‐ and posttest MTTO node counts at a.01 level (<emph>p</emph> < .1), while the difference between AllSoc pre‐ and postnode counts was significant at a.01 level (<emph>p</emph> < .01). Furthermore, the change in MTTO node count was significantly less than the change in AllSoc node count at a significance level of.05. It is also important to note that there was a significant difference in number of pre MTTO nodes compared to pre AllSoc nodes (<emph>p</emph> < .01), but this difference vanished in the postnodes (<emph>p</emph> > .1). Taken together, the previous information suggests a rebalancing of the students' representation of the social‐technical relationship that favors the AllSoc concepts. This approach captures learning outcomes that demonstrate how students blend their knowledge of technical and professional skills, including the social context and contemporary issues that inform engineering practice. Furthermore, by deploying this instrument within courses at different stages in the undergraduate engineering curriculum, the results are comparable and demonstrate knowledge acquisition over time.</p> <p>3 TABLESummary statistics of pre and post concept maps clustered by category codes, STS 4500/4600 2015–2017</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Node summary statistic</th><th align="left">MTTO<xref ref-type="fn" rid="tfn9" /> T0<xref ref-type="fn" rid="tfn11" /> (pre)</th><th align="left">MTTO T1<xref ref-type="fn" rid="tfn12" /> (post)</th><th align="left">MTTO difference</th><th align="left">AllSoc<xref ref-type="fn" rid="tfn10" /> T0 (pre)</th><th align="left">AllSoc T1 (post)</th><th align="left">AllSoc difference</th><th align="left">All T0 (pre)</th><th align="left">All T1 (post)</th></tr></thead><tbody valign="top"><tr><td>Mean (SD)</td><td>8.03 (5.80)</td><td>9.01 (6.63)</td><td>0.99<xref ref-type="fn" rid="tfn13" /> (6.56)</td><td>5.86 (4.18)</td><td>8.86 (6.36)</td><td>3<xref ref-type="fn" rid="tfn13" /> (6.45)</td><td>14.9 (5.89)</td><td>16.9 (9.49)</td></tr><tr><td>Median</td><td>7</td><td>7</td><td>1</td><td>5</td><td>8</td><td>2</td><td>13</td><td>15</td></tr></tbody></table> </ephtml> </p> <ulist> <item>9 a MTTO: additive combination of Materiality (M), Technoscience (T), and Technical Outcomes (TO) content codes. See Table 2 for more.</item> <item>10 b AllSoc: additive combination Ethics/Values; Nature, Environment, Ecology; Social Outcomes; Locality; Economics; Social Groups/Users/Non‐Users; STS Concepts; History, Future, or Temporality. See Table 2 for more.</item> <item>11 c T0: time of initial, pre concept map creation. See Table 1 for more.</item> <item>12 d T1: time of first post concept map creation. See Table 1 for more.</item> <item>13 e There was a significantly higher magnitude of change in the number of AllSoc nodes between pre and post concept maps compared to MTTO nodes (<emph>p</emph> < .05). Students had an average of three additional AllSoc nodes on their post concept maps compared to their pre concept map, while students only had an average of one more MTTO node on their post concept map compared to their pre concept map.</item> </ulist> <p>Figure 3 displays this through box plots transposed with violin plots. While box plots showcase key summary statistics (e.g., the median, 25th, and 75th percentile), they do not give any insight into the data's actual distribution. In contrast, violin plots capture how the data are "spread," which can differ greatly even between data sets with similar summary statistics (Hintze & Nelson, 1998). For example, while the distribution of STS 4500/4600 MTTO nodes are relatively similar between the pre and post concept maps, the distribution of AllSoc nodes between pre and post concept maps are notably different. In particular, the AllSoc node count in the postconcept map shows a wider distribution than the pre concept map, suggesting a higher diversity in the number of AllSoc nodes present in the posttest. This distribution may be indicative of students' enhanced awareness of sociotechnical concepts through their experiences in the course, the extent of which would not be captured through analysis of the box plots alone. This supports our interpretation that students are more likely to cognitively represent artifacts from a more constructivist perspective after the course, confronting what Cech (2014) called sociotechnical dualism, which is discussed further below.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6M4/01jan21/jee20373-fig-0003.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jee20373-fig-0003.jpg" title="3 MTTO and All Social codes for all STS 4500 and 4600 cohorts. Pretests are across the top row (a, b) and posttest are across the bottom row (c, d)" /> </p> <p></p> <p>As Hughes (1987) argued, a key characteristic of sociotechnical systems is that they are nested, dynamic, and complex. The concept maps were coded for structure to evaluate the students' ability to depict complexity in the maps. The pre‐ and posttest concept maps were coded using a 5‐point scale for structure as described in the methods. The linear regression analysis suggests that while most students created maps with little complexity at the onset of the course, maps created after the course were more complex (Figure 4). The difference of mean complexity between the pre‐ and posttests is statistically significant (<emph>p</emph> < .001). This suggests that not only are students expressing more connections between the social and technical aspects of an artifact but they are more capable of conceptualizing an artifact as situated within a complex sociotechnical system after taking the STS4500/4600 course sequence.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6M4/01jan21/jee20373-fig-0004.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jee20373-fig-0004.jpg" title="4 Complexity differences between pre and post concept maps. Blue bars denote counts of prenodes, while red bars denote counts of postnode. Dotted lines denote the line of best fit with corresponding R2 value (prenode unadjusted R2 = 0.923, postnode unadjusted R2 = 0.864)" /> </p> <p></p> <hd id="AN0148778237-22">Methodological findings of concept mapping experiments</hd> <p>The substantive findings above were derived from the data and interpreted while the research team grappled with methodological challenges. This section reports on experiments in the deployment and analysis of the concept maps. The aim here was to address the question: can concept mapping be time and cost efficient while still complying with demands for longitudinal evaluation of ABET's Criterion 4 Continuous Improvement? The initial pilot dataset captured in 2015–2016 indicated the assessment could prove productive, but methodological concerns were apparent and gave the research team pause entering 2016–2017. Application of the protocol across courses, between sections of the same course, and at different time points within the course revealed unforeseen challenges and opportunities. This section reports on four methodological questions that we pursued during the testing and development of the concept mapping activity.</p> <hd id="AN0148778237-23">Prompts with different objects</hd> <p>Diverging from traditional word‐based prompts, the research team selected two consumer goods as prompts for the concept mapping activity. The results indicate no significant difference in number of nodes expressed based on the artifact presented (Figure 5; <emph>p</emph> > .1). Even when student‐participants did not follow the instructions given and generated a concept maps of "other" objects, there was no significant change in the number of nodes. This indicates that separate instructors might select objects relevant to a particular course without sacrificing between‐course comparability across a curriculum. This is important for large courses with multiple sections and numerous instructors. An instructor might select a bicycle as an acceptable object to prompt consideration of global production, mechanics, and sustainable transportation, while a different instructor might select an airplane to solicit similar concepts in a course focused on aerospace and engineering ethics. The exploration of physical objects rather than words or concepts warrants further research. However, this style of prompt is a novel divergence from keyword‐based prompts. Selecting artifacts as prompts for concept mapping activities might afford greater connections between technical projects and the broader socioeconomic, environmental, or cultural context.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6M4/01jan21/jee20373-fig-0005.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jee20373-fig-0005.jpg" title="5 Node counts for pre‐ and post concept maps for STS 4500/ 4600 using a bike or iPod as the prompt. Rows indicate pre‐ or postnodes, while columns indicate item type. In particular, (a) represents bike prenode counts (n = 83), (b) represents iPod prenode counts (n = 81), (c) represents other prenode counts (n = 4), (d) represents bike postnode counts (n = 66), (e) represents iPod postnode counts (n = 68), and (f) represents other postnode counts (n = 34)" /> </p> <p></p> <hd id="AN0148778237-25">Time interval</hd> <p>The concept map activity was conducted to capture near‐term learning outcomes associated with specific courses rather than long‐term curricular or competency‐based learning. Thus, the time intervals between concept maps were explored in two courses, and the activity was conducted at different points in the semester. In both semesters, the concept map activity was performed on the first day of class to serve as the baseline pretest measure (T0). The students came into the course with an understanding of concept mapping (or the procedures are intuitive) as students expressed almost no confusion about the activity and generated maps that reflected their initial understanding of the artifact. In 2015–2016, the participants completed a concept map at the end of the fall semester (T1) and then again on the last day of the spring semester (T2). Using a paired <emph>t</emph> test, no statistical difference was observed between T1 and T2 although statistical significance is present between T0–T1 (<emph>p</emph> < .001) and T0–T2 (<emph>p</emph> < .001) in that cohort (Table 4).</p> <p>4 TABLESummary statistics of concept maps by time period, STS 4500/4600 2015–2017</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Course year</th><th align="left">T0<xref ref-type="fn" rid="tfn14" /> mean (<italic>SD</italic>)</th><th align="left">T1<xref ref-type="fn" rid="tfn15" /> mean (<italic>SD</italic>)</th><th align="left">T2<xref ref-type="fn" rid="tfn16" /> mean (<italic>SD</italic>)</th><th align="left">T0 median</th><th align="left">T1 median</th><th align="left">T1 median</th></tr></thead><tbody valign="top"><tr><td>2015–2016</td><td>14.1 (8.29)</td><td>19.8 (10.3)</td><td>23.4 (14.3)</td><td>12</td><td>18</td><td>19</td></tr><tr><td>2016–2017</td><td>17.2 (6.55)</td><td>20.6 (10.3)</td><td>—</td><td>15</td><td>19</td><td>—</td></tr></tbody></table> </ephtml> </p> <ulist> <item>14 a T0: time of initial, pre concept map creation. See Table 1 for more.</item> <item>15 b T1: time of first post concept map creation. See Table 1 for more.</item> <item>16 c T2: time of second post concept map creation. See Table 1 for more.</item> </ulist> <p>The focus groups, which consisted of seven males and eight females that identified as 66% White, 20% Asian, and 13% Non‐White/Non‐Asian, revealed several factors that influenced participation in the concept mapping activity. In Spring 2016, the posttest concept maps in STS4600 were influenced by proximity to graduation. Focus group participants stated that they were either "burnt out" or had already turned to future career opportunities and were, as a result, not entirely focused on the class. The focus group pointed to three factors that influenced the activity on the last day of classes before graduation: student motivation, excessive exhaustion, and survey fatigue as students are issued numerous exit surveys by SEAS; see prior work by Ferguson and Foley (2017). Following this feedback, the researchers moved the posttest (T1) to the week after spring break, which was 6 weeks before the end of the semester in 2016–2017 (Figure 1). The concept mapping activity generated results consistent with the 2015–2016 posttests (T1 and T2) while reducing the burden of testing on the students and the burden of managing compromised data during analysis.</p> <hd id="AN0148778237-26">Direct versus indirect administration</hd> <p>The concept maps were also not immune to the manner by which the assessment tool was deployed. Inconsistent time allotted to the concept mapping activity became an unexpected variable for the STS 1500 mapping activities (Table 5). During the pretest in the first section (<reflink idref="bib1" id="ref1">1</reflink>), a member of the research team was present and supported the administration and collection of the concept maps. Afterward, the concept mapping activity was administered by instructors in the nine different STS 1500 discussion sections, and the time allocated was recorded by an URA. The amount of time allocated to the test was highly variable, and in two sections the posttest was not even administered (Table 5). Those STS 1500 sections had lost a day to inclement weather, and the instructors prioritized other activities on the final class of the semester. URAs communicated this to the research team, and the research team did not seek to interfere in their decision. Prioritization decisions were understandable, if unfortunate, since the activity was not deemed to be part of the core course material and was viewed as a nonessential task. Depending on the class, students were given anywhere from 4 min to the full‐time of 15 min. There was a statistically significant difference in the number of pretest nodes across time intervals; specifically, for every 3 min of additional time students were given, an average of one new node was created (<emph>p</emph> =.038). However, this statistical difference was not observed in the posttest. Future methodological investigation could consider how to compare concept maps produced within different time intervals as well as whether the time difference induces significant differences in counts, links, and complexity.</p> <p>5 TABLETime in minutes allotted for concept map activity under indirect test administration</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">STS 1500 section</th><th align="left">1</th><th align="left">2</th><th align="left">3</th><th align="left">4</th><th align="left">5</th><th align="left">6</th><th align="left">7</th><th align="left">8</th><th align="left">9</th></tr></thead><tbody valign="top"><tr><td>T0<xref ref-type="fn" rid="tfn17" /> (pretest)</td><td>15</td><td>10</td><td>13.25</td><td>9</td><td>9.5</td><td>14</td><td>9.5</td><td>12.25</td><td>14.25</td></tr><tr><td>T1<xref ref-type="fn" rid="tfn18" /> (posttest)</td><td>10</td><td>8</td><td>10</td><td>4</td><td>—</td><td>—</td><td>5.25</td><td>8.25</td><td>8</td></tr></tbody></table> </ephtml> </p> <ulist> <item>17 a T0: time of initial pre concept map creation. See Table 1 for more.</item> <item>18 b T1: time of first post concept map creation. See Table 1 for more.</item> </ulist> <p>The indirect administration of the activity posed challenges about how seriously the students perceived the activity. The researchers had to give up control of the activity except for the guidance in the printed material and influence of passive observations by URAs present in the classroom during testing. During focus group meetings, students associated the concept maps activity with a "busy work" assignment, claiming this decreased motivation. One student participant declared, "That was for real? We all thought drawing pictures of bikes was a joke." Others treated the activity like any other assignment and put in a level of effort equivalent to their other coursework. This raises questions about how the instructor‐student relationship and the students' perception of the salience of the course can impact the results. Coupling concept maps with brief surveys about the students' motivation may help address this open question.</p> <hd id="AN0148778237-27">Intercoder reliability</hd> <p>Training people to code concept maps is an essential requirement for ensuring reliable outcomes, especially for concept maps where an ideal version does not exist. The first concept mapping cohort (2015‐16 STS 4500/4600) demanded that the co‐PIs (Foley and Ferguson) address any ambiguity of the coding scheme developed as well as a check on intercoder reliability via consensus‐based coding. After coding the first year of concept maps, the decision was made to collapse Materiality (M) and Technoscience (T) into "MT" due to an inability to differentiate "M" versus "T" between the two PIs (see Ferguson et al., 2018). After this, the difference between a single URA who was given the coding guidelines but no hands‐on training was explored.</p> <p>Table 6 compares Foley against the untrained URA and the four trained and supervised URA. For Foley and the supervised URAs, the Krippendorff's alpha scores show high levels of consistency for all the codes and nearly identical assignment of MT and ECON codes. The lower reliability with one untrained URA demonstrates that substantive coding cannot be handled without appropriate training. Coding 89 concept maps, based on the codes defined in Table 1, took one 2‐hr work session (10 total hours of labor), including the training. The individual URA who was untrained took almost 30 hr to code the same concept maps. Thus, the training added accuracy and decreased the amount of time spent on assigning codes.</p> <p>6 TABLEKrippendorff's alpha scores between primary investigator and undergraduate sssistants</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Coding type</th><th align="left">MT<xref ref-type="fn" rid="tfn20" /></th><th align="left">TO</th><th align="left">N</th><th align="left">SO</th><th align="left">E/V</th><th align="left">L</th><th align="left">ECON</th><th align="left">SU</th><th align="left">P</th></tr></thead><tbody valign="top"><tr><td>PI (Ferguson) versus four trained UTA supervised by PI (Foley)</td><td>0.99</td><td>0.91</td><td>0.96</td><td>0.91</td><td>0.83</td><td>0.87</td><td>0.99</td><td>0.95</td><td>0.9</td></tr><tr><td>PI (Ferguson) versus untrained UTA</td><td>0.7</td><td>0.23</td><td>0.59</td><td>0.47</td><td>0.11</td><td>−0.05<xref ref-type="fn" rid="tfn21" /></td><td>0.09</td><td>0.45</td><td>0.73</td></tr></tbody></table> </ephtml> </p> <ulist> <item>19 Abbreviations: E/V, ethics/values; ECON, economics; L, locality; N, nature/environment/ecology; P, policy/politics/regulation; PI, primary investigator; SO, social outcomes; SU, social groups/users/non‐users; TO, technical outcomes; UTA, undergraduate teaching assistant(s).</item> <item>20 a MT: additive combination of Materiality and Technoscience. See Table 2 for more.</item> <item>21 b A negative Krippendorff's alpha suggests that the codes assigned by Ferguson and the untrained UTA performed worse than if the category were randomly assigned.</item> </ulist> <p>Due to the continued interest in efficiency, the research team tested ABBYY FineReader's (2016) text recognition software to determine the viability of automating some of the coding with off‐the‐shelf software. The output suggests that hand‐coding is the only viable option for paper‐based mapping without further improvement in digital handwriting recognition. Figure 6 shows an image of the output from the software and illustrates the machine's difficulty in differentiating handwritten words as well as preprinted computer‐generated text. Roughly 25% of the text was recognized as text, and none of the recognized words were processed accurately, including the preprinted words and instructions, based on a sample of 25 scanned concept maps.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6M4/01jan21/jee20373-fig-0006.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jee20373-fig-0006.jpg" title="6 Output file from ABBYY FineReader (V12). Gray boxes represent software generated selections for possible text containing regions. The gray areas with a black triangle were not recognized as containing text translatable by the software. No statistical testing was conducted as the researchers viewed the initial results as inadequate" /> </p> <p></p> <hd id="AN0148778237-29">DISCUSSION</hd> <p>Cech (2014) cautioned that, after people graduate from undergraduate engineering programs, they place a greater emphasis on the technical aspects of problems and separate out societal elements. Her research showed that engineering programs perpetuate dualism by downplaying (or even actively dismissing) societal context. This practice highly correlates to a declining concern for public welfare. The concept mapping activity shared in this article was designed to track changes in students' representations of a physical artifact from simple linear concept maps that have a strong technical focus to more complex and dynamic concept maps that show greater balance between the societal and technical elements. This result appears to push back against the notion of dualism offered by Cech (2014) by offering an initial counter‐narrative as the engineering students expressed more balanced sociotechnical systems thinking with a near 50:50 ratio of MTTO to AllSoc nodes in the post concept maps. The concept maps are cognitive representations of physical artifacts and how those objects connect to social groups, economic forces, ethics and values, policies and politics, geographic locations, and the environment. In this way, the concept maps are flexible enough to capture assessment data about the changes in knowledge for students in courses that address the social contexts and ethical implications of engineering.</p> <hd id="AN0148778237-30">Limitations</hd> <p>While the results appear promising, the findings presented here are not a definitive counterargument to Cech's (2014) work on cultures of disengagement, and there are key limitations that warrant attention. First, this study was conducted at one university, and we expect the results are influenced by the institutional culture and curriculum developed across the long history of UVA's Engineering program. The UVA Engineering school incorporates communication, humanities, policy, and ethics throughout a student's experience, including the requirement for an interdisciplinary research thesis since the early 1900s (Pfaffenberger, 2012). Another limitation is the recruitment of students. While the student participants in STS 1500 spanned the entire school, only 5 of the 18 different sections of STS 4500 and STS 4600 courses were sampled. Furthermore, we have little understanding about the selection pressure placed upon students enrolled in STS 1500 and STS 4500/4600 who might be predisposed to thinking about engineering within a broader social context. The SEAS curriculum is designed to integrate STS into engineering, and the lack of a comparison to another university is a major drawback. An important limitation was the lack of consistent link labeling and directional arrows on the concept maps. Computer‐based programs can assure that all links are labeled and have arrows, which allows for more sophisticated interpretation of the relationships between two nodes. Nonetheless, concept maps offered a complementary means of tracking progress within an engineering program.</p> <hd id="AN0148778237-31">Future research directions</hd> <p>The STS faculty at UVA works to tailor the curriculum and classroom experiences to the needs of engineers‐in‐training and support positive engagement with the humanities and social sciences. With access to all engineering students across the entire undergraduate experience, the STS program is well situated to assess knowledge reformation as students' progress through the curriculum. We acknowledge that most engineering programs outsource the ethics, liberal arts, and humanities credits to other schools within their universities. While the inclusion of the STS program within SEAS is somewhat unique, we intentionally selected an assessment vehicle that might allow for translating knowledge assessment strategies to comparable schools that are also evaluating learning outcomes related to sociotechnical thinking. This points to the need to expand this assessment program and attempt to account for the influence of the broader institutional conditions and learning community. The STS program at UVA is not alone in its endeavor to engage with the rightful critiques offered by Slaton (2015) and Cech (2014).</p> <p>Several schools actively engage in the teaching of sociotechnical thinking across disciplines. For example, the University of Maryland, Stanford University, Rensselaer Polytechnic Institute, James Madison University, Vanderbilt, Virginia Polytechnic Institute and State University, Olin College, and The Citadel (to name a few) all offer STS courses within engineering programs. Thus, the implications of gender, race, and major on knowledge acquisition could be explored by partnering with institutions and generating a more representative sample across a larger population of engineering schools. There is also ample room to continue to refine this assessment tool alongside other methodologies such as reflection essays and research reports.</p> <p>Sharing experiences and comparing results across universities would support an understanding of how differences in institutional structures and curriculum impact knowledge formation among engineering students. One experiment that would be worthwhile across the schools would be deploying concept mapping tools that are software‐based in some classrooms while using paper and pencil in other classrooms. Such an experiment could expand our understanding of the trade‐offs and considerations between those two instruments. By comparing this assessment strategy with other approaches taken by engineering schools with programs that focus on sociotechnical thinking and interdisciplinary scholarship, we might arrive at a more refined list of best practices that could be shared across institutions, both nationally and internationally. Concept mapping could be coupled to discrete interventions to locate moments where and when students express new knowledge or worldviews. One additional means of experimentation across engineering programs might be to allow students to select any physical object as the starting point for their concept maps. ABET could serve to aggregate and catalog assessment approaches for technical and nontechnical aspects of the curriculum with associated contextual characteristics of the assessors and participants.</p> <hd id="AN0148778237-32">Concept mapping for assessment of sociotechnical thinking</hd> <p>This study shows that the instrument of concept mapping can serve to compare learning outcomes from different courses that address professional skills development. ABET coordinators would be well‐served to review both the opportunities and challenges of this approach as they work to evaluate the learning outcomes associated with student's professional development, ethical reasoning, and recognition of social context. The opportunities and challenges for this activity are quite varied yet can be addressed (Table 7). Concept mapping offers a transparent and robust approach to evaluate the learning outcomes for courses where there are multiple solutions and students' diverse experiences are drawn into the course content. While software programs are highly effective, the literature predominately contains examples where considerable time is required for student training and concept maps are central to the course (Watson & Barrella, 2017). The simplicity of paper and pencil, rather than software programs, places less administrative burden on faculty and students, potentially encouraging more consistent assessment and yielding more comparable results without concern regarding software skill level. Free concept mapping software programs do exist, but there is a hurdle for training and supply of the computers necessary to run the software. In contrast, simpler strategies maximize student expression of concepts from the entire corpus of their learning experiences, be they course‐related or external lived‐experiences.</p> <p>7 TABLEOpportunities and challenges of using concept mapping activities to assess learning outcomes</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Category</th><th align="left">Opportunity</th><th align="left">Challenge</th></tr></thead><tbody valign="top"><tr><td>Test procedure</td><td>+ Portability, as the activity can be readily executed in different classrooms (no special equipment required)+ Adaptability to course or curriculum based upon the prompt (word or object), iPod might work for computer and electrical engineering, while bicycles might serve civil engineering+ Comparability, prompt should not influence the evaluation of learning outcomes across social context courses that focus on different subject areas+ Flexibility, students can express their knowledge as it relates to any number of topics, including chocolate (see Supplementary Material, Map A123)</td><td>– Consistency across classrooms in terms of time allocation and instructions can render the results irreconcilable– Variability of artifacts in the research reported here has been shown to have no significant difference, yet future research is needed with a greater diversity of artifacts and prompts across institutions.</td></tr><tr><td>Activity format</td><td>+ Rapid, inexpensive deployment of paper concept mapping activity enables assessment in a variety of scenarios where computers are unavailable+ Unlike software‐based concept map activities, language differences are mitigated, and comparability can be achieved after researcher analysis+ Freedom of expression can lead to novel depictions of knowledge through symbols, drawings, and phrases</td><td>– Analysis of hand‐drawn concept maps require more time, yet this needs to be considered as a trade‐off for the upfront time of training students to use computer‐based programs– Paper‐based concept maps can be incomplete with nodes that are not connected and connections that are not fully labeled, which complicates more holistic or relational analysis– Computer programs offer sophisticated analysis of structure and form– Legibility can be a challenge for analysis</td></tr><tr><td>Population sampling and analysis</td><td>+ For accreditation and school‐level assessment, this method is possible at large scale with relatively few resources+ Multiple codes can be successful if offered training and oversight+ Insights into high school achievement, differences between majors, and broader curriculum might well be gleaned by conducting pretests at the time of enrollment and posttests at graduation (and intermittent testing)</td><td>– Without automated text analysis and natural language processing to conduct the substantive coding, the increase in sampling of a population can become a time and resource burden– There is a paucity of comparative research between instructors and their influence on concept maps</td></tr><tr><td>Automated text analysis</td><td>+ Rapid processing of paper‐generated maps for near real‐time assessment</td><td>– Commercially available optical recognition and natural language processing of handwritten words is currently technologically infeasible.</td></tr></tbody></table> </ephtml> </p> <p>Minimizing the time and resources needed to deploy the concept maps was important even before issues of validity and reliability could be addressed. Class time is a precious resource, so this activity lasted for only 15 min at the start and end of the semester. Even with a template to follow, this study showed that instructor priorities and variability in presenting the activity complicate consistency in timing and interpretation of the activity by students. As a result, the expectations set by the instructor could shift the distribution of codes. Furthermore, while this study did not deploy more frequent mapping, quick snapshots over time could provide a more granular understanding of student learning within a course, such as after a significant change is made in the curriculum. The lack of structure (compared to surveys) and expectation to use material from a specific course may give rise to more expansive findings of students' knowledge and experiences.</p> <p>While not accounted for in either literature reviews by Jesiek et al. (2018) or Henri et al. (2017), concept mapping appears to assess students' cognitive ability to represent complex relationships and span boundaries between technological, political, ethical, and social groups (and users) as well as geographic context. Similarly, engineering competencies have also been referred to in the sustainability literature as a key element of systems thinking (Warren et al., 2014; Wiek et al., 2016). This study reinforces the balance between technoscience and societal aspects of complex systems. Based on the pretests from the current study, students who enroll in engineering programs initially offer technically oriented representations. This pretest is important as it provides evidence of students' preconceived notions, heuristics, and assumptions (Garlock et al., 2017). However, posttests after the completing of a sociotechnical course indicate a reorientation away from decontextualized engineering practice and technological artifacts.</p> <p>Given more time and resources, concept maps might well be integrated with other assessment strategies. For example, after an initial pilot study on fourth‐ and first‐year students, several students were asked to participate in a focus group. The focus groups, similar to an exit interview, uncovered instances where the participants were expressing meta‐cognitive processes and thinking. They noted ontological shifts about the fundamental principles of engineering and how they relate to the profession, in addition to the ways in which their recent, lived experiences shaped how they deploy their knowledge on that day. While high‐level thinking is present, the research team needs to reflect on how the lack of link labeling may have impacted the total node counts and how to better facilitate link labeling during the activity.</p> <p>Coupling concept maps with the surveys used by Leydens et al. (2018) might identify correlations between the results. One additional means of experimentation across engineering programs might be to allow students to select a meaningful physical object from their lived experiences as the starting point for their concept maps. Concept mapping literature contains little guidance on emotional attachment to objects or how prior lived experiences might influence mapping activity. Furthermore, using this assessment technique within nonengineering programs could offer insights into the institutional context and population dynamics between engineering and non‐engineering programs. Concept maps are but one tool to assess learning outcomes. While this research suggests opportunities for concept maps to demonstrate changes in knowledge, further research is needed to integrate concept maps with other assessment options that would yield insights into course, curriculum, or institutional evaluation.</p> <hd id="AN0148778237-33">CONCLUSION</hd> <p>Cech's (2014) past research pointed to a concerning trend toward dualism and disengagement even as programs restructured curriculums toward developing professional engineers whoare more capable of managing complex, interdependent sociotechnical concerns. We know that an undergraduate engineering degree must include more than technical content, and our research questions explore one approach for assessing learning outcomes addressing sociotechnical thinking. In response to Question One, our findings show that concept mapping is a viable option for capturing students' knowledge constructs and evolution in sociotechnical thinking and professional skills. Curricula in U.S. engineering programs need to be designed in a manner that integrates sociotechnical thinking alongside robust assessment and evaluation of these learning outcomes for internal and external review. We transparently reflected on the opportunities and challenges associated with this evaluation tool across different courses while maintaining comparability. Concept maps offer key insights into whether students integrate knowledge of social and technical context throughout their educational experience. This approach can be deployed to address the demands of accreditation responsibilities in a time and resource efficient manner. Overall, this research encourages deliberation on institutional differences and assessment strategies, while seeking to inform best practices in engineering education and assessment. Work remains to test this approach in diverse institutional and programmatic contexts and to directly compare this paper‐pencil approach against software‐based tools for concept mapping.</p> <p>Future steps include adopting the concept mapping strategy as a means to assess changes over longer periods of time and measure learning outcomes throughout a students' educational experience from their first day on campus to postgraduation. Capturing baseline data prior to the start of an undergraduate engineer's education would provide insights into long‐term trends in knowledge acquisition and perhaps reveal the direct effects of targeted educational interventions in the curriculum. Assessment across multiple years would support cohort level understanding. Are there shared experiences that shape students' learning? Does disciplinary enculturation by major influence the integration of sociotechnical thinking (and, if so, how)? A multiuniversity study is perhaps the most important future research needed to recruit diverse participants from across the nation and generate cross‐campus comparisons.</p> <hd id="AN0148778237-34">ACKNOWLEDGMENTS</hd> <p>This research was supported with the assistance of undergraduate researchers Daniel Chen, John Eshirow, David Moran, Deidre O'Hara, Heather Reid, and Niaannette Thomas. Funding was provided by the University of Virginia's School of Engineering and Applied Science as well as by the 4VA Fund for Collaborative Research within the Commonwealth of Virginia. Portions of the Research Design section (3.0) have been previously published in the <emph>Proceedings of the ASEE Annual Conference</emph> by Foley, Ferguson, and Pollack in 2017 and 2018. No materials presented in the introduction, background, findings, discussion or conclusion are repurposed text. A copyright release for a portion of text in the methods and the repurposing of one figure was been received by the authors from the ASEE per Appendix E of the 2018 ASEE Authorship Guide, ASEE Official Plagiarism Policy.</p> <p>GRAPH: AppendixS1. Supporting Information.</p> <ref id="AN0148778237-35"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Funding information 4‐VA Commonwealth Fund; University of Virginia's School of Engineering and Applied Science</bibtext> </blist> </ref> <ref id="AN0148778237-36"> <title> REFERENCES </title> <blist> <bibtext> ABBYY FineReader. (2016). Abbyy Fine Reader (Version 12.1.625). Retrieved from https://<ulink href="http://www.abbyy.com/">www.abbyy.com/</ulink></bibtext> </blist> <blist> <bibl id="bib2" type="bt">2</bibl> <bibtext> Callon, M. (1987). Society in the making: The study of technology as a tool for sociological analysis. In W. Bijker, T. P. Hughes, & T. J. Pinch (Eds.), The social construction of technological systems: New directions in the sociology and history of technology (pp. 77 – 98). MIT Press.</bibtext> </blist> <blist> <bibl id="bib3" type="bt">3</bibl> <bibtext> Carlson, B. (1991). Innovation as social process: Elihu Thomson and the rise of General Electric (pp. 1870 – 1900). Cambridge University Press.</bibtext> </blist> <blist> <bibl id="bib4" type="bt">4</bibl> <bibtext> Cech, E. A. (2014). Culture of disengagement in engineering education? Science, Technology, & Human Values, 39 (1), 42 – 72. https://doi.org/10.1177/0162243913504305</bibtext> </blist> <blist> <bibl id="bib5" type="bt">5</bibl> <bibtext> Conley, S. N., Foley, R. W., Gorman, M. E., Denham, J., & Coleman, K. (2017). Acquisition of t‐shaped expertise: An exploratory study. Social Epistemology, 31 (2), 165 – 183. https://doi.org/10.1080/02691728.2016.1249435</bibtext> </blist> <blist> <bibl id="bib6" type="bt">6</bibl> <bibtext> David, B., & Marshall, J. (2017). Resolving epistemological tension in project‐based introductory engineering. Proceedings of the ASEE Annual Conference & Exposition. Retrieved from https://peer.asee.org/28800</bibtext> </blist> <blist> <bibl id="bib7" type="bt">7</bibl> <bibtext> Douglas, K. A., & Purzer, S. (2015). Validity: Meaning and relevancy in assessment for engineering education research. Journal of Engineering Education, 104 (2), 108 – 118. https://doi.org/10.1002/jee.20070</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> Downey, G. L., & Lucena, J. C. (2005). National identities in multinational worlds: Engineers and 'engineering cultures.' International Journal of Continuing Engineering Education and Lifelong Learning, 15 (3), 252 – 260. https://doi.org/10.1504/IJCEELL.2005.007714</bibtext> </blist> <blist> <bibl id="bib9" type="bt">9</bibl> <bibtext> Ferguson, S. M., & Foley, R. W. (2017). Assessment of student learning outcomes and ABET accreditation: A pilot study of fourth‐year engineering students using longitudinal concept maps. Proceedings of the American Society for Engineering Education Annual Conference & Exposition. Retrieved from https://peer.asee.org/27641</bibtext> </blist> <blist> <bibtext> Ferguson, S. M., Foley, R. W., Eshirow, J., & Pollack, C. C. (2018). Refining concept maps as method to assess learning outcomes among engineering students. Proceedings of the American Society for Engineering Education Annual Conference & Exposition. Retrieved from https://peer.asee.org/30924</bibtext> </blist> <blist> <bibtext> Foley, R. W., & Gibbs, B. (2019). Connecting engineering processes with responsible innovation: A response to macro‐ethical challenges. Engineering Studies, 11 (1), 9 – 33. https://doi.org/10.1080/19378629.2019.1576693</bibtext> </blist> <blist> <bibtext> Foley, R. W., Warren, A. E., Archambault, L. M., & Dong, H.‐K. (2017). Learning outcomes in sustainability education among future elementary school teachers. Journal of Education for Sustainable Development, 11 (1), 33 – 51. https://doi.org/10.1177/0973408217725861</bibtext> </blist> <blist> <bibtext> Garlock, M. E., Bhatai, A., & Laffey E. H. (2017). Effective approaches for teaching STEM‐literacy for all majors: The example of resonance. Proceedings of the American Society for Engineering Education Annual Conference & Exposition. Retrieved from https://peer.asee.org/28205</bibtext> </blist> <blist> <bibtext> Harsh, M., Bernstein, M. J., Wetmore, J. M., Cozzens, S., Woodson, T., & Castillo, R. (2017). Preparing engineers for the challenges of community engagement. European Journal of Engineering Education, 42 (6), 1154 – 1173. https://doi.org/10.1080/03043797.2016.127090</bibtext> </blist> <blist> <bibtext> Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1 (1), 77 – 89. https://doi.org/10.1080/19312450709336664</bibtext> </blist> <blist> <bibtext> Henri, M., Johnson, M. D., & Nepal, B. (2017). A review of competency‐based learning: Tools, assessments, and recommendations. Journal of Engineering Education, 106 (4), 607 – 638. https://doi.org/10.1002/jee.20180</bibtext> </blist> <blist> <bibtext> Herkert, J. R. (2001). Future directions in engineering ethics research: Microethics, macroethics and the role of professional societies. Science and Engineering Ethics, 7 (3), 403 – 414. https://doi.org/10.1007/s11948-001-0062-2</bibtext> </blist> <blist> <bibtext> Hintze, J. L., & Nelson, R. D. (1998). Violin plots: A box plot‐density trace synergism. The American Statistician, 52 (2), 181 – 184. https://doi.org/10.1080/00031305.1998.10480559</bibtext> </blist> <blist> <bibtext> Hughes, T. P. (1987). The evolution of large technological systems. In W. Bijker, T. Hughes, & T. Pinch (Eds.), The social construction of technological systems: New directions in the sociology and history of technology (pp. 46 – 76). MIT Press.</bibtext> </blist> <blist> <bibtext> Jablokow, K. W., DeFranco, J. F., & Richmond S. S. (2013). A statistical study of concept mapping metrics. Proceedings of the American Society for Engineering Education Annual Conference & Exposition. Retrieved from https://peer.asee.org/19119</bibtext> </blist> <blist> <bibtext> Jesiek, B. K., Mazzurco, A., Buswell, N. T., & Thompson, J. D. (2018). Boundary spanning and engineering: A qualitative systematic review. Journal of Engineering Education, 107 (3), 380 – 413. https://doi.org/10.1002/jee.20219</bibtext> </blist> <blist> <bibtext> Kinchin, M., Hay, D. B., & Adams, A. (2000). How a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development. Educational Research, 42 (1), 43 – 57. https://doi.org/10.1080/001318800363908</bibtext> </blist> <blist> <bibtext> Leydens, J. A., Johnson, K., Claussen, S., Blacklock, J., Moskal, B. M., & Cordova, O. (2018). Measuring change over time in sociotechnical thinking: A survey/validation model for sociotechnical habits of mind. Proceedings of the American Society for Engineering Education Annual Conference & Exposition. Retrieved from https://peer.asee.org/30794</bibtext> </blist> <blist> <bibtext> Mayes, T., & Bennett, J. (2005). ABET best practices: Results from interviews with 27 peer institutions. Proceedings of the American Society for Engineering Education Annual Conference & Exposition. Retrieved from https://peer.asee.org/14785</bibtext> </blist> <blist> <bibtext> McClure, J. R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36 (4), 475 – 492. https://doi.org/10.1002/(SICI)1098-2736(199904)36:4<475::AID-TEA5>3.0.CO;2-O</bibtext> </blist> <blist> <bibtext> Mertens, D. (2005). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. Sage Publications.</bibtext> </blist> <blist> <bibtext> Muryanto, S. (2006). Concept mapping: An interesting and useful learning tool for chemical engineering laboratories. International Journal of Engineering Education, 22 (5), 979 – 985.</bibtext> </blist> <blist> <bibtext> National Academy of Engineering. (2016). Infusing ethics into the development of engineers: Exemplary education activities and programs. The National Academies Press. https://doi.org/10.17226/21889</bibtext> </blist> <blist> <bibtext> Novak, J. D. (1990). Concept mapping: A useful tool for science education. Journal of Research in Science Teaching, 27 (10), 937 – 949. https://doi.org/10.1002/tea.3660271003</bibtext> </blist> <blist> <bibtext> Novak, J. D., & Cañas A. J. (2008). The theory underlying concept maps and how to construct them (Technical Report IHMC CmapTools 2006‐01 Rev 01‐2008). Florida Institute for Human and Machine Cognition. Retrieved from <ulink href="http://cmap.ihmc.us/Publications/ResearchPapers/TheoryUnderlyingConceptMaps.pdf">http://cmap.ihmc.us/Publications/ResearchPapers/TheoryUnderlyingConceptMaps.pdf</ulink></bibtext> </blist> <blist> <bibtext> Pfaffenberger, B. (2012). Undergraduate thesis manual. University of Virginia.</bibtext> </blist> <blist> <bibtext> Pinch, T., & Bijker, W. (1987). The social construction of facts and artifacts. In W. Bijker, T. P. Hughes, & T. J. Pinch (Eds.), The social construction of technological systems (pp. 17 – 50). MIT Press.</bibtext> </blist> <blist> <bibtext> Rittel, H. J., & Webber, M. M. (1973). Dilemmas in the general theory of planning. Policy Science, 4, 155 – 169. https://doi.org/10.1007/BF01405730</bibtext> </blist> <blist> <bibtext> Segalás, J., Ferrer‐Balas, D., & Mulder, F. K. (2008). Conceptual maps: Measuring learning processes of engineering students concerning sustainable development. European Journal of Engineering Education, 33 (3), 297 – 306. https://doi.org/10.1080/03043790802088610</bibtext> </blist> <blist> <bibtext> Shallcross, D. T. (2016). Concept maps for evaluating learning of sustainable development. Journal of Education for Sustainable Development, 10 (1), 160 – 177. https://doi.org/10.1177%2F0973408215625551</bibtext> </blist> <blist> <bibtext> Shankar, R. T., Mitsova, D., Sapat, A., & Terrell, D. J. (2017). A case‐study approach to interlink humanities with engineering education. Proceedings of the American Society for Engineering Education Annual Conference & Exposition. Retrieved from https://peer.asee.org/27447</bibtext> </blist> <blist> <bibtext> Shavelson, R. J., & Ruiz‐Primo, M. A. (2005). On the psychometrics of assessing science understanding. In J. J. Mintzes, J. H. Wandersee, & J. D. Novak (Eds.), Assessing science understanding: A human constructivist view (pp. 303 – 341). Academic Press. https://doi.org/10.1016/B978-0-12-498365-6.X5000-8</bibtext> </blist> <blist> <bibtext> Slaton, A. (2015). Meritocracy, technocracy, democracy: Understandings of racial and gender equity in American engineering education. In S. H. Christensen, C. Didier, A. Jamison, M. Meganck, C. Mitcham, & B. Newberry (Eds.), International perspectives on engineering education: Engineering education and practice in context (pp. 171 – 189). Springer Switzerland AG. https://doi.org/10.1007/978-3-319-16169-3_8</bibtext> </blist> <blist> <bibtext> Stirling, A. (2008). "Opening up" and "closing down" power, participation, and pluralism in the social appraisal of technology. Science, Technology and Human Values, 33 (2), 262 – 294. https://doi.org/10.1177/0162243907311265</bibtext> </blist> <blist> <bibtext> Turns, J., Atman, C. J., & Adams, R. (2000). Concept maps for engineering education: A cognitively motivated tool supporting varied assessment functions. IEEE Transactions on Education, 43 (2), 164 – 173. https://doi.org/10.1109/13.848069</bibtext> </blist> <blist> <bibtext> Warren, A. E., Archambault, L. M., & Foley, R. W. (2014). Sustainability education framework for teachers: Developing sustainability literacy though futures, values, systems, and strategic thinking. Journal of Sustainability Education, 6 (1), 1 – 14 Retrieved from <ulink href="http://www.susted.com/wordpress/content/sustainability-education-framework-for-teachers-developing-sustainability-literacy-through-futures-values-systems-and-strategic-thinking%5f2015%5f01/">http://www.susted.com/wordpress/content/sustainability-education-framework-for-teachers-developing-sustainability-literacy-through-futures-values-systems-and-strategic-thinking%5f2015%5f01/</ulink></bibtext> </blist> <blist> <bibtext> Watson, M. K., & Barrella, E. (2017). Using concept maps to explore the impacts of a learning‐cycle‐based sustainability module implemented in two institutional contexts. Journal of Professional Issues in Engineering Education and Practice, 143 (2), D4016001‐11. https://doi.org/10.1061/(ASCE)EI.1943-5541.0000304</bibtext> </blist> <blist> <bibtext> Wiek, A., Bernstein, M. J., Foley, R. W., Cohen, M., Forrest, N., Kuzdas, C., Kay, B. & Withycombe‐Keeler, L. (2016). Operationalising competencies in higher education for sustainable development. In M. Barth, G. Michelsen, M. Rieckmann, & I. Thomas (Eds.), Handbook of higher education for sustainable development (pp. 241 – 260). Routledge.</bibtext> </blist> <blist> <bibtext> Weinberg, A. M. (1967). Can technology replace social engineering? American Behavioral Scientist, 10 (9), 7 – 7. https://doi.org/10.1177/0002764201000903</bibtext> </blist> <blist> <bibtext> York, E. (2018). Doing STS in STEM spaces: Experiments in critical participation. Engineering Studies, 10 (1), 66 – 84. https://doi.org/10.1080/19378629.2018.1447576</bibtext> </blist> </ref> <aug> <p>By Rider W. Foley; Sean M. Ferguson and Catherine C. Pollack</p> <p>Reported by Author; Author; Author</p> <p></p> <p>Rider W. Foley is an Associate Professor in the Department of Engineering and Society at the University of Virginia, 351 McCormick Drive, Charlottesville, VA, 22904‐4744;</p> <p>Sean M. Ferguson is an Assistant Professor in the Department of Engineering and Society at the University of Virginia, 351 McCormick Drive, Charlottesville, VA 22904‐4744;</p> <p>Catherine C. Pollack is a Doctoral Candidate in the Quantitative Biomedical Sciences Program at Dartmouth College, 1 Medical Center Drive, Lebanon, NH 03756;</p> </aug>
Header DbId: eric
DbLabel: ERIC
An: EJ1286729
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Measuring the Amorphous: Substantive and Methodological Outcomes from Concept Maps
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Foley%2C+Rider+W%2E%22">Foley, Rider W.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-9362-8790">0000-0002-9362-8790</externalLink>)<br /><searchLink fieldCode="AR" term="%22Ferguson%2C+Sean+M%2E%22">Ferguson, Sean M.</searchLink><br /><searchLink fieldCode="AR" term="%22Pollack%2C+Catherine+C%2E%22">Pollack, Catherine C.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7434-5306">0000-0002-7434-5306</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Engineering+Education%22"><i>Journal of Engineering Education</i></searchLink>. Jan 2021 110(1):161-183.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 23
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2021
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Engineering+Education%22">Engineering Education</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Concept+Mapping%22">Concept Mapping</searchLink><br /><searchLink fieldCode="DE" term="%22Concept+Formation%22">Concept Formation</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+Solving%22">Problem Solving</searchLink><br /><searchLink fieldCode="DE" term="%22Thinking+Skills%22">Thinking Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Accreditation+%28Institutions%29%22">Accreditation (Institutions)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jee.20373
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1069-4730
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Accreditation organizations broadened program assessment criteria in ways that present challenges for the evaluation of learning outcomes. This is especially the case in courses where there are not narrowly defined questions or definitive solutions, such as engineering ethics. While protocols for learning assessment exist, there is limited research exploring sociotechnical learning outcomes in a manner that combines theoretical, empirical, and procedural aspects of assessment. Purpose/Hypothesis: This paper shares 3 years of research into the effectiveness of concept maps as an assessment tool for engineering students in courses that prioritize professional skills development and sociotechnical thinking. We show that concept maps can offer evidence of knowledge formation and learning outcomes associated with courses that introduce complex problems with multiple possible interpretations or viable solutions. Design/Method: A concept mapping activity was completed by 614 undergraduate engineering students at the start and end of three different courses to evaluate sociotechnical thinking. Student-level longitudinal changes were evaluated using paired t tests and simple linear regression. The concept mapping activity was improved iteratively in response to pilot tests and focus groups. Results: Undergraduate engineers demonstrated greater complexity in the representation of sociotechnical relationships as observed in the structure and content of the concept maps. The methodological results offer lessons about the administration and analysis of concept maps often excluded from conversations on course and curriculum level assessment. Conclusions: Concept maps offer insights into student learning outcomes, and they can be deployed and analyzed with minimal resources. However, assessments must be carefully designed to account for administrative and analytical challenges.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1286729
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1286729
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jee.20373
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 23
        StartPage: 161
    Subjects:
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Engineering Education
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Concept Mapping
        Type: general
      – SubjectFull: Concept Formation
        Type: general
      – SubjectFull: Problem Solving
        Type: general
      – SubjectFull: Thinking Skills
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Accreditation (Institutions)
        Type: general
    Titles:
      – TitleFull: Measuring the Amorphous: Substantive and Methodological Outcomes from Concept Maps
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Foley, Rider W.
      – PersonEntity:
          Name:
            NameFull: Ferguson, Sean M.
      – PersonEntity:
          Name:
            NameFull: Pollack, Catherine C.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-print
              Value: 1069-4730
          Numbering:
            – Type: volume
              Value: 110
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Engineering Education
              Type: main
ResultId 1