View in EDS HTML Full Text PDF Full Text

Modeling Student Negotiation of Assessment-Related Epistemological Messages in a College Science Course

Saved in:
Bibliographic Details
Title: Modeling Student Negotiation of Assessment-Related Epistemological Messages in a College Science Course
Language: English
Authors: Cara E. Schwarz, Kimberly S. DeGlopper, Nicole C. Greco, Rosemary S. Russ, Ryan L. Stowe (ORCID 0000-0002-5548-495X)
Source: Science Education. 2025 109(2):429-447.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 19
Publication Date: 2025
Sponsoring Agency: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)
National Science Foundation (NSF), Division of Undergraduate Education (DUE)
Contract Number: 2003680
2225025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: College Science, Modeling (Psychology), Undergraduate Students, Organic Chemistry, Courses, STEM Education, Student Attitudes, Evaluation Methods, Change Strategies, Epistemology, Learning, Feedback (Response)
DOI: 10.1002/sce.21914
ISSN: 0036-8326
1098-237X
Abstract: To prepare students to use science knowledge in their later personal or professional lives, we must attend to what they believe it means to know and learn science (i.e., epistemology). Unfortunately, we have little understanding of how students' epistemologies shift and are stabilized as they navigate their science courses. Researchers have made intuitive arguments that many microscale epistemological messages sum over time to give rise to macro-scale understandings of knowing and learning, but we have no theoretical model for how this sum unfolds. Here, we begin to build such a theoretical model. To do so, we focus on assessments and related materials in a college chemistry course as potentially consequential sources of messages about valued knowledge products and processes. We then elicited students' evolving understandings of assessment-related epistemological messages in several one-on-one interviews conducted throughout the semester. Analysis of how three students experienced, negotiated, and responded to assessment-related messages showed that interactions with the course system stabilized a consistent, well-resolved picture of the ways of knowing and learning that counted in the focal course. Specifically, good knowledge must have specific authority-mandated features and knowledge is justified primarily via alignment with an instructor-authored key. Students found utility in different (reliable) processes for achieving the aim of authorized knowledge, and some of these differences were maintained throughout the semester. Implications for modeling students' experience with course-embedded epistemological messages over time and how this work might inform practice are discussed.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1460580
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFXW67hdCb2i3UN-eS-Ua-LAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDFdSsHfJRqJhb8ydbQIBEICBm8YWnf8nGz4bcEiw9ayR8uKBiMY4WHeD9J_pCji8Tl38yaI_zjP_4xdRHnGVYJ4FEBp06Df_aoneWoKSkuF7srCg6Ew1yEpN9Aao2lbUsL7VujCN9uC7wgxg4rDoxTLm1FrewLBeuqYUMv0XGosYY710-u0GTzcQxT-kJk2l9KRq-_phuOlUxMgGL7GB4eNNVs-T7w8AQYSKpL9H
Text:
  Availability: 1
  Value: <anid>AN0183867415;sed01mar.25;2025Mar21.07:44;v2.2.500</anid> <title id="AN0183867415-1">Modeling Student Negotiation of Assessment‐Related Epistemological Messages in a College Science Course </title> <p>To prepare students to use science knowledge in their later personal or professional lives, we must attend to what they believe it means to know and learn science (i.e., epistemology). Unfortunately, we have little understanding of how students' epistemologies shift and are stabilized as they navigate their science courses. Researchers have made intuitive arguments that many microscale epistemological messages sum over time to give rise to macro‐scale understandings of knowing and learning, but we have no theoretical model for how this sum unfolds. Here, we begin to build such a theoretical model. To do so, we focus on assessments and related materials in a college chemistry course as potentially consequential sources of messages about valued knowledge products and processes. We then elicited students' evolving understandings of assessment‐related epistemological messages in several one‐on‐one interviews conducted throughout the semester. Analysis of how three students experienced, negotiated, and responded to assessment‐related messages showed that interactions with the course system stabilized a consistent, well‐resolved picture of the ways of knowing and learning that counted in the focal course. Specifically, good knowledge must have specific authority‐mandated features and knowledge is justified primarily via alignment with an instructor‐authored key. Students found utility in different (reliable) processes for achieving the aim of authorized knowledge, and some of these differences were maintained throughout the semester. Implications for modeling students' experience with course‐embedded epistemological messages over time and how this work might inform practice are discussed.</p> <p>Keywords: assessment; chemistry; epistemology; learning; science</p> <hd id="AN0183867415-2">Introduction</hd> <p>Across the United States, policy makers, universities, education researchers, and instructors are dedicating a substantial amount of time, financial resources, and person‐hours to reforming undergraduate science instruction (Kozminski et al. [<reflink idref="bib41" id="ref1">41</reflink>]; Bauerle et al. [<reflink idref="bib3" id="ref2">3</reflink>]). These reforms are motivated, at least tacitly, by a desire for science education to help learners use science knowledge in their later personal and professional lives. For example, <emph>A Framework for K‐12 Science Education</emph>, which underpins many ongoing reforms in higher education (Cooper et al. [<reflink idref="bib13" id="ref3">13</reflink>]; Matz et al. [<reflink idref="bib46" id="ref4">46</reflink>]), begins by emphasizing the importance of scientific and engineering knowledge in engaging "with the major public policy issues of today as well as to make informed everyday decisions" (The National Research Council [<reflink idref="bib52" id="ref5">52</reflink>], p. 7). AAAS' Vision and Change, which articulates a vision for undergraduate biology education, likewise notes that, "as a growing number of societal challenges, from preserving the environment to advancing human health and quality of life, intersect with biology, future scientists and nonscientists alike must become adept at making connections among seemingly disparate pieces of information, concepts, and questions, as well as be able to understand and evaluate evidence" (Bauerle et al. [<reflink idref="bib3" id="ref6">3</reflink>], p. 3). We claim that goals of this sort are intentionally and explicitly not about accumulating certain authorized science facts. Instead, these goals are about what students believe it means to know and learn science (i.e., epistemologies; Hofer and Pintrich [<reflink idref="bib35" id="ref7">35</reflink>]).</p> <p>Although our field espouses goals for science learning that are largely epistemological, approaches to course reform, particularly in chemistry, have focused on organizing teaching practices and curricula to support construction of expert‐like answers. Claims that reforms are working are often supported by data showing students enrolled in reformed courses construct more canonically correct answers than students enrolled in comparable, more traditionally structured courses (e.g., Cooper and Klymkowsky [<reflink idref="bib12" id="ref8">12</reflink>]; Talanquer and Pollard [<reflink idref="bib76" id="ref9">76</reflink>]; DeGlopper et al. [<reflink idref="bib20" id="ref10">20</reflink>]; Vincent‐Ruz et al. [<reflink idref="bib78" id="ref11">78</reflink>], Shwartz et al. [<reflink idref="bib73" id="ref12">73</reflink>]). Research in science education suggests that we should not assume useful epistemological outcomes will automatically result from courses optimized to support construction of scientifically accurate answers (Berland and Hammer [<reflink idref="bib4" id="ref13">4</reflink>]; Gouvea and Passmore [<reflink idref="bib30" id="ref14">30</reflink>]; Kuhn and Pease [<reflink idref="bib42" id="ref15">42</reflink>]; Manz [<reflink idref="bib45" id="ref16">45</reflink>]; McNeill et al. [<reflink idref="bib47" id="ref17">47</reflink>]). Decades of scholarship have demonstrated that, even in reformed courses, students often see the purpose of class work as accumulating disconnected, authority‐approved facts (Redish, Saul, and Steinberg [<reflink idref="bib55" id="ref18">55</reflink>]). This leads to class engagement in "doing the lesson" (Jiménez‐Aleixandre, Rodríguez, Duschl, [<reflink idref="bib38" id="ref19">38</reflink>])—performatively constructing knowledge products to please the teacher. This form of engagement—<emph>I constructed this knowledge because #4 said "</emph>Draw the Lewis structure for water"—is very different from why scientists and citizens might construct knowledge—<emph>to make progress</emph> "on communal objects and meet evolving goals<emph>"</emph> (Manz [<reflink idref="bib45" id="ref20">45</reflink>], p. 555). There is a mismatch between our goal of making science useful later in personal and professional life and the kinds of work students may do to achieve correct answers in our science classrooms. To prepare scientists and scientifically literate citizens, we have to attend to and work to develop students' epistemologies, not simply gauge the extent to which they write something an expert would deem correct.</p> <p>Although we know that students often emerge from our science courses with epistemologies of limited utility in postschool life, we do not know the mechanism(s) by which these epistemologies are affected by interactions with the course system. There is some literature that documents the way classroom interactions lead to students performatively engaging in science work in individual moments (Berland and Reiser [<reflink idref="bib5" id="ref21">5</reflink>]; Berland and Hammer [<reflink idref="bib4" id="ref22">4</reflink>]; Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref23">56</reflink>]; Manz [<reflink idref="bib45" id="ref24">45</reflink>]). To account for how these interactions lead to specific sorts of student engagement, some scholars have suggested that instruction sends tacit "meta‐messages" (Berland et al. 2015, p. 1084) about how students should understand and engage in knowledge construction. Specifically, they hypothesize the impact of instructor epistemological messages—messages about "what constitutes knowledge or learning in the immediate moment" (Russ [<reflink idref="bib58" id="ref25">58</reflink>], p. 99)—on student science learning.</p> <p>One limitation of these studies of epistemological messaging is that they take the form of case studies at a very fine grain size lasting less than 5 min (Berland and Hammer [<reflink idref="bib4" id="ref26">4</reflink>]; Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref27">56</reflink>]; Scherr and Hammer [<reflink idref="bib66" id="ref28">66</reflink>]; Shar, Russ, and Laverty [<reflink idref="bib71" id="ref29">71</reflink>]; Irving, Martinuk, and Sayre [<reflink idref="bib37" id="ref30">37</reflink>]). While these studies offer nuanced understandings of individual moments of epistemological messaging, they do not provide empirical evidence for the impact of those messages over longer timescales. Instead, researchers typically theoretically assert that it is the sum of these individual moments over time that give rise to these course‐level outcomes (Ke and Schwarz [<reflink idref="bib39" id="ref31">39</reflink>]). For example, Russ et al. ([<reflink idref="bib59" id="ref32">59</reflink>]) suggest that, "This single episode may not have a long‐standing influence on [the students]; however, an accumulation of school experiences will likely shape [student] ideas of what it means to do science" (p. 889). However, we have no empirical backing for such claims. We do not know whether or how individual moments synthesize across the many interactions in a semester.</p> <p>Our goal in this work, then, is to begin to build a bridge between these individual moments and the survey work examining students' epistemologies before and after a semester of instruction (Adams et al. [<reflink idref="bib2" id="ref33">2</reflink>]; Redish, Saul, and Steinberg [<reflink idref="bib55" id="ref34">55</reflink>]). However, it is analytically impossible to account for how every aspect of a course over an entire semester affects students' epistemologies. As a result, we must "place bets," both on which parts of the learning environment are most consequential for student epistemologies and which times during the semester are epistemologically meaningful for students.</p> <p>In our work here we place our bets on assessment to determine both where and when to look for moments of epistemological consequence. Specifically, we map students' experiences of messages about knowledge and knowing (i.e., epistemological messages) related to course exams and quizzes (hereafter referred to simply as "assessments"). We focus on assessment‐related epistemological messages for three reasons: (<reflink idref="bib1" id="ref35">1</reflink>) assessments powerfully affect what students believe matters in a course (Crooks [<reflink idref="bib19" id="ref36">19</reflink>]; Entwistle [<reflink idref="bib26" id="ref37">26</reflink>]; Momsen et al. [<reflink idref="bib51" id="ref38">51</reflink>]; Scouller [<reflink idref="bib69" id="ref39">69</reflink>]; Scouller and Prosser [<reflink idref="bib70" id="ref40">70</reflink>]; Snyder [<reflink idref="bib74" id="ref41">74</reflink>]), (<reflink idref="bib2" id="ref42">2</reflink>) instructors can (in principle) alter their assessment practices in response to the findings from studies like this one, and (<reflink idref="bib3" id="ref43">3</reflink>) little is known about how students experience epistemological messages related to assessments (Shar, Russ, and Laverty [<reflink idref="bib71" id="ref44">71</reflink>]). By tracking how students understand course assessments at different points in the semester, we begin the work of understanding how students' end‐of‐course epistemologies emerge from local interactions with the course system. This in turn will allow researchers and practitioners to more effectively design for and assess progress toward epistemological outcomes.</p> <p>To advance this work, we pose the following research question:</p> <p>How do assessment‐related epistemological messages influence students' perspectives on valued ways of knowing and learning in a semester‐long college STEM course?</p> <p>We address this question by eliciting students' understandings of assessment‐embedded epistemological messages in several one‐on‐one interviews conducted throughout the semester. We believe that, ultimately, these localized interviews will allow us to understand how assessments repeatedly stimulate particular epistemologies over the course of the semester to produce relatively stable understandings of science and science learning in the course. This and subsequent work aimed at bringing the epistemological landscape of courses into focus will support inferences about how course components impact student epistemological learning.</p> <hd id="AN0183867415-3">Prior Research and Conceptual Framework</hd> <p>If we as science educators take seriously our goal for science learning to be useful in daily life, then we must pay attention to what students believe it means to know and do science. Specifically, unless students see science as something that is relevant outside of the classroom and as something they can learn without needing an authority (like a teacher) to monitor them, they will not draw on the constructs and practices of science when they encounter situations and issues related to science in their lives. These ideas about science and science learning that we want students to use in their daily lives are epistemological in nature.</p> <p>In this work, we draw on the construct of epistemologies as it is used in the education and science education literature rather than as it is used in philosophy. Specifically, we follow scholarship that suggests that "what individuals believe about knowledge and knowing and how they think and reason about epistemological aspects of knowing are all part of a psychological construct" called epistemic cognition (Hofer, [<reflink idref="bib34" id="ref45">34</reflink>], p. 19). As Elby, Macrander, and Hammer ([<reflink idref="bib24" id="ref46">24</reflink>]) articulate, this theorization of epistemology merges research traditions in personal epistemology (an individuals' beliefs about their own knowledge and knowing) and the nature of science (an individuals' beliefs about knowledge and knowing in professional science).</p> <p>Students' epistemologies thus defined are important because they have implications for transfer of learning to everyday life. In her study of fifth graders learning about species survival, Engle ([<reflink idref="bib25" id="ref47">25</reflink>]) demonstrates that transfer occurs when students understand themselves as knowledge producers in a larger intellectual community that is temporally connected to contexts in which they will use their knowledge. Hammer and his colleagues (2005) further flesh out the mechanisms by which epistemology may underpin, or drive, these local temporal connections. In this sense, epistemologies support what Bransford and Schwartz ([<reflink idref="bib8" id="ref48">8</reflink>]) call "preparation for future learning," which we would reframe as "preparation for future learning in the everyday world." Taken together, this work indicates that epistemologies are essential for students seeing their learning in one context (i.e., school) as relevant to other contexts (i.e., personal or professional life). Developing epistemologies must be a key outcome if our goals for science education involve students using science in their postschool lives.</p> <hd id="AN0183867415-4">The Need to Explicitly Attend to Student Epistemologies in College Science Learning</hd> <p>Despite broad agreement among science education scholars that epistemology is important (Greene, Sandoval, and Bråten [<reflink idref="bib31" id="ref49">31</reflink>]), there is very little work that seeks to explicitly improve students' epistemological outcomes. Specifically in college chemistry courses, reform efforts rarely focus on <emph>why</emph> enrolled students construct knowledge. Instead, these efforts aim to improve course grades, drop/fail/withdraw rates, or scores on researcher‐created instruments by changing what is taught (curriculum) and/or how classes are taught (instructional practices). For example, a comprehensive review on the efficacy of Peer‐Led Team Learning in college STEM found that the vast majority of studies reported course grades, scores on nationally normed exams, and/or retention as a measure of success (Wilson and Varma‐Nelson [<reflink idref="bib81" id="ref50">81</reflink>]). Studies evaluating curricular changes in general chemistry (Williams et al. [<reflink idref="bib80" id="ref51">80</reflink>]; Cooper et al. [<reflink idref="bib16" id="ref52">16</reflink>]; Talanquer and Pollard [<reflink idref="bib76" id="ref53">76</reflink>]) and organic chemistry (Crandell, Lockhart, and Cooper [<reflink idref="bib17" id="ref54">17</reflink>]; DeGlopper et al. [<reflink idref="bib20" id="ref55">20</reflink>]) likewise report the percentage of students who respond to content‐focused instruments in more or less expert‐like ways. Success in college chemistry (and STEM more broadly) is often determined by students' conceptual, rather than epistemological, understandings.</p> <p>This is not to say that no studies on the epistemological outcomes of college chemistry courses exist. A few studies in chemistry education have evaluated the impact of curricular changes on students' beliefs about learning (Bowen, Flaherty, and Cooper [<reflink idref="bib7" id="ref56">7</reflink>]) or their understanding of the nature of science (NOS; Russell and Weaver [<reflink idref="bib61" id="ref57">61</reflink>]; Flaherty [<reflink idref="bib27" id="ref58">27</reflink>]). For example, Flaherty ([<reflink idref="bib27" id="ref59">27</reflink>]) interviewed students about the structure and development of knowledge in science generally and in their core‐idea focused organic chemistry course specifically. She found that students viewed scientists as people who interrogate the data behind claims and generate knowledge through experiments involving trial and error. Similarly, students saw the need to understand how and why a reaction occurs rather than memorize what happens and to practice applying foundational concepts to new problems. Although in this example, the epistemological outcomes were largely positive, Flaherty argues that future work in science education should not treat epistemological outcomes as secondary to conceptual understanding in undergraduate science courses.</p> <p>There is other research from K‐12 science education that reinforces the idea that epistemological outcomes do not merely come along for the ride just because students construct expert‐like knowledge products. Although this work comes from the K‐12 rather than the undergraduate context, we assume the theoretical mechanisms connecting epistemology and conceptual knowledge at play are the same. Specifically, we follow the model in which epistemology and conceptual knowledge are distinct cognitive entities, and in which epistemology acts as a control structure that "choose[s] how an individual will construct [conceptual] knowledge in a particular situation" (Redish, [<reflink idref="bib54" id="ref60">54</reflink>], p.23). We are not alone in this assumption; there is a long history of work, particularly in the field of undergraduate physics education, that relies on theories and mechanisms of learning developed in the K‐12 context (Bing & Reddish, [<reflink idref="bib6" id="ref61">6</reflink>]; Hammer, Elby, Scherr, & Reddish, 2005; Hutchison and Hammer [<reflink idref="bib36" id="ref62">36</reflink>]; Lising & Elby, [<reflink idref="bib44" id="ref63">44</reflink>]; Odden & Russ, [<reflink idref="bib53" id="ref64">53</reflink>]; Redish, [<reflink idref="bib54" id="ref65">54</reflink>]; Scherr and Hammer [<reflink idref="bib66" id="ref66">66</reflink>]; Shar, Russ, and Laverty [<reflink idref="bib71" id="ref67">71</reflink>]; Tuminaro & Redish, [<reflink idref="bib77" id="ref68">77</reflink>]). Although the specifics of the content vary dramatically across these two contexts, the notion that epistemologies are a distinct set of knowledge from conceptual knowledge applies to both.</p> <p>Beyond these theoretical assumptions about their distinctness, empirical studies also demonstrate that conceptual and epistemological knowledge do not necessarily codevelop. In fact, many have argued that it is inappropriate to assume canonically correct answers are underpinned by epistemologies that are potentially useful in later life (Berland and Hammer [<reflink idref="bib4" id="ref69">4</reflink>]; Gouvea and Passmore [<reflink idref="bib30" id="ref70">30</reflink>]; Kuhn and Pease [<reflink idref="bib42" id="ref71">42</reflink>]; Manz [<reflink idref="bib45" id="ref72">45</reflink>]; McNeill et al. [<reflink idref="bib47" id="ref73">47</reflink>]). Students often performatively engage in science practices to please the teacher rather than to make sense of their own ideas and experiences. Such engagement may result in expert‐like knowledge products (e.g., explanations, arguments) that mean little to students. Performative practice engagement is not useful in daily life and thus learning environments which promote this sort of activity are not well aligned with our field's goals. It is incumbent on those of us who study science learning, then, to work to understand how we can create classes in which students experience the purpose of their work as something sensible and useful for them instead of just jumping through teacher‐imposed hoops. We must then attend to, and work to affect, students' epistemologies in and of themselves outside of attention to or focus on students' conceptual understandings.</p> <hd id="AN0183867415-5">Epistemological Resources</hd> <p>If we want to explicitly attend to epistemologies in our course design and enactments, we must have a way to model their ontology such that we can identify and track them. Some researchers model epistemologies through developmental stages (e.g., King and Kitchener [<reflink idref="bib40" id="ref74">40</reflink>]) that act as traits that students have. In this approach, epistemological traits are inherent to the student and are something they use across contexts in their lives. These traits may change (or develop) over time (e.g., from high school to college) in stages, but at any given time they are relatively constant and fixed. Other models adopt a view of epistemologies as theories or systems that students consciously possess and apply in their lives (Hofer and Pintrich [<reflink idref="bib35" id="ref75">35</reflink>]; Schommer‐Aikins, Bird, and Bakken [<reflink idref="bib67" id="ref76">67</reflink>]). Here, rather than being traits inherent to the student, epistemologies are ideas students take on and then use as they make sense of the world. In both of these models however, students possess a priori epistemological theories that they then apply globally, or consistently across many contexts.</p> <p>In our work we take a different view of epistemologies. While we agree with the above researchers that epistemologies are a cognitive, psychological construct, we view the ontology of that construct quite differently. Specifically, we understand epistemology as made up of many, fine‐grained pieces, (DiSessa [<reflink idref="bib22" id="ref77">22</reflink>]; Minstrell [<reflink idref="bib50" id="ref78">50</reflink>]; Sherin [<reflink idref="bib72" id="ref79">72</reflink>]) and adopt Hammer and Elby's ([<reflink idref="bib32" id="ref80">32</reflink>]) "framework of epistemological resources [that are] smaller and more general than theories or traits" (p. 176). Instead of people having "precompiled" (Hammer et al. [<reflink idref="bib33" id="ref81">33</reflink>], p. 95) epistemologies that they apply across different contexts, we assume that people draw on many epistemological pieces and put them together locally in the moments when they are needed for knowledge construction. That is, instead of students possessing a priori traits or theories about epistemologies that they apply wholesale to new situations, we assume that they have many general ways of thinking about knowledge that, when faced with a new context or situation, they assemble into a local coherence that they use in that moment alone (Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref82">56</reflink>]). After that context or situation is over, the local coherence they assembled is broken up again into its pieces to be recompiled again later in a new way in response to a new situation. This model of epistemologies is consistent with recent work around epistemic cognition (Greene, Sandoval, and Bråten, [<reflink idref="bib31" id="ref83">31</reflink>]; Elby, Macrander, and Hammer, [<reflink idref="bib24" id="ref84">24</reflink>]) and data from undergraduate science courses (e.g., Irving, Martinuk, and Sayre [<reflink idref="bib37" id="ref85">37</reflink>]; Scherr and Hammer [<reflink idref="bib66" id="ref86">66</reflink>]; Shar, Russ, and Laverty [<reflink idref="bib71" id="ref87">71</reflink>]) that show transitions between epistemologies often occur over minutes (rather than the hours, days, or years assumed in more stable models of cognition).</p> <p>One of the key premises of a fine‐grained model of epistemological resources is that different situations call for different epistemologies (Elby and Hammer [<reflink idref="bib23" id="ref88">23</reflink>]). That is, epistemological resources are constantly being recompiled in response to the context such that epistemological coherences are necessarily situated in context (Sandoval [<reflink idref="bib64" id="ref89">64</reflink>], [<reflink idref="bib63" id="ref90">63</reflink>]). For example, a student may in the morning treat their doctor as an authority on knowledge when they are seeking help for an infection but may instead seek to draw on their own experiences when trying to make sense of data in their physics lab later that day. In both contexts, students will construct conceptual knowledge in those individual moments based on the epistemological resources they have activated. That construction is not necessarily long‐lasting or durable but is rather an in‐the‐moment construction that supports their reasoning in that moment.</p> <p>It is this variability in utility across contexts that makes it impossible to say that some epistemologies are always better than others; it is not the case that using your own experience (e.g., in lab) is always better than receiving knowledge from authority (e.g., the doctor's office). As such, we reject models in which epistemological sophistication is universally defined by a single set of expert views. Instead, epistemological sophistication—the sort of epistemological outcome we believe supports students in using science in daily life—is the ability to "explore and discuss the differences between knowledge in multiple contexts" (Elby and Hammer [<reflink idref="bib23" id="ref91">23</reflink>]). That is, sophistication involves the recognition of variability and the judicious use of epistemological resources as dictated by the context. A resources model of epistemologies demands that we attend to students' changing use of a multitude of fine‐grained epistemologies over time.</p> <hd id="AN0183867415-6">A Need to Bridge Existing Research</hd> <p>As evidenced above, we are certainly not the first scholars to suggest that our field needs to focus on epistemologies and epistemological outcomes for our students. There have been two major threads of this work, one at what we think of as the micro level and one at the macro level. At the micro level, scholars have attended closely to the dynamics of student epistemologies as they unfold over a time scale of minutes of classroom instruction (Berland and Hammer [<reflink idref="bib4" id="ref92">4</reflink>]; Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref93">56</reflink>]; Scherr and Hammer [<reflink idref="bib66" id="ref94">66</reflink>]; Shar, Russ, and Laverty [<reflink idref="bib71" id="ref95">71</reflink>]; Irving, Martinuk, and Sayre [<reflink idref="bib37" id="ref96">37</reflink>]). For example, Rosenberg and his colleagues (2006) present a rich analysis of how a group of eighth graders drastically shift from reasoning about the rock cycle through "cut[ting] and past[ing]" (p. 283) random, disconnected facts from class and worksheets to "storytelling" (p. 285) using their everyday experiences of heat and pressure. Scherr and Hammer ([<reflink idref="bib66" id="ref97">66</reflink>]) describe shifts in student reasoning in a college physics class as they transition between working on a worksheet, discussing with one another, working with the TA, and joking around. When students shift between these different activities, they activate and use different sets of epistemologies. Our field has been successful in understanding these micro‐moments of change in student epistemologies.</p> <p>On the other hand, our field has also been quite successful in modeling macro level changes in student epistemologies across semesters and years of instruction. These studies typically use a pre– post survey design to assess students' espoused, or formal (Sandoval [<reflink idref="bib65" id="ref98">65</reflink>]), epistemologies. While theoretically we agree with Sandoval ([<reflink idref="bib65" id="ref99">65</reflink>]) that such surveys may underrepresent the breadth of student epistemologies and overrepresent their stability, these types of surveys can still provide some measure of how students understand knowledge construction in these courses. For example, in their study of college physics classes across the country (both traditional and reformed), Redish, Saul and Steinberg ([<reflink idref="bib55" id="ref100">55</reflink>]) found that, "[a]t <emph>every</emph> school we studied, the overall results [of student epistemologies] deteriorated as the result of one semester of instruction" (p. 224, emphasis ours). Similarly, when the Colorado Learning Attitudes about Science Survey (adapted for chemistry) was administered to students in the first semester of general chemistry, students' posttest scores were lower on average—farther away from desirable—than their pre‐test scores (Adams et al. [<reflink idref="bib2" id="ref101">2</reflink>]). These results demonstrate that student epistemologies do change across a semester.</p> <p>How do the types of in‐the‐moment shifts described in the micro‐scale studies relate to the survey‐level shifts we see across semesters of undergraduate science courses? As a field, we have only theoretical claims that the micro‐shifts add up in some way to the kinds of larger shifts observed in surveys. Our work here is an attempt to build an empirical bridge between these two kinds of studies by drawing on another theoretical construct: epistemological messaging.</p> <hd id="AN0183867415-7">Epistemological Messages</hd> <p>What leads students to compile their epistemological understandings in a particular way in a given moment and ultimately across a semester? To answer this question, we draw on work by Russ that argues that, during instruction, teachers tacitly communicate to students the type of scientific thinking that is valued in that moment. For example, a science teacher who listens to a student's idea and says, "Oh, it's interesting how you brought in what we learned from the last chapter," sends the message that drawing on prior knowledge is an important part of science (Russ [<reflink idref="bib58" id="ref102">58</reflink>], p. 98). These types of messages are epistemic, or epistemological in nature (e.g., Berland et al. 2015; Christodoulou and Osborne [<reflink idref="bib11" id="ref103">11</reflink>]) because they are about knowledge and knowledge construction. (Some scholars distinguish between epistemic and epistemological (Sandoval [<reflink idref="bib63" id="ref104">63</reflink>]); here, however, we refer to all messages about knowledge and knowing as epistemological.) It is these messages that convey to students which epistemological resources they should draw on during individual moments of classroom learning.</p> <p>Russ ([<reflink idref="bib58" id="ref105">58</reflink>]) originally introduced the notion of epistemological messages being present in high school teachers' in‐the‐moment responses to student ideas during class. Ke and Schwarz ([<reflink idref="bib39" id="ref106">39</reflink>]) took up that perspective and investigated the messages elementary school teachers send to their students over the course of a single, model‐based unit. Each of these lines of work focused exclusively on the messages embedded in student–teacher verbal interactions.</p> <p>Here, we extend the construct of epistemological messages to think about how they are present in the course design itself. There are many different aspects of epistemological messaging that we could pay attention to. We could attend to the instructors' intended epistemological messages—what they want students to take away from course elements. We could pay attention to how students experience, negotiate, and respond to messages embedded in course elements. We could pay attention to the alignment (or lack thereof) of instructor and students' epistemological understandings of the course elements. Although instructors may intend specific messages in their course elements, those intentions mean little if the students do not take them up in that way. As a result, since it is students' experiences and understandings of course elements that ultimately drive how they engage in science learning (Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref107">56</reflink>]; Russ [<reflink idref="bib58" id="ref108">58</reflink>]; Ke and Schwarz [<reflink idref="bib39" id="ref109">39</reflink>]), here we focus on unpacking the epistemological resources students use to make sense of messages in course materials rather than focusing on instructor intent.</p> <p>Specifically, in this work we place our bets on messages students experience from credit‐earning assessments (i.e., quizzes and exams) given throughout the course. We do so because there is a great deal of literature that demonstrates how such assessments drive much of what students believe matters in a course (Crooks [<reflink idref="bib19" id="ref110">19</reflink>]; Entwistle [<reflink idref="bib26" id="ref111">26</reflink>]; Momsen et al. [<reflink idref="bib51" id="ref112">51</reflink>]; Scouller [<reflink idref="bib69" id="ref113">69</reflink>]; Scouller and Prosser [<reflink idref="bib70" id="ref114">70</reflink>]; Snyder [<reflink idref="bib74" id="ref115">74</reflink>]). Additionally, some research suggests that the specific nature of individual assessment tasks can also cue and shift student epistemological resources (Shar, Russ, and Laverty [<reflink idref="bib71" id="ref116">71</reflink>]). Thus, in addition to assessments obtaining information from students, we also hypothesize that assessments, especially if they are high‐stakes, can send powerful epistemological messages to students about what type of knowledge is worth knowing, whose knowledge is worth knowing, and how knowledge should be constructed and evaluated. Our methods reflect how we have placed our bets on high‐stakes assessments as a key source of epistemological messaging—and thus as a key source of information for students' shifting epistemologies—in undergraduate chemistry courses.</p> <hd id="AN0183867415-8">Research Question</hd> <p>If we want students to use the concepts and practices of science they have learned in school in their postschool lives, developing their epistemologies must be a key outcome for our classes. Specifically, we cannot focus all of our efforts on conceptual understanding and just assume that students' epistemologies will come along for the ride. In this work, we explore whether and how aspects of our courses send messages to students about particular epistemologies they should adopt and use in their learning. We specifically place bets on understanding how high‐stakes assessments—long shown to influence student thinking and learning in courses—impact student epistemologies. To do so, we seek to answer the following research question:</p> <p>How do assessment‐related epistemological messages influence students' perspectives on valued ways of knowing and learning in a semester‐long college STEM course?</p> <p>In answering this question, we seek to build an initial model for how epistemological messages can bridge in‐the‐moment studies of student epistemologies with findings from semester‐long surveys about student understandings of science and science learning.</p> <hd id="AN0183867415-9">Methods</hd> <p></p> <hd id="AN0183867415-10">Course Context and Participants</hd> <p>Students were recruited from a large‐enrollment, undergraduate organic chemistry course at a large midwestern university. This course was designed to embody an emerging consensus on best practices in college STEM instruction (Bauerle et al. [<reflink idref="bib3" id="ref117">3</reflink>]; Cooper et al. [<reflink idref="bib13" id="ref118">13</reflink>], [<reflink idref="bib15" id="ref119">15</reflink>]; DeGlopper et al. [<reflink idref="bib20" id="ref120">20</reflink>]; Stowe et al. [<reflink idref="bib75" id="ref121">75</reflink>]). Instruction and assessments were structured around scaffolded progressions of core ideas (e.g., energy, bonding; Cooper, Posey, and Underwood [<reflink idref="bib14" id="ref122">14</reflink>]) that build in complexity as students predict, explain, and model phenomena. Accordingly, high‐stakes assessments placed substantial emphasis (e.g., 35%–60% of total points on exams) on students connecting energy and bonding to construct normative explanations for how and why observable events unfolded as they did (DeGlopper et al. [<reflink idref="bib20" id="ref123">20</reflink>]). Graded assessments included four exams and three quizzes, which accounted for 87% and 13% of the students' total scores, respectively. Problem sets, discussion activities, and practice exams and quizzes were provided to support students' learning but were not graded. Instead, answer keys were made available so that students could assess their own work. In‐class interactions were consistent with focus on causal accounts for phenomena—instructors and/or students spent ~80% of class time engaged in this sort of work (Schwarz et al. [<reflink idref="bib68" id="ref124">68</reflink>]).</p> <p>This study involved two phases: a pilot consisting of one interview per student during an 8‐week summer 2022 Organic II course, and a full study involving three interviews per student over the course of a fall 2022 Organic I class. Four students volunteered to participate in the pilot study, and ten students volunteered to participate in the full study. Eight students completed all three interviews, and two students decided to end their participation after the first or second interview. All students who volunteered to participate in the study identified as women.</p> <hd id="AN0183867415-11">Data Collection</hd> <p></p> <hd id="AN0183867415-12">Interview Design</hd> <p>Our goal in this work is to explore how students understand epistemological messages related to course assessments and how those local understandings build to larger, semester‐long understandings. Typical survey measures of student epistemologies (e.g., Barbera et al. 2008; Redish, Saul, and Steinberg [<reflink idref="bib55" id="ref125">55</reflink>]; White et al. [<reflink idref="bib79" id="ref126">79</reflink>]) are insufficiently fine‐grained to capture the sort of situated, in‐the‐moment understanding of messages our theoretical framework predicts. Thus, ideally, we would capture these dynamic understandings in situ as messages appeared in course materials; we would like to capture their practical epistemologies (Sandoval [<reflink idref="bib65" id="ref127">65</reflink>]). This ideal approach would involve observing students as they experienced a message and subsequently made sense of it.</p> <p>However, this approach is not ideal for messages embedded in written assessments for a variety of reasons. First, asking students to stop and speak with us about their responses to exam‐embedded messages would distract them and prevent them from performing on the assessment. Second, we theorize along with Russ ([<reflink idref="bib58" id="ref128">58</reflink>]) that students' experiences of epistemological messages are cognitive and may occur without any visible action or behavior that would indicate their thinking. Third, even if there were observable behaviors that resulted from their understanding of those messages, our field has yet to theorize the timescale along which we might expect to see those behaviors. We would not know when or where to look for our signal. Finally, some messages about assessments are found in course materials that students access outside of class time. While it is becoming increasingly feasible with today's recording technology to have students capture their own data outside of class, the resulting data would likely include substantial noise and very little signal. Given the diffuse and tacit nature of students' understandings of epistemological messages, observations are not a feasible data collection strategy to address our research question in our study context.</p> <p>As a result of these challenges, we designed and conducted semi‐structured, one‐on‐one, qualitative interviews to elicit student epistemological resources around assessment messages. In designing our interviews, we drew questions from several major types of interviews used in the literature.</p> <p></p> <ulist> <item> 1. Think aloud interviews (Charters [<reflink idref="bib9" id="ref129">9</reflink>]) allow participants to verbalize their thinking, thereby giving interviewers access to their ways of making sense that would usually be hidden. For example, we asked students to think aloud as they read through the syllabus for the first time, particularly focusing on places in which epistemological information is given.</item> <p></p> <item> 2. Task‐based, cognitive clinical interviews (Ginsburg [<reflink idref="bib28" id="ref130">28</reflink>]; Russ, Lee, and Sherin [<reflink idref="bib60" id="ref131">60</reflink>]) involve designing tasks that students might experience outside of the interview and then encouraging them to share their thinking as they go about completing the task. For example, we provided students with a course assessment question and the answer key and asked them to talk through not only their solution but why they chose to draw on and use the knowledge they did in their solution.</item> <p></p> <item> 3. Stimulated recall interviews (Dempsey [<reflink idref="bib21" id="ref132">21</reflink>]) present participants with recorded data of events that happened in the past and ask participants to comment on what happened. For example, we presented students with a post from the online class forum with epistemological import and asked them to identify and talk through how they understood that post.</item> </ulist> <p>Our goal with each of these interview types was to generate tasks for the interview protocol that simulate contexts students would experience in their science courses. This process allowed us to, as closely as possible, approximate students' practical epistemologies (Sandoval [<reflink idref="bib65" id="ref133">65</reflink>]) in the interview setting.</p> <p>Although these interview types give us the form of interview questions, we turn to multidimensional models of epistemology to provide the substance of our questions. Specifically, we adopted the AIR model of epistemology proposed by Chinn, Rinehart, and Buckland ([<reflink idref="bib10" id="ref134">10</reflink>]) to guide construction of our interview protocol and interpret dialog elicited by this protocol. We attended to three key dimensions of epistemology:</p> <p></p> <ulist> <item> 1. <emph>Epistemic aims</emph> are the ends to which other aspects of epistemic cognition are directed. We anticipate aims such as understanding or true beliefs. <emph>Epistemic values</emph> define the relative worth of different aims. In some contexts, knowing appropriate facts may be more useful than a causal explanation and vice versa.</item> <p></p> <item> 2. <emph>Epistemic Ideals</emph> are "the criteria or standards that must be met for [one] to judge that their epistemic ends have been achieved" (Chinn, Rinehart, and Buckland [<reflink idref="bib10" id="ref135">10</reflink>], p. 426). These may include coherence with other accepted knowledge, alignment with testimony from a trusted expert, or agreement with observation.</item> <p></p> <item> 3. <emph>Reliable Processes</emph> refer to the actions a learner takes toward or away from achieving epistemic aims; they are answers to the question "What are you doing?" For example, students might describe processes such as telling a story or recalling facts (Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref136">56</reflink>]).</item> </ulist> <p>Protocol questions were designed to probe particular dimensions of epistemology identified by Chinn and his colleagues. Our semi‐structured interviews allowed us to capture the nuance of student thinking as they encountered and made sense of the epistemological messages embedded in course assessments.</p> <p>Once we had constructed the initial interview protocol, we piloted the interview mid‐semester (after the second exam) in a summer Organic Chemistry II course. We chose to conduct the interviews at this time in the semester because we anticipated that students would have encountered, developed, and acted on understandings of assessment‐embedded messages. This in turn may have led to them receiving additional feedback/data about the appropriateness of those understandings. The interviews had three parts: explication of exam preparation, reflecting on/answering an exam question, and evaluating sample exam responses. Pilot interviews ranged in length from 30 to 61 min.</p> <p>The most important outcome of the pilot interviews was the finding that the types of interview prompts we designed successfully elicited student understandings of knowledge and knowing as represented in course assessments. Specifically, these questions gave valuable information about students' thoughts and experiences of several dimensions of epistemology. For example, one student's statement that, "you can memorize all you want, but you're not gonna really know what's going on," suggests that memorizing facts is an unreliable process for achieving the epistemic aim of understanding. However, one challenge in the pilot interviews was the students' repeated use of the word "understanding" without clarifying what understanding meant to them. As a result, the refined protocol directly and indirectly probed student use of that word.</p> <hd id="AN0183867415-13">Multi‐Interview Study</hd> <p>Given that our goal is to understand how individual moments of epistemological messaging give rise to larger, semester‐long epistemological changes, we designed our study around three interviews that occurred at different time points in the semester. Interviews were conducted shortly after the course began, approximately halfway through the term, and near the end of the term (see Figure 1). The first interview was designed to elicit information on what students expected to be valued knowledge products and processes in their organic chemistry class. The second interview intended to probe how students' views on knowledge and knowing were affected by interactions with course assessments. The final interview was meant to prompt reflection on how/whether students' epistemological resources changed and/or were stabilized across a semester of instruction in response to interactions with the course system.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/SED/01mar25/sce21914-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="sce21914-fig-0001.jpg" title="1 A timeline showing approximately when exams and interviews occurred throughout the semester." /> </p> <p></p> <p>Sample interview prompts, and the dimension(s) of epistemology they were meant to probe, are shown in Table 1. Notice that each of the questions is continuous with the kinds of things students might reasonably do on their own in the context of the course, allowing us access to their situated, in‐the‐moment epistemological resources. To enable inferences about how particular dimensions of epistemology (e.g., reliable process) were affected by assessment‐embedded messages, questions pertaining to target dimensions were asked in all three interviews. As an example of this, Table 1 shows questions about reliable processes asked in each interview. Full interview protocols can be found in the Supplemental Information.</p> <p>1 Table Interview questions used to elicit student responses to course‐related epistemological messages. Cell color indicates the interview in which the question was asked.</p> <p> <ephtml> <table><thead valign="bottom"><tr valign="bottom"><th>Interview Question</th><th>Target Epistemological Dimension(s)</th></tr></thead><tbody valign="top"><tr><td>How do you plan to approach studying for assessments in this class?</td><td>Reliable Process</td></tr><tr><td><list list-type="Bullet"><list-item><p>a. What specific kinds of things do you plan to do?</p></list-item><list-item><p>b. (For each thing they list) why do you plan to do "x"?</p></list-item><list-item><p>c. Where did you get the idea to do "x"?</p></list-item></list></td></tr><tr><td>In a course forum post, your instructor said that "narrating how and why each step of a reaction happens would help you 'predict the product'" [show forum post]. Is that consistent with your experience?</td><td>Reliable Process</td></tr><tr><td>Some people in chemistry have talked about the difference between the "what" of chemistry and the "why/how" of chemistry. With this question [from a recent exam] is your instructor asking about the what or the why/how?</td><td>Epistemic Aims and Values</td></tr><tr><td>a. Why do you think your instructor cares about that?</td></tr><tr><td>b. Do you think [whichever they said] is what is important for you?</td></tr><tr><td>Here are three potential answers to the exam question.</td><td>Epistemic Ideals:</td></tr><tr><td><list list-type="Bullet"><list-item><p>a. What ideas do you see each person using in their response?</p></list-item><list-item><p>b. Where do you think they got these ideas?</p></list-item><list-item><p>c. Do you think these responses provide convincing evidence that the person understands what the instructors were intending to assess? Why or why not?</p></list-item></list></td><td>Structure of Knowledge (a)Source of Knowledge (b)Justification (c)</td></tr><tr><td>If you plan to take more organic chemistry courses, how will your experience in this course influence your actions in these future courses?</td><td>Reliable Process</td></tr><tr><td>This course covers a lot of material. How would you classify the knowledge you gained from this class?</td><td>Structure of Knowledge</td></tr><tr><td><list list-type="Bullet"><list-item><p>a. Do pieces of knowledge seem organized/disorganized?</p></list-item><list-item><p>b. Do pieces fit together/connect or seem separate/random? If pieces seem connected, what are they connected to (e.g., things in real life, other things in the course)? Can you give an example?</p></list-item></list></td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note:</emph> Yellow cells contain questions asked in interview 1, green cells contain questions asked in interview 2, and blue cells contain questions asked in interview 3.</p> <p>Interviews varied in length from 23 to 80 min, depending on the student. Interviews were semi‐structured, and follow‐up questions were added to the protocol in the moment where appropriate. Audio recordings were collected during each interview with consent from the participants, and we obtained transcripts from the automated transcription service Temi. Interviewers cleaned up the transcripts, which were then used for analyses. Some filler words (e.g., um, like) were removed for clarity. Descriptive information on each focal student interviewed, and the length of each interview, can be found in Table 2.</p> <p>2 Table Focal students and their interview details.</p> <p> <ephtml> <table><thead valign="bottom"><tr valign="bottom"><th>Student</th><th>Major</th><th>Career Plans</th><th>Interview</th><th>Interview Length</th></tr></thead><tbody valign="top"><tr><td>Ella</td><td>Chemistry</td><td>Pharmacist</td><td>1</td><td>52 min</td></tr><tr><td align="center">2</td><td>1 h 8 min</td></tr><tr><td align="center">3</td><td>59 min</td></tr><tr><td>Lexi</td><td>Neurobiology</td><td>Physician</td><td>1</td><td>36 min</td></tr><tr><td align="center">2</td><td>39 min</td></tr><tr><td align="center">3</td><td>31 min</td></tr><tr><td>Sarah</td><td>Chemistry</td><td>Chemist</td><td>1</td><td>39 min</td></tr><tr><td align="center">2</td><td>36 min</td></tr><tr><td align="center">3</td><td>31 min</td></tr></tbody></table> </ephtml> </p> <hd id="AN0183867415-15">Data Analysis</hd> <p></p> <hd id="AN0183867415-16">Phase 1: Initial Data Reduction</hd> <p>In this analysis, we focus on the epistemological messages experienced by a subset of students who were interviewed at all three time points. To aid in selection of focal students, researchers who interviewed each student crafted analytic memos (Miles and Huberman [<reflink idref="bib49" id="ref137">49</reflink>]) describing how students' understandings of knowing and learning evolved over the course of the semester. These memos were informed by the AIR model of epistemology proposed by Chinn, Rinehart, and Buckland ([<reflink idref="bib10" id="ref138">10</reflink>]) and focused on how (or whether) aims, ideals, and reliable processes changed throughout the semester. The research team read each memo and selected focal students who were representative of several interviewed students and/or who expressed an especially rich array of epistemological resources. We narrowed our sample to three students whose interviews we analyzed further: Lexi, a pre‐med biology major; Ella, a pre‐pharmacy chemistry major; and Sarah, a chemistry major who plans to pursue graduate school in chemistry. All students were sophomores at the time of the study and earned a B or higher in the course.</p> <p>Due to the fact that well over half of the course enrollment is taking organic chemistry to satisfy a medical school prerequisite, we felt it important to have a pre‐med student among our focal students. The epistemological resources expressed by Lexi were similar to views on knowledge and knowing articulated by other pre‐med students interviewed (i.e., typical case sampling; Creswell and Poth [<reflink idref="bib18" id="ref139">18</reflink>]). Sarah was selected in part due to the anticipated overlap between course content and her future career aspirations. Many chemistry courses, including our focal course, are designed with a tacit commitment to preparing future chemists (ACS Committee on Professional Training [<reflink idref="bib1" id="ref140">1</reflink>]). Accordingly, we expected Sarah's epistemological resources would differ in important and interesting ways from those inferable from Lexi's and Ella's dialog. Finally, Ella was selected due to the rich array of epistemological resources she expressed during her interviews (i.e., theory‐based sampling; Creswell and Poth [<reflink idref="bib18" id="ref141">18</reflink>]). We found that Ella's elaborated reflections on knowing and learning added depth to themes expressed by the others.</p> <hd id="AN0183867415-17">Phase 2: Identifying and Describing Epistemological Messaging Segments in Transcripts</hd> <p>After selecting our focal students, we examined each transcript line‐by‐line to identify segments related to students' experience, negotiation, and/or response to course‐related epistemological messages. These segments constituted our unit of analysis for coding. Transcript segments were selected for further analysis if they: (<reflink idref="bib1" id="ref142">1</reflink>) related to students' experiences with knowing and learning in the course and (<reflink idref="bib2" id="ref143">2</reflink>) suggested source(s) where the message originated from. In line with our theoretical framework, we expect students' emerging epistemological understandings of the course will involve complex, dynamic negotiations between their prior epistemological resources, messages from the class itself, their experiences in prior courses, their understanding of the content, etc. As such, we anticipated assessment‐embedded epistemological messages would be entangled with messages from a variety of other sources within and beyond the course. To account for this, our analysis attended to any course‐related message for which we could infer a source, even if that source was outside the course system (e.g., a friend who took the course last year).</p> <p>For each episode, we constructed analytic memos (Miles and Huberman [<reflink idref="bib49" id="ref144">49</reflink>]) to summarize the focus of the epistemological message(s), where in the course system the message(s) originated from, and how students responded to messages they experienced. Message focus was operationalized in terms of the categories proposed by Chinn, Rinehart, and Buckland ([<reflink idref="bib10" id="ref145">10</reflink>]) AIR model of epistemic cognition. We descriptively coded (Saldaña [<reflink idref="bib62" id="ref146">62</reflink>]) dimension(s) (e.g., types of epistemic aims, reliable processes, ideals, etc.) embedded in each messaging episode. To do so, two authors independently coded episodes in each transcript and met to resolve differences in coding. Following this, the entire team met to review and discuss coding across all transcripts. Message source and students' responses to messages they experienced were described in parallel to message focus and discussed during the same research team meetings.</p> <hd id="AN0183867415-18">Phase 3: Identifying Patterns Across Summaries of Epistemological Messaging</hd> <p>To support inferences about how students' epistemological resources evolve through interactions with the course system, we conducted within‐case and cross‐case analyses on patterns of epistemological messages embedded in our data (Merriam [<reflink idref="bib48" id="ref147">48</reflink>]). An individual student's experiences with course‐related messages across all three interviews is considered a case in this study. We first looked across summaries of messages for each student using a constant comparative approach (Glaser and Strauss [<reflink idref="bib29" id="ref148">29</reflink>]) in which we iteratively compared and contrasted patterns to revise, combine, and collapse them into general themes that are distinct from one another. We then conducted cross‐case analyses to determine any potential similarities or differences across cases. Consequential similarities and differences across cases were identified via a constant comparative approach (Glaser and Strauss [<reflink idref="bib29" id="ref149">29</reflink>]). Specifically, we noticed that students' experiences with valued ways of knowing and learning in the course acted to homogenize their epistemic aims and the criteria by which those aims are achieved (i.e., epistemic ideals). By contrast, reliable processes for aim achievement were less uniform across our three focal students. In the sections that follow, we will trace how Ella's, Sarah's, and Lexi's experiences with course‐related epistemological messages led to calcification of a narrow set of epistemic aims and ideals and allowed differences in reliable processes to grow and persist.</p> <hd id="AN0183867415-19">Results and Discussion</hd> <p>Recall that our goal in this analysis is to begin to understand how students move from local interactions around knowledge and learning to larger, course‐wide epistemologies. Importantly, it is <emph>not</emph> our goal to describe all epistemological messages all students experienced, negotiated, and responded to while navigating an organic chemistry course. That is, what follows is not intended to be a comprehensive and complete set of student epistemological resources. Instead, our results present one analysis that explores how assessment‐related messages might synthesize (or not) over the course of a semester to affect students' understandings of knowing and learning in a college STEM course. Work on epistemological resources—both its original conception (Hammer and Elby [<reflink idref="bib32" id="ref150">32</reflink>]) and in descriptions of their dynamics (Lising and Elby [<reflink idref="bib44" id="ref151">44</reflink>]; Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref152">56</reflink>])—typically does not attempt to create an exhaustive list of all resources. Instead, scholars document the resources at play in particular moments of learning for the purpose of examining their role in student learning.</p> <p>Our goal then can be understood as an attempt to develop what Rosenberg, Hammer and Phelan ([<reflink idref="bib56" id="ref153">56</reflink>]) refer to as a "toy model" of meso‐level student epistemological resources. Although toy models are deliberately simplified in their mechanisms, "studying a toy model is an effort to understand and help develop the approach to modeling" (p. 282). Namely, the analysis we describe illustrates a type of analysis that could unfold if researchers worked to attend to aggregation of epistemological messages over the timescale of a course. We do not claim it is the only way to conduct such an analysis, but rather that doing so affords us some understanding of how student epistemological resources emerge from—and potentially stabilize within—course components. Our study design enables us to infer what sources of epistemological messages were consequential to students over time and how messages experienced from these sources shaped meso‐level epistemological resources related to epistemic aims, ideals, and reliable processes. However, the long duration between interviews means that we cannot make detailed claims about micro‐level shifts in students' epistemological resources moment‐to‐moment. This loss of resolution is necessary if we want to consider epistemological dynamics over the course of a semester (or longer).</p> <hd id="AN0183867415-20">Course‐Related Epistemological Messages Acted to Homogenize Students' Epistemic Aims and Idea...</hd> <p>Here, we trace how epistemological messages embedded in and related to course assessments restricted the epistemic aims and ideals available to our focal students. As Sarah, Ella, and Lexi engaged with their organic chemistry course, they experienced a narrow set of epistemic aims as allowed. Criteria for aim achievement were likewise narrow and consistent within and across students' experiences. Our results are organized temporally, with quotes presented from each interview in sequence. This allows us to infer how (or whether) students' sense of valued ways of knowing and learning changed throughout the course.</p> <hd id="AN0183867415-21">Students Expected to Construct Knowledge for the Purpose of Being Evaluated by an Authority</hd> <p>Due at least in part to her prior experiences with similar courses, Ella entered organic chemistry expecting to accumulate knowledge from authorities. Specifically, in her first interview at the start of the semester she said, "I do wanna try, at least a little bit, <bold>to understand whatever my professors are telling me about</bold> and whatever the TAs try to tell us about." Ella's sense for the sort of knowledge "professors are telling me about" was likely informed by her experiences in the course so far—including both lecture dialog and written materials (e.g., problem sets, syllabi). These experiences suggested to her that, "they put an emphasis on, not just memorization, but <bold>more understanding how processes work and why they work</bold>." Thus, before engaging in any assessments, or getting feedback on her assessment responses, Ella anticipated that she would need to construct causal accounts for phenomena to be successful in organic chemistry. Course assessments, she expected, would be spaces to, "prove that you, you know, actually learn something instead of get quote unquote free credit." Assessments are not, despite what the syllabus says, meant to support future learning. When queried about this directly Ella responded, "I really <bold>don't think it's helpful to learn from the exam</bold>. Like, <bold>you're supposed to learn for the exam</bold>."</p> <p>Due, in part, to the relatively stable epistemic norms for school science (Lemke [<reflink idref="bib43" id="ref154">43</reflink>]; Hutchison and Hammer [<reflink idref="bib36" id="ref155">36</reflink>]; Rosenberg, Hammer, and Phelan [<reflink idref="bib56" id="ref156">56</reflink>]), Sarah and Lexi also entered their organic chemistry course expecting to construct authorized knowledge for the purpose of pleasing the instructor. Lexi, like Ella, saw exams as judged performances, saying "when I'm studying for an exam, I don't think of it as a teaching tool. You know what I mean? And then <bold>after the exam, I kind of forget whatever was on it and focus on the new stuff</bold>." If knowledge constructed for an assessment is not expected to be useful in later life or later in the course—as has likely been Lexi's experience in previous science courses—it seems reasonable to clear your mind after an exam and move on to "new stuff." Sarah, at this early point in the course, already had an expectation for the sort of "stuff" that would be recognized as valid by course instructors. She said, "and our lecturer has been <bold>giving us buzzwords that we could answer questions</bold> with, that kind of means that <bold>we're gonna be having explaining questions</bold>." Sarah clearly anticipated that instructor‐provided buzzwords would be useful in answering "explaining questions." It is unclear, however, whether she saw utility in using specialist language beyond pleasing the teacher. As we will see, Sarah's perspective on what "explain" means in organic chemistry was resolved as she interacted further with the course system.</p> <hd id="AN0183867415-22">Epistemic Aims and Ideals Calcified Through Engagement With the Course Ecosystem</hd> <p>Since our mid‐semester interview occurred after administration of several exams and quizzes, we knew that students would have received substantial feedback on the appropriateness of their emerging understandings of relevant epistemic aims and ideals. They may have in turn resolved their sense of what constitutes good knowledge in organic chemistry. Ella's experience with assessment‐related epistemological messages led her to see constructing authorized knowledge (i.e., knowledge that matches an instructor‐authored key) as a higher status aim than constructing causal accounts that made sense to her. This was clearly evident in Ella's response to a question regarding the use of everyday language in chemistry explanations. Students were asked to comment on an instructor post from the course forum that began by saying, "We're not big fans of the use of the word <emph>attack</emph> by chemists to describe molecules. It seems to be anthropomorphizing atoms and molecules in a weirdly violent manner." This post implied everyday language was not valuable in the course, and Ella responded by saying,</p> <p>I think explaining how electrons are donated in a, like <bold>a real scientific explanation</bold> is important [for the class/exam]. But using words like "attack" or "want" is okay if you're <bold>explaining it to yourself</bold> or maybe to other people in a <bold>less academic way</bold>, like group study maybe.</p> <p>Clearly, Ella possessed multiple epistemological resources related to aims and ideals—she mentioned the possibility of using everyday language to construct knowledge that is understandable to a peer group. However, in the focal course, "real scientific explanations" had more worth than "explaining it to yourself or maybe to other people." This perception may be why Ella used extensive domain‐specific vocabulary when thinking aloud as she solved an exam‐like problem.</p> <p>We see further evidence of the suite of epistemological resources available to Ella in her response to a question about working out a problem she did not remember the answer to. Ella stated, "I think I could probably look at this maybe a little bit and, and try and formulate an answer that <bold>makes sense chemically</bold>, but <bold>isn't probably what they're looking for</bold>." This quote demonstrates an important negotiation between two different possible justifications for knowledge: something that makes sense to her versus what she expects the instructors are looking for. On assessments, Ella may have seen <emph>making sense to her</emph> as a less useful epistemic ideal than <emph>closely matches the answer key</emph>.</p> <p>Given that "they" (the instructors) are in charge of justifying knowledge, it is reasonable to wonder how Ella and her colleagues arrived at an understanding of what "they" (the instructors) are looking for. While we lack the resolution to specify all of the course‐related messages students experienced about the characteristics of good knowledge, we can say with confidence that exam answer keys sent consequential messages about the structure of valued knowledge products. As evidence for this claim, consider Ella's evaluation of a sample student response during the mid‐semester interview. This response did not include specialist jargon common on course answer keys but was otherwise canonically correct. While reflecting on whether using this jargon was likely to be important on exams, Ella said:</p> <p>Yeah. Maybe [a response that lacks jargon] is a viable response that would earn them at least most of the credit because <bold>looking at the answer key for my exam, there were multiple ways to get all the points.</bold> And the orbital thing [jargon] was one of them.</p> <p>This remark makes it clear that assessment answer keys act to calibrate Ella's perspective on what structural elements are important for knowledge to be considered valid. Interestingly, when she was evaluating whether the sample response provided evidence of student understanding, Ella remarked:</p> <p>I think <bold>they understand how the um, how the stabilization via electron donating works to some extent</bold>. Because I feel like <bold>knowing that certain electrons are donated into certain orbitals is kind of advanced</bold> and I feel like people might have not really known that and I, <bold>I might have not known it either if I didn't look at that one.</bold> Uh, if I didn't look at that one part of my notes that, that's, that said all that.</p> <p>This suggests that Ella did not see using specialist terms in certain ways (i.e., naming donor and acceptor orbitals) as necessary in construction of a plausible explanation. However, since the answer key privileged use of certain terms, she may have experienced the need to construct buzzword‐laden phrases on assessments.</p> <p>Sarah and Lexi compiled their understanding of worthwhile epistemic aims in similar ways to Ella. Both experienced aligning with authority expectations as the main way knowledge was justified in class. For example, when thinking through how she would address an exam‐like prompt, Sarah noted that she "would probably add a sentence or two about how because it's polar and protic and can donate electrons, this ethanol is gonna stabilize this better and like reduce the overall energy." She would connect ideas in this way because the resulting knowledge product would contain "the key word that they're looking for." How (or whether) it was important for her explanation to make sense to her was not mentioned.</p> <p>Lexi also experienced that it was important to include keywords in authorized knowledge products. Specifically, she stated:</p> <p>I remember there was one, er, cause in like a certain type of problem, like it would ask about stability, and <bold>they really wanted you to say "hyperconjugation" cause like in all the keys it was like, "hyperconjugation,"</bold> and I kept forgetting to say that. < Laugh> I would just be like, "It's more stable." And then on the exam, I literally saw a question that was asking about it. So I just wrote "hyperconjugation" like the first thing, like when I did it, so I would know to say it. And then I got that question right for the answer. Cause <bold>they wanted that word, I think.</bold></p> <p>Course answer keys, which commonly specified points associated with certain words/phrases (see Figure 2), communicated to students that good knowledge had to include certain structural features—keywords, in the case of written explanations or arguments. When engaging with high‐stakes assessments, a central goal was to construct knowledge that included all of the words/phrases/drawings that instructors decided were necessary. In Lexi's example, it was useful to plug the science word "hyperconjugation" into a sentence when asked about the stability of certain species in a chemical system.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/SED/01mar25/sce21914-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="sce21914-fig-0002.jpg" title="2 An example of a practice exam answer key showing how points are assigned to specific words/phrases/ideas." /> </p> <p></p> <p>Across all of our focal students, engagement with assessment‐related epistemological messages led to a fairly consistent perspective on valued epistemic aims and ideals by our second interview. Specifically, students all experienced the aim of engaging with assessments as constructing knowledge that aligned with authority expectations. These expectations stipulated that good knowledge has a very specific structure, which can be extrapolated from assessment‐related materials (e.g., answer keys). We will turn our attention to how students constructed authorized knowledge in Section 5.2, which is dedicated to compilation of reliable processes.</p> <hd id="AN0183867415-24">Epistemic Aims and Ideals Were Consistent between Interviews 2 and 3</hd> <p>Students' perspective on worthwhile epistemic aims and ideals in organic chemistry was remarkably consistent between interviews 2 and 3. At the end of the semester, Ella continued to look to the answer key to get a read on the precise structure of good knowledge in class. This sometimes created moments where she experienced dissonance between what one authoritative source (e.g., the lecturer) was telling her and what was written on an answer key. For example:</p> <p>On one of the problem sets, I noticed that, or, I answered a question that was like, I made it the most substituted alkene because it made more sense, and <bold>that's what we'd been taught in lecture</bold>. But then <bold>going through the key, I found that the actual answer wasn't the most substituted one</bold>.</p> <p>Here, Ella attempted to apply a heuristic learned in lecture to a new problem and found, after consulting the key, that what she wrote was not consistent with the authorized answer. This experience reinforced to Ella that chemistry is made up of rules and exceptions—a perspective that she recalled one of her high school teachers expressing by saying, "every general rule in chemistry, there's always some exception to the rule and every single, every single thing has some exception."</p> <p>Lexi and Sarah also experienced the message that knowledge product validity is determined by the extent to which these products map onto instructor‐authored keys. Such a message, if taken up, restricts students' options for justifying knowledge. When all that matters is how/whether what you wrote looks like what your instructor wrote, there is little room for negotiation. It is irrelevant, for example, whether your response makes sense to you, is convincing to a peer community, is consistent with observation, etc. For Lexi, this narrow path toward authorized knowledge sometimes led to frustration. She said, "Sometimes when you draw the products, I drew different wedge and dashes in different spots. And <bold>it was sometimes confusing to know if my answer was right too, if it didn't match the keys</bold>." Sarah experienced a similar frustration, saying,</p> <p>One problem that I had was <bold>I would know what happens and I would know why</bold>, but <bold>I would leave out a keyword</bold> or like one out of three factors in the short answer <bold>and lose a lot of points from that</bold>, which is a little bit frustrating.</p> <p>These experiences suggested that other ways of justifying knowledge (e.g., considering whether a claim makes sense to you) were only allowed insofar as what you wrote mapped onto some predetermined ideal answer you had no say in constructing.</p> <p>When taken together, Ella, Sarah, and Lexi's experiences with assessment‐related epistemological messages tell a very similar story. All three students entered class expecting to construct knowledge for the purpose of being evaluated by the instructor for correctness. This is unsurprising as "[Due to many possible, interrelated, contextual factors]... science classrooms by and large simply have not been hospitable environments for teaching that ventures beyond the traditional emphasis on mastering science content knowledge" (Rudolph [<reflink idref="bib57" id="ref157">57</reflink>], pp. 1072–1073). We would therefore expect many of these students' prior experiences in science classes to center on demonstrating to teachers that they could construct canonical answers.</p> <p>Ella, Sarah, and Lexi's perspectives on what good knowledge looks like in organic chemistry, as well as how one should go about justifying whether their knowledge product is good, resolved over the course of the semester through engaging with various aspects of the course system. We found that instructor‐authored answer keys were a particularly consequential source of messages about epistemic ideals, specifically the structure of knowledge that would count. Relatedly, knowledge was justified via alignment with these instructor‐authored keys—other ways of justifying knowledge were seen as less central to success in the focal course.</p> <hd id="AN0183867415-25">Students Compiled Their Reliable Processes for Authorized Knowledge Differently</hd> <p>In the preceding section, we saw that experiences with course‐related messages about valued epistemic aims and ideals led all three students to a similar, well‐resolved picture of the knowledge that counted. This suggests that consequential messages from a course can act to homogenize ways of knowing and learning students see as allowed. However, since students' emerging epistemic understandings of a course likely involve dynamic negotiations between (different) experiences in prior courses, course‐embedded messages, messages from outside school etc., we anticipated that there would be some notable differences in epistemological resources used across time and across students. In our data set, these differences were most apparent in the ways students sought to achieve their epistemic aims (i.e., reliable processes). Below, we will consider how students experienced epistemological messages related to reliable processes at each interview timepoint. Doing so will let us surface distinctions between students at the beginning, middle, and end of the course.</p> <hd id="AN0183867415-26">Processes Students Expected Would Be Reliable</hd> <p>From the first day of class, students had the potential to experience epistemological messages about (un)reliable processes for achieving epistemic aims of value in the course. For example, the very first lecture slide of the semester had a bullet point that read, "Do NOT rely on memorization" (emphasis theirs). The course syllabus potentially communicated a similar message when it read, "We will always try to separate those that are trying to memorize patterns or use mnemonic devices from those who understand the content". This suggests a hierarchy of both aims (understanding vs. accumulated facts) and (un)reliable processes (memorization being unreliable for achieving understanding).</p> <p>Students interpreted these messages in light of their prior experiences with similar courses. For example, when considering how she was likely to study for organic chemistry, Lexi noted, "I don't really do, like the vocab, I don't really memorize vocab for chem." This suggests that memorizing particular words/phrases had not helped Lexi achieve valued aims in prior chemistry courses. That said, Lexi recognized that there are situations in which memorization might be a reliable process. She indicated this by saying, "depending on what it is, I'll either memorize it or try to understand it really well so I don't have to memorize it." The notion that memorization is not a reliable process for achieving understanding was also echoed by Sarah when she said:</p> <p>When I learned hybridization in high school, it was completely over my head, did not get it at all. I memorized like if it has this many connections, it's, you know, sp<sups>3</sups>, sp<sups>2</sups>, whatever, but I didn't really understand how that worked. ... but then a couple, like last week we did the unit on it and I went into office hours and I rewatched more videos and I took double the amount of notes I usually do in class. And I actually understood how the bond, like the orbitals themselves overlapped to create this and why it works and all the exceptions to the rules. And I think that makes it much more of a full understanding as opposed to memorizing.</p> <p>Here, Sarah explicitly contrasted accumulating memorized facts/skills (which was useful in high school) and constructing a causal account for how orbitals overlap to create bonding interactions (which she expected to be a high value aim in the present course).</p> <p>While memorization was not expected to be helpful (due to some combination of past experience and course‐embedded messages), connecting past and present knowledge was forecast to be useful by interview participants. Ella noted, "it makes my understanding easier if I connect to something I already understand." Sarah was more specific about the sort of prior knowledge she would seek to recruit to support new understanding, saying, "ochem is built on everything in [general] chem ... every unit that we've done so far has exactly correlated to stuff that we're learning or going to learn." Sarah also provided some insight into how she built functional connections between pieces of knowledge in prior courses when she said,</p> <p>When I write ... flow charts like that, <bold>I try to make them ... general as possible so that I can apply it to as much things as possible</bold>. And then I remember the basic outline, I can apply that to a specific problem that I'm trying to work on.</p> <p>Sarah tried to abstract connections between knowledge pieces that were broadly useful (vs. only useful in a very specific context). She then used these connection approaches across many in‐class tasks. Presumably, this had been a productive approach in other classes that seem similar to the present class.</p> <p>Looking across Ella, Lexi, and Sarah's expectations for reliable processes useful in achieving valued aims, we see many similarities. All anticipate that memorization will be less‐than‐useful, and all made some reference to the utility of connecting past and present knowledge. However, this first interview occurred before substantial engagement with the focal course—students had not yet had many opportunities to act on course‐embedded messages about reliable processes and get feedback on their understandings of these messages (e.g., via assessment feedback).</p> <hd id="AN0183867415-27">Reliable Processes Resolved Through Experiences With the Course Ecosystem</hd> <p>Before our second interview, students had engaged with two quizzes and exams, and gotten feedback on the knowledge they constructed for these assessments. As we noted previously, this feedback, and epistemological messages embedded in other exam‐related materials (e.g., answer keys to practice exams) communicated that good knowledge in organic chemistry has certain authority‐mandated structural features (e.g., canonical orbital nomenclature, lone pairs drawn explicitly on atoms active in a given step of an electron‐pushing mechanism). How, one might wonder, do students go about preparing to construct such good knowledge on assessments? Lexi provided a cogent answer:</p> <p>I mean after I did a lot of problems, <bold>I kind of picked up on what they wanted us to say, and then I kind of just memorized that</bold> and was like, I just have to say this for the exam. Um, but when I was first learning it and trying to understand everything, it was hard cause I would get points off for not saying the right words even though I thought I was understanding the concepts. But then I <bold>feel like if you practice a lot then you kind of pick up on it.</bold></p> <p>Despite early expectations that memorization was not likely to be useful in organic chemistry, by midway through the semester, Lexi found that she <emph>had</emph> to memorize buzzwords to succeed on assessments. If all that matters is that you transcribe the words likely to appear on an answer key down onto the page, then it is irrelevant whether you "understand everything" in a way that is sensible to you—you need to put the "right words" in the right place or you will not pass the course, which will in turn prevent you from pursuing your desired career (a physician, in Lexi's case).</p> <p>Sarah and Ella, like Lexi, both found some utility in memorizing and recalling knowledge. For example, when reflecting on what the instructor was trying to get them to understand with an assessment item, Sarah said: "I think that the main thing would be remembering what polar, protic, and electron donating is because without that you don't have any of the reason why solvents would be different." This suggests that an acceptable reason needed to contain certain words and that it was fruitful to memorize and then recall these words to construct good knowledge on an assessment. Similarly, when reflecting on an instructor's urging to not memorize lists of reactions, Ella responded:</p> <p>But <bold>I kind of just didn't really study why and more studied the fact that it happens. So basically memorization.</bold> And they tell, they always tell you not to do memorization, but sometimes it's just easier I think ... Yeah. Like this last sentence by, I assume [instructor], says that "it's not a very productive strategy to make a list of each reaction's characteristics in terms of regiochem and stereochem," but <bold>I guess that depends because I basically did that. I know it's very helpful.</bold></p> <p>Here, we see Ella's experience in the course standing in contrast to an explicit epistemological message from the instructor about reliable processes. Ella was able to construct authorized knowledge without narrating how and why phenomena happen—instead, she memorized lists of reactions and a few choice characteristics of each reaction.</p> <p>Although all three students saw memorization as a reliable process, during this interview Sarah and Ella began to diverge from Lexi with regard to the value they placed on reusable approaches to connecting knowledge. From the first interview to the second, Sarah maintained that it is worthwhile to figure out broadly useful ways of connecting pieces of knowledge. In the second interview, she noted, "they're (the instructors) trying to teach us how all these little pieces fit together and how you can do different stuff with different pieces because of different things." This suggests that one can construct good knowledge across assessment contexts by finding reusable ways of fitting knowledge pieces together. Ella was more specific about what connection approaches she expected to be reusable across assessments when she said:</p> <p> <bold>We were expected</bold> [on this assessment] <bold>to explain how those stabilizing factors work</bold> because I guess [on earlier assessments] we were just never asked, or it was good enough to say that the solvent stabilizes that [molecule]. But <bold>now we have to explain how exactly that stabilization happens and a lot of it is how electrons are donated or moved from one orbital to another</bold>, which is actually something I used a lot on the exam.</p> <p>Here, Ella discussed that students were expected to connect structure and energy ideas to explain the outcomes of reactions by the time they take the second exam, and these connections go beyond the claims (e.g., "the solvent stabilizes the molecule") that were expected on earlier course assessments. This suggests that Ella had experienced connecting structural and energetic ideas as a reliable process to constructing authorized knowledge. Indeed, when evaluating sample student responses to an exam prompt or thinking aloud as she approached a class‐embedded task, Ella regularly connected the structure of species in the reaction system to their relative energies. Lexi, by contrast, did not deviate substantially from memorizing words/phrases/drawings from assessment‐related materials and recalling them. Connecting structure and energy ideas across contexts did not surface as a reliable process for her in this interview.</p> <hd id="AN0183867415-28">Distinctions Between Students' Reliable Processes Stabilized</hd> <p>In our third and final interview, which took place around the time the course final exam was to be administered, we see calcification of the reliable process differences hinted at during interview 2. Specifically, Sarah and Ella both articulated the utility in connecting specific phenomena to more fundamental ideas. Ella articulated this by saying, "I think the most helpful thing you could learn in [organic I] is understanding that kind of, like, fundamental knowledge and using it to kind of explain everything else pretty much in the rest of the class." The utility of this approach to connecting knowledge was driven by how Ella experienced the expectations on course assessments. Through engaging in three quizzes and three exams, Ella had learned that "when you had to explain why this one's gonna hap‐, this reac‐, or this product is major and this one's minor, et cetera. ... you had to use this concept that you learned in the first chapter to explain it." This contrasted with Ella's experience in an analytical chemistry course where she "didn't really need to know anything at all from like a previous module, in like, as in, like pre exam, like for exam two, I didn't need to know anything from exam one at all." One might predict that Ella would compile her reliable processes for this analytical course in a manner qualitatively distinct from how she approached achieving valued epistemic aims in organic chemistry.</p> <p>Sarah's interview dialog also suggests that she valued reusable approaches to connecting knowledge pieces throughout her time in organic chemistry. She noted that the most important thing she learned in the course was "the reasoning behind all the mechanisms" which was useful because it is "so much more applicable and you can take that [reasoning] and apply to so many more different things than just [discrete factoids]." Assessments and instructor rhetoric suggested which knowledge construction strategies were likely to be reusable. Certain tasks in which one is asked to propose an approach to synthesizing a molecule required that you "put all of the pieces that we learned together." The ways of putting knowledge together that lead to success on these problems were seen as worthwhile because the instructor "was talking about how [these types of questions are] gonna be on every test in [the next chemistry course]."</p> <p>Lexi's final interview, by contrast, made little mention of functionally useful connections that enabled construction of authorized knowledge across contexts. Her experience attempting to succeed in organic chemistry can be well encapsulated by the following quote:</p> <p>Cause that was when <bold>we all of a sudden started learning so many mechanisms so fast.</bold>Then I was like, what is hap—like I literally was like, didn't know how to study for it 'cause it was so many. And then someone in my class was like, <bold>there's literally no way to study for the third exam 'cause it's too much info</bold>, like you can't know everything. And I was like, that's kind of true, I feel like. But then that's when <bold>I started doing the note cards 'cause it helped me learn it faster.</bold></p> <p>From Lexi's perspective, accumulating disaggregated facts/skills/heuristics/drawings was the purpose of the focal course. Once she experienced acceleration in the pace of new content she was expected to know, she turned to memorizing flashcards as a way to cope. Others in her class reinforced Lexi's perception that there is cause to shortcut "knowing everything." She experienced this reliable process as useful in her pursuit of canonical facts/skills/drawings on assessments, saying:</p> <p>I would write, like if I didn't remember something about a mechanism, I write it on the note card, and then either draw the mechanism on the back or write what I forgot. And [my friend] was like, they don't want you to memorize things like that. < Laugh> She was like, that's what they tell you not to do. But I feel like that actually helped me a lot. 'Cause like when I made it in a note card, I remembered it. So that was just a weird study thing I did. But it worked for me so I was like, whatever.</p> <p>Recall that Lexi did not expect memorizing vocabulary to be useful in this course. What we see here, then, is a notable shift in reliable processes for achieving valued epistemic aims that was induced, in part, by course‐related messages. Lexi could not see how to "understand everything" apart from memorizing true beliefs—an experience shared by at least a few of her peers. We should stress here that the reliable processes Lexi compiled for this course seemed sensible to adopt, given the course's emphasis on very specific knowledge products justified by aligning with authority expectations. She is not wrong for having adopted these processes.</p> <hd id="AN0183867415-29">Putting Together a "Toy Model" for Students' Experiences With Course‐Embedded Epistemological...</hd> <p>Our study design enables us to begin to piece together a simple model of how assessment‐related epistemological messages may synthesize to affect students' meso‐level understandings of knowing and learning in a college STEM course. We see, unsurprisingly, that students entered the focal course expecting to "do the lesson"—they anticipated that knowledge would be justified by aligning with authority expectations, class would be the sole useful source of knowledge, etc. For the most part, these expectations were reinforced by the structure of their organic chemistry course. Specifically, answer keys and exam feedback communicated to students that allowable knowledge products have very specific structures that are determined by authority figures. Since allowed knowledge products often had similar features (e.g., connection between molecular structure and relative energy), some students saw utility in connecting phenomena to a small set of fundamental ideas. However, students did not experience the agency to decide how and why connections between ideas should be made—they were required to connect what instructors said they had to connect. Students expressed fairly consistent views on relevant epistemic aims and ideals within and across interviews 2 and 3. This suggests that epistemological resources related to these dimensions of epistemology stabilized before our second interview and remained useful to students through the end of the course. Our findings may indicate that messages embedded across practice assessments, answer keys, course exams, and exam feedback all contributed to this stabilization.</p> <p>It is likely that many readers of this <emph>Journal</emph> are unsurprised that students experienced a college chemistry class as an opportunity to "do the lesson" (Lemke [<reflink idref="bib43" id="ref158">43</reflink>]). This is a well‐documented and common approach to school science (Berland and Hammer [<reflink idref="bib4" id="ref159">4</reflink>]; Gouvea and Passmore [<reflink idref="bib30" id="ref160">30</reflink>]; Kuhn and Pease [<reflink idref="bib42" id="ref161">42</reflink>]; Manz [<reflink idref="bib45" id="ref162">45</reflink>]; McNeill et al. [<reflink idref="bib47" id="ref163">47</reflink>]). What we contribute in this study is a theoretical mechanism by which epistemological resources associated with "doing the lesson" are stabilized in a given course—via aggregation of epistemological messages from a variety of sources (including assessments!). We then begin to explore this model via analysis of dialog from three temporally spaced interviews that are contextualized by assessment‐embedded epistemological messages. This analytic approach enabled us to pinpoint sources of epistemological messages that mattered to interviewed students (e.g., practice exam answer keys) and to describe how students experienced, negotiated, and responded to messages from these sources. Our findings make it clear that our bets paid off; high‐stakes assessments did indeed send consequential epistemological messages that acted to shape students' approaches to knowing and learning. We expect others could undertake a similarly structured analysis to explore the epistemological messaging landscape of virtually any college STEM course. Such analyses open up possibilities for what could be considered markers of success in course transformation projects. For example, if a course transformation had "open space for student agency" as a central goal, a study similar to that described here could be carried out to explore the extent to which students saw themselves as authorities on useful knowledge across different assessment contexts.</p> <hd id="AN0183867415-30">Conclusions and Implications</hd> <p>To better understand how students' epistemological resources are influenced by course structure and materials, particularly assessments, we conducted interviews in which students reflected on their expectations for and experiences with knowledge and knowing in an introductory organic chemistry course. We found that most students entered the course expecting to construct knowledge on assessments for the purpose of being evaluated by the instructor. Students' past experiences in STEM courses had taught them that success in such courses requires learning authorized content. Other goals which appear in reform documents (e.g., making sense of everyday experiences, negotiating how and why to construct knowledge as a class) were not voiced by the students we spoke with. In later interviews, it became evident that students' epistemological resources, particularly their views regarding worthwhile epistemic aims and ideals, were significantly influenced by assessment answer keys and feedback they received on their assessment responses. Students experienced a need to include certain terms in their responses, and in some cases, they adjusted their study strategies to better anticipate which terms were deemed important by the instructors. Students who we interviewed did not experience the course as a place where they could decide the characteristics of good knowledge. As such, how or whether specific terminology was needed was determined by the structure of answer keys and how those keys were used.</p> <p>It is important to emphasize that our work here should <emph>not</emph> be read as suggesting that students failed to adopt the right epistemological resources. Indeed, all three students whose dialog we report here earned a B or higher in the course—they successfully did the work that was asked of them. The resources model of epistemic cognition that informs this study assumes there are no universally right views on knowledge and knowing. As a result, we reject the notion that some epistemological resources are advanced while others are not. Instead, all epistemological resources have utility and productivity in some context. Students adopt epistemological resources they (tacitly) anticipate will be useful in navigating the context within which they find themselves. Lexi, Ella, and Sarah's approaches to constructing and refining knowledge are reasonable in light of the emphasis their course placed on aligning with authority expectations.</p> <p>The challenge, then, is to envision college STEM learning environments that communicate the value of other, potentially more useful, epistemological resources. This will require that we attend to and intentionally work to foster epistemological learning that aligns with our goals. To do so, we must figure out how course components communicate messages about valued knowledge products and processes to students, and how students experience, negotiate, and respond to these messages. This understanding will let us tune course systems so that they signal the importance of epistemological resources better aligned with our stated values. The analytic and theoretical machinery we have started to build enables these sorts of explorations. In particular, we found utility in interviewing students at several time points throughout the course and contextualizing interview protocols using course‐embedded epistemological messages we hypothesized might matter to students.</p> <p>As an exploratory study, this work was limited in several important ways. In focusing solely on assessment‐related messages, we ignored other potentially consequential sources of epistemological messages (e.g., class dialog). It would be worthwhile for scholars in the field to create more nuanced models of course epistemological messaging which attend to multiple message sources (e.g., class dialog, peer interactions). Such models may support inferences about how students experience and negotiate potentially conflicting messages they experience from a variety of sources in the course system. Additionally, the students who we interviewed were a fairly homogeneous group—all were women who were STEM majors and who earned a C or better in the class. Future work should attend to the epistemological resources of a wider range of enrolled students (e.g., students who did not succeed in the course).</p> <p>While it is satisfying and useful for researchers to build theoretical and analytical tools that enable the modeling of epistemological messaging in STEM courses, this will not in and of itself lead to the changes we hope for. For students' experiences with course‐embedded epistemological messages to drive widespread, practical course changes, we must build the capacity of practitioner communities to engage in work similar to what we report here. This will likely require both creation of new tools (e.g., prompts to elicit students' experiences with course‐embedded messages) and communities focused on surfacing and responding to students' experiences with knowing and learning in STEM classes. It is only by explicitly attending in our classes to multiple dimensions of epistemological resources that we can support students in adopting an approach to science that makes it useful for their everyday lives.</p> <hd id="AN0183867415-31">Acknowledgments</hd> <p>We would like to acknowledge all of the students who shared their experiences with course‐embedded epistemological messages. We also gratefully acknowledge funding from the National Science Foundation for CES (DRL 2003680) and NCG (DUE 2225025). Any opinions, findings, and conclusions or recommendations expressed in this submission are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Support was also provided by the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin–Madison with funding from the Wisconsin Alumni Research Foundation.</p> <hd id="AN0183867415-32">Ethics Statement</hd> <p>The study presented here was approved by the University of Wisconsin‐Madison IRB, ID 2022‐0687.</p> <hd id="AN0183867415-33">Conflicts of Interest</hd> <p>The authors declare no conflicts of interest.</p> <hd id="AN0183867415-34">Data Availability Statement</hd> <p>The data that support the findings of this study are available from the corresponding author upon reasonable request.</p> <p>GRAPH: Supporting information.</p> <ref id="AN0183867415-35"> <title> References </title> <blist> <bibl id="bib1" idref="ref35" type="bt">1</bibl> <bibtext> ACS Committee on Professional Training. 2023. ACS Guidelines for Bachelor's Degree Programs. Washington, D.C.: American Chemical Society. https://<ulink href="http://www.acs.org/education/policies/acs-approval-program/guidelines.html">www.acs.org/education/policies/acs-approval-program/guidelines.html</ulink>.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref33" type="bt">2</bibl> <bibtext> Adams, W. K., C. E. Wieman, K. K. Perkins, and J. Barbera. 2008. " Modifying and Validating the Colorado Learning Attitudes About Science Survey for use in Chemistry." Journal of Chemical Education 85, no. 10 : 1435 – 1439. https://doi.org/10.1021/ed085p1435.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref2" type="bt">3</bibl> <bibtext> Bauerle, C., A. DePass, D. Lynn, et al. 2011. Vision and Change in Undergraduate Biology Education: A Call to Action. Washington, D.C.: American Association for the Advancement of Science.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref13" type="bt">4</bibl> <bibtext> Berland, L. K., and D. Hammer. 2012. " Framing for Scientific Argumentation." Journal of Research in Science Teaching 49, no. 1 : 68 – 94. https://doi.org/10.1002/tea.20446.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref21" type="bt">5</bibl> <bibtext> Berland, L. K., and B. J. Reiser. 2011. " Classroom Communities' Adaptations of the Practice of Scientific Argumentation." Science Education 95, no. 2 : 191 – 216. https://doi.org/10.1002/sce.20420.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref61" type="bt">6</bibl> <bibtext> Bing, T. J., and E. F. Redish. 2009. " Analyzing Problem Solving Using Math in Physics: Epistemological Framing Via Warrants." Physical Review Special Topics ‐ Physics Education Research 5, no. 2 : 020108. https://doi.org/10.1103/PhysRevSTPER.5.020108.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref56" type="bt">7</bibl> <bibtext> Bowen, R. S., A. A. Flaherty, and M. M. Cooper. 2022. " Investigating Student Perceptions of Transformational Intent and Classroom Culture in Organic Chemistry Courses." Chemistry Education Research and Practice 23, no. 3 : 560 – 581. https://doi.org/10.1039/D2RP00010E.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref48" type="bt">8</bibl> <bibtext> Bransford, J. D., and D. L. Schwartz. 1999. " Chapter 3: Rethinking Transfer: A Simple Proposal With Multiple Implications." Review of Research in Education 24, no. 1 : 61 – 100.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref129" type="bt">9</bibl> <bibtext> Charters, E. 2003. " The Use of Think‐Aloud Methods in Qualitative Research an Introduction to Think‐Aloud Methods." Brock Education Journal 12, no. 2. https://doi.org/10.26522/brocked.v12i2.38.</bibtext> </blist> <blist> <bibtext> Chinn, C. A., R. W. Rinehart, and L. A. Buckland. 2014. " Epistemic Cognition and Evaluating Information: Applying the AIR Model of Epistemic Cognition." In Processing Inaccurate Information: Theoretical and Applied Perspectives From Cognitive Science and the Educational Sciences, edited by D. N. Rapp and J. L. G. Braasch, 425 – 453. The MIT Press.</bibtext> </blist> <blist> <bibtext> Christodoulou, A., and J. Osborne. 2014. " The Science Classroom as a Site of Epistemic Talk: A Case Study of a Teacher's Attempts to Teach Science Based on Argument: The Science Classroom as a Site oF Epistemic Talk." Journal of Research in Science Teaching 51, no. 10 : 1275 – 1300.</bibtext> </blist> <blist> <bibtext> Cooper, M., and M. Klymkowsky. 2013. " Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform." Journal of Chemical Education 90, no. 9 : 1116 – 1122. https://doi.org/10.1021/ed300456y.</bibtext> </blist> <blist> <bibtext> Cooper, M. M., M. D. Caballero, D. Ebert‐May, et al. 2015. " Challenge Faculty to Transform Stem Learning." Science 350, no. 6258 : 281 – 282. https://doi.org/10.1126/science.aab0933.</bibtext> </blist> <blist> <bibtext> Cooper, M. M., L. A. Posey, and S. M. Underwood. 2017. " Core Ideas and Topics: Building Up or Drilling Down? " Journal of Chemical Education 94, no. 5 : 541 – 548. https://doi.org/10.1021/acs.jchemed.6b00900.</bibtext> </blist> <blist> <bibtext> Cooper, M. M., R. L. Stowe, O. M. Crandell, and M. W. Klymkowsky. 2019. " Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum." Journal of Chemical Education 96, no. 9 : 1858 – 1872. https://doi.org/10.1021/acs.jchemed.9b00401.</bibtext> </blist> <blist> <bibtext> Cooper, M. M., S. M. Underwood, C. Z. Hilley, and M. W. Klymkowsky. 2012. " Development and Assessment of a Molecular Structure and Properties Learning Progression." Journal of Chemical Education 89, no. 11 : 1351 – 1357. https://doi.org/10.1021/ed300083a.</bibtext> </blist> <blist> <bibtext> Crandell, O. M., M. A. Lockhart, and M. M. Cooper. 2020. " Arrows on the Page Are Not a Good Gauge: Evidence for the Importance of Causal Mechanistic Explanations About Nucleophilic Substitution in Organic Chemistry." Journal of Chemical Education 97, no. 2 : 313 – 327. https://doi.org/10.1021/acs.jchemed.9b00815.</bibtext> </blist> <blist> <bibtext> Creswell, J., and C. Poth. 2017. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. London : Sage.</bibtext> </blist> <blist> <bibtext> Crooks, T. J. 1988. " The Impact of Classroom Evaluation Practices on Students." Review of Educational Research 58, no. 4 : 438 – 481. https://doi.org/10.3102/00346543058004438.</bibtext> </blist> <blist> <bibtext> DeGlopper, K. S., C. E. Schwarz, N. J. Ellias, and R. L. Stowe. 2022. " Impact of Assessment Emphasis on Organic Chemistry Students' Explanations for an Alkene Addition Reaction." Journal of Chemical Education 99, no. 3 : 1368 – 1382.</bibtext> </blist> <blist> <bibtext> Dempsey, N. P. 2010. " Stimulated Recall Interviews in Ethnography." Qualitative sociology 33, no. 3 : 349 – 367.</bibtext> </blist> <blist> <bibtext> DiSessa, A. A. 1993. " Toward an Epistemology of Physics." Cognition and Instruction 10, no. 2–3 : 105 – 225.</bibtext> </blist> <blist> <bibtext> Elby, A., and D. Hammer. 2001. " On the Substance of a Sophisticated Epistemology." Science Education 85, no. 5 : 554 – 567. https://doi.org/10.1002/sce.1023.</bibtext> </blist> <blist> <bibtext> Elby, A., C. Macrander, and D. Hammer. 2016. " Epistemic Cognition in Science." In Handbook of Epistemic Cognition, edited by J. A. Greene, W. A. Sandoval, and I. Bråten, 110 – 127). Routledge.</bibtext> </blist> <blist> <bibtext> Engle, R. A. 2006. " Framing Interactions to Foster Generative Learning: A Situative Explanation of Transfer in a Community of Learners Classroom." Journal of the Learning Sciences 15, no. 4 : 451 – 498.</bibtext> </blist> <blist> <bibtext> Entwistle, N. J. 1991. " Approaches to Learning and Perceptions of the Learning Environment." Higher Education 22, no. 3 : 201 – 204.</bibtext> </blist> <blist> <bibtext> Flaherty, A. A. 2020. " Investigating Perceptions of the Structure and Development of Scientific Knowledge in the Context of a Transformed Organic Chemistry Lecture Course." Chemistry Education Research and Practice 21, no. 2 : 570 – 581. https://doi.org/10.1039/C9RP00201D.</bibtext> </blist> <blist> <bibtext> Ginsburg, H. 1997. Entering the Child's Mind: The Clinical Interview in Psychological Research and Practice. New York, NY, USA : Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Glaser, B. G., and A. L. Strauss. 1967. The Discovery of Grounded Theory. Strategies for Qualitative Research. Chicago : Aldine.</bibtext> </blist> <blist> <bibtext> Gouvea, J., and C. Passmore. 2017. " Models Of' Versus 'Models For': Toward an Agent‐Based Conception of Modeling in the Science Classroom." Science & Education 26, no. 1–2 : 49 – 63. https://doi.org/10.1007/s11191-017-9884-4.</bibtext> </blist> <blist> <bibtext> Greene, J. A., Sandoval, W. A., and Bråten, I., ed. 2016. Handbook of Epistemic Cognition. New York, NY, USA : Routledge.</bibtext> </blist> <blist> <bibtext> Hammer, D., and A. Elby. 2002. " On the Form of a Personal Epistemology." In Personal Epistemology: The Psychology of Beliefs About Knowledge and Knowing, edited by B. K. Hofer and P. R. Pintrich, 169190. Mahway, New Jersey : Lawrence Erlbaum Associates, Inc.</bibtext> </blist> <blist> <bibtext> Hammer, D., A. Elby, R. E. Scherr, and E. F. Redish. 2005. " Resources, Framing, and Transfer." In Transfer of Learning from a Modern Multidisciplinary Perspective, edited by J. P. Mestre, 89 – 119. IAP.</bibtext> </blist> <blist> <bibtext> Hofer, B. 2016. " Epistemic Cognition as a Psychological Construct." In Handbook of Epistemic Cognition, edited by J. A. Greene, W. A. Sandoval, and I. Bråten, 19 – 38). Routledge.</bibtext> </blist> <blist> <bibtext> Hofer, B. K., and P. R. Pintrich. 1997. " The Development of Epistemological Theories: Beliefs About Knowledge and Knowing and Their Relation to Learning." Review of Educational Research 67, no. 1 : 88 – 140.</bibtext> </blist> <blist> <bibtext> Hutchison, P., and D. Hammer. 2010. " Attending to Student Epistemological Framing in a Science Classroom." Science Education 94, no. 3 : 506 – 524. https://doi.org/10.1002/sce.20373.</bibtext> </blist> <blist> <bibtext> Irving, P. W., M. S. Martinuk, and E. C. Sayre. 2013. " Transitions in Students' Epistemic Framing Along Two Axes." Physical Review Special Topics‐Physics Education Research 9, no. 1 : 010111.</bibtext> </blist> <blist> <bibtext> Jiménez‐Aleixandre, M. P., A. Bugallo Rodríguez, and R. A. Duschl. 2000. " 'Doing the Lesson' or 'Doing Science': Argument in High School Genetics." Science Education 84 : 757 – 792. https://doi.org/10.1002/1098-237X(200011)84:6<757::AID-SCE5>3.0.CO;2-F.</bibtext> </blist> <blist> <bibtext> Ke, L., and C. V. Schwarz. 2021. " Supporting Students' Meaningful Engagement in Scientific Modeling Through Epistemological Messages: A Case Study of Contrasting Teaching Approaches." Journal of Research in Science Teaching 58, no. 3 : 335 – 365.</bibtext> </blist> <blist> <bibtext> King, P. M., and K. S. Kitchener. 2004. " Reflective Judgment: Theory and Research on the Development of Epistemic Assumptions Through Adulthood." Educational psychologist 39, no. 1 : 5 – 18.</bibtext> </blist> <blist> <bibtext> Kozminski, J., H. Lewandowski, and N. Beverly, et al. 2014. AAPT Recommendations for the Undergraduate Physics Laboratory Curriculum. College Park, MD : American Association of Physics Teachers, 29.</bibtext> </blist> <blist> <bibtext> Kuhn, D., and M. Pease. 2008. " What Needs to Develop in the Development of Inquiry Skills? " Cognition and Instruction 26, no. 4 : 512 – 559.</bibtext> </blist> <blist> <bibtext> Lemke, J. L. 1990. Talking Science: Language, Learning and Values. Norwood, NJ : Ablex.</bibtext> </blist> <blist> <bibtext> Lising, L., and A. Elby. 2005. " The Impact of Epistemology on Learning: A Case Study from Introductory Physics." American Journal of Physics 73, no. 4 : 372 – 382. https://doi.org/10.1119/1.1848115.</bibtext> </blist> <blist> <bibtext> Manz, E. 2015. " Representing Student Argumentation as Functionally Emergent From Scientific Activity." Review of Educational Research 85, no. 4 : 553 – 590.</bibtext> </blist> <blist> <bibtext> Matz, R. L., C. L. Fata‐Hartley, L. A. Posey, et al. 2018. " Evaluating the Extent of a Large‐Scale Transformation in Gateway Science Courses." Science Advances 4, no. 10 : eaau0554. https://doi.org/10.1126/sciadv.aau0554.</bibtext> </blist> <blist> <bibtext> McNeill, K. L., M. González‐Howard, R. Katsh‐Singer, and S. Loper. 2016. " Pedagogical Content Knowledge of Argumentation: Using Classroom Contexts to Assess High‐Quality PCK Rather Than Pseudoargumentation." Journal of Research in Science Teaching 53, no. 2 : 261 – 290.</bibtext> </blist> <blist> <bibtext> Merriam, S. 1998. Qualitative Research and Case Study Applications in Education. San Francisco, CA : Jossey‐Bass.</bibtext> </blist> <blist> <bibtext> Miles, M. B., and A. M. Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA : Sage Publications.</bibtext> </blist> <blist> <bibtext> Minstrell, J. 1992. " Facets of Students' Knowledge and Relevant Instruction ". In Research in Physics Learning: Theoretical Issues and Empirical Studies, edited by R. Duit, F. Goldberg, and H. Niedderer, 110 – 128. Kiel : IPN.</bibtext> </blist> <blist> <bibtext> Momsen, J., E. Offerdahl, M. Kryjevskaia, L. Montplaisir, E. Anderson, and N. Grosz. 2013. " Using Assessments to Investigate and Compare the Nature of Learning in Undergraduate Science Courses." CBE—Life Sciences Education 12 : 239 – 249. https://doi.org/10.1187/cbe.12-08-0130.</bibtext> </blist> <blist> <bibtext> National Research Council. 2012. A Framework for K‐12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, DC : National Academies Press.</bibtext> </blist> <blist> <bibtext> Odden, T. O. B., and R. S. Russ. 2019. " Defining Sense Making: Bringing Clarity to a Fragmented Theoretical Construct." Science Education 103 : 187 – 205.</bibtext> </blist> <blist> <bibtext> Redish, E. F. 2004. " A Theoretical Framework for Physics Education Research: Modeling Student Thinking." ArXiv Preprint Physics/0411149. https://arxiv.org/abs/physics/0411149.</bibtext> </blist> <blist> <bibtext> Redish, E. F., J. M. Saul, and R. N. Steinberg. 1998. " Student Expectations in Introductory Physics." American journal of physics 66, no. 3 : 212 – 224.</bibtext> </blist> <blist> <bibtext> Rosenberg, S., D. Hammer, and J. Phelan. 2006. " Multiple Epistemological Coherences in an Eighth‐Grade Discussion of the Rock Cycle." Journal of the Learning Sciences 15, no. 2 : 261 – 292.</bibtext> </blist> <blist> <bibtext> Rudolph, J. L. 2014. " Dewey's "Science As Method" a Century Later: Reviving Science Education for Civic Ends." American Educational Research Journal 51, no. 6 : 1056 – 1083. https://doi.org/10.3102/0002831214554277.</bibtext> </blist> <blist> <bibtext> Russ, R. S. 2018. " Characterizing Teacher Attention to Student Thinking: A Role for Epistemological Messages." Journal of Research in Science Teaching 55, no. 1 : 94 – 120.</bibtext> </blist> <blist> <bibtext> Russ, R. S., J. E. Coffey, D. Hammer, and P. Hutchison. 2009. " Making Classroom Assessment More Accountable to Scientific Reasoning: A Case for Attending to Mechanistic Thinking." Science Education 93, no. 5 : 875 – 891. https://doi.org/10.1002/sce.20320.</bibtext> </blist> <blist> <bibtext> Russ, R. S., V. R. Lee, and B. L. Sherin. 2012. " Framing in Cognitive Clinical Interviews About Intuitive Science Knowledge: Dynamic Student Understandings of the Discourse Interaction." Science Education 96, no. 4 : 573 – 599.</bibtext> </blist> <blist> <bibtext> Russell, C. B., and G. C. Weaver. 2011. " A Comparative Study of Traditional, Inquiry‐Based, and Research‐Based Laboratory Curricula: Impacts on Understanding of the Nature of Science." Chem. Educ. Res. Pract. 12, no. 1 : 57 – 67. https://doi.org/10.1039/C1RP90008K.</bibtext> </blist> <blist> <bibtext> Saldaña, J. 2015. The Coding Manual for Qualitative Researchers. Los Angeles, CA : Sage.</bibtext> </blist> <blist> <bibtext> Sandoval, W. 2014. " Science Education's Need for a Theory of Epistemological Development." Science Education 98, no. 3 : 383 – 387.</bibtext> </blist> <blist> <bibtext> Sandoval, W. 2012. " Situating epistemological development." In The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012) – Volume 1, Full Papers, edited by J. van Aalst, K. Thompson, M. J. Jacobson, and P. Reimann, 347 – 354. Sydney, NSW : International Society of the Learning Sciences.</bibtext> </blist> <blist> <bibtext> Sandoval, W. A. 2005. " Understanding Students' Practical Epistemologies and Their Influence on Learning Through Inquiry." Science Education 89 : 634 – 656.</bibtext> </blist> <blist> <bibtext> Scherr, R. E., and D. Hammer. 2009. " Student Behavior and Epistemological Framing: Examples From Collaborative Active‐Learning Activities in Physics." Cognition and Instruction 27, no. 2 : 147 – 174. https://doi.org/10.1080/07370000902797379.</bibtext> </blist> <blist> <bibtext> Schommer‐Aikins, M., M. Bird, and L. Bakken. 2010. " Manifestations of an Epistemological Belief System in Preschool to Grade Twelve cCassrooms." In Personal Epistemology in the Classroom: Theory, Research, and Implications for Practice, edited by L. D. Bendixen, and F. C. Feucht, 31 – 54. Cambridge University Press. https://doi.org/10.1017/CBO9780511691904.002.</bibtext> </blist> <blist> <bibtext> Schwarz, C. E., K. S. DeGlopper, B. J. Esselman, and R. L. Stowe. 2024. "Tweaking Instructional Practices was Not the Answer: How Increasing the Interactivity of a Model‐Centered Organic Chemistry Course Affected Student Outcomes." Journal of Chemical Education, Under Review.</bibtext> </blist> <blist> <bibtext> Scouller, K. 1998. " The Influence of Assessment Method on Students' Learning Approaches: Multiple Choice Question Examination Versus Assignment Essay." Higher Education 35 : 453 – 472. https://doi.org/10.1023/A:1003196224280.</bibtext> </blist> <blist> <bibtext> Scouller, K. M., and M. Prosser. 1994. " Students' Experiences in Studying for Multiple Choice Question Examinations." Studies in Higher Education 19, no. 3 : 267 – 279.</bibtext> </blist> <blist> <bibtext> Shar, K., R. S. Russ, and J. T. Laverty. 2020. " Student Epistemological Framing on Paper‐Based Assessments." Physical Review Physics Education Research 16, no. 2 : 020101. https://doi.org/10.1103/PhysRevPhysEducRes.16.020101.</bibtext> </blist> <blist> <bibtext> Sherin, B. 2006. " Common Sense Clarified: The Role of Intuitive Knowledge in Physics Problem Solving." Journal of Research in Science Teaching 43, no. 6 : 535 – 555.</bibtext> </blist> <blist> <bibtext> Shwartz, Y., A. Weizman, D. Fortus, J. Krajcik, and B. Reiser. 2008. " The IQWST Experience: Using Coherence as a Design Principle for a Middle School Science Curriculum." The Elementary School Journal 109, no. 2 : 199 – 219. https://doi.org/10.1086/590526.</bibtext> </blist> <blist> <bibtext> Snyder, B. 1973. The Hidden Curriculum. Cambridge, MA : The MIT Press.</bibtext> </blist> <blist> <bibtext> Stowe, R. L., B. J. Esselman, V. R. Ralph, et al. 2020. " Impact of Maintaining Assessment Emphasis on Three‐Dimensional Learning as Organic Chemistry Moved Online." Journal of Chemical Education 97, no. 9 : 2408 – 2420. https://doi.org/10.1021/acs.jchemed.0c00757.</bibtext> </blist> <blist> <bibtext> Talanquer, V., and J. Pollard. 2017. " Reforming a Large Foundational Course: Successes and Challenges." Journal of Chemical Education 94, no. 12 : 1844 – 1851. https://doi.org/10.1021/acs.jchemed.7b00397.</bibtext> </blist> <blist> <bibtext> Tuminaro, J., and E. F. Redish. 2007. " Elements of a Cognitive Model of Physics Problem Solving: Epistemic Games." Physical Review Physics Education Research 3, no. 2 : 020101.</bibtext> </blist> <blist> <bibtext> Vincent‐Ruz, P., T. Meyer, S. G. Roe, and C. D. Schunn. 2020. " Short‐Term and Long‐Term Effects of POGILin a Large‐Enrollment General Chemistry Course." Journal of Chemical Education 97, no. 5 : 1228 – 1238. https://doi.org/10.1021/acs.jchemed.9b01052.</bibtext> </blist> <blist> <bibtext> White, B. Y., A. Elby, J. R. Fredriksen, and C. Schwarz. 1999. "The Epistemological Beliefs Assessment for Physical Students." (Unpublished). Presented at the 1999 AERA Session 'Assessing and Changing Students' Epistemological Beliefs About Science and Learning', Montreal, 1999.</bibtext> </blist> <blist> <bibtext> Williams, L. C., S. M. Underwood, M. W. Klymkowsky, and M. M. Cooper. 2015. " Are Noncovalent Interactions an Achilles Heel in Chemistry Education? A Comparison of Instructional Approaches." Journal of Chemical Education 92, no. 12 : 1979 – 1987. https://doi.org/10.1021/acs.jchemed.5b00619.</bibtext> </blist> <blist> <bibtext> Wilson, S. B., and P. Varma‐Nelson. 2016. " Small Groups, Significant Impact: A Review of Peer‐Led Team Learning Research with Implications for STEM Education Researchers and Faculty." Journal of Chemical Education 93, no. 10 : 1686 – 1702. https://doi.org/10.1021/acs.jchemed.5b00862.</bibtext> </blist> </ref> <aug> <p>By Cara E. Schwarz; Kimberly S. DeGlopper; Nicole C. Greco; Rosemary S. Russ and Ryan L. Stowe</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib41" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib13" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib46" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib52" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib35" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib12" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib76" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib78" firstref="ref11"></nolink> <nolink nlid="nl10" bibid="bib73" firstref="ref12"></nolink> <nolink nlid="nl11" bibid="bib30" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib42" firstref="ref15"></nolink> <nolink nlid="nl13" bibid="bib45" firstref="ref16"></nolink> <nolink nlid="nl14" bibid="bib47" firstref="ref17"></nolink> <nolink nlid="nl15" bibid="bib55" firstref="ref18"></nolink> <nolink nlid="nl16" bibid="bib38" firstref="ref19"></nolink> <nolink nlid="nl17" bibid="bib56" firstref="ref23"></nolink> <nolink nlid="nl18" bibid="bib58" firstref="ref25"></nolink> <nolink nlid="nl19" bibid="bib66" firstref="ref28"></nolink> <nolink nlid="nl20" bibid="bib71" firstref="ref29"></nolink> <nolink nlid="nl21" bibid="bib37" firstref="ref30"></nolink> <nolink nlid="nl22" bibid="bib39" firstref="ref31"></nolink> <nolink nlid="nl23" bibid="bib59" firstref="ref32"></nolink> <nolink nlid="nl24" bibid="bib19" firstref="ref36"></nolink> <nolink nlid="nl25" bibid="bib26" firstref="ref37"></nolink> <nolink nlid="nl26" bibid="bib51" firstref="ref38"></nolink> <nolink nlid="nl27" bibid="bib69" firstref="ref39"></nolink> <nolink nlid="nl28" bibid="bib70" firstref="ref40"></nolink> <nolink nlid="nl29" bibid="bib74" firstref="ref41"></nolink> <nolink nlid="nl30" bibid="bib34" firstref="ref45"></nolink> <nolink nlid="nl31" bibid="bib24" firstref="ref46"></nolink> <nolink nlid="nl32" bibid="bib25" firstref="ref47"></nolink> <nolink nlid="nl33" bibid="bib31" firstref="ref49"></nolink> <nolink nlid="nl34" bibid="bib81" firstref="ref50"></nolink> <nolink nlid="nl35" bibid="bib80" firstref="ref51"></nolink> <nolink nlid="nl36" bibid="bib16" firstref="ref52"></nolink> <nolink nlid="nl37" bibid="bib17" firstref="ref54"></nolink> <nolink nlid="nl38" bibid="bib61" firstref="ref57"></nolink> <nolink nlid="nl39" bibid="bib27" firstref="ref58"></nolink> <nolink nlid="nl40" bibid="bib54" firstref="ref60"></nolink> <nolink nlid="nl41" bibid="bib36" firstref="ref62"></nolink> <nolink nlid="nl42" bibid="bib44" firstref="ref63"></nolink> <nolink nlid="nl43" bibid="bib53" firstref="ref64"></nolink> <nolink nlid="nl44" bibid="bib77" firstref="ref68"></nolink> <nolink nlid="nl45" bibid="bib40" firstref="ref74"></nolink> <nolink nlid="nl46" bibid="bib67" firstref="ref76"></nolink> <nolink nlid="nl47" bibid="bib22" firstref="ref77"></nolink> <nolink nlid="nl48" bibid="bib50" firstref="ref78"></nolink> <nolink nlid="nl49" bibid="bib72" firstref="ref79"></nolink> <nolink nlid="nl50" bibid="bib32" firstref="ref80"></nolink> <nolink nlid="nl51" bibid="bib33" firstref="ref81"></nolink> <nolink nlid="nl52" bibid="bib23" firstref="ref88"></nolink> <nolink nlid="nl53" bibid="bib64" firstref="ref89"></nolink> <nolink nlid="nl54" bibid="bib63" firstref="ref90"></nolink> <nolink nlid="nl55" bibid="bib65" firstref="ref98"></nolink> <nolink nlid="nl56" bibid="bib11" firstref="ref103"></nolink> <nolink nlid="nl57" bibid="bib15" firstref="ref119"></nolink> <nolink nlid="nl58" bibid="bib75" firstref="ref121"></nolink> <nolink nlid="nl59" bibid="bib14" firstref="ref122"></nolink> <nolink nlid="nl60" bibid="bib68" firstref="ref124"></nolink> <nolink nlid="nl61" bibid="bib79" firstref="ref126"></nolink> <nolink nlid="nl62" bibid="bib28" firstref="ref130"></nolink> <nolink nlid="nl63" bibid="bib60" firstref="ref131"></nolink> <nolink nlid="nl64" bibid="bib21" firstref="ref132"></nolink> <nolink nlid="nl65" bibid="bib10" firstref="ref134"></nolink> <nolink nlid="nl66" bibid="bib49" firstref="ref137"></nolink> <nolink nlid="nl67" bibid="bib18" firstref="ref139"></nolink> <nolink nlid="nl68" bibid="bib62" firstref="ref146"></nolink> <nolink nlid="nl69" bibid="bib48" firstref="ref147"></nolink> <nolink nlid="nl70" bibid="bib29" firstref="ref148"></nolink> <nolink nlid="nl71" bibid="bib43" firstref="ref154"></nolink> <nolink nlid="nl72" bibid="bib57" firstref="ref157"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1460580
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Modeling Student Negotiation of Assessment-Related Epistemological Messages in a College Science Course
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Cara+E%2E+Schwarz%22">Cara E. Schwarz</searchLink><br /><searchLink fieldCode="AR" term="%22Kimberly+S%2E+DeGlopper%22">Kimberly S. DeGlopper</searchLink><br /><searchLink fieldCode="AR" term="%22Nicole+C%2E+Greco%22">Nicole C. Greco</searchLink><br /><searchLink fieldCode="AR" term="%22Rosemary+S%2E+Russ%22">Rosemary S. Russ</searchLink><br /><searchLink fieldCode="AR" term="%22Ryan+L%2E+Stowe%22">Ryan L. Stowe</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-5548-495X">0000-0002-5548-495X</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Science+Education%22"><i>Science Education</i></searchLink>. 2025 109(2):429-447.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 19
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)<br />National Science Foundation (NSF), Division of Undergraduate Education (DUE)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 2003680<br />2225025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22College+Science%22">College Science</searchLink><br /><searchLink fieldCode="DE" term="%22Modeling+%28Psychology%29%22">Modeling (Psychology)</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Organic+Chemistry%22">Organic Chemistry</searchLink><br /><searchLink fieldCode="DE" term="%22Courses%22">Courses</searchLink><br /><searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Attitudes%22">Student Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Change+Strategies%22">Change Strategies</searchLink><br /><searchLink fieldCode="DE" term="%22Epistemology%22">Epistemology</searchLink><br /><searchLink fieldCode="DE" term="%22Learning%22">Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/sce.21914
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0036-8326<br />1098-237X
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: To prepare students to use science knowledge in their later personal or professional lives, we must attend to what they believe it means to know and learn science (i.e., epistemology). Unfortunately, we have little understanding of how students' epistemologies shift and are stabilized as they navigate their science courses. Researchers have made intuitive arguments that many microscale epistemological messages sum over time to give rise to macro-scale understandings of knowing and learning, but we have no theoretical model for how this sum unfolds. Here, we begin to build such a theoretical model. To do so, we focus on assessments and related materials in a college chemistry course as potentially consequential sources of messages about valued knowledge products and processes. We then elicited students' evolving understandings of assessment-related epistemological messages in several one-on-one interviews conducted throughout the semester. Analysis of how three students experienced, negotiated, and responded to assessment-related messages showed that interactions with the course system stabilized a consistent, well-resolved picture of the ways of knowing and learning that counted in the focal course. Specifically, good knowledge must have specific authority-mandated features and knowledge is justified primarily via alignment with an instructor-authored key. Students found utility in different (reliable) processes for achieving the aim of authorized knowledge, and some of these differences were maintained throughout the semester. Implications for modeling students' experience with course-embedded epistemological messages over time and how this work might inform practice are discussed.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1460580
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1460580
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/sce.21914
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 19
        StartPage: 429
    Subjects:
      – SubjectFull: College Science
        Type: general
      – SubjectFull: Modeling (Psychology)
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Organic Chemistry
        Type: general
      – SubjectFull: Courses
        Type: general
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: Student Attitudes
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Change Strategies
        Type: general
      – SubjectFull: Epistemology
        Type: general
      – SubjectFull: Learning
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
    Titles:
      – TitleFull: Modeling Student Negotiation of Assessment-Related Epistemological Messages in a College Science Course
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Cara E. Schwarz
      – PersonEntity:
          Name:
            NameFull: Kimberly S. DeGlopper
      – PersonEntity:
          Name:
            NameFull: Nicole C. Greco
      – PersonEntity:
          Name:
            NameFull: Rosemary S. Russ
      – PersonEntity:
          Name:
            NameFull: Ryan L. Stowe
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0036-8326
            – Type: issn-electronic
              Value: 1098-237X
          Numbering:
            – Type: volume
              Value: 109
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Science Education
              Type: main
ResultId 1