View in EDS HTML Full Text PDF Full Text

Exploring STEM Students' Model Validation across Two Data-Rich Task Environments

Saved in:
Bibliographic Details
Title: Exploring STEM Students' Model Validation across Two Data-Rich Task Environments
Language: English
Authors: Jooyoung Park (ORCID 0000-0003-0427-4706), Jennifer A. Czocher, Ryan T. White
Source: International Journal of Science and Mathematics Education. 2025 23(8):3181-3204.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 24
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: STEM Education, Models, Validity, Data Analysis, Competence, Undergraduate Students
DOI: 10.1007/s10763-025-10580-9
ISSN: 1571-0068
1573-1774
Abstract: Proficiency in data analysis and modeling is increasingly essential across science, technology, engineering, and mathematics (STEM) disciplines. A critical component of these skills is model validation--assessing how well a model's predictions align with empirical data and the theoretical assumptions underlying the model. This study examines the model validation competency of undergraduate STEM students as they engaged with two data-rich modeling tasks situated in different academic domains. Students employed a range of modeling strategies, most commonly developing black-box or white-box models, with some constructing grey-box models that integrated data and theoretical reasoning. The findings suggest that students' validation approaches were shaped by specific features of the tasks, such as the availability of data and the degree of ambiguity in the problem context. The study highlights how data-rich tasks can stimulate validation activities, underscoring the importance of educators' deliberate selection of validation techniques to ensure tasks effectively achieve their intended objectives.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1501315
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHZteS93XwwzqZPH8y8wRfSAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDBOnrelD0Vq7zrLiigIBEICBm50r2CgafOot0zWRGSh-BcpLVFbVHkhL0YUsca7o0Gh9Vz3wSjb9ewe6fS2J4QvsuzOmQer0PbDDUkNyQubGDNAijYjiPtojbcLo_45r2I5jvr7jhDFjCAFKfx-dsAnazI14ZXG9qM1p2gGnte5UtQ6XrUNYosUMVnev4_1o-dHZTLkzV8PpLA7DFYv8JgxVhA4iGkCEeRcTtfKb
Text:
  Availability: 1
  Value: <anid>AN0190589616;[3d0g]01dec.25;2026Jan02.04:32;v2.2.500</anid> <title id="AN0190589616-1">Exploring STEM Students' Model Validation across Two Data-Rich Task Environments </title> <p>Proficiency in data analysis and modeling is increasingly essential across science, technology, engineering, and mathematics (STEM) disciplines. A critical component of these skills is model validation—assessing how well a model's predictions align with empirical data and the theoretical assumptions underlying the model. This study examines the model validation competency of undergraduate STEM students as they engaged with two data-rich modeling tasks situated in different academic domains. Students employed a range of modeling strategies, most commonly developing black-box or white-box models, with some constructing grey-box models that integrated data and theoretical reasoning. The findings suggest that students' validation approaches were shaped by specific features of the tasks, such as the availability of data and the degree of ambiguity in the problem context. The study highlights how data-rich tasks can stimulate validation activities, underscoring the importance of educators' deliberate selection of validation techniques to ensure tasks effectively achieve their intended objectives.</p> <p>Keywords: Modeling with data; Model validation; Data-rich task; STEM undergraduates</p> <p>Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</p> <hd id="AN0190589616-2">Introduction</hd> <p>The recent and rapid increase in the availability of measurement data of physical systems has spurred the development of many data-driven methods for modeling and predicting dynamics in Science and Engineering (Montáns et al., [<reflink idref="bib29" id="ref1">29</reflink>]). Many of the critical issues in science, technology, engineering, and mathematics (STEM) application areas where data-driven approaches (e.g., Machine Learning) have proven successful are challenges for which black-box methods relying solely on data are insufficient; rather, these problems call for a combination of modern data-driven and classical theory-based (e.g., physics-based) perspectives (e.g., Montáns et al., [<reflink idref="bib29" id="ref2">29</reflink>]). A purely data-driven approach faces difficulties in considering all aspects of modeling in engineering and science, leading to a resurging interest in taking advantage of the learnings from the classical, theory-driven approaches; together, these trends are currently motivating a mixed approach in which data-driven modeling is guided by some physical insight (Karpatne et al., [<reflink idref="bib26" id="ref3">26</reflink>]). Theory-driven modeling means constructing or developing a model from scientific principles, like using Newtonian mechanics to estimate the stopping distance for a car by considering initial velocity, weight, and friction. In contrast, goals of data-driven modeling typically include obtaining empirical relationships among variables, like fitting a curve to a set of stopping times and vehicle weights measured by manufacturers.</p> <p>Modeling real-world situations can fall along a spectrum from entirely data-driven to entirely theory-driven (Bautista et al., [<reflink idref="bib8" id="ref4">8</reflink>]). In a classroom setting, Stillman and Brown ([<reflink idref="bib40" id="ref5">40</reflink>]) argued that data are integral to both approaches. In the data-driven approach, data are used to generate a "fitted" model, which can be used for prediction (either interpolation or extrapolation), and prediction errors can be checked against a testing subset of the data. Data is a fundamental aspect of the model construction phase in data-driven modeling (Velten, [<reflink idref="bib44" id="ref6">44</reflink>]). In theory-driven modeling, data can be used to check the validity of results and the accuracy of predictions from the resulting mathematical model (Pfannkuch et al., [<reflink idref="bib32" id="ref7">32</reflink>]; Stillman & Brown, [<reflink idref="bib40" id="ref8">40</reflink>]).</p> <p>There are many skills associated with generating such models. A data-driven approach and learning process involves developing cognitive skills (e.g., analytical thinking: Grover & Pea, [<reflink idref="bib19" id="ref9">19</reflink>]; Wing, [<reflink idref="bib47" id="ref10">47</reflink>]), statistical thinking, such as considering random behaviors in real-world systems and simulating data (Engel & Kuntze, [<reflink idref="bib12" id="ref11">12</reflink>]), getting a comprehensive insight into the usefulness of data to draw effective problem solutions; data collection; data analysis and processing; data visualization by creating representations like charts, maps, or graphs; and in-sample or out-of-sample validations (e.g., Grover & Pea, [<reflink idref="bib19" id="ref12">19</reflink>]; Wing, [<reflink idref="bib47" id="ref13">47</reflink>]).</p> <p>These skills are inseparably intertwined with modeling competencies like making assumptions and validation because decisions in data collection, wrangling, and analysis shape the resulting model and its empirical adequacy. Mathematics education research on theory-driven modeling has produced mathematical modeling cycles (MMCs), which describe the competencies associated with constructing a mathematical model for a real-world scenario based on mathematical knowledge and knowledge about how the world works (Blum & Leiss, [<reflink idref="bib9" id="ref14">9</reflink>]). Studies have indicated that validating a model is challenging for students because of how the individual perceives and resolves a cognitive conflict between their expectations of their model (e.g., predictions) and outcomes (e.g., empirical observations) (e.g., Czocher, [<reflink idref="bib4" id="ref15">4</reflink>]). They may fail to notice something is amiss, perceive difficulties that do not exist, provide an inadequate response, or even change the problem to suit their readily available knowledge (Stillman, [<reflink idref="bib39" id="ref16">39</reflink>]). Indeed, some have observed that validating is a "uniform shortcoming" of students'mathematical modeling because students do not reflect on improving their models (e.g., Blum & Leiss, [<reflink idref="bib9" id="ref17">9</reflink>]). Despite the increasing importance of modeling in STEM contexts, students often struggle to critically validate their models—particularly in data-rich tasks that blur the lines between empirical and theoretical approaches. This challenge underscores the need to better understand how undergraduate students engage in validation when navigating modeling tasks that combine data analysis with conceptual reasoning. While validation is central to both theory- and data-driven modeling, the specific ways in which task features promote or hinder this process remain underexplored. This study addresses that gap by examining how task design can influence students' validation behaviors and foster deeper engagement in revising and refining their models.</p> <p>While there has been extensive study of students' validating competencies in theory-driven modeling (e.g., Czocher, [<reflink idref="bib7" id="ref18">7</reflink>]; Ishibashi & Uegatani, [<reflink idref="bib22" id="ref19">22</reflink>]; Vorhölter, [<reflink idref="bib45" id="ref20">45</reflink>]) there has been comparatively less in settings calling for data-driven modeling, despite the fact that there are non-overlapping goals, approaches, and skills associated with each paradigm. Even fewer attempts have been made to research undergraduate students' modeling competencies in data-rich task situations (Hallström & Schönborn, [<reflink idref="bib21" id="ref21">21</reflink>]). Empirical research about interdependencies among statistics, inclusion of data, and theory-driven modeling competencies and effective ways of eliciting and fostering these competencies is needed.</p> <p>Recognizing the importance of the model validation competency to developing both data-driven and theory-driven modeling skills, this study explored task features that can elicit and encourage student engagement in validating while working on data-rich tasks. Data-rich tasks are hybrids; they are neither entirely data-driven nor entirely theory-driven but rather occupy a position on "a continuum along 'theory-driven' to 'data-driven'" (Bautista et al., [<reflink idref="bib8" id="ref22">8</reflink>], p. 9). The field has reached a consensus that task design for fostering students' modeling competencies must focus first on whether the tasks can evoke modeling processes (Blum & Leiss, [<reflink idref="bib9" id="ref23">9</reflink>]; Zbiek & Conner, [<reflink idref="bib48" id="ref24">48</reflink>]) before assessing whether learning results from using them. With this in mind, our goals in this study were to (<reflink idref="bib1" id="ref25">1</reflink>) design such data-rich tasks and (<reflink idref="bib2" id="ref26">2</reflink>) investigate their capacity to evoke modeling competencies, particularly model validation. Hence, this paper reports on how undergraduate STEM students engaged in model validation when working on such tasks.</p> <hd id="AN0190589616-3">Literature Review</hd> <p>The interplay between modeling competencies, different modeling approaches (e.g., data-driven, theory-driven, or in-between), and validation practices underscore the complexity of mathematical modeling and its pivotal role in understanding real-world phenomena. In this section, we discuss a nuanced exploration of these elements, aiming to elucidate the underlying processes driving effective modeling practices and pave the way for enhanced educational strategies and interdisciplinary insights.</p> <p>The notion of modeling competency is based on theoretical considerations by Blum and Kaiser (cited in Maaß, [<reflink idref="bib28" id="ref27">28</reflink>]), who specify modeling competencies by a detailed list of abilities (competencies) that are related to carrying out a modeling process (see Fig. 1). It is common to use mathematical modeling cycles (MMCs) as an analytic framework for task creation and analyzing students' productions in mathematical modeling (Kaiser, [<reflink idref="bib25" id="ref28">25</reflink>]). MMCs are useful for describing features of learning environments (e.g., Czocher, [<reflink idref="bib6" id="ref29">6</reflink>]) and features of modeling tasks (e.g., Maaß, [<reflink idref="bib28" id="ref30">28</reflink>]). Coordinating task design, research setting, and analysis through an MMC permits conclusions about the extent to which certain features of the learning environment elicit or improve specific modeling competencies. Based on the common language of the MMCs, the field has developed task-design heuristics (e.g., Maaß, [<reflink idref="bib28" id="ref31">28</reflink>]) and means for examining in detail the modeling process and the <emph>what</emph> and <emph>how</emph> of validating. We introduce the conceptual framework in this section to facilitate connections with prior research. We recognize and acknowledge that in statistics modeling and data science, different cycles are used to highlight aspects of working with large data sets and variability and those cycles may more closely match the traditions of statistics education (see, for example, Pfannkuch et al., [<reflink idref="bib32" id="ref32">32</reflink>]; Patel & Pfannkuch, [<reflink idref="bib30" id="ref33">30</reflink>]).</p> <p>Graph: Fig. 1 A mathematical modeling cycle (Blum & Leiss, [<reflink idref="bib9" id="ref34">9</reflink>])</p> <p>In modeling cycles, such as those described by Blum and Leiss ([<reflink idref="bib9" id="ref35">9</reflink>]) and illustrated in Fig. 1, the process distinguishes between the real world and the mathematical world. The modeling process begins in the real world by identifying a real-world problem to solve (understanding), structuring that problem through declaring variables to include and imposing assumptions on them (simplifying/structuring), developing a conventional mathematical representation for the relationships among variables (mathematizing), solving the resulting mathematical problem (working mathematically), interpreting the results in terms of real-world constraints (interpreting), and then assessing the adequacy of the model based on the implications of the real-world results (validating). In this study, we focus on the relationship between the amount of information given in a task statement, which has implications for <emph>making assumptions</emph> and students' approaches to <emph>validating</emph>.</p> <hd id="AN0190589616-4">Skills Necessary for Modeling in the Continuum Between Theory-Driven and Data-Scenarios</hd> <p>Bautista et al. ([<reflink idref="bib8" id="ref36">8</reflink>]) conceptualized "modeling activities as a continuum along 'data-driven' to 'theory-driven' situations" (p. 9). In theory-driven situations, the purpose of the model is to explain causal mechanisms and predict phenomena with a close connection to the real-world mechanisms governing the situation that permit an explanation of why or how some event happened. In contrast, data-driven situations call for models that fit functions to empirically gathered data. The fitted models often lack a theoretical foundation, and it may be difficult to infer causal mechanisms. The fitted models are useful for prediction (say, based on statistically identified causal factors), but can fall short of explanatory capabilities.</p> <p>In both theory-driven and data-driven situations, information about the real world – data – plays a pivotal role. However, the role data plays in modeling differs across stages of the modeling process. Indeed, empirical data can be used to evaluate the adequacy of a model even though it is not used to generate the model. In a data-driven situation, constructing a model involves checking statistical assumptions to determine whether a method can be applied to the empirical data unless it involves high dimensional data. The distinctions in how data is used are tied to the model construction goals and, therefore, anticipate that differing skills are needed to use the data.</p> <hd id="AN0190589616-5">Not All Model Types are Equally Transparent</hd> <p>The models that arise from the continuum from theory-driven to data-driven paradigms reflect the needs of the disciplines that create them. Velten ([<reflink idref="bib44" id="ref37">44</reflink>]) observed that scientific disciplines tend to use particular kinds of mathematical content to build models, in concert with the a priori knowledge about the system of interest, to address a range of purposes one might wish to model (see Fig. 2). Systems can be found within any field of inquiry, such as climate, tax policy, educational measurement, natural habitats, and so on. The purpose of the model and the amount of a priori information available about the structure of the system will guide the type of modeling that results. One way to think about these model types is in terms of their transparency: black, white, or grey. A mathematical model is called a <emph>black box model (defined as a data-driven model in the previous section)</emph> when its construction is based on experimental data only, using no a priori information about the system to be modeled. A mathematical model is called a <emph>white box model (defined as theory-driven model in the previous section)</emph> when mathematical statements used to describe the system are based solely on a priori information about the system. In between those end points, there are also <emph>grey box models, w</emph>here the modeler has some prior information about the system but also uses experimental data in their construction.</p> <p>Graph: Fig. 2 Classification of mathematical models between black and white box models (Velten, [<reflink idref="bib44" id="ref38">44</reflink>], p. 40). Note: the three dimensions of a mathematical model (S, Q, M) can be seen in the figure: the systems (S), a list of objectives that mathematical models can be used for (Q), and mathematical structures (M) ranging from algebraic equations (AEs) to differential equations (DEs)</p> <p>Building on these characterizations, Frejd and Bergsten ([<reflink idref="bib13" id="ref39">13</reflink>]) interviewed professional modelers from STEM fields to connect their self-described modeling practices to educational practices. The professional modelers'efforts resulted in the development of both black box and grey box models derived from social, economic, biological, and physical systems, as well as white-box models from climate and physical systems. Additionally, the professional modelers discussed model-generated modeling, which the authors compared to common educational modeling activities. Model-generated modeling involves using and adapting pre-existing models to fit specific systems, resulting in both grey box and white box models. The role of data was central to all of the professional modelers' activities and often connected to their validation approach, a connection we elaborate on in the next section. For both white box and grey box models, data supported the evaluation of the effectiveness of the resulting models, while in black-box modeling, data was instrumental to the utility of the resulting model. Frejd and Bergman's ([<reflink idref="bib13" id="ref40">13</reflink>]) study offers a framework for considering the key activities essential to modeling in data-rich environments: the interaction between data, making assumptions, and validation. The following discussion will focus on these three aspects of modeling, incorporating both statisticians' perspectives and studies of classroom modeling situations involving data, where applicable.</p> <hd id="AN0190589616-6">Validation Plays a Crucial Role in Both Data-Driven and Theory-Driven Modeling Processes</hd> <p>Validation is an important part of modeling because it examines the accuracy and robustness of the model by checking if the model performs as intended and whether the real-world problem is solved satisfactorily by using the model. In this section, we review empirical and theoretical literature related to students' validation of models.</p> <p>According to MMC's, a model is validated by determining whether its predictions are reasonably accurate (Blum & Leiss, [<reflink idref="bib9" id="ref41">9</reflink>]). Based on these descriptions, validating seems to be analogous to the problem-solving phases of looking back (Pólya, [<reflink idref="bib33" id="ref42">33</reflink>]) or verification (Schoenfeld, [<reflink idref="bib37" id="ref43">37</reflink>]), where the problem solver reviews her completed solution. In the empirical literature, there is a lack of agreement about whether and how validation judgments occur during modeling (Czocher, [<reflink idref="bib7" id="ref44">7</reflink>]), and others have noted it shares only some features in common with problem-solving (Lesh & Zawojewski, [<reflink idref="bib27" id="ref45">27</reflink>]).</p> <p>Many studies have highlighted students'challenges in validating their models. Blum and Leiss ([<reflink idref="bib9" id="ref46">9</reflink>]) noted that students often fail to validate or reflect on their models, while Stillman and Galbraith ([<reflink idref="bib41" id="ref47">41</reflink>]) found that many students either did not see the need or lacked the time to check their models. Borromeo Ferri ([<reflink idref="bib10" id="ref48">10</reflink>]) identified two types of validation used by modelers: knowledge-based validation, which involves conscious decisions based on real-world knowledge, and intuitive validation, an unconscious sense that results might be wrong. This intuitive approach underscores that students may struggle to articulate the rationale behind their judgments.</p> <p>Previous research focused on secondary students and the correctness of their final answers to determine validation (Goos & Galbraith, [<reflink idref="bib15" id="ref49">15</reflink>]; Stillman, [<reflink idref="bib39" id="ref50">39</reflink>]). Validation has been used to capture holistic reflection on the modeling process (Grunewald, [<reflink idref="bib20" id="ref51">20</reflink>]) and to compare different solution methods (Schukajlow & Krug, [<reflink idref="bib38" id="ref52">38</reflink>]). Supporting Borromeo Ferri's ([<reflink idref="bib10" id="ref53">10</reflink>]) findings, Czocher ([<reflink idref="bib4" id="ref54">4</reflink>], [<reflink idref="bib5" id="ref55">5</reflink>], [<reflink idref="bib7" id="ref56">7</reflink>]) found that validation played a significant role in the modeling processes of middle, secondary, and undergraduate students, regardless of whether the model was correct. Ishibashi and Uegatani ([<reflink idref="bib22" id="ref57">22</reflink>]) found that middle-grade students validated their models by imagining scenarios in which the model would be valid, suggesting a connection between validation and the conditions assumed by the modeler. Validation processes in data-driven and theory-driven contexts share similarities. In data-driven contexts, a model can be evaluated operationally by checking if its output aligns with observed data and conceptually by assessing the justification of the underlying statistical theory and assumptions (Rykiel, [<reflink idref="bib34" id="ref58">34</reflink>]; Sargent, [<reflink idref="bib35" id="ref59">35</reflink>]). Statisticians distinguish between model validation (ensuring a model performs within a given error tolerance) and model verification (ensuring the programming and implementation are correct) (Sargent, [<reflink idref="bib35" id="ref60">35</reflink>]; Schlesinger et al., [<reflink idref="bib36" id="ref61">36</reflink>]). In classrooms, students use trustworthy software, so verification focuses on selecting the correct statistical procedures. Statisticians also ensure data validity, meaning datasets are adequate and satisfy the required assumptions for the intended statistical model.</p> <p>In data-driven situations, various tests and evaluations (e.g., hypothesis tests, goodness-of-fit tests, confidence intervals, residuals) are conducted to consider a model valid for its purpose (Sargent, [<reflink idref="bib35" id="ref62">35</reflink>]). Sensitivity analysis, which assesses the influence of parameters, initial conditions, and assumptions on model predictions, is essential for validation (Peck, [<reflink idref="bib31" id="ref63">31</reflink>]). Data-driven models must ultimately be validated by comparing their predictions to real-world experimental data. There are two main approaches: in-sample validation, which examines how well the model fits the data, and out-of-sample validation, which compares the model's output to observations not included in the dataset used to fit the model. Out-of-sample validation is often considered the most robust method for testing a model's predictive viability.</p> <hd id="AN0190589616-7">The Validation of Models Varies Across Theory-Driven and Data-Driven Approaches Based on How...</hd> <p>In a modeling process, assumptions about the scenario and available information shape and constrain the models that can be produced. Therefore, it is necessary to consider validation in relation to making assumptions, which depends on the information the modeler has available. Modeling problems with missing information can be classified as ill-defined problems, where the data, goals, and operators are not clearly specified (Jonassen, [<reflink idref="bib23" id="ref64">23</reflink>]). Schukajlow and Krug ([<reflink idref="bib38" id="ref65">38</reflink>]) identified the benefits of prompting students to find multiple solutions to modeling problems with missing data on students' interest and subjective experience of modeling competency, although they did not explicitly connect the condition of missing data to validating competency.</p> <p>A key distinction between theory-driven and data-driven situations lies in the types and roles of assumptions in each situation. In theory-driven situations, making assumptions involves introducing and prioritizing key variables, designating some variables as parameters and potentially assigning them values, creating quantitative relationships among variables based on knowledge about how the world works, and sometimes deliberately ignoring aspects of the scenario to make model development manageable. In contrast, in data-driven situations, statistical methods derive relationships between variables analytically. Thus, making assumptions involves selecting relevant parameters, measuring them appropriately (e.g., ordinal levels), and choosing a specific statistical method.</p> <p>We build on this body of work by examining the extent to which data-rich tasks, with varying degrees and types of information, elicit validating competencies and impact the nature of validating opportunities they provide to students. The guiding questions for the study were: <emph>What approaches did students take to modeling the scenarios in two data-rich tasks? and What distinctions in student validating activities arise from the two tasks?</emph></p> <hd id="AN0190589616-8">Method</hd> <p>We designed data-rich modeling tasks that could be tested for eliciting theory-driven and data-driven validating competencies. For descriptive analysis, we compared levels of validation and validation approaches observed from the participants' written work on these tasks. In this section, we describe the research setting, the data-rich modeling tasks, and the research design.</p> <hd id="AN0190589616-9">Research Setting</hd> <p>The modeling tasks were facilitated in a Statistics class, a required course for some engineering and computer science majors at a private university in the southeast USA. The course lists integral calculus as a prerequisite. It covers topics of probability, inferential statistics, and regression analysis, and students engage in class activities using the R programming language. Forty-five students enrolled in the course, which was taught by mathematics faculty in the Department of Mathematics and Systems Engineering. The data-driven modeling tasks (described below) were given as in-class activities. Approximately half of the participating students were computer science majors, while the rest were engineering majors. Among the engineering students, the majority specialized in mechanical engineering, with a smaller proportion in biomedical engineering.</p> <hd id="AN0190589616-10">Task Design</hd> <p>We needed to create tasks that satisfied two criteria important to the research design. The tasks needed to be data-rich, and thus located towards the data-driven end of the continuum. The tasks also needed to draw on students' real-world knowledge about relationships among variables, so the students would not be precluded from taking a theory-driven approach to finding a solution (Brown, [<reflink idref="bib11" id="ref66">11</reflink>]; Greefrath, [<reflink idref="bib17" id="ref67">17</reflink>]; Zbiek & Conner, [<reflink idref="bib48" id="ref68">48</reflink>]). To guide task design, we synthesized Maaß's ([<reflink idref="bib28" id="ref69">28</reflink>]) task framework, which characterizes the scope of the modeling process, with Velten's ([<reflink idref="bib44" id="ref70">44</reflink>]) framework describing systems explored in different academic domains and types of mathematical models (see Fig. 2). Accordingly, two tasks were designed (see Fig. 3): the Stopping Distance Task (Mechanical System) and the U.S. Coal Task (Economical System). Important to the research design, both tasks were intended to evoke the entire modeling cycle and both are data-rich. Thus, the scenarios are located towards the data-driven end of the continuum.</p> <p>Graph: Fig. 3 Two data-rich tasks used as comparison conditions</p> <p>The Stopping Distance Task provides students with inadequate information, which allows them to seek out an extra data that was not given. The task presents students with data for multiple vehicle models before asking them to build a model of the stopping distance. The task also included information consistent across the cars, such as the car speed and the distance to the child. The academic domain and system for the task is mechanical engineering or physics relevant. In the U.S. Coal Task, students were provided historical data on coal consumption, import, export, and production. It asked students to generate a function to model the data. In contrast to the Stopping Distance Task, the U.S. Coal Task asked substantive follow-up questions prompting students to answer predicted outcomes based on their models. The U.S. Coal Task is situated in the Economics Domain.</p> <p>Both tasks prompted students to explain how well their models worked and requested students share the limitations of their models. The tasks were checked for face and construct validity by a panel of mathematics educators. The educators were asked to use Maaß's ([<reflink idref="bib28" id="ref71">28</reflink>]) and Velten's ([<reflink idref="bib44" id="ref72">44</reflink>]) framework to evaluate the tasks by attending to: academic domains, whether the tasks would elicit each competency of the MMC, and types of mathematical models (black box or data-driven, white box or theory-driven or grey box).</p> <hd id="AN0190589616-11">Data Collection</hd> <p>Task 1 was assigned at the outset of the semester. However, no feedback was given on students'work for Task 1 before Task 2 was introduced to the class midway through the semester. Both tasks were undertaken individually, and upon the conclusion of the class session, participants submitted IRB consent forms along with their written responses to the tasks. The class comprised 45 students; thirty-seven partially or fully completed Task 1, while forty-four students did so for Task 2. Following the elimination of invalid and incomplete responses, a total of 35 responses were included in the study analysis.</p> <hd id="AN0190589616-12">Data Analysis</hd> <p>We used descriptive statistics to address the research questions. We first characterized students' validation approaches. We then assessed their responses to the tasks using a validating-specific rubric (described below) to generate an ordinal score corresponding to each response's level of validating competency.</p> <hd id="AN0190589616-13">Model Choices and Model Validation Approaches</hd> <p>Because the literature review suggested that the opportunities for demonstrating validating competency may depend on the students' approach to modeling and their disposition toward the data included in the problem, we categorized students' approaches to creating the model as black box, white box, or grey box (Velten, [<reflink idref="bib44" id="ref73">44</reflink>]), and noted which approaches or techniques of validation they employed, if any, for example, Fig. 4 shows a black box model choice. The student fit a polynomial to the model and compared the results to the empirical data using R<sups>2</sups>, which evidences a model-fit, in-sample validation. The response also acknowledged the randomness of the phenomenon and the variation and variability of data.</p> <p>Graph: Fig. 4 Sample response: a black-box model in the US Coal Task situation</p> <hd id="AN0190589616-14">Rubric for Levels of Validating Competency</hd> <p>To assess students' validating competency on the two data-rich tasks, we adapted the <emph>Mathematical Modeling Competency (MMC) Assessment Rubric</emph> (Chan et al., [<reflink idref="bib2" id="ref74">2</reflink>]; Galbraith & Stillman, [<reflink idref="bib14" id="ref75">14</reflink>]; Greefrath et al., [<reflink idref="bib18" id="ref76">18</reflink>]; Greefrath, [<reflink idref="bib16" id="ref77">16</reflink>]; Tekin-Dede & Bukova-Guzel, [<reflink idref="bib42" id="ref78">42</reflink>]). The scoring articulates the expectations for a task by listing the criteria and describing levels of quality (Andrade & Du, [<reflink idref="bib1" id="ref79">1</reflink>]). In the <emph>MMC Assessment Rubric</emph>, qualities of validating include identifying the influence of relevant constraints/real-world aspects on the mathematical results, checking the real-world interpretation of results against the situation model for adequacy, and ascertaining the plausibility of the model's predictions (Greefrath et al., [<reflink idref="bib18" id="ref80">18</reflink>]; Greefrath, [<reflink idref="bib16" id="ref81">16</reflink>]; Tekin-Dede & Bukova-Guzel, [<reflink idref="bib42" id="ref82">42</reflink>]). These factors which align well with the qualities of validating synthesized in our review of literature. The assessment framework evaluates the process of validation, the accuracy of validation assessments, and the effort to rectify inadequate models.</p> <p>Given that the data primarily consisted of written responses, distinguishing between some individual levels reliably was not feasible. Consequently, we collapsed levels 1 and 2, as well as levels 4 and 5, from the original rubric devised by Tekin-dede and Bukova-Guzel ([<reflink idref="bib42" id="ref83">42</reflink>]), yielding a 4-level scale:</p> <p></p> <ulist> <item> Level A: Not validating or making a wrong validation. Validating partially, not correcting the determined mistakes.</item> <p></p> <item> Level B: Validating partially, correcting the determined mistakes to some extent.</item> <p></p> <item> Level C: Validating partially, correcting the determined mistakes. Validating completely, not correcting the determined mistakes.</item> <p></p> <item> Level D: Validating completely, correcting the determined mistakes to some extent.</item> </ulist> <p>Since validating may evaluate the adequacy of any of the steps in the MMC (Czocher, [<reflink idref="bib7" id="ref84">7</reflink>]), raters considered students' responses taking the whole solution into account rather than focusing on whether each step of the modeling process was validated. Statements about ideal results, assessment of their answers or models, comparing their answers to theoretical or empirical results, and checking their models against the given real-world context were indicators of validating activities. In the context of the data sources for this study, we interpreted <emph>partial validation</emph> to mean using statistical tests without reference to real-world considerations. For example, if a student did check a goodness of fit measure (e.g., R<sups>2</sups>) but did not refer back to the real-world aspects of the data (e.g., trends) that would be marked as <emph>partial validation</emph>. In contrast, the integration of both real-world and statistical validation aspects was categorized as complete validation.</p> <p>Two raters examined a calibration set comprising 35 responses for each data-rich task and categorized each response into its corresponding level. The raters convened to compare their assessments on the calibration set and achieved an initial consensus, resolving any discrepancies through discussion. Upon reaching an agreement, the raters independently applied the rubric to the remaining responses, and the ongoing agreement was assessed using Cohen's kappa (Cohen, [<reflink idref="bib3" id="ref85">3</reflink>]). For each task, the raters met reliability standards (Stopping Distance Task Validating: κ = 0.96; US Coal Task Validating: κ = 0.93). Tables 1 and 2 present key excerpts from sample responses exemplifying the validating component. In the Stopping Distance task, responses demonstrating complete validation did not include any corrections for identified limitations or errors.</p> <p>Table 1 Sample responses from the Stopping Distance Task rated at each level of validating competency: Purely text responses are reproduced verbatim</p> <p> <ephtml> <table rules="groups"><thead><tr><th align="left"><p>Level</p></th><th align="left"><p>Sample of student work</p></th><th align="left"><p>Justification for level</p></th></tr></thead><tbody><tr><td align="left"><p>Level A</p></td><td align="left"><p><italic>The child is moving or there is something that will slow down the car, the result would not be accurate. Something could be wrong with the car that affects breaking distance too</italic></p></td><td align="left"><p>The student points out how the solution may be wrong—validating partially, but does not offer remedies</p></td></tr><tr><td align="left"><p>Level B</p></td><td align="left"><p><italic>The equation is vague and does not take into account when the driver will notice the kid and begin breaking. So, the actual breaking distance is different than the one calculated. The equation also does not factor in environmental factors such as weather, debris on the road, and road conditions (e.g., cracks, potholes, </italic>etc<italic>.)</italic></p></td><td align="left"><p>The student partially validated by pointing out specific shortcomings in the solution (e.g., mentioning reaction time for the model) but did not attempt to solve these problems</p></td></tr><tr><td align="left"><p>Level C</p></td><td align="left"><p><italic>If the road is wet, the speed will be half of the current speed. We can then see different scenarios based on the speed, the movement of the child, and the road conditions</italic></p><p><italic>The model can only calculate the static values that are given and measurable. It can compute outcomes for both wet and dry roads based on the situation. However, the limitation is that the model cannot calculate human responses, emotions, or states of mind</italic></p></td><td align="left"><p>The student partially validated by giving specific problems with the solution and indicating the direction of the error if these problems were considered, but the student did not do so</p></td></tr><tr><td align="left"><p>Level D</p></td><td align="left"><p><italic>The model is limited as it does not consider external forces like wind and only represents the braking distances of specific car models, which may vary in braking system quality. These limitations are likely negligible for a vehicle traveling at 55 km/h needing to stop within 40 m, as the greatest stopping distance is 19.1 m less than 40 m. However, with a smaller coefficient of friction, such as for a larger vehicle, these limitations would need careful consideration to ensure the braking distance does not exceed 40 m</italic></p></td><td align="left"><p>The student points out problems with the solution and argues why the changes would have a negligible effect on the estimate and hence provides a solution in such a case: the student validated the model by stating the results of considering external factors (e.g., wind) and braking system (friction) and corrected mistakes by providing the results from the correction</p></td></tr></tbody></table> </ephtml> </p> <p>Graph</p> <hd id="AN0190589616-15">Results</hd> <p></p> <hd id="AN0190589616-16">What Approaches did Students Take to Modeling the Scenarios in Data-Rich Tasks?</hd> <p>Across both tasks, approximately half of the responses opted for black box models employing the provided data, while around 30% developed white-box models rooted in physics principles but did not rely on computerized models or data. The remaining responses represented data-informed physics models, where data played an integral role in the modeling process, which we characterize as grey box models. Specifically, regarding the US Coal Task situation, all responses resulted in black-box models, primarily fitted curves like regression models (e.g., linear or polynomial). These models underwent verification through computer computations. As anticipated, students exhibited diverse approaches based on data utilization within the modeling process. For instance, in the Stopping Distance Task, Student Chris (see Fig. 5) opted for a fully data-driven (a black box) and computerized model, selecting a fitted parabolic function correlating car speed with stopping distance, as indicated by the data plotted. The student appropriately verified that the data met the statistical assumptions for linear regression but overlooked several practical considerations. Notably, the driver's reaction time and road conditions were not considered. Chris'analysis primarily focused on in-sample validation, assessing the model's adequacy in representing the given data, examining parameters'relationships, and conducting residual analysis.</p> <p>Graph: Fig. 5 Chris'work on the Stopping Distance Task situation demonstrates a black-box model and in-sample validation</p> <p>Dana employed a white box model on the Stopping Distance Task (see Fig. 6) and evidenced realistic considerations by incorporating various factors inherent to car mechanics (such as friction, deceleration, vehicle weight, etc.) alongside aspects like driver reaction time and road conditions, into the modeling process. However, it's worth noting that Dana's approach did not involve integrating data for model construction or validation through computer simulation.</p> <p>Graph: Fig. 6 Dana's work on the Stopping Distance Task situation generates a white-box model</p> <p>In contrast, Evan adopted an approach that combined theoretical insights with the utilization of data points for computational analysis (see Fig. 7). Ensuring the adequacy of the data for statistical procedures through a data validity check, Evan conducted statistical model validation. Evan expanded the number of identifiable parameters by incorporating additional experimental data (such as out-of-sample braking distance), thereby attempting out-of-sample validation and model calibration. Moreover, Evan conducted a sensitivity analysis to assess the relative influence of parameters, particularly focusing on the friction coefficient and initial conditions. Although this attempt was somewhat limited and did not fully align with the iterative process typical of disciplinary paradigms, it nonetheless demonstrates sophistication not commonly observed in other validation efforts.</p> <p>Graph: Fig. 7 Evan's work on the Stopping Distance Task situation demonstrates a grey-box model and out-of sample validation approach</p> <hd id="AN0190589616-17">Do Validating Levels Differ Based on the Amount of Information Given?</hd> <p>The students'responses underwent evaluation using the MMC Validating Rubric (e.g., Tekin-Dede & Bukova-Guzel, [<reflink idref="bib42" id="ref86">42</reflink>]), with the corresponding relative frequency distribution presented in Table 3. It was observed that the median level of validating competency was higher in the Stopping Distance Task (Median = 3) compared to the U.S. Coal Task (Median = 1) when scaled on a range from 1 to 4. However, the extent to which students validated and subsequently corrected their models based on these validation judgments was limited in both tasks.</p> <p>Table 3 Relative frequency of validating levels, by task</p> <p> <ephtml> <table rules="groups"><thead><tr><th align="left" /><th align="left"><p>Level A</p></th><th align="left"><p>Level B</p></th><th align="left"><p>Level C</p></th><th align="left"><p>Level D</p></th></tr></thead><tbody><tr><td align="left"><p>Stopping Distance (<math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi>n</mi><mo>=</mo><mn>35</mn><mo stretchy="false">)</mo></mrow></math></p></td><td char="." align="char"><p>14.3%</p></td><td align="left"><p>8.6%</p></td><td char="." align="char"><p>62.8%</p></td><td char="." align="char"><p>14.3%</p></td></tr><tr><td align="left"><p>US Coal <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mo stretchy="false">(</mo><mi>n</mi><mo>=</mo><mn>35</mn><mo stretchy="false">)</mo></mrow></math></p></td><td char="." align="char"><p>74.3%</p></td><td align="left"><p>0</p></td><td char="." align="char"><p>11.4%</p></td><td char="." align="char"><p>14.3%</p></td></tr></tbody></table> </ephtml> </p> <p>In the Stopping Distance Task, a typical Level A response in this task acknowledged some model limitations but did not rectify errors or offer a conclusive judgment regarding the model's adequacy. Most students (62.8%) validated their models but refrained from revising or providing corrections after identifying limitations. Such responses were categorized as Level C. However, Finley provided one of the few Level D responses (see Fig. 8). Student Finley validated the student's model by delineating imperfect assumptions and subsequently proposed specific resolutions or argued why certain aspects of the scenario were inconsequential. They heavily relied on factual information about the particular cars (e.g., comparing design features of the Ferrari to the Toyota), introduced simplifying assumptions to reduce variation (e.g., assuming equal braking and reaction times for drivers), controlled environmental conditions (e.g., disregarding wind resistance), and observed that their chosen models were not sufficiently sophisticated to accurately represent driving conditions (i.e., 1-D Newtonian kinematics).</p> <p>Graph: Fig. 8 Finley validates his grey-box model in the Stopping Distance Task situation: Purely text responses are reproduced verbatim</p> <p>In comparison, in the U.S. Coal Task situation, the validation of models was primarily confined to verifying the application of statistical tests or procedures, such as ensuring compliance with statistical assumptions, evaluating model fit, or examining the coefficient of variation. Given that all student responses in the U.S. Coal Task were data-driven black box models, it is not surprising that validation techniques were predominantly centered around model verification. Additionally, we did not observe any Level B responses in the U.S. Coal Task, which required some model modification. Across both tasks, students who opted for a black box model typically did not engage in further calibration of their models once the model was generated by the software. This could be attributed to the students potentially lacking tools to modify a data-driven model, although they could still critique its fit. Notably, all Level D responses in the U.S. Coal Task were characterized as black box models, whereas in the Stopping Distance task, 80% of all the responses were identified as grey box models. The majority of student responses manifested as data-driven black box models, effectively leveraging the provided dataset in relation to parameters for the U.S. Coal task. Grey box models, on the other hand, emerged in Levels C and D for the Stopping Distance Task, where responses incorporated both the provided dataset and out-of-sample data. When analyzing the information used by students to validate their models, a notable disparity emerged between the Stopping Distance Task and the U.S. Coal Task. The sources of information employed by students for model validation differed across these tasks. In the U.S. Coal Task, students relied on provided information, which included multiple datasets with values for relevant variables, to examine the relationships among variables, calculate R<sups>2</sups>, and verify adherence to statistical assumptions. An example of such a response was shown earlier in the data analysis section, as depicted in Fig. 4. We hypothesized that students predominantly employed in-sample validation techniques when abundant data were provided. Conversely, in the Stopping Distance Task, the second most prevalent approach involved seeking a best-fit model utilizing the enclosed dataset, akin to the approach adopted by Student Chris, as illustrated in Fig. 5.</p> <p>However, the predominant approach in the Stopping Distance Task involved validation through acknowledgment of real-world conditions influencing the predictions of their models, along with an examination of how the specified conditions (see Fig. 9) affected the parameters within the models. For instance, students actively sought external information regarding friction coefficient values for the provided cars, mirroring the approach taken by Student Evan, as depicted in Fig. 7, which we deem an instance of out-of-sample validation. Similarly, Chris, illustrated in Fig. 5, pursued validation by seeking external information about the realistic assumptions they made regarding factors related to car mechanics (such as friction, deceleration, and vehicle weight), the driver, and environmental conditions. Harper (in Fig. 9) generated a grey box model grounded in established theory, evidenced by their utilization of entries from the dataset and external information to refine their physics-based model.</p> <p>Graph: Fig. 9 Student Harper's response to the Stopping Distance Task, which includes seeking external information to validate the model. Student H's response to the Stopping Distance Task, which includes seeking</p> <hd id="AN0190589616-18">Discussion and Conclusion</hd> <p>Ideally, model validity is assessed along two main dimensions: verifying whether the model's output aligns with observed data and evaluating the justifiability of the underlying theory and assumptions (e.g., Rykiel, [<reflink idref="bib34" id="ref87">34</reflink>]; Sargent, [<reflink idref="bib35" id="ref88">35</reflink>]). However, due to the scarce literature in educational research regarding systematic task-based studies examining students'approaches to model validation with data, this study aimed to investigate students'validation approaches concerning models created within the context of data-rich tasks, by considering academic domains and types of model dimensions (black box or data-driven, white box or theory-driven model, and grey box which falls between these extremes). The findings from the current study are consistent with previous assertions that students rarely spontaneously refine their models after generating initial models (Blum & Leiss, [<reflink idref="bib9" id="ref89">9</reflink>]) an important quality for classifying responses at higher levels of the validating assessment rubric. Various aspects of the research setting influenced our results, such as students'assumption-making and validation competencies elicited when responding to prompting questions. It may be possible to encourage a higher level of validating activity by incorporating more explicit prompts, such as requesting an examination of other potential parameters, initial conditions, and alternative assumptions on model predictions. Additionally, explicitly requesting modifications to the models could foster increased engagement. However, our findings also suggest that in data-rich contexts or data-driven modeling, merely asking students to validate may not suffice to achieve higher levels of validation. Engaging in validation during data-rich tasks requires students to recognize missing information or ambiguous task statements and then seek to address these gaps by adding information or making assumptions to resolve ambiguity. It appears that students'choices depend on their extra-mathematical knowledge, perceptions of task expectations, and general experiences with mathematical modeling. The task prompts asked students about their assumptions, leading to considerations of model fit (U.S. Coal Task) and integration of assumptions into their models (Stopping Distance Task). Neither task explicitly prompted students to revise their models based on validation results, which may explain why few students attempted model adjustments. Future studies might explore the effects of adding targeted follow-up prompts that directly request model modification or re-analysis based on students' own critiques, potentially encouraging deeper engagement. Exploring alternative prompts could focus students on revising models, although educators should reflect on the meaning of"modification"of a statistical model on a task-by-task basis and ensure that revision techniques are accessible to students.</p> <p>In our study, both tasks were characterized as data-rich modeling tasks. We find that the task attributes of the Stopping Distance Task facilitated both modeling the data and modeling the phenomenon (Stillman & Brown, [<reflink idref="bib40" id="ref90">40</reflink>]), thus allowing for a wider array of validating approaches compared to the U.S. Coal Task. We believe that data-rich tasks are a nexus for interaction among data collected from real-world systems, mathematical analysis, and statistics. Challenges in data modeling frequently unveil students'fundamental conceptual insights into statistical reasoning, encompassing aspects such as managing variation, data transformation, assessing statistical models, and amalgamating contextual and statistical elements within the problem (Wild & Pfannkuch, [<reflink idref="bib46" id="ref91">46</reflink>]). A purely data-driven approach encounters challenges in comprehensively considering all facets of modeling in engineering and science. Current trends seek a mixed approach, wherein data-driven modeling is informed by physical insights. Consequently, data-rich tasks can aid students in eliciting such modeling competency, particularly during the validating process. In the Stopping Distance Task, students' model choices were white box, grey box, or black box models, suggesting that this task offered opportunities to differentiate between modeling the phenomenon and modeling the data (Stillman & Brown, [<reflink idref="bib40" id="ref92">40</reflink>]). In contrast, responses to the U.S. Coal Task primarily demonstrated exclusively data-driven modeling approaches. This task provided students with comprehensive data about coal consumption, production, import, and export, enabling them to primarily engage in in-sample validation without reflecting on the model's suitability for real-world contexts. Another plausible explanation could be the unfamiliarity of the academic domain (economics), leading students to consider black box models predominantly. These interpretations align with Patel and Pfannkuch's ([<reflink idref="bib30" id="ref93">30</reflink>]) observation that students may neglect to conduct out-of-sample validation in data-rich modeling situations. However, this study suggests that insufficient data or ill-defined information may prompt students to seek additional data sets beyond the provided data, depending on their knowledge of theories in the academic domain (mechanical engineering or physics) to identify other parameters and conduct out-of-sample validation. While this validation aspect is relevant to data validity, ensuring that the data necessary for model building, evaluation, and testing are adequate and accurate (Sargent, [<reflink idref="bib35" id="ref94">35</reflink>]), it represents only the data-driven aspects of model validation. As prior studies showed, the students sometimes did not consider the whole data set or ignored some parts of the data and thus failed to test the model adequately (e.g., Jones et al., [<reflink idref="bib24" id="ref95">24</reflink>]; Türker Biber et al., [<reflink idref="bib43" id="ref96">43</reflink>]).</p> <p>We attribute the observed contrasts in student validation levels to two task characteristics: the encouragement of white box, black box, or grey box modeling approaches and academic domains. In the Stopping Distance Task, students' validation was linked to the real-world context, as evidenced by their considerations of road, weather, and tire conditions, and discussions on drivers' reaction times. Depending on students' extra-mathematical knowledge, some attempted out-of-sample validation using data not included in the dataset used to construct the model. In contrast, responses to the U.S. Coal Task were predominantly focused on modeling the provided dataset. With ample data about coal consumption, production, import, and export, all students opted for a data-driven approach and did not utilize out-of-sample information to reflect on the real-world context during their validation activity. Our findings further elaborate on Stillman and Brown's ([<reflink idref="bib40" id="ref97">40</reflink>]) observations that in data-rich task situations, students' choice of models and validation approaches with data hinge on task characteristics and features.</p> <p>Although we intentionally selected two tasks that differed in domain and structure to reflect a range of data-rich task characteristics, we acknowledge that the differences between the tasks may also contribute to the variation in student responses in ways unrelated to task design. For example, while the Stopping Distance Task invited both theory- and data-driven approaches, the U.S. Coal Task was strictly empirical and perhaps too narrowly scoped to prompt higher-order validation. Thus, while we interpret the variation as emerging from task characteristics, we cannot definitively exclude other explanatory variables, such as topic familiarity or perceived task expectation. One notable structural difference between the tasks involved the nature of the follow-up questions. Task 2 included explicit follow-up questions that guided students to engage with predictions and model fit, whereas Task 1 left more open-ended opportunities for model reflection. For instance, Task 2 prompted students with specific items such as "How would you measure the accuracy of the model?" and "How quickly was consumption and production decreasing in 2017?", which encouraged operational forms of validation but did not require students to revise or refine their models. In contrast, Task 1 asked students to "share the limitations of your model" and "list factors you did not consider," which invited more open-ended reflection, creating space for broader and more interpretive reflection on the modeling process. However, these structural differences did not necessarily translate into higher levels of validation. Despite Task 2's more directive prompts, students rarely attempted model refinement, possibly because the prompts stopped short of explicitly asking for it.</p> <p>Although our study focused on tertiary-level students, model validation is a competency that should be developed gradually throughout a student's education. In light of this, we recommend that future work explore how task design at the primary and secondary school levels can be structured to introduce foundational concepts of model validation. Early exposure to reasoning about assumptions, data fit, and real-world constraints could better prepare students to approach complex modeling in postsecondary STEM education and beyond. Such vertical alignment of modeling competencies would enhance students' ability to think critically about models across all levels of schooling.</p> <p>We conclude that the learning environments fostered by data-rich tasks have the potential to stimulate validating activity. However, we emphasize the importance of educators'deliberate consideration regarding the available techniques for validation. Such intentionality is essential in ensuring that the tasks effectively fulfill their intended purpose.</p> <hd id="AN0190589616-19">Authors'Contributions</hd> <p>Conceptualization, data collection, formal analysis, investigation, and writing-original draft: Joo young Park; Validation: Joo young Park, Jennifer Czocher, & Ryan White; Writing-review & editing: Joo young Park, Jennifer Czocher, & Ryan white.</p> <hd id="AN0190589616-20">Funding</hd> <p>No funding was received for conducting this study.</p> <hd id="AN0190589616-21">Data Availability</hd> <p>Compliance with Ethical Consent. Informed consent was obtained from all participants. Permissions were obtained from the appropriate ethics committees.</p> <hd id="AN0190589616-22">Code Availability (Software Application or Custom Code)</hd> <p>Not applicable.</p> <hd id="AN0190589616-23">Declarations</hd> <p></p> <hd id="AN0190589616-24">Conflicts of interest/Competing interests</hd> <p>All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.</p> <hd id="AN0190589616-25">Originality</hd> <p>We hereby certify that this chapter is not being submitted for consideration for publication elsewhere and is an original, unpublished work.</p> <hd id="AN0190589616-26">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0190589616-27"> <title> References </title> <blist> <bibl id="bib1" idref="ref25" type="bt">1</bibl> <bibtext> Andrade H, Du Y. Student perspectives on rubric-referenced assessment. Practical Assessment, Research & Evaluation. 2005; 10: 1-11. 10.1080/02602930801955986</bibtext> </blist> <blist> <bibl id="bib2" idref="ref26" type="bt">2</bibl> <bibtext> Chan CME, Ng KED, Widjaja W, Seto C. Assessment of primary 5 students' mathematical modelling competencies. Journal of Science and Mathematics Education in Southeast Asia. 2012; 35; 2: 146-178</bibtext> </blist> <blist> <bibl id="bib3" idref="ref85" type="bt">3</bibl> <bibtext> Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960; 20; 1: 37-46. 10.1177/001316446002000104</bibtext> </blist> <blist> <bibl id="bib4" idref="ref15" type="bt">4</bibl> <bibtext> Czocher, J. A. (2015). Competing conceptual systems and their impact on generating mathematical models. In K. Krainer & N. Vondrová (Eds.), Proceedings of the 9th Congress of European Research in Mathematics Education (pp. 841–847). CERME.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref55" type="bt">5</bibl> <bibtext> Czocher, J. A. (2016). Introducing modeling transition diagrams as a tool to connect mathematical modeling to mathematical thinking. Mathematical Thinking and Learning, 18(2), 77–106. https://doi.org/10.1080/10986065.2016.1148530</bibtext> </blist> <blist> <bibl id="bib6" idref="ref29" type="bt">6</bibl> <bibtext> Czocher, J. A. (2017). Mathematical modeling cycles as a task design heuristics. The Mathematics Enthusiast, 14(1–3), 129–140. https://scholarworks.umt.edu/tme/vol14/iss1/9</bibtext> </blist> <blist> <bibl id="bib7" idref="ref18" type="bt">7</bibl> <bibtext> Czocher, J. A. (2018). How does validating contribute to the modeling process. Educational Studies in Mathematics, 99(2), 137–159. https://doi.org/10.1080/10511970.2018.1472158</bibtext> </blist> <blist> <bibl id="bib8" idref="ref4" type="bt">8</bibl> <bibtext> Bautista A, Wilkerson-Jerde MH, Tobin RG, Brizuela BM. Mathematics teachers' ideas about mathematical models: A diverse landscape. PNA. 2014; 9; 1: 1-28. 10.30827/pna.v9i1.6107</bibtext> </blist> <blist> <bibl id="bib9" idref="ref14" type="bt">9</bibl> <bibtext> Blum W, Leiss DHaines C, Galbraith PL, Blum W, Khan S. How do students and teachers deal with mathematical modelling problems? The example sugarloaf and the DISUM project. Mathematical modelling: Education, engineering and economics. 2007; Horwood: 222-231. 10.1533/9780857099419.5.221</bibtext> </blist> <blist> <bibtext> Borromeo Ferri, R. (2006). Theoretical and empirical differentiations of phases in the modelling process. ZDM – Mathematics Education,38(2), 86–95.</bibtext> </blist> <blist> <bibtext> Brown J. Complexities of digital technology use and the teaching and learning of function. Computers and Education. 2015; 87: 112-122. 10.1016/j.compedu.2015.03.022</bibtext> </blist> <blist> <bibtext> Engel, J, & Kuntze, S. (2011). From data to functions: Connecting modelling competencies and statistical literacy. In G. Kaiser, W. Blum, R. Borromeo Ferri, & G. Stillman (Eds.), Trends in teaching and learning of mathematical modelling International perspectives on the teaching and learning of mathematical modelling (vol. 1, pp. 305–314). Springer. https://doi.org/10.1007/978-94-007-0910-2_39</bibtext> </blist> <blist> <bibtext> Frejd, P, & Bergsten, C. (2016). Mathematical modelling as a professional task. Educational Studies in Mathematics, 91(1), 11–35. https://doi.org/10.1007/s10649-015-9654-7</bibtext> </blist> <blist> <bibtext> Galbraith, P, & Stillman, G. (2006). A framework for identifying student blockages during transitions in the modelling process. ZDM – Mathematics Education,38, 143–162.</bibtext> </blist> <blist> <bibtext> Goos M, Galbraith P. Do it this way! Metacognitive strategies in collaborative mathematical problem solving. Educational Studies in Mathematics. 1996; 30; 3: 229-260. 10.1007/BF00304567</bibtext> </blist> <blist> <bibtext> Greefrath GStillman G, Blum W, Biembengut MS. Problem solving methods for mathematical modeling. Mathematical modelling in education research and practice: Cultural, social and cognitive influences ICTMA 16. 2015; Springer: 173-183. 10.1007/978-3-319-18272-8_13</bibtext> </blist> <blist> <bibtext> Greefrath G. Mathematical modelling competence: Selected current research developments. Avances De Investigación En Educación Matemática. 2020; 17: 38-51</bibtext> </blist> <blist> <bibtext> Greefrath, G, Kaiser, G, Blum, W, & Borromeo Ferri, R. (2013). Mathematisches Modellieren—Eine Einführung in theoretische und didaktische Hintergründe [Mathematical modeling – An introduction to theoretical and didactic background]. In R. Borromeo Ferri, G. Greefrath, & G. Kaiser (Eds.), Mathematisches Modellieren für Schule und Hochschule: Theoretische und didaktische Hintergründe (pp. 11–37). Springer Spektrum.</bibtext> </blist> <blist> <bibtext> Grover S, Pea R. Computational thinking in K-12: A review of the state of the field. Educational Researcher. 2013; 42; 1: 38-43. 10.3102/0013189X12463051</bibtext> </blist> <blist> <bibtext> Grunewald, S. (2013). The development of modelling competencies by year 9 students: Effects of a modelling project. In G. Stillman, G. Kaiser, W. Blum, & J. Brown (Eds.), Teaching mathematical modelling: Connecting to research and practice (pp. 185–194). Springer. https://doi.org/10.1007/978-94-007-6540-5_16</bibtext> </blist> <blist> <bibtext> Hallström, J, & Schönborn, K. J. (2019). Models and modelling for authentic STEM education: Reinforcing the argument. International Journal of STEM education, 6, Article 22. https://doi.org/10.1186/s40594-019-0178-z</bibtext> </blist> <blist> <bibtext> Ishibashi, I, & Uegatani, Y. (2022). Cultural relevance of validation during mathematical modeling and word problem-solving: Reconceptualizing validation as an integration of possible fictional worlds. Journal of Mathematical Behavior, 66, 100934. https://doi.org/10.1016/j.jmathb.2022.100934</bibtext> </blist> <blist> <bibtext> Jonassen DH. Toward a design theory of problem solving. Educational Technology Research and Development. 2000; 48; 4: 63-85. 10.1007/BF02300500</bibtext> </blist> <blist> <bibtext> Jones C, Carvalho L, Higgins W, Waliser D, Schemm J. A statistical forecast model of tropical intraseasonal convective anomalies. Journal of Climate. 2004; 17; 10: 2078-2095. 10.1175/1520-0442(2004)017<2078:ASFMOT>2.0.CO;2</bibtext> </blist> <blist> <bibtext> Kaiser GCai J. The teaching and learning of mathematical modeling. Compendium for research in mathematics education. 2017; National Council of Teachers of Mathematics: 267-291</bibtext> </blist> <blist> <bibtext> Karpatne A, Alturi G, Faghmous JH, Steinbach A, Banerjee A, Ganguly A, Shekhar S, Samatova N, Kumar V. Theory-guided data science: A new paradigm for scientific discovery from data. IEEE Transactions on Knowledge and Data Engineering. 2017; 29: 2318-2331. 10.1109/TKDE.2017.2720168</bibtext> </blist> <blist> <bibtext> Lesh R, Zawojewski JSLester F. Problem solving and modeling. Second handbook of research on mathematics teaching and learning. 2007; Information Age Publishing: 763-802</bibtext> </blist> <blist> <bibtext> Maaß K. A classification scheme for modelling tasks. Journal Für Mathematik-Didaktik. 2010; 31; 2: 285-311. 10.1007/s13138-010-0010-2</bibtext> </blist> <blist> <bibtext> Montáns F, Chinesta F, Gómez-Bombarelli R, Kutz J. Data-driven modeling and learning in science and engineering. Comptes Rendus Mécanique. 2019; 347; 11: 845-855. 10.1016/j.crme.2019.11.009</bibtext> </blist> <blist> <bibtext> Patel, A, & Pfannkuch, M. (2018). Developing a statistical modeling framework to characterize Year 7 students' reasoning. ZDM – Mathematics Education,50, 1197–1212. https://doi.org/10.1007/s11858-018-0960-2</bibtext> </blist> <blist> <bibtext> Peck SL. Simulation as experiment: A philosophical reassessment for biological modeling. Trends in Ecology & Evolution. 2004; 19; 10: 530-534. 10.1016/j.tree.2004.07.019</bibtext> </blist> <blist> <bibtext> Pfannkuch, M, Budgett, S, Fewster, R, Fitch, M, Pattenwise, S, Wild, C, & Ziedins, I. (2016). Probability modeling and thinking: What can we learn from practice? Statistics Education Research Journal, 15(2), 11–37.</bibtext> </blist> <blist> <bibtext> Pólya, G. (1973). How to solve it: A new aspect of mathematical method. Princeton University Press.</bibtext> </blist> <blist> <bibtext> Rykiel EJ. Testing ecological models: The meaning of validation. Ecological Modelling. 1996; 90: 229-244. 10.1016/0304-3800(95)00152-2</bibtext> </blist> <blist> <bibtext> Sargent, R. (1984). A tutorial on verification and validation of simulation models. In S. Sheppard, U. Pooch, & D. Pegden (Eds.), Proceedings of the winter simulation conference (pp. 115–122). IEEE. https://doi.org/10.1109/WSC.1984.7453877</bibtext> </blist> <blist> <bibtext> Schlesinger, S, Crosbie, R. E, Gagne, R. E, Innis, G. S, Lalwani, C. S, Loch, J, Sylvester, R. J, Wright, R. D, Kheir, N, & Bartos, D. (1979). Terminology for model credibility. Simulation, 32(3), 103–104. https://doi.org/10.1177/003754977903200304</bibtext> </blist> <blist> <bibtext> Schoenfeld AH. Mathematical problem solving. 1985; Academic Press</bibtext> </blist> <blist> <bibtext> Schukajlow S, Krug A. Do multiple solutions matter? Prompting multiple solutions, interest, competence, and autonomy. Journal for Research in Mathematics Education. 2014; 45; 4: 497-533. 10.5951/jresematheduc.45.4.0497</bibtext> </blist> <blist> <bibtext> Stillman, G. (2011). Applying metacognitive knowledge and strategies in applications and modelling tasks at secondary school. In G. Kaiser, W. Blum, R. Borromeo Ferri, & G. Stillman (Eds.), Trends in teaching and learning of mathematical modelling (pp. 165–180). Springer.</bibtext> </blist> <blist> <bibtext> Stillman, G, & Brown, J. (2023). Modeling the phenomenon versus modeling the data set. Mathematical Thinking and Learning, 25(3), 270–295. https://doi.org/10.1080/10986065.2021.2013144</bibtext> </blist> <blist> <bibtext> Stillman G, Galbraith P. Applying mathematics with real world connections: Metacognitive characteristics of secondary students. Educational Studies in Mathematics. 1998; 36; 2: 157-195. 10.1023/A:1003246329257</bibtext> </blist> <blist> <bibtext> Tekin-Dede, A, & Bukova-Güzel, E. (2018). A rubric development study for the assessment of modeling skills. The Mathematics Educator, 27(2), 33–72.</bibtext> </blist> <blist> <bibtext> Türker Biber, B, Yetkin Özdemir, İE, & Lesh, R. (2023). Teacher noticing of students' thinking in the context of mathematical modeling activities related to statistics. International Journal of Science and Mathematics Education,21(6), 1797–1818. https://doi.org/10.1007/s10763-022-10328-9</bibtext> </blist> <blist> <bibtext> Velten, K. (2009). Mathematical modeling and simulation: Introduction for scientists and engineers. Wiley-VCH.</bibtext> </blist> <blist> <bibtext> Vorhölter, K. (2018). Conceptualization and measuring of metacognitive modelling competencies: Empirical verification of theoretical assumptions. ZDM – Mathematics Education,50, 343–354. https://doi.org/10.1007/s11858-017-0909-x</bibtext> </blist> <blist> <bibtext> Wild CJ, Pfannkuch M. Statistical thinking in empirical enquiry. International Statistical Review. 1999; 67: 223-265. 10.1111/j.1751-5823.1999.tb00442.x</bibtext> </blist> <blist> <bibtext> Wing JM. Computational thinking. Communications of the ACM. 2006; 49; 3: 33-35. 10.1145/1118178.1118215</bibtext> </blist> <blist> <bibtext> Zbiek RM, Conner A. Beyond motivation: Exploring mathematical modeling as a context for deepening students' understandings of curricular mathematics. Educational Studies in Mathematics. 2006; 63; 1: 89-112. 10.1007/s10649-005-9002-4</bibtext> </blist> </ref> <aug> <p>By Jooyoung Park; Jennifer A. Czocher and Ryan T. White</p> <p>Reported by Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib29" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib26" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib40" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib44" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib32" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib19" firstref="ref9"></nolink> <nolink nlid="nl7" bibid="bib47" firstref="ref10"></nolink> <nolink nlid="nl8" bibid="bib12" firstref="ref11"></nolink> <nolink nlid="nl9" bibid="bib39" firstref="ref16"></nolink> <nolink nlid="nl10" bibid="bib22" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib45" firstref="ref20"></nolink> <nolink nlid="nl12" bibid="bib21" firstref="ref21"></nolink> <nolink nlid="nl13" bibid="bib48" firstref="ref24"></nolink> <nolink nlid="nl14" bibid="bib28" firstref="ref27"></nolink> <nolink nlid="nl15" bibid="bib25" firstref="ref28"></nolink> <nolink nlid="nl16" bibid="bib30" firstref="ref33"></nolink> <nolink nlid="nl17" bibid="bib13" firstref="ref39"></nolink> <nolink nlid="nl18" bibid="bib33" firstref="ref42"></nolink> <nolink nlid="nl19" bibid="bib37" firstref="ref43"></nolink> <nolink nlid="nl20" bibid="bib27" firstref="ref45"></nolink> <nolink nlid="nl21" bibid="bib41" firstref="ref47"></nolink> <nolink nlid="nl22" bibid="bib10" firstref="ref48"></nolink> <nolink nlid="nl23" bibid="bib15" firstref="ref49"></nolink> <nolink nlid="nl24" bibid="bib20" firstref="ref51"></nolink> <nolink nlid="nl25" bibid="bib38" firstref="ref52"></nolink> <nolink nlid="nl26" bibid="bib34" firstref="ref58"></nolink> <nolink nlid="nl27" bibid="bib35" firstref="ref59"></nolink> <nolink nlid="nl28" bibid="bib36" firstref="ref61"></nolink> <nolink nlid="nl29" bibid="bib31" firstref="ref63"></nolink> <nolink nlid="nl30" bibid="bib23" firstref="ref64"></nolink> <nolink nlid="nl31" bibid="bib11" firstref="ref66"></nolink> <nolink nlid="nl32" bibid="bib17" firstref="ref67"></nolink> <nolink nlid="nl33" bibid="bib14" firstref="ref75"></nolink> <nolink nlid="nl34" bibid="bib18" firstref="ref76"></nolink> <nolink nlid="nl35" bibid="bib16" firstref="ref77"></nolink> <nolink nlid="nl36" bibid="bib42" firstref="ref78"></nolink> <nolink nlid="nl37" bibid="bib46" firstref="ref91"></nolink> <nolink nlid="nl38" bibid="bib24" firstref="ref95"></nolink> <nolink nlid="nl39" bibid="bib43" firstref="ref96"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1501315
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Exploring STEM Students' Model Validation across Two Data-Rich Task Environments
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Jooyoung+Park%22">Jooyoung Park</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-0427-4706">0000-0003-0427-4706</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jennifer+A%2E+Czocher%22">Jennifer A. Czocher</searchLink><br /><searchLink fieldCode="AR" term="%22Ryan+T%2E+White%22">Ryan T. White</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Science+and+Mathematics+Education%22"><i>International Journal of Science and Mathematics Education</i></searchLink>. 2025 23(8):3181-3204.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Validity%22">Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Analysis%22">Data Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Competence%22">Competence</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s10763-025-10580-9
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1571-0068<br />1573-1774
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Proficiency in data analysis and modeling is increasingly essential across science, technology, engineering, and mathematics (STEM) disciplines. A critical component of these skills is model validation--assessing how well a model's predictions align with empirical data and the theoretical assumptions underlying the model. This study examines the model validation competency of undergraduate STEM students as they engaged with two data-rich modeling tasks situated in different academic domains. Students employed a range of modeling strategies, most commonly developing black-box or white-box models, with some constructing grey-box models that integrated data and theoretical reasoning. The findings suggest that students' validation approaches were shaped by specific features of the tasks, such as the availability of data and the degree of ambiguity in the problem context. The study highlights how data-rich tasks can stimulate validation activities, underscoring the importance of educators' deliberate selection of validation techniques to ensure tasks effectively achieve their intended objectives.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1501315
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1501315
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10763-025-10580-9
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 3181
    Subjects:
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Validity
        Type: general
      – SubjectFull: Data Analysis
        Type: general
      – SubjectFull: Competence
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
    Titles:
      – TitleFull: Exploring STEM Students' Model Validation across Two Data-Rich Task Environments
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Jooyoung Park
      – PersonEntity:
          Name:
            NameFull: Jennifer A. Czocher
      – PersonEntity:
          Name:
            NameFull: Ryan T. White
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1571-0068
            – Type: issn-electronic
              Value: 1573-1774
          Numbering:
            – Type: volume
              Value: 23
            – Type: issue
              Value: 8
          Titles:
            – TitleFull: International Journal of Science and Mathematics Education
              Type: main
ResultId 1