View in EDS HTML Full Text PDF Full Text

Assessing the Degree of Ecological Validity of Your Study: Introducing the Multidimensional Assessment of Research in Context (MARC) Tool

Saved in:
Bibliographic Details
Title: Assessing the Degree of Ecological Validity of Your Study: Introducing the Multidimensional Assessment of Research in Context (MARC) Tool
Language: English
Authors: Naumann, Sandra (ORCID 0000-0001-8992-2930), Byrne, Michelle L. (ORCID 0000-0002-4180-8095), de la Fuente, Alethia (ORCID 0000-0002-5689-1709), Harrewijn, Anita (ORCID 0000-0001-9938-4048), Nugiel, Tehila (ORCID 0000-0001-5547-1852), Rosen, Maya (ORCID 0000-0002-4649-9965), van Atteveldt, Nienke (ORCID 0000-0002-3387-6151), Matusz, Pawel J. (ORCID 0000-0003-1721-4868)
Source: Mind, Brain, and Education. Aug 2022 16(3):228-238.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 11
Publication Date: 2022
Document Type: Journal Articles
Reports - Descriptive
Tests/Questionnaires
Descriptors: Evaluation Methods, Neurosciences, Educational Research, Validity, Research Methodology
DOI: 10.1111/mbe.12318
ISSN: 1751-2271
1751-228X
Abstract: In cognitive neurosciences, fundamental principles of mental processes and functional brain organization have been established with highly controlled tasks and testing environments. Recent technical advances allowed the investigation of these functions and their brain mechanisms in naturalistic settings. The diversity in those approaches have been recently (Matusz, P. J., Dikker, S., Huth, A. G., & Perrodin, C. (2019). Are we ready for real-world neuroscience? "Journal of Cognitive Neuroscience," 31(3), 327-338. doi:10.1162/jocn_e_01276) classified via a three-category cycle, including "controlled laboratory," "partially naturalistic laboratory," and "naturalistic real-world research." Based on this model, we developed the Multidimensional Assessment of Research in Context (MARC) tool to easily delineate the approach researchers have taken in their study. MARC provides means to describe the degree of ecological validity for each component of a study (e.g., sample, stimuli, measures, etc.), and the study's location on the cycle. The tool comprises seven questions concerning a study's characteristics. It outputs a summary of those and a compass plot, which can be used for presentations, preregistration, grant proposals, and papers. It aims to improve drawing conclusions across studies and raise awareness about the generalizability of research findings.
Abstractor: As Provided
Entry Date: 2022
Accession Number: EJ1343744
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFwac0lcwrE1F95NAO6PidkAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCz_Qs8ShACeOIcAfAIBEICBmw0Zmyvc4rtHvjkxuUfHd07m9VD8UIZopocbjPcBL38-CDnNLTMuQK8h2gGtloo_tsZdYx1pFYAdZmT_yWXpwA63D9NxHCtcZqg8Nxs76L1cfpWmCbDKjPx_EDyWF_rPExRlGpjigC30UWwaq9iAVQeRe7mhhCwZ9MIvNutAcLUpV9Mp8-7WXD8ltvbW8XOKMN8JiDm5-Y2bif17
Text:
  Availability: 1
  Value: <anid>AN0158316337;[309x]01aug.22;2022Aug04.05:55;v2.2.500</anid> <title id="AN0158316337-1">Assessing the Degree of Ecological Validity of Your Study: Introducing the Multidimensional Assessment of Research in Context (MARC) Tool </title> <p>In cognitive neurosciences, fundamental principles of mental processes and functional brain organization have been established with highly controlled tasks and testing environments. Recent technical advances allowed the investigation of these functions and their brain mechanisms in naturalistic settings. The diversity in those approaches have been recently (Matusz, P. J., Dikker, S., Huth, A. G., & Perrodin, C. (2019). Are we ready for real‐world neuroscience? Journal of Cognitive Neuroscience, 31(<reflink idref="bib3" id="ref1">3</reflink>), 327–338. doi:10.1162/jocn_e_01276) classified via a three‐category cycle, including controlled laboratory, partially naturalistic laboratory, and naturalistic real‐world research. Based on this model, we developed the Multidimensional Assessment of Research in Context (MARC) tool to easily delineate the approach researchers have taken in their study. MARC provides means to describe the degree of ecological validity for each component of a study (e.g., sample, stimuli, measures, etc.), and the study's location on the cycle. The tool comprises seven questions concerning a study's characteristics. It outputs a summary of those and a compass plot, which can be used for presentations, preregistration, grant proposals, and papers. It aims to improve drawing conclusions across studies and raise awareness about the generalizability of research findings.</p> <p>We developed a questionnaire—the Multidimensional Assessment of Research in Context (MARC) tool—to assess the ecological validity of psychological and neuroscientific studies (the degree to which they capture the real‐world behavior they aim to capture). MARC enables researchers to explicitly report the level of ecological validity of each aspect of their study by answering seven questions, for example, about the task they used or the involvement of nonresearch experts. The use of MARC should improve transparency, result interpretation, and theory development.</p> <p>Cognitive neuroscience has taught us a lot about cognitive functions and how these are represented in the brain. Most of these studies, especially the early ones, have used basic computerized tasks and simple stimuli to exert as much experimental control over the studied process as possible. This type of research has provided a lot of detailed information about cognitive processes and their underlying brain mechanisms, but it also has certain drawbacks. One of the main challenges in cognitive neuroscience is the low ecological validity of paradigms for a range of behaviors and cognitive processes naturally occurring typically outside the laboratory‐like contexts (Dziobek, 2012; Matusz, Dikker, Huth, & Perrodin, 2019; Shamay‐Tsoory & Mendelsohn, 2019; van Atteveldt, van Kesteren, Braams, & Krabbendam, 2018), limiting the interpretation of the results with regard to real‐life functioning. Our working definition of ecological validity is a quality of capturing the specific real‐world behavioral and environmental factors a study aims to understand (Dunlosky, Bottiroli, & Hartwig, 2009; Holleman et al., 2020). Low ecological validity could be related to both person‐dependent factors (the limited active role of the participants in lab‐based paradigms may e.g., interfere with their sense of agency and with the embodiment of their information processing) and situation‐dependent factors (artificial, decontextualized environments may not represent real‐world interactions) (for more in‐depth discussion, see Shamay‐Tsoory & Mendelsohn, 2019).</p> <p>Brunswik (1943, 1955) proposed that to achieve generalizability of results from psychological studies, stimuli and tasks should be sampled just like participants are sampled in psychological studies, that is, in a way that represents the distribution and intercorrelations of ecological variables in the real‐world. Notably, he defined this quality of a study to reflect the variability present in the outside world as "representative design." Brunswick was de facto the originator of the term "ecological validity." However, his definition of ecological validity (Brunswik, 1955) was not only much more precise than the currently existing, more or less explicit, definitions of this construct. It also described a different construct to the one invoked by this term today. Namely ecological validity according to Brunswik is a relationship between a proximal cue delivered to the senses (e.g., stimulation on the retina) and a distal, present in the outside environment, object. Thus, the term describes the potential utility to the organism of a given cue in achieving its behavioral goals. As such, Brunswik's definition of ecological validity is unlike the modern definitions where the term is used interchangeably with "real‐world" research.</p> <p>What Brunswick also recognized, and what has attracted a lot of interest in psychological research in recent years, is that the external environment is uncertain, which has a bearing on some cues being more relevant to one's behavioral goals than others in a given context. The psychological and cognitive neuroscience research has built on this realization. It recognized the importance of information uncertainty as well as the fact that the utility of a given piece of information can be relative. Namely, relevant dimensions of the environment ("signal") are intertwined with the nonrelevant ones ("noise"). Recent accounts point out that to guide behavior effectively, the brain needs to continue actively weighing and reweigh particular dimensions rather than outright ignore the nonrelevant ones (Nastase, Goldstein, & Hasson, 2020). Our brains have been shaped to utilize this multidimensionality, which quality may be critical to emulate to understand many behaviors. However, it has not been consistently recognized throughout the history of cognitive research, thus limiting the generalizability of at least some of this work. This has been recently changing. Nastase et al. (2020) has emphasized the added value of researchers identifying explicitly those manipulations that characterize the boundary conditions of behaviors that occur in the environments outside of the laboratory setting. Notably, as Holleman et al. (2020) has recently pointed out, the behavior of interest should always be specified with regard to its context of interest, as opposed to vague terms of "real‐life," "ecological validity," or "naturalistic." As an example, instead of studying "real‐world social attention," researchers should aim to study attention in a situation of baking a cake, of sharing a meal, or of waiting in a waiting room. Only studies within such a well‐defined context can shed light on context‐specific and context‐generic processes governing attention, and other mental processes, in social situations (Holleman et al. 2020). In other words, even a behavior of attending to stimuli on a screen, in a laboratory setting, with limited number and variability of stimuli and task demands, might be characterized by high ecological validity. If one is interested in processes governing attentive behavior in such a setting, those settings will constitute ecologically valid settings.</p> <p>Whichever processes and contexts are of interest, ecological validity can be assumed to be particularly low in neuroimaging studies, compared to behavioral experimental studies, because of the lengthy, highly controlled tasks and stimuli, as well as the artificial and isolated environment, such as an Magnetic Resonance Imaging (MRI) facility, in which the testing takes place (van Atteveldt et al., 2018). Van Atteveldt et al. (2018) describe four approaches to increase the ecological validity of neuroimaging studies. One approach focuses on using more naturalistic tasks and stimuli, such as videos and social interactions. Such stimuli may better reflect some of the behaviors in the outside world, compared to the stimuli in highly controlled lab experiments, as they typically have meaning and vary in the senses they engage from moment to moment. A second approach involves moving the research to environments outside the laboratory by using portable neuroimaging devices, such as electroencephalography (EEG), functional near‐infrared spectroscopy, or wearable technology. A third approach focuses on combining tightly controlled lab‐based neuroimaging measurements with real‐life variables and follow‐up studies conducted "in the field," for example, in the classroom, for those studies where behaviors and cognition inside the classroom are of interest. Lastly, one can improve the ecological validity of neuroimaging studies by including stakeholders (e.g., teachers and students in the case of studies on learning and education), and doing so at most or all stages of the research (van Atteveldt, Tijsma, Janssen, & Kupper, 2019). All of these approaches help to bring the research closer to understanding information processing and the involved brain mechanisms in everyday environments, as well as defining relevant research questions. Finally, from a more analytical viewpoint, Nastase et al. (2020) has suggested that the ecological validity of investigations into some behaviors can be improved by formalizing hypotheses as explicit models that can offer quantitative predictions of neural activity under the most naturalistic conditions (for those well‐defined behaviors) that are possible, and using findings to generate new predictions tested in naturalistic or more controlled contexts.</p> <p>Recent technical advances, such as increased computational power and better brain mapping tools, have actually provided researchers with the opportunity to more efficiently analyze data from paradigms and experiments in which behaviors occurring outside the laboratory settings are of interest (Bevilacqua et al., 2019; Rosenblau, O'Connell, Heekeren, & Dziobek, 2019; Vanderwal, Eilbott, & Castellanos, 2019). Indeed, many recent cognitive neuroscience studies are now starting to use paradigms reflecting and emulating behaviors occurring outside the laboratory (e.g., Föcker, Mortazavi, Khoe, Hillyard, & Bavelier, 2019; Matusz, Turoman, Tivadar, Retsa, & Murray, 2019; Peelen & Kastner, 2014; Vanderwal et al., 2019), as compared to the studies that first pioneered the field, using limited ranges of stimuli and tasks. The former represent different approaches aimed at increasing ecological validity of investigations of behaviors more characteristics of outside laboratory settings: by making use of dynamic stimuli, such as naturalistic movies (e.g., Vanderwal et al., 2019), or audio‐visual objects varied in task‐relevance, in multistimulus displays (Alsius & Soto‐Faraco, 2011; Cavallina, Puccio, Capurso, Bremner, & Santangelo, 2018; Matusz, Turoman, et al., 2019; Turoman et al., 2020a, 2020b); or tasks where attention changes dynamically between focused versus divided attention (Föcker et al., 2019) or of real‐world scenes (Peelen & Kastner, 2014); or even by studying how brain research impacts perceptions of adolescents and their parents (Altikulaç et al., 2019). These recent studies are valuable as they help bridge research on behaviors characteristics of the environments outside the laboratory, like on the high‐street or inside the classroom, across more traditional approaches and those involving the study of these behaviors in veridical external environments (Matusz, Dikker, et al., 2019; Nastase et al., 2020).</p> <hd id="AN0158316337-2">ASSESSING ECOLOGICAL VALIDITY</hd> <p>Now that behaviors and cognitive processes are being studied with paradigms that vary in the amount of control over the stimuli as well as over the environment in which they are studied, it would be helpful to explicitly report the levels of this control and 'naturalness' that researchers chose for their study. Explicitly reporting these qualities for stimuli, task, population sampling, etc., in the study takes the burden off the reader, as it is immediately clear what type of research it is and where the study is positioned in the field. In addition, such assessment improves comparing results of different studies, drawing conclusions based on these studies, and identifying gaps for future research. Furthermore, assessing the level of control and the 'naturalness' of their own work encourages researchers to think about them at the design stage of their future studies, and make them generally more aware of the extent to which the experimental setup they have designed offers high levels of ecological validity for the behavior they are interested in studying. These qualities have the potential to increase the ease of interpretation and so the applicability of cognitive neuroscience studies overall. Therefore, we developed the Multidimensional Assessment of Research in Context (MARC) tool to assess the ecological validity of each component part (sample, stimuli, setting, measures, stakeholders, etc.), of psychological and neuroscientific studies. This tool can easily be used for communication among researchers and with reviewers, preregistration, grant proposals, papers, and meta‐analyses.</p> <p>We based the MARC tool on a conceptualization of neuroscientific investigations as a three‐category cycle, as proposed by Matusz et al. (2019). They argue that neurocognitive processes of interest can be studied using three different approaches that complement one another and are each of equal importance: a controlled laboratory research approach, a partially naturalistic laboratory research approach, and a naturalistic real‐world research approach. In the controlled laboratory research approach, the process of interest is studied in the lab, where it is isolated by manipulating only a minimum number of factors in a specific experimental design, and all other factors are held constant. This approach provides maximal control over stimuli and environment, which enables testing hypotheses about behaviors and cognitive processes in specific and highly defined contexts, and doing so with maximized statistical power. In the partially naturalistic laboratory research approach, these process‐specific tasks are still administered in lab‐like settings but where those settings resemble everyday situations in which many behaviors may occur. This could be done by selecting different stimuli (e.g., naturalistic movies, multisensory stimuli, or including goal‐irrelevant distractors), task conditions (e.g., dynamically changing task difficulty and familiarity, or giving the impression that the participant is being watched by a peer), and/or lab designs (e.g., a lab that is set up to look like a classroom, or virtual reality). This approach provides a closer approximation of how the cognitive and/or socio‐emotional process of interest might operate in the real world, while maintaining a certain level of control over it and the contexts within which it is gauged. The results of experiments carried out within this partially naturalistic laboratory approach show how well the hypotheses developed within simplified tasks and with simple stimuli hold in contexts more resembling the real‐world. In the naturalistic real‐world research approach, the process of interest is measured in real‐world situations. This approach enables direct testing of the extent to which lab‐generated models hold outside traditional laboratory investigations. It likewise allows researchers to uncover new mechanisms supporting cognitive functions in everyday situations or new factors modulating those functions (Matusz, Dikker, et al., 2019). All three of these categories are of critical importance to the goal of creating more ecologically valid research. Studying a process of interest across all three approaches is important because only together they can provide a more complete understanding of the process of interest and generate hypotheses for its further investigations.</p> <p>We want to emphasize here that, in line with our working definition, the controlled laboratory research approach can offer "ecologically valid" evidence regarding a behavior of interest, as it is really the nature of that behavior that determines whether a given context is "ecologically valid." Consequently, studies of behaviors in the environments and situations outside the laboratory do not automatically constitute more ecologically valid contexts for investigation of any behaviors, for example, looking through a person's pictures on social media will not be forcibly instantly studied in more naturalistic fashion if done outside the laboratory. In this context, the MARC tool can be used to assess whether the study represents a more controlled laboratory approach, partially naturalistic laboratory approach, or more naturalistic approach, but without making a judgment about the study's ecological validity.</p> <p>MARC consists of a set of seven questions about the design, tasks, stimuli, measures, participant sampling, and stakeholder involvement that will be answered by the researcher. For each question, the researcher self‐assesses if this part of the study is best described as a 'controlled laboratory‐based', 'partially naturalistic', or 'naturalistic' approach. The tool provides examples of answers that would reflect one of these three approaches. Explicit description of key aspects of a study has the potential to improve comparisons between the study in question and other similar research, which in turn should lead to more ecologically valid research (Schmuckler, 2001). We propose that researchers include the results of MARC in their presentations, preregistration, grant proposals, meta‐analyses, and/or as part of the Methods section of their manuscripts, and this way allow others to understand more easily what category their study belongs to.</p> <hd id="AN0158316337-3">WHY VARIETY IN CATEGORIES OF ECOLOGICAL VALIDITY MATTER IN ADOLESCENT RISK‐TAKING RESEARCH</hd> <p>Choosing the components and tasks that are most appropriate for any given research question requires the researcher to pay special attention to the ecological validity and the construct validity. Different approaches and testing environments are appropriate for different research questions. Consider a construct that has been of great interest in the field of developmental cognitive neuroscience for over two decades: adolescent risk‐taking (e.g., Casey et al., 1997; Galvan et al., 2006; Steinberg, 2005). While one is ultimately interested in understanding the causes and consequences of adolescent risky behavior in the real‐world, there are advantages of studying predictors and outcomes related to risk‐taking in a controlled laboratory, partially naturalistic, and fully naturalistic manner. One task that is often used in studies of risk‐taking is the Balloon Analog Risk Taking (BART) task, which involves having participants inflate a balloon to earn points. But the more they pump the balloon the greater risk they take of the balloon popping and losing all of their points (LeJuez et al., 2002). This task is administered in a controlled laboratory setting and experimenters can manipulate the parameters to make the task more or less risky. This allows a high level of control when considering the implications of the results. However, how much is this controlled task 1) representative of the specific real‐world behavior researchers are interested in (ecological validity), and 2) correlated with other predictors or indicators of that same behavior (convergent validity, or more broadly, construct validity)? We can also measure adolescent risk‐taking in a partially naturalistic manner by asking participants directly about their real‐world behavior (for example, through self‐ or parent‐reports about unsafe levels of alcohol consumption or crossing red traffic lights; e.g., Domain‐Specific Risk‐Taking Scale or DOSPERT, Figner & Weber, 2011). We note that although questionnaires may be less naturalistic measures of behavior than actual behavior measured in observational or experimental studies, they are still useful measures of real‐world 'behavior' (while the limitations characterizing inference from self‐report to behavior, such as demands on introspection, should be always kept in mind). This is especially the case in situations where questionnaires or self‐reports, as sparse measures of real‐life behavior, are utilized to improve the ecological validity of neural measures of real‐life behavior by assessing and improving an association between the two (for details, see section 5.1 in van Atteveldt et al., 2018).</p> <p>Critically, risky behavior on the BART task has a low‐ to medium‐strength link (r =0.243) to risky real‐world behavior related to motor vehicle safety (Vaca et al., 2013), suggesting that this task may also have a high level of construct validity for that particular risk behavior. In other words, the task produces observed behavior in the laboratory that correlates well with observed real‐world behavior outside of the lab; therefore, the task appears to measure what it was designed to measure (high construct validity). In this case, tasks that are more naturalistic also have a higher level of construct validity if the construct we are interested in is real‐world behavior. We usually want our tasks to correlate well with real‐world behavior, to predict and prevent adverse outcomes in real‐life. Laboratory tasks can be made more naturalistic by adding components such as peer presence. Recent work has explored the impact of peers and parents on adolescent risk‐taking behavior and neural responses in the Stoplight task (Chein, Albert, O'Brien, Uckert, & Steinberg, 2011), or the adapted Yellow Light Game (Op de Macks et al., 2018; van Hoorn, McCormick, Rogers, Ivory, & Telzer, 2018), in which participants play a simulated driving game and must decide whether to complete the game faster by speeding through yellow lights at the risk of crashing or stopping. Finally, adolescent risk‐taking behavior can be explored in an even more naturalistic manner outside of the lab through the use of ecological momentary assessment (EMA), which uses mobile devices through which individuals report about their behavior, which could include risk‐taking behavior, and emotions in real‐time (Kenny, Dooley, & Fitzgerald, 2016). To capitalize on multi‐method approaches, some adolescent risk‐taking studies have used factor analysis to combine multiple indicators, including several self‐report measures and behavioral tasks, of the risk‐taking construct (Harden et al., 2017), with results suggesting the need for further multi‐method assessments of psychological constructs.</p> <p>These examples serve to illustrate that cognitive psychology and/or neuroscience research benefits from diversity in the level of ecological validity (i.e., a spectrum of study design from high control to more naturalistic settings) of tasks and materials. The controlled laboratory research approach (in our example, the BART) provides maximal control over stimuli and environment, which here enabled the testing of specific and highly detailed hypotheses with maximized statistical power. The naturalistic laboratory research approach (in our example, the Stoplight task or the Yellow Light Game) provided a closer approximation of how the cognitive and/or socio‐emotional process of interest might operate in the real‐world, while maintaining a certain level of control over the stimuli and the environment. The naturalistic real‐world research approach (in our example, the EMA) enabled direct testing of the extent to which lab‐generated models hold outside traditional laboratory investigations. MARC is a tool to allow researchers to consider and clearly justify where their study (or various study components) lies in terms of ecological validity, all the way from the design to reporting stages.</p> <hd id="AN0158316337-4">HOW TO USE THE MARC TOOL</hd> <p>You can use MARC at any point in time in the process of conducting your research ‐ when writing a preregistration, before beginning data collection, or when submitting a manuscript. The tool can be found online at: https://marcform‐git.herokuapp.com/; (Source code is available at https://github.com/Lauralethia/MARCWeb; compute.py is the script to compute the scores and make the graph). When you begin, think of responding to the MARC questions as for a single manuscript. With that in mind proceed through each question. Each question is accompanied by sets of examples that fall into three categories: controlled, partially naturalistic, and naturalistic. Use these examples as a guide to help you categorize each component of your study. Please note that many projects will consist of components that fall on different points along the controlled, partially naturalistic to naturalistic research approach continuum. If you have multiple components for a particular question (e.g., more than one task), simply enter the number of components for that particular section (e.g., using multiple different stimuli, or carrying the study multiple contexts, for example, the laboratory and in the "field") and answer the question separately for each component. Be sure to accompany your response with a justification–description of each of your components in the space allotted.</p> <p>The MARC tool allows researchers to answer a series of questions about different aspects of their study (e.g., task, stimuli). When they finish filling out the questionnaire, MARC will produce a compass plot that consists of a "triangle" created by scores along the three axes on a circle (Figure 1). The header of the plot informs about the behavior of interest as well as of the context researchers want to generalize to; the footer includes detailed information as to which aspect of the study contributed to which of the three axes. Each point on the circle represents the extent (here: proportion) to which one's study encompasses each of the three categories (controlled, partially naturalistic, naturalistic). For example, a study could be 75% controlled and 25% partially naturalistic or 60% partially naturalistic, 20% naturalistic, and 20% controlled. The orange triangle is drawn by connecting the factor loadings for each category. The balance score (BS) is the normalized area of the triangle, ranging from 0 to 1. Then, a study designed to focus specifically on one factor–category will have a BS of 0, while a study designed to equally balance across all three categories will have a BS of 1. We stress that a BS value is not a judgment, that is, it does not have a 'good' or 'optimum' value. Rather, it represents the balance between the categories of the cycle, providing evidence for the study being more focused on any one category from the 3‐category model (Matusz, Dikker, et al., 2019) versus it being a more balanced mixture of the three categories. We refer the reader to the tool itself and to its online version for familiarizing themselves with the attributes that would classify an aspect of a study as belonging to one versus another of the three categories. Crucially, the MARC tool is organized so that your answers to all the questions can be saved as a separate document that you can then attach as part of a preregistration, or link to in a grant application or a manuscript.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/309X/01aug22/mbe12318-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="mbe12318-fig-0001.jpg" title="1 Exemplary compass plot as output of the MARC tool. BS = Balance score; CLR = controlled laboratory research; NRWRA = naturalistic real‐world research approach; PNLRA = partially naturalistic laboratory research approach. The footer of the figure provides information, which aspect of the study was evaluated as belonging to which part of the cycle." /> </p> <p></p> <hd id="AN0158316337-6">Informed Consent</hd> <p>No empirical data were collected for this theoretical manuscript, therefore, no human or animal ethics approval was required. Equally, there was no need for informed consent.</p> <hd id="AN0158316337-7">PILOTING THE MARC TOOL</hd> <p>In order to test the feasibility of the MARC tool, coauthors provided original research articles (<emph>N</emph> =9), which have been previously published from various fields and research topics. Every article was rated by two independent raters (See Appendix S1). Subsequently, Pearson correlations were calculated between the first and the second‐rater for each score provided by the MARC tool (controlled laboratory research, partially naturalistic laboratory research approach, naturalistic real‐world research approach score). Pearson correlations between the first and second‐rater can be considered large (controlled laboratory research: <emph>r</emph>(<reflink idref="bib7" id="ref2">7</reflink>) = .83, <emph>p</emph> < .01; partially naturalistic laboratory research approach: <emph>r</emph>(<reflink idref="bib7" id="ref3">7</reflink>) = .72, <emph>p</emph> < .01; naturalistic real‐world research approach: <emph>r</emph>(<reflink idref="bib7" id="ref4">7</reflink>) = .88, <emph>p</emph> < .01), indicating a sufficient inter‐rater reliability within the first pilot of the MARC tool (See Appendix S2).</p> <hd id="AN0158316337-8">Acknowledgments</hd> <p>This work was the result of the 2019 Flux Society Annual Meeting's Pre‐Conference workshop "Beyond the lab: Translating developmental neuroscience" organized by Lucia Magis Weinberg, & Natasha Duell & Jennifer Pfeifer, funded by HopeLab, Bezos Family Foundation & Center for Translational Neuroscience (University of Oregon). This work was supported by the Swiss National Science Foundation grant (no. PZ00P1_174150). We thank Isabel Dziobek for her extensive comments on the original version of this manuscript. Open access funding provided by Haute Ecole Specialisee de la Suisse Occidentale.</p> <hd id="AN0158316337-9">Conflict of Interest</hd> <p>The authors have stated explicitly that there are no conflicts of interest in connection with this article.</p> <p>A APPENDIX MULTIDIMENSIONAL ASSESSMENT OF RESEARCH IN CONTEXT (MARC) TOOL</p> <p>Below you find several short questions regarding your research project. After submitting the form, a summary with your answers and descriptions accompanied by a compass plot will appear. The graph indicates the match of your research idea with three categories that are controlled laboratory‐based, partially naturalistic laboratory and naturalistic real‐world research. For every category, we added one main example in brackets and some other examples in the tables. The tool can be found online at: https://marcform-git.herokuapp.com/</p> <p>1a) What behavior are you trying to observe?</p> <p>.....................................................................</p> <p>.....................................................................</p> <p>1b) What is the context you aim to generalize to?</p> <p>.....................................................................</p> <p>.....................................................................></p> <p>2. Thinking about the context you described in 1 a/b, which of the following best describes...</p> <p>... how your sample reflects that context?</p> <p> <emph>When answering this question, consider how you recruit participants for your study, and whether your sample is representative of, e.g., your region</emph>.</p> <p>[—] Controlled laboratory‐based [e.g. Convenience sample, such as undergraduate students at a university].</p> <p>[—] Partially naturalistic [e.g. Community‐based recruitment]</p> <p>[—] Naturalistic real‐world [e.g. A large, nationally representative sample of school districts in a city]</p> <p>More examples:</p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Controlled laboratory‐based</td><td><p>Control sample matched only by age and gender</p><p>Preclinical studies</p><p>Western, Educated, Industrialized, Rich, and Democratic (WEIRD) society</p><p>Recruiting children of professors at a university</p></td></tr><tr><td>Partially naturalistic</td><td><p>Recruiting from one or few local schools</p><p>Recruiting an aging sample from several nearby community living facilities</p></td></tr><tr><td>Naturalistic real‐world</td><td><p>Recruiting a large data sample that matches national demographic and socio‐economic status characteristic distribution</p><p>Large crowd sourced data from public databases (e.g. free narratives, subjective tags, etc.)</p></td></tr></tbody></table> </ephtml> </p> <p>3. Thinking about the context you described in 1 a/b, which of the following best describes...</p> <p>... how your testing site reflects that context?</p> <p>[—] Controlled laboratory‐based [e.g. Lab/ clinical testing room]</p> <p>[—] Partially naturalistic [e.g. Lab set up to look like a classroom]</p> <p>[—] Naturalistic real‐world [e.g. Classroom with little/no experimenter presence and interference into teaching activities]</p> <p>More examples:</p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Controlled laboratory‐based</td><td><p>In an MEG/EEG lab or an MRI scanner/facility</p><p>Lab testing room in wet‐lab facilities</p></td></tr><tr><td>Partially naturalistic</td><td><p>More naturalistic stimulation is delivered via VR goggles while wearing M/EEG/fNIRS</p><p>In schools and classrooms but not in typical classroom setting (during a normal lesson)</p><p>Measuring EEG simultaneously in two participants who are interacting in a lab</p><p>Testing ambulatory patients</p></td></tr><tr><td>Naturalistic real‐world</td><td><p>Where the real‐world behavior would take place (in the street, market, etc.)</p><p>At participant's home</p><p>Hyper‐scanning during a real concert</p></td></tr></tbody></table> </ephtml> </p> <p>4. Thinking about the context you described in 1 a/b, which of the following best describes...</p> <p>... how your task reflects that context?</p> <p> <emph>Here is a general explanation for what the different categories mean with regard to the task</emph>:</p> <p></p> <ulist> <item> Controlled laboratory‐based: the process of interest is isolated by manipulating only a minimum number of factors in a specific experimental design, and all other factors are held constant.</item> <p></p> <item> Partially naturalistic: process‐specific tasks are used in settings that resemble everyday situations.</item> <p></p> <item> Naturalistic real‐world: the process of interest is measured in real‐world situations.</item> </ulist> <p>[—] Controlled laboratory‐based [e.g. Working memory task for shapes presented on a screen].</p> <p>[—] Partially naturalistic [e.g. Test of memory after viewing a movie]</p> <p>[—] Naturalistic real‐world [e.g. Memory test of interaction after prolonged delay that involved other activities]</p> <p>More examples:</p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Controlled laboratory‐based</td><td><p>Spatial orienting task involving a single target stimulus (a shape) preceded by a single cue/distractor</p><p>Inhibiting a button press to a trained stimulus</p><p>Flanker task (responding to the direction of a middle arrow, that is displayed between other arrows pointing in a similar or opposite direction)</p><p>Oddball task (responding to a target stimulus in a stream of distractors)</p></td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Partially naturalistic</td><td><p>Selective attention task where both targets and distractors are presented in multi‐stimulus arrays (e.g. visual search) and vary across multiple dimensions</p><p>Tasks conducted in virtual reality or a room resembling a naturalistic context (e.g. kitchen, a flat or a simple shop)</p><p>Social interaction in the lab with a confederate</p><p>Clinical neuropsychological task to measure cognitive functions such as fluid Intelligence [Matrix Reasoning, etc], attention [Forward Digit span, etc], executive functions [Trail Making Test Part B, etc], memory [Rey Auditory Verbal Learning Test], social cognition [Reading the Mind in the Eyes Test, etc], etc.</p><p>Watching a movie in the MRI scanner</p><p>Peer presence during task ‐ giving a speech in the lab in front of confederates/video recording of an audience</p></td></tr><tr><td>Naturalistic real‐world</td><td><p>Selective attention task where both targets and distractors varying across multiple dimensions appear within a veridical external environment (classroom or public place like museum exhibitions)</p><p>Observing a child interact with a parent at home</p><p>Classroom based and teacher lead curriculum</p><p>Free narratives</p><p>Observing/transcribing videos of treatment sessions with clinician</p><p>Using a fitbit‐like motion sensor to measure daily activity</p><p>Social network analysis</p><p>In‐classroom behavior</p><p>EMA about social behavior</p></td></tr></tbody></table> </ephtml> </p> <p>5. Thinking about the context you described in 1 a/b, which of the following best describes...</p> <p>... how your stimuli reflect that context?</p> <p> <emph>Here is a general explanation for what the different categories mean with regard to the stimuli</emph>:</p> <p></p> <ulist> <item> Controlled laboratory‐based: maximum control over stimuli.</item> <p></p> <item> Partially naturalistic: some, but not total, control over stimuli.</item> <p></p> <item> Naturalistic real‐world: no control over stimuli.</item> </ulist> <p>[—] Controlled laboratory‐based [e.g. Static stimuli, typical for perceptual/cognitive studies, like face images].</p> <p>[—] Partially naturalistic [e.g. Dynamic stimuli, like dynamic faces on video]</p> <p>[—] Naturalistic real‐world [e.g. Fully naturalistically sampled stimuli: people during social interaction]</p> <p>More examples:</p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Controlled laboratory‐based</td><td><p>Simplistic stimuli presented multiple times</p><p>Stimuli presented one at a time, sequentially</p><p>Stimuli varying in their goal‐relevance to the performed task (distractor and target stimuli)</p><p>Audio clips of phonemes or words</p><p>Colored 2D shapes</p><p>Pictures of faces with different expressions</p><p>Unisensory stimuli</p></td></tr><tr><td>Partially naturalistic</td><td><p>Rich, naturalistic stimuli with whose properties and their distribution reflect those present in the relevant context</p><p>Stimuli varying in their goal‐relevance to the performed task (there are distractor and target stimuli) while varying also on other dimensions (see below)</p><p>Distractors or target stimulating many senses (visual/auditory, multisensory),</p><p>Distractors/targets varying in their familiarity to the observer, being unfamiliar or representing/being connected by a naturalistic object category (animate objects, tools, conspecifics)</p><p>Multisensory</p><p>Stimuli whose meaning is dependent on the context (audio or audiovisual clips of social scenarios; watching movies or listening to stories; stimuli presented in VR)</p></td></tr><tr><td>Naturalistic real‐world</td><td><p>Veridical real‐world stimuli whose properties and their distribution reflect those present in the relevant context</p><p>Disruptions from classmates during a lesson at school</p><p>In‐place experiences (such as subjective effect of a drug in natural settings, risk activity, etc.)</p></td></tr></tbody></table> </ephtml> </p> <p>6. Thinking about the context you described in 1 a/b, which of the following best describes...</p> <p>... how what is being measured reflects that behavior?</p> <p>[—] Controlled laboratory‐based [e.g. Well‐understood, well‐researched brain correlates of a specific cognitive process, such as the Event‐Related Potential (ERP) components P1 or N2 (or a cognitive contrast in fMRI), tested in typical conditions].</p> <p>[—] Partially naturalistic [e.g. Testing the canonical brain correlates in nontraditional laboratory settings and/or using more portable brain imaging tools, like EEG or fNIRS]</p> <p>[—] Naturalistic real‐world [e. g. Using portable brain imaging tools in veridical external environments to test for canonical brain EEG/ERP 'correlates' of cognitive processes or for spectral features as correlates of mental states (engagement)]</p> <p>More examples:</p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Controlled laboratory‐based</td><td><p>Response or accuracy time in rigorous, process‐specific lab‐based tasks</p><p>Environmental variables included only as covariates</p><p>Biomarkers such as cortisol level, RNA expression or DNA methylation, etc.</p></td></tr><tr><td>Partially naturalistic</td><td><p>Using a physical setup inside the testing room or virtual reality</p><p>Inter‐subject correlational analyses of brain mechanisms during movie watching</p><p>In a different / multiple senses</p><p>In nontraditional populations (across the lifespan; individuals with nontraditional experience like sensory impairment)</p><p>Questionnaire about real‐life risk‐taking</p><p>Self/parent report on outside lab behavior as variables of interest</p></td></tr><tr><td>Naturalistic real‐world</td><td><p>Impact of variable of interest for grades or standardized test scores</p><p>Topics in free reports extracted by natural language processing (LSA or speech graphs)</p><p>Ecological momentary assessment (EMA) ratings of anxiety symptoms</p><p>Behavior in classroom</p><p>Social network analysis</p><p>Real‐life behavior data like risk‐taking (e.g. alcohol use) or incarceration rates</p></td></tr></tbody></table> </ephtml> </p> <p>7. Are nonresearch stakeholders involved? (teachers, caretaker, institutions, clinicians)</p> <p>[—] Controlled laboratory‐based [e.g. Stakeholders only facilitate access to the sample].</p> <p>[—] Partially naturalistic [e.g. Stakeholders involved in conception OR interpretation/writing up the results]</p> <p>[—] Naturalistic real‐world [e.g. Involvement in conception of project AND interpretation/writing up the results]</p> <p>More examples:</p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Controlled laboratory‐based</td><td>Practitioners (clinicians, teachers, head teachers, speech therapists) are not involved or involved only through providing the access to the populations of interests</td></tr><tr><td>Partially naturalistic</td><td>Practitioners advise on and contribute at some but not all stages of the research project (e.g. result interpretation)</td></tr><tr><td>Naturalistic real‐world</td><td>Practitioners advise on and contribute to all stages of the research project (e.g. help design, implement, and report on study results)</td></tr></tbody></table> </ephtml> </p> <p>Is there an intervention component in the study</p> <p>[—] Yes</p> <p>[—] No</p> <p>8. Please indicate where your intervention fits in best.</p> <p>[—] Controlled laboratory‐based [e.g. Children play a game on laptop/ tablet at the lab/ clinic supervised by experimenters and/or parents]</p> <p>[—] Partially naturalistic [e.g. Children play a game on a laptop/ tablet at home supervised by parents]</p> <p>[—] Naturalistic real‐world [e.g. Children play an online application at home by themselves when they feel like it]</p> <p>More examples:</p> <p></p> <p> <ephtml> <table><tbody valign="top"><tr><td>Controlled laboratory‐based</td><td>Computer paradigm to train participants to look at neutral instead of negative faces</td></tr><tr><td>Partially naturalistic</td><td>Training in school with standardized training but outside the regular classroom activities</td></tr><tr><td>Naturalistic real‐world</td><td>Providing first‐line treatment for psychopathology by a trained clinician</td></tr></tbody></table> </ephtml> </p> <p>GRAPH: Appendix S1. MARC—tool pilot</p> <p>GRAPH: Appendix S2. Pilot study correlations</p> <ref id="AN0158316337-10"> <title> Footnotes </title> <blist> <bibl id="bib1" type="bt">1</bibl> <bibtext> All authors contributed equally.</bibtext> </blist> </ref> <ref id="AN0158316337-11"> <title> REFERENCES </title> <blist> <bibtext> Alsius, A., & Soto‐Faraco, S. (2011). Searching for audiovisual correspondence in multiple speaker scenarios. Experimental Brain Research, 213 (2–3), 175 – 183. https://doi.org/10.1007/s00221‐011‐2624‐0</bibtext> </blist> <blist> <bibl id="bib2" type="bt">2</bibl> <bibtext> Altikulaç, S., Lee, N. C., van der Veen, C., Benneker, I., Krabbendam, L., & van Atteveldt, N. (2019). The teenage brain: Public perceptions of neurocognitive development during adolescence. Journal of Cognitive Neuroscience, 31, 339 – 359. https://doi.org/10.1162/jocn_a_01332</bibtext> </blist> <blist> <bibl id="bib3" idref="ref1" type="bt">3</bibl> <bibtext> Bevilacqua, D., Davidesco, I., Wan, L., Chaloner, K., Rowland, J., Ding, M., ... Dikker, S. (2019). Brain‐to‐brain synchrony and learning outcomes vary by student‐teacher dynamics: Evidence from a real‐world classroom electroencephalography study. Journal of Cognitive Neuroscience, 31, 401 – 411. https://doi.org/10.1162/jocn_a_01274</bibtext> </blist> <blist> <bibl id="bib4" type="bt">4</bibl> <bibtext> Brunswik, E. (1943). Organismic achievement and environmental probability. Psychological Review, 50 (3), 255. https://doi.org/10.1037/h0060889</bibtext> </blist> <blist> <bibl id="bib5" type="bt">5</bibl> <bibtext> Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62 (3), 193. https://doi.org/10.1037/h0047470</bibtext> </blist> <blist> <bibl id="bib6" type="bt">6</bibl> <bibtext> Casey, B. J., Trainor, R. J., Orendi, J. L., Schubert, A. B., Nystrom, L. E., Giedd, J. N., ... Rapoport, J. L. (1997). A developmental functional MRI study of prefrontal activation during performance of a go‐no‐go task. Journal of Cognitive Neuroscience, 9 (6), 835 – 847. https://doi.org/10.1162/jocn.1997.9.6.835</bibtext> </blist> <blist> <bibl id="bib7" idref="ref2" type="bt">7</bibl> <bibtext> Cavallina, C., Puccio, G., Capurso, M., Bremner, A. J., & Santangelo, V. (2018). Cognitive development attenuates audiovisual distraction and promotes the selection of task‐relevant perceptual saliency during visual search on complex scenes. Cognition, 180, 91 – 98. https://doi.org/10.1016/j.cognition.2018.07.003</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> Chein, J., Albert, D., O'Brien, L., Uckert, K., & Steinberg, L. (2011). Peers increase adolescent risk taking by enhancing activity in the brain's reward circuitry. Developmental Science, 14 (2), F1 – F10. https://doi.org/10.1111/j.1467‐7687.2010.01035.x</bibtext> </blist> <blist> <bibl id="bib9" type="bt">9</bibl> <bibtext> Dunlosky, J., Bottiroli, S., & Hartwig, M. (2009). Sins committed in the name of ecological validity: A call for representative design in education science. In D. J. Hacker, J. Dunlosky & A. C. Graesser (Eds.), Handbook of metacognition in education. (pp. 442 – 452). Abingdon, England : Routledge.</bibtext> </blist> <blist> <bibtext> Dziobek, I. (2012). Comment: Towards a more ecologically valid assessment of empathy. Emotion Review, 4 (1), 18 – 19. https://doi.org/10.1177/1754073911421390</bibtext> </blist> <blist> <bibtext> Figner, B., & Weber, E. U. (2011). Who takes risks when and why? Current Directions in Psychological Science, 20 (4), 211 – 216. https://doi.org/10.1177/0963721411415790</bibtext> </blist> <blist> <bibtext> Föcker, J., Mortazavi, M., Khoe, W., Hillyard, S. A., & Bavelier, D. (2019). Neural correlates of enhanced visual attentional control in action video game players: An event‐related potential study. Journal of Cognitive Neuroscience, 31 (3), 377 – 389. https://doi.org/10.1162/jocn_a_01230</bibtext> </blist> <blist> <bibtext> Galvan, A., Hare, T. A., Parra, C. E., Penn, J., Voss, H., Glover, G., & Casey, B. J. (2006). Earlier development of the accumbens relative to orbitofrontal cortex might underlie risk‐taking behavior in adolescents. The Journal of Neuroscience, 26 (25), 6885 – 6892. https://doi.org/10.1523/JNEUROSCI.1062‐06.2006</bibtext> </blist> <blist> <bibtext> Harden, K. P., Kretsch, N., Mann, F. D., Herzhoff, K., Tackett, J. L., Steinberg, L., & Tucker‐Drob, E. M. (2017). Beyond dual systems: A genetically‐informed, latent factor model of behavioral and self‐report measures related to adolescent risk‐taking. Developmental Cognitive Neuroscience, 25, 221 – 234. https://doi.org/10.1016/j.dcn.2016.12.007</bibtext> </blist> <blist> <bibtext> Holleman, G. A., Hooge, I. T., Kemner, C., & Hessels, R. S. (2020a). The 'real‐world Approach'and its problems: A critique of the term ecological validity. Frontiers in Psychology, 11, 721. https://doi.org/10.3389/fpsyg.2020.00721</bibtext> </blist> <blist> <bibtext> Kenny, R., Dooley, B., & Fitzgerald, A. (2016). Ecological momentary assessment of adolescent problems, coping efficacy, and mood states using a mobile phone app: An exploratory study. JMIR Mental Health, 3 (4), e51. https://doi.org/10.2196/mental.6361</bibtext> </blist> <blist> <bibtext> Lejuez, C. W., Read, J. P., Kahler, C. W., Richards, J. B., Ramsey, S. E., Stuart, G. L., ... Brown, R. A. (2002). Evaluation of a behavioral measure of risk taking: The balloon analogue risk task (BART). Journal of Experimental Psychology: Applied, 8 (2), 75 – 84. https://doi.org/10.1037/1076‐898X.8.2.75</bibtext> </blist> <blist> <bibtext> Matusz, P. J., Dikker, S., Huth, A. G., & Perrodin, C. (2019). Are we ready for real‐world neuroscience? Journal of Cognitive Neuroscience, 31 (3), 327 – 338. https://doi.org/10.1162/jocn_e_01276</bibtext> </blist> <blist> <bibtext> Matusz, P. J., Turoman, N., Tivadar, R. I., Retsa, C., & Murray, M. M. (2019). Brain and cognitive mechanisms of top‐down attentional control in a multisensory world: Benefits of electrical neuroimaging. Journal of Cognitive Neuroscience, 31 (3), 412 – 430. https://doi.org/10.1162/jocn_a_01360</bibtext> </blist> <blist> <bibtext> Nastase, S. A., Goldstein, A., & Hasson, U. (2020). Keep it real: Rethinking the primacy of experimental control in cognitive neuroscience. Neuroimage, 117254, 222. https://doi.org/10.31234/osf.io/whn6d</bibtext> </blist> <blist> <bibtext> Op de Macks, Z. A., Flannery, J. E., Peake, S. J., Flournoy, J. C., Mobasser, A., Alberti, S. L., ... Pfeifer, J. H. (2018). Novel insights from the yellow light game: Safe and risky decisions differentially impact adolescent outcome‐related brain function. Neuroimage, 181, 568 – 581. https://doi.org/10.1016/j.neuroimage.2018.06.058</bibtext> </blist> <blist> <bibtext> Peelen, M. V., & Kastner, S. (2014). Attention in the real world: Toward understanding its neural basis. Trends in Cognitive Sciences, 18 (5), 242 – 250. https://doi.org/10.1016/j.tics.2014.02.004</bibtext> </blist> <blist> <bibtext> Rosenblau, G., O'Connell, G., Heekeren, H. R., & Dziobek, I. (2019). Neurobiological mechanisms of social cognition treatment in high‐functioning adults with autism spectrum disorder. Psychological Medicine, 50 (14), 2374 – 2384. https://doi.org/10.1017/S0033291719002472</bibtext> </blist> <blist> <bibtext> Schmuckler, M. A. (2001). What is ecological validity? A dimensional analysis. Infancy, 2 (4), 419 – 436. https://doi.org/10.1207/S15327078IN0204_02</bibtext> </blist> <blist> <bibtext> Shamay‐Tsoory, S. G., & Mendelsohn, A. (2019). Real‐life neuroscience: An ecological approach to brain and behavior research. Perspectives on Psychological Science, 14 (5), 841 – 859. https://doi.org/10.1177/1745691619856350</bibtext> </blist> <blist> <bibtext> Steinberg, L. (2005). Cognitive and affective development in adolescence. Trends in Cognitive Sciences, 9 (2), 69 – 74. https://doi.org/10.1016/j.tics.2004.12.005</bibtext> </blist> <blist> <bibtext> Turoman, N., Tivadar, R. I., Retsa, C., Maillard, A. M., Scerif, G., & Matusz, P. J. (2020a). The development of attentional control mechanisms in multisensory environments. BioRxiv. https://doi.org/10.1101/2020.06.23.166975.</bibtext> </blist> <blist> <bibtext> Turoman, N., Tivadar, R. I., Retsa, C., Maillard, A. M., Scerif, G., & Matusz, P. J. (2020b). Uncovering the mechanisms of real‐world attentional control over the course of primary education. BioRxiv. https://doi.org/10.1101/2020.10.20.342758</bibtext> </blist> <blist> <bibtext> Vaca, F. E., Walthall, J. M., Ryan, S., Moriarty‐Daley, A., Riera, A., Crowley, M. J., & Mayes, L. C. (2013). Adolescent balloon analog risk task and behaviors that influence risk of motor vehicle crash injury. Annals of Advances in Automotive Medicine, 57, 77 – 88.</bibtext> </blist> <blist> <bibtext> van Atteveldt, N., Tijsma, G., Janssen, T., & Kupper, F. (2019). Responsible research and innovation as a novel approach to guide educational impact of mind, brain, and education research. Mind, Brain, and Education, 13 (4), 279 – 287. https://doi.org/10.1111/mbe.12213</bibtext> </blist> <blist> <bibtext> van Atteveldt, N., van Kesteren, M. T. R., Braams, B., & Krabbendam, L. (2018). Neuroimaging of learning and development: Improving ecological validity. Frontline Learning Research, 6 (3), 186 – 203. https://doi.org/10.14786/flr.v6i3.366</bibtext> </blist> <blist> <bibtext> van Hoorn, J., McCormick, E. M., Rogers, C. R., Ivory, S. L., & Telzer, E. H. (2018). Differential effects of parent and peer presence on neural correlates of risk taking in adolescence. Social Cognitive and Affective Neuroscience, 13 (9), 945 – 955. https://doi.org/10.1093/scan/nsy071</bibtext> </blist> <blist> <bibtext> Vanderwal, T., Eilbott, J., & Castellanos, F. X. (2019). Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging. Developmental Cognitive Neuroscience, 36, 100600. https://doi.org/10.1016/j.dcn.2018.10.004</bibtext> </blist> </ref> <aug> <p>By Sandra Naumann; Michelle L. Byrne; Alethia de la Fuente; Anita Harrewijn; Tehila Nugiel; Maya Rosen; Nienke van Atteveldt and Pawel J. Matusz</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author; Author</p> </aug>
Header DbId: eric
DbLabel: ERIC
An: EJ1343744
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Assessing the Degree of Ecological Validity of Your Study: Introducing the Multidimensional Assessment of Research in Context (MARC) Tool
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Naumann%2C+Sandra%22">Naumann, Sandra</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8992-2930">0000-0001-8992-2930</externalLink>)<br /><searchLink fieldCode="AR" term="%22Byrne%2C+Michelle+L%2E%22">Byrne, Michelle L.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4180-8095">0000-0002-4180-8095</externalLink>)<br /><searchLink fieldCode="AR" term="%22de+la+Fuente%2C+Alethia%22">de la Fuente, Alethia</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-5689-1709">0000-0002-5689-1709</externalLink>)<br /><searchLink fieldCode="AR" term="%22Harrewijn%2C+Anita%22">Harrewijn, Anita</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9938-4048">0000-0001-9938-4048</externalLink>)<br /><searchLink fieldCode="AR" term="%22Nugiel%2C+Tehila%22">Nugiel, Tehila</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5547-1852">0000-0001-5547-1852</externalLink>)<br /><searchLink fieldCode="AR" term="%22Rosen%2C+Maya%22">Rosen, Maya</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4649-9965">0000-0002-4649-9965</externalLink>)<br /><searchLink fieldCode="AR" term="%22van+Atteveldt%2C+Nienke%22">van Atteveldt, Nienke</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-3387-6151">0000-0002-3387-6151</externalLink>)<br /><searchLink fieldCode="AR" term="%22Matusz%2C+Pawel+J%2E%22">Matusz, Pawel J.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1721-4868">0000-0003-1721-4868</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Mind%2C+Brain%2C+and+Education%22"><i>Mind, Brain, and Education</i></searchLink>. Aug 2022 16(3):228-238.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 11
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2022
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive<br />Tests/Questionnaires
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Neurosciences%22">Neurosciences</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Validity%22">Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/mbe.12318
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1751-2271<br />1751-228X
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In cognitive neurosciences, fundamental principles of mental processes and functional brain organization have been established with highly controlled tasks and testing environments. Recent technical advances allowed the investigation of these functions and their brain mechanisms in naturalistic settings. The diversity in those approaches have been recently (Matusz, P. J., Dikker, S., Huth, A. G., & Perrodin, C. (2019). Are we ready for real-world neuroscience? "Journal of Cognitive Neuroscience," 31(3), 327-338. doi:10.1162/jocn_e_01276) classified via a three-category cycle, including "controlled laboratory," "partially naturalistic laboratory," and "naturalistic real-world research." Based on this model, we developed the Multidimensional Assessment of Research in Context (MARC) tool to easily delineate the approach researchers have taken in their study. MARC provides means to describe the degree of ecological validity for each component of a study (e.g., sample, stimuli, measures, etc.), and the study's location on the cycle. The tool comprises seven questions concerning a study's characteristics. It outputs a summary of those and a compass plot, which can be used for presentations, preregistration, grant proposals, and papers. It aims to improve drawing conclusions across studies and raise awareness about the generalizability of research findings.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2022
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1343744
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1343744
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/mbe.12318
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
        StartPage: 228
    Subjects:
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Neurosciences
        Type: general
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Validity
        Type: general
      – SubjectFull: Research Methodology
        Type: general
    Titles:
      – TitleFull: Assessing the Degree of Ecological Validity of Your Study: Introducing the Multidimensional Assessment of Research in Context (MARC) Tool
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Naumann, Sandra
      – PersonEntity:
          Name:
            NameFull: Byrne, Michelle L.
      – PersonEntity:
          Name:
            NameFull: de la Fuente, Alethia
      – PersonEntity:
          Name:
            NameFull: Harrewijn, Anita
      – PersonEntity:
          Name:
            NameFull: Nugiel, Tehila
      – PersonEntity:
          Name:
            NameFull: Rosen, Maya
      – PersonEntity:
          Name:
            NameFull: van Atteveldt, Nienke
      – PersonEntity:
          Name:
            NameFull: Matusz, Pawel J.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2022
          Identifiers:
            – Type: issn-print
              Value: 1751-2271
            – Type: issn-electronic
              Value: 1751-228X
          Numbering:
            – Type: volume
              Value: 16
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Mind, Brain, and Education
              Type: main
ResultId 1