View in EDS HTML Full Text PDF Full Text

Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions

Saved in:
Bibliographic Details
Title: Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions
Language: English
Authors: Pamela R. Buckley (ORCID 0000-0002-8268-3524), Katie Massey Combs, Karen M. Drewelow, Brittany L. Hubler, Marion Amanda Lain
Source: Evaluation Review. 2025 49(2):237-269.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 33
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
Descriptors: Middle School Students, Middle School Teachers, Evidence Based Practice, Program Development, Program Effectiveness, Program Evaluation, Program Implementation, Prevention, Drug Abuse, Fidelity, Program Validation, Generalization, Intervention, Scoring, Test Use, Predictive Validity
DOI: 10.1177/0193841X241248864
ISSN: 0193-841X
1552-3926
Abstract: As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers' ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane's (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1466339
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEqrt6-riJpPnfnJuZnftMaAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI8Pb21uYXGWn82mDgIBEICBmgARDTZQ1b2xlP1ZxlJV2fSCc2E01lwQG1atbzvDYXh-rvW1I5aV24iQqZaOYv7EX32N0ato0_M4VTSjpMTeMpk047BzXudkJDxFawCrgKfL_4phXQxHQQh7punPJpl5M4S-bfPcxHPlUdBo3qFVmEFGfQZ2-RdnfWGfy0ZadNT3B8h3OnZTdSU0WFPCT6cbudpV1Dyer6UO3-o=
Text:
  Availability: 1
  Value: <anid>AN0183370751;evr01apr.25;2025Mar04.03:58;v2.2.500</anid> <title id="AN0183370751-1">Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions </title> <p>As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers' ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane's (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.</p> <p>Keywords: program implementation; scale-up; evidence-based interventions; dissemination and implementation; fidelity of implementation; measurement of program fidelity; observation protocols</p> <hd id="AN0183370751-2">Introduction</hd> <p>Numerous interventions have been evaluated following rigorous scientific methods and shown to prevent a range of behavioral health problems ([<reflink idref="bib37" id="ref1">37</reflink>]; [<reflink idref="bib45" id="ref2">45</reflink>]). As these evidence-based interventions (EBIs) get widely disseminated, fidelity of implementation (FOI), and thus effectiveness, often wanes ([<reflink idref="bib18" id="ref3">18</reflink>]; [<reflink idref="bib32" id="ref4">32</reflink>]; [<reflink idref="bib43" id="ref5">43</reflink>]; [<reflink idref="bib75" id="ref6">75</reflink>]). To understand this phenomenon, it is essential to have high-quality measurement processes for assessing fidelity to the intervention's model ([<reflink idref="bib40" id="ref7">40</reflink>]; [<reflink idref="bib68" id="ref8">68</reflink>]; [<reflink idref="bib78" id="ref9">78</reflink>]). Measuring FOI can be used to: (<reflink idref="bib1" id="ref10">1</reflink>) determine whether poor behavioral or clinical outcomes reflect a failure of the model or failure to implement the model as intended ([<reflink idref="bib15" id="ref11">15</reflink>]; [<reflink idref="bib20" id="ref12">20</reflink>]; [<reflink idref="bib49" id="ref13">49</reflink>]; [<reflink idref="bib66" id="ref14">66</reflink>]; [<reflink idref="bib78" id="ref15">78</reflink>]), (<reflink idref="bib2" id="ref16">2</reflink>) provide adequate guidelines for replicating EBIs ([<reflink idref="bib81" id="ref17">81</reflink>]), and/or (<reflink idref="bib3" id="ref18">3</reflink>) help explain variance in treatment effects ([<reflink idref="bib15" id="ref19">15</reflink>]; [<reflink idref="bib81" id="ref20">81</reflink>]). Validating FOI measures, however, is a complicated and iterative task that most intervention research studies do not undertake ([<reflink idref="bib15" id="ref21">15</reflink>]; [<reflink idref="bib68" id="ref22">68</reflink>]; [<reflink idref="bib72" id="ref23">72</reflink>]). That is, few published studies measuring FOI provide details on the construction and validation of FOI measures ([<reflink idref="bib68" id="ref24">68</reflink>]), though some examples exist ([<reflink idref="bib15" id="ref25">15</reflink>]; [<reflink idref="bib47" id="ref26">47</reflink>]; [<reflink idref="bib62" id="ref27">62</reflink>]). As such, further attention to the use, measurement and refinement of fidelity criteria is important to evaluation practice ([<reflink idref="bib68" id="ref28">68</reflink>]; [<reflink idref="bib72" id="ref29">72</reflink>]).</p> <p>Botvin <emph>LifeSkills Training</emph> (LST) middle school program is a universal prevention EBI for secondary students that has been shown in several randomized control trials to reduce risky behaviors with results maintained over time ([<reflink idref="bib8" id="ref30">8</reflink>], [<reflink idref="bib10" id="ref31">10</reflink>]). LST is classroom-based and includes 30 sessions divided into three levels taught in sequence over three consecutive school years. As of 2017, the program had been adopted by over 1200 communities across the United States, serving more than one million youth with a cost benefit ratio of $13.49 for every $1 spent ([<reflink idref="bib86" id="ref32">86</reflink>]). Cultural adaptations have been disseminated internationally in countries such as Italy ([<reflink idref="bib84" id="ref33">84</reflink>]), China ([<reflink idref="bib52" id="ref34">52</reflink>]), Taiwan ([<reflink idref="bib58" id="ref35">58</reflink>]) and Iran ([<reflink idref="bib53" id="ref36">53</reflink>]). Despite extensive evaluation and broad implementation, to our knowledge, the validity of observational protocols used to measure FOI to the LST model has never been documented.</p> <p>The validity literature describes an integrated approach to articulate the degree to which a specific measure has been validated for a particular purpose ([<reflink idref="bib2" id="ref37">2</reflink>]; [<reflink idref="bib48" id="ref38">48</reflink>]; [<reflink idref="bib54" id="ref39">54</reflink>]; Standards: [<reflink idref="bib1" id="ref40">1</reflink>]). Using data collected by 279 raters involved in a dissemination project in which LST was scaled-up within 379 socioeconomically diverse middle schools in 14 states across the U.S., this study tested assumptions for a validity argument of items within the Session Review Form (SRF) used to measure fidelity to interactive drug prevention programs such as LST. Our goals were to: (<reflink idref="bib1" id="ref41">1</reflink>) establish evidence of validity of SRF scores for measuring FOI of the LST program; (<reflink idref="bib2" id="ref42">2</reflink>) expand upon the modest FOI measurement literature by offering an example of how validity evidence is built; and (<reflink idref="bib3" id="ref43">3</reflink>) engage the field in a discussion about the potential uses of valid FOI scores when conducting research on EBIs.</p> <hd id="AN0183370751-3">Defining Fidelity of Implementation</hd> <p>Fidelity is defined as the extent to which an intervention is delivered as intended ([<reflink idref="bib27" id="ref44">27</reflink>]; [<reflink idref="bib62" id="ref45">62</reflink>]; [<reflink idref="bib72" id="ref46">72</reflink>]). Researchers measure fidelity to maximize program effectiveness and to guide the successful dissemination and sustainability of EBIs in natural settings. Fidelity measures are any tools that assess the faithfulness in implementation of program elements ([<reflink idref="bib6" id="ref47">6</reflink>]). Typically, scales are developed to generate scores that quantify behaviors comparing an intervention as implemented to the intended model on which it was based ([<reflink idref="bib28" id="ref48">28</reflink>]).</p> <p>Fidelity has various domains ([<reflink idref="bib26" id="ref49">26</reflink>]; [<reflink idref="bib73" id="ref50">73</reflink>]), and many focus on the following five: (<reflink idref="bib1" id="ref51">1</reflink>) adherence, or whether program components are delivered as prescribed, (<reflink idref="bib2" id="ref52">2</reflink>) dosage, as in the frequency or duration of program delivery, (<reflink idref="bib3" id="ref53">3</reflink>) quality of delivery, such as how well the program material is implemented, (<reflink idref="bib4" id="ref54">4</reflink>) participant responsiveness, meaning how well the intervention is received or perceived, and/or (<reflink idref="bib5" id="ref55">5</reflink>) program differentiation, as in the degree of contrast between treatment and control activities ([<reflink idref="bib17" id="ref56">17</reflink>]; [<reflink idref="bib27" id="ref57">27</reflink>]; [<reflink idref="bib31" id="ref58">31</reflink>]). These criteria can be generally divided into two groups: (<reflink idref="bib1" id="ref59">1</reflink>) fidelity to structure (i.e., adherence and dosage), and (<reflink idref="bib2" id="ref60">2</reflink>) fidelity to process (i.e., quality of delivery and program differentiation), with participant responsiveness taking on characteristics of both ([<reflink idref="bib68" id="ref61">68</reflink>]; [<reflink idref="bib72" id="ref62">72</reflink>]).</p> <p>Fidelity to structure is more frequently reported in the literature, likely because these criteria are easier to measure and interpret since they commonly consist of counts, for example, of minutes, curriculum points, or sessions ([<reflink idref="bib17" id="ref63">17</reflink>]; [<reflink idref="bib23" id="ref64">23</reflink>], [<reflink idref="bib25" id="ref65">25</reflink>]; [<reflink idref="bib55" id="ref66">55</reflink>]). Our focus is on fidelity to process occurring through direct methods that include observation of practitioners' delivery of the intervention, which is typically considered the "gold standard" for measuring fidelity ([<reflink idref="bib46" id="ref67">46</reflink>]; [<reflink idref="bib55" id="ref68">55</reflink>]). Doing so can be problematic, however, as many difficult-to-measure features of instruction outside the prescribed intervention may influence treatment effects and participants' experience with the intervention, such as the extent to which "good teaching" interacts with implementation of the program's instructional model ([<reflink idref="bib72" id="ref69">72</reflink>]).</p> <hd id="AN0183370751-4">Process Data Purposes</hd> <p>[<reflink idref="bib27" id="ref70">27</reflink>] strongly recommend that researchers measure all five dimensions of fidelity (dosage, adherence, quality, responsiveness, and program differentiation) to provide a comprehensive picture of program integrity. However, since this dissemination project involved a process evaluation, there was no control group and program differentiation (i.e., the degree of contrast between treatment and control activities) data were not collected. In addition, no student behavioral outcome data (e.g., youth self-reported tobacco, alcohol, or drug use) were collected. Rather, the project used the other four FOI domains (dosage, adherence, quality, and responsiveness) as implementation outcome data to measure the degree to which the LST program was delivered as intended. These implementation outcomes were provided as mean (average) scores in site reports to the funder and as feedback to describe the extent to which instructors within the participating schools delivered LST fully and according to program guidelines. Responses were then used to provide coaching designed to, for instance, increase content coverage, diversify teaching strategies, assist with time management, reduce implementation challenges, and/or guide modifications. More generally, one or more of the aggregate FOI scores measuring dosage, adherence, quality, responsiveness, and/or program differentiation have been used as implementation outcome variables in scientific studies to inform scale-up and sustainability of classroom-based drug prevention programs, such as LST (e.g., [<reflink idref="bib24" id="ref71">24</reflink>], [<reflink idref="bib23" id="ref72">23</reflink>], [<reflink idref="bib25" id="ref73">25</reflink>]). In addition, other academic papers have used these FOI scores as moderators of LST treatment effects ([<reflink idref="bib12" id="ref74">12</reflink>]; [<reflink idref="bib38" id="ref75">38</reflink>]).</p> <hd id="AN0183370751-5">Session Review Form Measuring Fidelity to Interactive Techniques</hd> <p>Many school-based prevention programs, such as LST, include interactive techniques that promote skills or specific attitudes and beliefs. More than simply performing from a script, these methods rely heavily on the implementer acting as a facilitator and coach; [<reflink idref="bib82" id="ref76">82</reflink>] identified interactivity as a key for successful drug prevention. The quality of interaction, such as how well facilitators follow the program's structure and engage students as intended, is thus important to measure when analyzing FOI to school-based drug prevention EBIs ([<reflink idref="bib31" id="ref77">31</reflink>]). LST is a universal prevention EBI generally facilitated by classroom teachers using a range of interactive teaching techniques, including didactic instruction, discussion, demonstration, and behavior skill rehearsals to teach personal self-management skills (e.g., self-esteem, problem solving, and coping), social skills (e.g., communication and building relationships), and drug resistance skills (e.g., consequences of drug use and refusal skills; [<reflink idref="bib11" id="ref78">11</reflink>]). The LST dissemination project described in this paper administered an observation protocol to capture the use of interactive teaching techniques.</p> <p>Referred to as the Session Review Form (SRF), the instrument was developed collaboratively by two senior implementation scientists intimately familiar with survey design, program replication, and the core components of LST—one an internationally renowned criminologist and sociologist with over 30 years of research experience ([<reflink idref="bib33" id="ref79">33</reflink>]) and the other a sociologist and implementation scientist ([<reflink idref="bib35" id="ref80">35</reflink>]). Based on their own experiences and content expertise, as well as project documentation, program records and site observations from an LST dissemination project that started in 1999, this team developed the SRF to quantify "fidelity to process" for LST. The instrument included items that assess interactivity given the research showing that interactive drug prevention programs are consistently more effective than the non-interactive programs ([<reflink idref="bib82" id="ref81">82</reflink>]). This resulted in two subscales on the SRF, with items measuring "quality of delivery" and "participant responsiveness."</p> <hd id="AN0183370751-6">SRF Construct Definitions</hd> <p>Quality of delivery refers to how well an intervention is carried out (beyond the prescribed content) by the individuals responsible for its implementation using overall processes or strategies prescribed by developers. It therefore encompasses competence and skill level of the implementers and the appropriateness of the delivery method ([<reflink idref="bib17" id="ref82">17</reflink>]; [<reflink idref="bib27" id="ref83">27</reflink>]; [<reflink idref="bib31" id="ref84">31</reflink>]). Examples provided by [<reflink idref="bib27" id="ref85">27</reflink>] include "implementor enthusiasm, leader preparedness, global estimates of session effectiveness, and leader attitudes toward program" (p. 45). Evaluating the quality of delivery helps researchers and practitioners understand the extent to which the intervention is implemented as originally designed and whether any adaptations or modifications are necessary for optimal effectiveness ([<reflink idref="bib4" id="ref86">4</reflink>]). Drawing from this definition, the SRF measures <emph>quality of delivery</emph> through seven items requiring observers to rate the instructor's delivery of lessons including teachers' knowledge of the program, enthusiasm, poise and confidence, rapport and communication, classroom management, ability to address questions and concerns, and overall quality of the lesson.</p> <p>Participant responsiveness refers to the extent to which participants actively and positively engage with the program components. It includes participants' willingness, enthusiasm, and cooperation in participating in the intervention activities and their receptiveness to the intervention's content and delivery methods ([<reflink idref="bib17" id="ref87">17</reflink>]; [<reflink idref="bib27" id="ref88">27</reflink>]; [<reflink idref="bib30" id="ref89">30</reflink>]; [<reflink idref="bib31" id="ref90">31</reflink>]; [<reflink idref="bib72" id="ref91">72</reflink>]). Fidelity assessment in this context aims to ensure that the program resonates with the participants effectively and as intended. This construct is vital in understanding how well the intervention is received by its target audience and whether participants are actively involved and motivated to benefit from the intervention's intended behavioral or clinical outcomes ([<reflink idref="bib4" id="ref92">4</reflink>]). The SRF measures <emph>participant responsiveness</emph> by having observers rate sessions based on three items asking how well students understood, participated in, and responded to the lesson.</p> <hd id="AN0183370751-7">SRF Scoring Procedures</hd> <p>A total of 279 raters (one to five per school district) were hired for this dissemination project and trained to administer FOI measures in classrooms where the LST program was being implemented. Raters scored two count measures of FOI: (<reflink idref="bib1" id="ref93">1</reflink>) <emph>Dosage</emph> or the average number of minutes spent on an LST lesson ([<reflink idref="bib9" id="ref94">9</reflink>]; [<reflink idref="bib67" id="ref95">67</reflink>]). Observers were trained to report the precise lesson start time, lesson end time, and note any interruptions to instruction (e.g., announcements, fire drills, and redirecting student misbehavior) and subtract this time from the total number of minutes. The dosage measure represented the time specifically dedicated to lesson delivery; and (<reflink idref="bib2" id="ref96">2</reflink>) <emph>Adherence</emph> or the percentage of core activities and lesson points in the LST teacher's manual that an instructor covered (https://<ulink href="http://www.lifeskillstraining.com/lst-fidelity-checklists/">www.lifeskillstraining.com/lst-fidelity-checklists/</ulink>). Raters used the adherence checklist created by the developer of LST ([<reflink idref="bib71" id="ref97">71</reflink>]) to assess whether the content was being delivered accurately. This included ensuring that the lesson observed covered the intended topics and that the activities aligned with the curriculum.</p> <p>In addition, these same raters administered the SRF observational protocol, in which they viewed program delivery and how the LST instructors interacted with students. To rate items measuring (<reflink idref="bib3" id="ref98">3</reflink>) <emph>Quality of delivery</emph>, raters received training in assessing whether the facilitator accurately conveyed LST's program content, concepts, and key messages without misinterpretation or distortion. They also were trained to assess the facilitator's skills in engaging students (e.g., encouraging questions, discussion, and critical thinking) and providing timely feedback. In addition, observers were trained to assess how well the facilitator maintained a professional demeanor and managed classroom behavior (e.g., handled disruptions and fostered a safe and respectful learning environment). Item response options were on a Likert scale (1 = poor to 5 = excellent) and were averaged by researchers (not the raters) to create a subscale score. Items on the SRF measuring (<reflink idref="bib4" id="ref99">4</reflink>) <emph>Student responsiveness</emph> involved assessing how actively and positively participants engaged with and responded to LST's content and activities. Raters were trained to observe whether participants were attentive, participated in discussions, completed activities, and/or showed enthusiasm. They were also trained to look for specific behavioral indicators of responsiveness, such as maintaining eye contact, nodding in agreement, and volunteering to answer questions. Item response options were on a Likert scale (1 = poor to 5 = excellent) and researchers averaged the items to create a subscale score.</p> <hd id="AN0183370751-8">Current Study</hd> <p>The psychometric literature uses validity theory to guide the development and evaluation of measurement instruments ([<reflink idref="bib1" id="ref100">1</reflink>]; [<reflink idref="bib54" id="ref101">54</reflink>]; [<reflink idref="bib89" id="ref102">89</reflink>]). Validity is defined as "the degree to which evidence and theory support the interpretations of test scores for proposed uses" ([<reflink idref="bib1" id="ref103">1</reflink>], p. 11). Validating proposed uses and interpretations of scores requires constructing an evidence-based case for defending the appropriateness of a test for its intended uses. One framework for validating scores is the argument-based approach to validity articulated by [<reflink idref="bib54" id="ref104">54</reflink>]. According to this method, the interpretation and planned use of scores are specified and empirical evidence is gathered using a variety of methods to support a linked set of assumptions that must be met to validate the interpretation and use of the proposed measurement score(s). [<reflink idref="bib2" id="ref105">2</reflink>] used Kane's validity argument to assess teacher quality for high-stakes personnel and professional development decisions while [<reflink idref="bib48" id="ref106">48</reflink>] used Kane to assess the quality of teachers' enactment of mathematics instruction. They identified four areas requiring investigation to build validity evidence:</p> <p></p> <ulist> <item> "(<reflink idref="bib1" id="ref107">1</reflink>) Assumptions involving <emph>scoring</emph>, typically focused around whether items are being used consistently and yield accurate and desired information;</item> <p></p> <item> (<reflink idref="bib2" id="ref108">2</reflink>) Assumptions involving <emph>generalization</emph>, typically focused around whether the sample of tasks and/or observations adequately represents the universe of potential observations;</item> <p></p> <item> (<reflink idref="bib3" id="ref109">3</reflink>) Assumptions involving <emph>extrapolation</emph>, typically focused around whether the assessment represents the constructs intended and the measure as a whole aligns with external indicators of examinee success in the domain(s) of interest; and</item> <p></p> <item> (<reflink idref="bib4" id="ref110">4</reflink>) Assumptions involving <emph>decisions</emph>, typically focused around whether consequences based on scores from the instrument are appropriate ([<reflink idref="bib48" id="ref111">48</reflink>], p. 90)."</item> </ulist> <p>[<reflink idref="bib54" id="ref112">54</reflink>] framework emphasizes the importance of ensuring that the <emph>scoring</emph> process is accurate, fair, and reliable. This would involve gathering evidence such as rater consistency (e.g., interrater agreement or interrater reliability) to obtain a precise and valid representation of the SRF items measuring fidelity to process. Kane's framework also requires gathering evidence to support the argument that the test measures what it is intended to measure (i.e., generalization). <emph>Generalization</emph> evidence can come from various sources, such as by examining whether FOI scores on the SRF are correlated with similar measures (e.g., whether SRF scores assessing fidelity to process predict scores from other validated instruments that assess the same construct). The <emph>decisions</emph> component of Kane's framework involves using scores to make important decisions, such as using SRF responses to design coaching aimed at improving delivery of the LST curriculum and aid in scaling and sustaining the program. Doing so requires providing evidence that coaching decisions are valid and defensible.</p> <p>While each of these forms of validity are important to consider, the present paper focused on the <emph>extrapolation</emph> assumption since we were interested in gathering evidence of the stability and consistency of the underlying constructs on the SRF measuring fidelity to process to compare aggregated domain scores assessing FOI levels across different settings. Specifically, we sought to increase confidence in future reports to funders and schools detailing the extent to which instructors delivered LST according to program guidelines. In addition, we sought to build evidence of validity for future scientific papers in which these FOI scores are used as implementation outcomes informing scale-up and/or as moderators of LST treatment effects. Also—importantly—we used a historical dataset that included: (<reflink idref="bib1" id="ref113">1</reflink>) one rater per observation (prohibiting us from assessing the scoring assumption); (<reflink idref="bib2" id="ref114">2</reflink>) no additional measures collected on the sample reported in this dissemination project (for evaluating the generalizability assumption); (<reflink idref="bib3" id="ref115">3</reflink>) no outcome data (needed to test additional elements of the extrapolation assumption, such as whether FOI scores on the SRF are correlated with other related variables in hypothesized ways); and (<reflink idref="bib4" id="ref116">4</reflink>) no qualitative performance data or coaching records (that could be used for assessing the decisions assumption).</p> <p>To assess the extrapolation assumption, we posed four research questions: (<reflink idref="bib1" id="ref117">1</reflink>) Does the dimensionality of items on the Session Review Form (SRF) measuring "fidelity to process" of the LST middle school program conform to the hypothesized (i.e., a priori) two-factor structure (with dimensions "quality of delivery" and "participant responsiveness")? (<reflink idref="bib2" id="ref118">2</reflink>) Does group invariance of this structure hold across grade level and schools with high, medium, and low proportions of students experiencing socioeconomic disadvantage? (<reflink idref="bib3" id="ref119">3</reflink>) Can a reliable aggregate score (e.g., a total score) for each two-factor structure be reasonably achieved? (<reflink idref="bib4" id="ref120">4</reflink>) Were there any systematic differences across settings (e.g., grade level and schools) in the average score for each structure?</p> <hd id="AN0183370751-9">Method</hd> <p></p> <hd id="AN0183370751-10">Sample</hd> <p>A total of 379 schools participated in the project (representing 127 school districts), and 1626 teachers taught the LST curriculum. All participating schools received the same support and implemented the program between academic years 2016–2017 and 2018–2019, with 90% delivering LST in grades 6–8 and 10% in grades 7–9. Following the prescribed LST middle school model, Level 1 (15 foundational sessions) was taught to 6<sups>th</sups> grade students (or 7<sups>th</sups> in the 7–9 implementation plan) in 2016–2017, with LST taught one to five times per week (45 to 50 minutes per session). In 2017–2018, these students received Level 2 (10 sessions, 45–50 minutes per session), while an incoming cohort of 6<sups>th</sups> (or 7<sups>th</sups>) grade students received Level 1. In 2018–2019, 8<sups>th</sups> (or 9<sups>th</sups>) grade students received Level 3 (five 45–50-min sessions), while 7<sups>th</sups> (or 8<sups>th</sups>) grade students received Level 2, and an incoming cohort of 6<sups>th</sups> (or 7<sups>th</sups>) grade students received Level 1. Counting only Level 1 students, the study enrolled a total of 192,423 unduplicated students across three cohorts. Cohort 1 (<emph>n</emph> = 67,823 students) received Level 1 in 2016–2017, Cohort 2 (<emph>n</emph> = 62,590 students) received Level 1 in 2017–2018, and Cohort 3 (<emph>n</emph> = 62,010 students) received Level 1 in 2018–19. During the study period, Cohort 1 students received the full three-year model, whereas Cohort 2 students received two years, and Cohort 3 received one year.</p> <p>Most schools (<emph>n</emph> = 211, 56%) had three or fewer instructors to teach all three LST levels. That is, of the 379 schools, 22% (<emph>n</emph> = 84 schools) had one teacher, 20% (<emph>n</emph> = 76 schools) had two teachers, 32% (<emph>n</emph> = 123) had three to five teachers, and 25% (<emph>n</emph> = 96) had six or more teachers to teach all three LST levels. Though health teachers comprised the largest portion of LST facilitators, this varied by school and included social studies, science, math, computer science, and language arts teachers, as well as school counselors.</p> <hd id="AN0183370751-11">Fidelity Data Collection</hd> <p>Raters selected at random the dates/lessons for observation to collect dosage and adherence data and administer the SRF (all of which were collected during the same session). Teachers typically did not have advance notice of the observation, and raters observed the full class period (or the entire LST lesson taught that day). Based on a structured job description, raters were recommended by school district personnel, with the explicit guidance that raters should not be school staff or familiar with the teachers implementing LST to reduce bias. Because raters were hired for a specific district or school, teachers were generally observed by the same rater except in cases of rater turnover.</p> <p>Raters were required to attend a one- to two-day training workshop in the first year of implementation and were offered optional one-day booster training in the following year(s). This training was conducted by certified LST trainers within the program developer's training firm, National Health Promotion Associates ([<reflink idref="bib71" id="ref121">71</reflink>]). Additionally, raters were required to participate in a 60–90-min training session dedicated to the procedures and protocols of the rater role. This training was conducted by the project coordination staff who provided guidance and oversight for LST implementation. Project coordinators also conducted annual site visits in which they co-observed a lesson with most raters. During these in-person observations, raters identified the program level of the classroom observed, completed the LST checklist to report dosage and adherence, and filled out the SRF. As part of their training, the project coordinators reviewed SRF responses and followed up with questions or feedback to promote consistency and reduce drift.</p> <p>A total of 6219 observations of LST lessons were conducted over the 3-year project for the 1626 teachers. This equated to an average of 3.8 observations (<emph>SD</emph> = 3.3, range = 1–14) per teacher (or cluster), depending on which level(s) or grades of LST the instructor taught and how many years they taught LST, as well as scheduling conflicts, teacher turnover, and other challenges that may have prevented the above proposed number of observations to occur. Each of the 279 raters conducted an average of 22.3 classroom observations (<emph>SD</emph> = 20.2, range: 1–112) and observed lessons for an average of 5.8 teachers. Observed sessions lasted an average of 43.4 min (<emph>SD</emph> = 12.6, range: 13 – 135), with 24 observations shorter than 20 min and 21 observations longer than 90 min. Our general rule was that observations needed to last at least 20 min to be included in the analysis, with exceptions made when teachers' schedules were such that all their lessons were under 20 min. Each year, raters were instructed to observe each Level 1 teacher four times (27% of the 15 Level 1 core sessions), each Level 2 teacher three times (30% of the 10 Level 2 core sessions), and each Level 3 teacher two times (40% of the 5 Level 3 core sessions).</p> <hd id="AN0183370751-12">Study Context</hd> <p>Schools were provided with training, technical assistance, material support, and fidelity monitoring at no cost to encourage and facilitate high-quality LST implementation so that students received the information and practice to achieve optimal behavioral outcomes associated with skill building and substance abuse prevention programming. All teachers received a one- or two-day training workshop in their first year of implementation and were offered an optional one-day booster training in the following year(s). Like the rater training, the LST instructor training was conducted by a cadre of certified LST trainers within the program developer's training firm, National Health Promotion Associates ([<reflink idref="bib71" id="ref122">71</reflink>]). Training was required for all LST instructors and encouraged for school administrators and support staff. Technical assistance was provided throughout the project, which included regular correspondence and annual visits with "project coordinators" who were employed with the dissemination project to oversee, coordinate, and support LST implementation and who met with LST teachers, classroom raters, and school administration throughout the study period to discuss implementation progress and problems. Phone-based and on-site technical assistance was offered as needed, generally after teachers started delivering lessons and had identified specific questions, and project coordinators often attended training and technical assistance workshops alongside school district personnel as part of comprehensive implementation support and guidance. Additionally, project coordinators provided a formal report to the school district at the end of each year summarizing process evaluation results.</p> <hd id="AN0183370751-13">Analysis</hd> <p>First, descriptive statistics were conducted in SPSS 29 to report school demographics. The percentage of students receiving free or reduced-price lunch (FRL) was used as a proxy for students experiencing socioeconomic disadvantage. School-level FRL data were collected using 2019-2020 datasets administered and made publicly available by the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education's Institute of Education Sciences. School-level characteristics such as race and ethnicity, student-teacher ratio, and locale were also collected via NCES 2019–2020 datasets. Though 2018–2019 NCES data would have been best (to match the period of this study), after the study was completed and the manuscript was written, we compared the 2018–2020 NCES data to the 2018–2019 NCES data and no major differences were found.</p> <p>Second, to assess the factor structure of the FOI items on the SRF, confirmatory factor analysis (CFA) and measurement invariance analyses were conducted in Mplus 8.1 ([<reflink idref="bib70" id="ref123">70</reflink>]). We used CFA to test the fit of the two-factor structure of "quality of delivery" and "participant responsiveness" for the full sample of observations. Because teachers had multiple observations of lessons, we adjusted standard errors to account for clustering among teachers. Teachers were also clustered within schools and thus we considered accounting for clustering at this second, higher level. However, given that over half of schools had three or fewer LST facilitators for the duration of the project (so likely were too small to provide a reliable cluster estimates) and since adding this second level would greatly increase the complexity of our models, we determined that parsimony was a stronger option. Also, in most cases the "teacher" represented the school or a specific grade within a school. Thus, we felt that accounting for clustering within person (at the teacher level) was most critical and sufficiently addressed our clustered data concerns.</p> <p>The CFA and measurement invariance analyses used robust maximum likelihood (MLR estimation; [<reflink idref="bib59" id="ref124">59</reflink>]). MLR is recommended with categorical variables, such as Likert scale responses, because it accounts for the categorical nature of the variables and provides more robust parameter estimates and standard errors under conditions where the assumption of normality may be violated. ([<reflink idref="bib3" id="ref125">3</reflink>]; [<reflink idref="bib60" id="ref126">60</reflink>]; [<reflink idref="bib76" id="ref127">76</reflink>]). All Likert scale variables were roughly normal (though positively skewed) and treated as continuous. For the CFA, the first loading of each factor was constrained to one, and all nonfixed unstandardized factor loadings were freely estimated. Good model fit was established a priori and included: non-statistically significant chi-square statistic, comparative fit index (CFI) and Tucker–Lewis Index (TLI) ≥ 0.95, root mean square error of approximation (RMSEA) ≤ 0.08, and SRMR ≤ 0.06. Because the chi-square is influenced by large sample sizes ([<reflink idref="bib56" id="ref128">56</reflink>]), the CFI, TLI, RMSEA, and SRMR were prioritized. Cutoff criteria to accept more constrained models in measurement invariance analyses were: ΔCFI ≤ −.010, ΔRMSEA <.015, and ΔSRMR greater than or equal to.030 (for metric invariance) and greater than or equal to.010 (for scalar invariance) ([<reflink idref="bib19" id="ref129">19</reflink>]; [<reflink idref="bib21" id="ref130">21</reflink>]; [<reflink idref="bib51" id="ref131">51</reflink>]).</p> <p>After fitting the CFA for the full sample, we used multigroup CFA ([<reflink idref="bib39" id="ref132">39</reflink>]) to test the measurement structure across grade level, and proportion of students receiving FRL at the school-level. Because subgroups within a population are often heterogeneous in terms of the parameter values of a model, assuming homogeneity of the population is not appropriate ([<reflink idref="bib69" id="ref133">69</reflink>]). Tests of measurement invariance using the multigroup technique were conducted to examine whether the factor structure of the SRF scores was equivalent across classrooms from high, middle, and low SES schools (in which the distribution of each school's percentage of students receiving FRL was used to create these tertial SES groups) as well as the three LST program levels (reflecting implementation plans of 6<sups>th</sups>–8<sups>th</sups> or 7<sups>th</sups>–9<sups>th</sups>). Specifically, we conducted tests of configural, metric, and scalar invariance ([<reflink idref="bib74" id="ref134">74</reflink>]; [<reflink idref="bib88" id="ref135">88</reflink>]) to evaluate the extent to which raters were using the SRF observation protocol in the same way across different school settings and hence whether FOI scores could be compared across different school settings. Configural invariance tested whether the same factor structure existed between groups (i.e., if the same items loaded onto the respective factors across the groups). Metric invariance tested whether factor loadings were equivalent across groups and hence whether factor variances could be compared. Scalar invariance tested whether item intercepts were equal across groups and hence whether factor means could be compared across groups.</p> <p>Finally, we used SPSS 29 to assess internal consistency (i.e., Cronbach's alpha) of each subscale on the SRF measuring "quality of delivery" and "student responsiveness." SPSS 29 was also used to report means and standard deviations of the fidelity items on the SRF observation protocol (overall, as well as by grade level and school setting).</p> <hd id="AN0183370751-14">Results</hd> <p></p> <hd id="AN0183370751-15">Sample Description</hd> <p>Table 1 displays school-level demographics for the full sample (<emph>n</emph> = 379 schools) as well as by high (<emph>n</emph> = 126 schools), medium (<emph>n</emph> = 127 schools), and low (<emph>n</emph> = 126 schools) SES schools. On average, schools represented in the full sample were mid-size (enrolled around 675 students), had a ratio of 15 students per teacher (which is consistent with the current national average in the United States). For the overall sample, 60% of the schools were in suburban/urban communities and 40% were in rural/town communities. In addition, on average, schools served 59% FRL-eligible students, 52% White students, 24% Black or African American students, and 17% Hispanic or Latino students.</p> <p>Table 1. School Descriptives for Full Sample and SES Subgroups.</p> <p>Graph</p> <p></p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" rowspan="2" /><th align="center">Total (<italic>N</italic> = 379)</th><th align="center">Low SES (<italic>N</italic> = 126)</th><th align="center">Med SES (<italic>N</italic> = 127)</th><th align="center">High SES (<italic>N</italic> = 126)</th><th align="center" rowspan="2"><italic>F Test/Chi-Square</italic></th></tr><tr><th align="left"><italic>%/Mean (SD)</italic></th><th align="center"><italic>%/Mean (SD)</italic></th><th align="center"><italic>%/Mean (SD)</italic></th><th align="center"><italic>%/Mean (SD)</italic></th></tr></thead><tbody valign="top"><tr><td align="left">Total enrollment</td><td align="char" char="(">675.0 (357.0)</td><td align="char" char="(">604.9 (295.4)</td><td align="char" char="(">676.4 (327.5)</td><td align="char" char="(">744.70 (424.3)</td><td align="char" char=".">5.0**</td></tr><tr><td align="left">Student-teacher ratio</td><td align="char" char="(">15.4 (3.4)</td><td align="char" char="(">15.0 (3.7)</td><td align="char" char="(">15.8 (3.1)</td><td align="char" char="(">15.43 (3.2)</td><td align="char" char=".">2.0</td></tr><tr><td align="left">% of students with FRL</td><td align="char" char="(">59.1 (28.0)</td><td align="char" char="(">91.4 (8.6)</td><td align="char" char="(">58.5 (8.1)</td><td align="char" char="(">27.5 (12.9)</td><td align="char" char=".">1254.4***</td></tr><tr><td align="left" colspan="6">Locale</td></tr><tr><td align="left"> Rural</td><td align="center">29.6%</td><td align="center">17.6%</td><td align="center">43.5%%</td><td align="center">27.8%</td><td align="char" char=".">20.4***</td></tr><tr><td align="left"> Suburban/urban</td><td align="center">70.4%</td><td align="center">82.4%</td><td align="center">56.5%</td><td align="center">72.2%</td><td align="left" /></tr><tr><td align="left" colspan="6">Race and ethnicity</td></tr><tr><td align="left"> American Indian</td><td align="char" char="(">0.4 (1.5)</td><td align="char" char="(">0.2 (0.3)</td><td align="char" char="(">0.6 (1.9)</td><td align="char" char="(">0.3 (1.7)</td><td align="char" char=".">2.2</td></tr><tr><td align="left"> Asian</td><td align="char" char="(">2.1 (3.7)</td><td align="char" char="(">1.6 (3.8)</td><td align="char" char="(">1.5 (2.7)</td><td align="char" char="(">3.1 (4.2)</td><td align="char" char=".">7.3***</td></tr><tr><td align="left"> Black</td><td align="char" char="(">24.1 (27.3)</td><td align="char" char="(">41.7 (30.7)</td><td align="char" char="(">20.4 (22.9)</td><td align="char" char="(">10.2 (16.3)</td><td align="char" char=".">56.6***</td></tr><tr><td align="left"> Hispanic</td><td align="char" char="(">17.4 (20.7)</td><td align="char" char="(">29.1 (27.5)</td><td align="char" char="(">14.3 (15.0)</td><td align="char" char="(">8.8 (9.0)</td><td align="char" char=".">39.2***</td></tr><tr><td align="left"> Native Hawaiian/Pacific Islander</td><td align="char" char="(">0.1 (0.2)</td><td align="char" char="(">0.1 (0.2)</td><td align="char" char="(">0.1 (0.2)</td><td align="char" char="(">0.1 (0.2)</td><td align="char" char=".">2.2</td></tr><tr><td align="left"> White</td><td align="char" char="(">52.0 (33.0)</td><td align="char" char="(">23.6 (25.5)</td><td align="char" char="(">58.8 (28.9)</td><td align="char" char="(">73.5 (22.1)</td><td align="char" char=".">126.3***</td></tr><tr><td align="left"> Multiracial</td><td align="char" char="(">4.0 (2.9)</td><td align="char" char="(">3.6 (3.5)</td><td align="char" char="(">4.2 (2.7)</td><td align="char" char="(" style="color:#010205">4.02 (2.4)</td><td align="char" char=".">1.4</td></tr></tbody></table> </ephtml> </p> <p>1 Notes. SES, socio-economic status.</p> <ulist> <item>2 <emph>F</emph> statistic and <emph>chi-square</emph> tests for differences across the three SES groups.</item> <item>3 **<emph>p</emph> <.01, ***<emph>p</emph> <.001.</item> </ulist> <hd id="AN0183370751-16">Model Fit</hd> <p>One way to evaluate model fit is to assess the extent to which the data support the protocol's hypothesized domain structure ([<reflink idref="bib2" id="ref136">2</reflink>]). CFA was performed on the full sample to verify the two-factor structure within the SRF that measured "quality of delivery" and "student responsiveness." Though initial results suggested adequate model fit, the modification indices suggested that allowing several items within the "quality of delivery" factor to covary (i.e., correlating their residuals) would improve fit. Specifically, we specified that knowledge covary with overall quality of session and rapport, and that classroom management covary with knowledge and enthusiasm. Given the theoretical similarity of these items, these modifications were reasonable. With these modifications, strong model fit was achieved on four of the five fit statistics (Table 2). Though the chi-square test of model fit was statistically significant (<emph>χ</emph><sups><emph>2</emph></sups>(<reflink idref="bib28" id="ref137">28</reflink>) = 397.1, <emph>p</emph> <.001), suggesting that that there was significant difference in the patterns observed in the data versus the model specified, the chi-square statistic is influenced by large sample sizes (e.g., >400; [<reflink idref="bib56" id="ref138">56</reflink>]). Given the large size of this sample, other fit statistics were examined and prioritized. The CFI/TLI were well above >.95, the RMSEA below.05, and the SRMR below.04, and thus within the specified limits that suggest strong model fit. All factor loadings (see Table 2) were statistically significant at <emph>p</emph> <.001. Overall, factor loadings were consistently high, with standardized loadings on the "quality of delivery" factor ranging from.84 to.90, and the standardized loadings on the "participant responsiveness" factor ranging from.82 to.88. In addition, for all indicators, at least two-thirds of the variance was explained by the underlying factor. Specifically, <emph>R</emph><sups>2</sups> values for indicators ranged from.67 to.82. The correlation between the two factors was <emph>r</emph> = 0.84. Since four of the five fit statistics demonstrated good model fit and standardized factor loadings were uniformly high (above.82), these results indicate that items correlate within each factor and support the interpretation that the observation protocol measured two dimensions of FOI (quality of delivery and participant responsiveness).</p> <p>Table 2. Items and Factors Loadings for Confirmatory Factor Analysis.</p> <p>Graph</p> <p></p> <p> <ephtml> <table><thead valign="top"><tr><th align="left">Construct/Items within Construct</th><th align="center">unstandardized</th><th align="center">SE</th><th align="center">Standardized</th><th align="center">SE</th></tr></thead><tbody valign="top"><tr><td align="left" colspan="5">Quality of delivery</td></tr><tr><td align="left"> Knowledge of program/lesson content</td><td align="char" char=".">1.00</td><td align="char" char=".">0.00</td><td align="char" char=".">0.84</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> Level of enthusiasm</td><td align="char" char=".">1.01</td><td align="char" char=".">0.02</td><td align="char" char=".">0.87</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> Poise and confidence</td><td align="char" char=".">0.96</td><td align="char" char=".">0.02</td><td align="char" char=".">0.90</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> Rapport and communication with students</td><td align="char" char=".">1.05</td><td align="char" char=".">0.02</td><td align="char" char=".">0.90</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> Classroom management</td><td align="char" char=".">1.14</td><td align="char" char=".">0.03</td><td align="char" char=".">0.84</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> Effectively addressed question/concerns</td><td align="char" char=".">1.05</td><td align="char" char=".">0.02</td><td align="char" char=".">0.89</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> Overall quality of the program session</td><td align="char" char=".">1.14</td><td align="char" char=".">0.02</td><td align="char" char=".">0.90</td><td align="char" char=".">0.01</td></tr><tr><td align="left" colspan="5">Participant responsiveness</td></tr><tr><td align="left"> How well students responded to the session</td><td align="char" char=".">1.00</td><td align="char" char=".">0.00</td><td align="char" char=".">0.84</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> How actively students participated in discussion/activities</td><td align="char" char=".">0.99</td><td align="char" char=".">0.01</td><td align="char" char=".">0.82</td><td align="char" char=".">0.01</td></tr><tr><td align="left"> How well students understood the lesson</td><td align="char" char=".">1.04</td><td align="char" char=".">0.02</td><td align="char" char=".">0.88</td><td align="char" char=".">0.01</td></tr><tr><td align="left">Quality of delivery with student responsiveness</td><td align="char" char=".">0.43</td><td align="char" char=".">0.20</td><td align="char" char=".">0.84</td><td align="char" char=".">0.01</td></tr></tbody></table> </ephtml> </p> <ulist> <item>4 Note. All factor loading estimates <emph>p</emph> <.001.</item> <item>5 Note. Fit statistics for the CFA: <emph>chi-square</emph> = 397.1, df = 28, <emph>p</emph> <.001; <emph>TLI</emph> = 0.982; <emph>CFI</emph> = 0.989; <emph>RMSEA</emph> = 0.046; <emph>SRMR</emph> = 0.015.</item> </ulist> <hd id="AN0183370751-17">Measurement Invariance</hd> <p>The CFA model was also used to test measurement invariance across high, medium, and low SES subgroups of schools (i.e., percentage of students receiving FRL) and across the three LST levels in all schools. As shown in Table 3, configural, metric, and scalar invariance was achieved across groups as fit statistics remained similar when the models were increasingly constrained. Specifically, configural invariance was first examined by specifying the same factor loading structure across different groups while allowing all other parameters to differ. Except for the chi-square statistic, the other four fit indices suggested strong fit across SES subgroups and grade levels with the CFI/TLI remaining above.95, RMSEA below.05, and SRMR below.04. Metric invariance was tested by requiring the same factor structure and equal factor loadings across groups while all other parameters were allowed to differ. Scalar invariance was tested by constraining equal item intercepts and factor loadings across different groups while allowing other parameters to differ. In these increasingly constrained models, fit statistics remained within thresholds, and the change in CFI, RMSEA, and SRMR were under the cutoffs. Items therefore showed full configural, metric, and scalar invariance, suggesting the two-domain structure (quality of delivery and student responsiveness) was generalizable across LST levels and diverse economic school environments.</p> <p>Table 3. Fit Statistics for Full Sample CFA and Measurement Invariance Tests.</p> <p>Graph</p> <p></p> <p> <ephtml> <table><thead valign="top"><tr><th align="left">Model</th><th align="center">Chi-Square</th><th align="center">df</th><th align="center">TLI</th><th align="center">CFI</th><th align="center">ΔCFI</th><th align="center">RMSEA</th><th align="center">ΔRMSEA</th><th align="center">SRMR</th><th align="center">ΔSRMR</th></tr></thead><tbody valign="top"><tr><td align="left" colspan="10">SES groups</td></tr><tr><td align="left"> Configural</td><td align="char" char=".">477.2***</td><td align="char" char=".">84</td><td align="char" char=".">0.981</td><td align="char" char=".">0.988</td><td align="char" char=".">---</td><td align="char" char=".">0.048</td><td align="char" char=".">---</td><td align="char" char=".">0.017</td><td align="char" char=".">---</td></tr><tr><td align="left"> Metric</td><td align="char" char=".">502.6***</td><td align="char" char=".">100</td><td align="char" char=".">0.984</td><td align="char" char=".">0.988</td><td align="char" char=".">0.003</td><td align="char" char=".">0.044</td><td align="char" char=".">−0.004</td><td align="char" char=".">0.035</td><td align="char" char=".">0.018</td></tr><tr><td align="left"> Scalar</td><td align="char" char=".">545.4***</td><td align="char" char=".">116</td><td align="char" char=".">0.985</td><td align="char" char=".">0.987</td><td align="char" char=".">0.001</td><td align="char" char=".">0.042</td><td align="char" char=".">−0.002</td><td align="char" char=".">0.04</td><td align="char" char=".">0.005</td></tr><tr><td align="left" colspan="10">LST level (i.e., grade level)</td></tr><tr><td align="left"> Configural</td><td align="char" char=".">484.8***</td><td align="char" char=".">84</td><td align="char" char=".">0.981</td><td align="char" char=".">0.988</td><td align="char" char=".">---</td><td align="char" char=".">0.048</td><td align="char" char=".">---</td><td align="char" char=".">0.016</td><td align="char" char=".">---</td></tr><tr><td align="left"> Metric</td><td align="char" char=".">506.6***</td><td align="char" char=".">100</td><td align="char" char=".">0.984</td><td align="char" char=".">0.988</td><td align="char" char="."><0.001</td><td align="char" char=".">0.044</td><td align="char" char=".">−0.004</td><td align="char" char=".">0.023</td><td align="char" char=".">0.007</td></tr><tr><td align="left"> Scalar</td><td align="char" char=".">562.7***</td><td align="char" char=".">116</td><td align="char" char=".">0.985</td><td align="char" char=".">0.987</td><td align="char" char="."><0.001</td><td align="char" char=".">0.043</td><td align="char" char=".">−0.001</td><td align="char" char=".">0.026</td><td align="char" char=".">0.003</td></tr></tbody></table> </ephtml> </p> <p>6 Note. *** indicates <emph>p</emph> <.001. CFA, confirmatory factor analysis.</p> <hd id="AN0183370751-18">Reliability</hd> <p>"Quality of delivery" and "participant responsiveness" dimensions both had strong internal consistency for the sample (α =.92 and.86, respectively), indicating that the required training and follow-up coaching helped generate reliable estimates for the FOI constructs within the SRF.</p> <hd id="AN0183370751-19">Variance in Scores Across Grade Level and School Setting</hd> <p>Table 4 displays descriptive statistics of each item measuring "fidelity to process" for the total sample and for each group (schools with high, medium, and low proportions of students on FRL). Average item scores ranged from 4.18 to 4.45 for the overall sample, indicating high fidelity. Still, results showed sufficient variance among aggregate groups, such as at the instructor, LST program (i.e., grade), and school-levels. For example, the mean "quality of delivery" score across instructors was 4.35 (SD = 0.80), and the mean "participant responsiveness" score was 4.31 (SD = 0.75). The low standard deviation (SD) shows that the data were clustered closely around the mean and suggests little difference across classrooms. Meanwhile, means across the SES groups ranged from 3.96 to 4.28 for low SES schools, 4.22 to 4.46 for medium SES schools, and 4.34 to 4.58 for high SES schools, while means across LST levels ranged from 4.17 to 4.43 for Level 1 (typically implemented in grade 6), 4.23 to 4.47 for Level 2 (generally implemented in grade 7), and 4.18 to 4.46 for Level 3 (usually implemented in grade 8). All item scores were statistically significantly different between SES groups (<emph>p</emph> <.05), except for the scores on knowledge between low and medium SES schools. Related to LST levels, no item scores were statistically significant except for knowledge between Levels 1 and 2. Thus, variability in the average score was systematically higher for schools with a lower proportion of FRL students relative to schools serving a larger proportion of FRL students—whereas no similar pattern was detected by grade level.</p> <p>Table 4. Item Descriptives for Full Sample and Subgroups (N = 6219 Observations).</p> <p>Graph</p> <p></p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" rowspan="2" /><th align="center" colspan="2">Total (<italic>N</italic> = 6219)</th><th align="center" colspan="2">Low SES (<italic>n</italic> = 1814)</th><th align="center" colspan="2">Med SES (<italic>n</italic> = 2346)</th><th align="center" colspan="2">High SES (<italic>n</italic> = 2059)</th><th align="center" colspan="2">Level 1<xref ref-type="table-fn" rid="tfn8">a</xref> (<italic>n</italic> = 3970)</th><th align="center" colspan="2">Level 2<xref ref-type="table-fn" rid="tfn8">a</xref> (<italic>n</italic> = 1668)</th><th align="center" colspan="2">Level 3<xref ref-type="table-fn" rid="tfn8">a</xref> (<italic>n</italic> = 581)</th></tr><tr><th align="left">M</th><th align="center">sd</th><th align="center">M</th><th align="center">sd</th><th align="center">M</th><th align="center">sd</th><th align="center">M</th><th align="center">sd</th><th align="center">M</th><th align="center">sd</th><th align="center">M</th><th align="center">sd</th><th align="center">M</th><th align="center">sd</th></tr></thead><tbody valign="top"><tr><td align="left">Quality of delivery</td><td align="left" colspan="14">Prompt: <italic>"On the following scale, rate the implementer on the following qualities..."</italic></td></tr><tr><td align="left"> Knowledge of program/lesson content</td><td align="char" char=".">4.36</td><td align="char" char=".">0.98</td><td align="char" char=".">4.15</td><td align="char" char=".">1.14</td><td align="char" char=".">4.41</td><td align="char" char=".">0.94</td><td align="char" char=".">4.48</td><td align="char" char=".">0.82</td><td align="char" char=".">4.32</td><td align="char" char=".">1.02</td><td align="char" char=".">4.43</td><td align="char" char=".">1.41</td><td align="char" char=".">4.39</td><td align="char" char=".">0.86</td></tr><tr><td align="left"> Level of enthusiasm</td><td align="char" char=".">4.34</td><td align="char" char=".">0.98</td><td align="char" char=".">4.14</td><td align="char" char=".">1.10</td><td align="char" char=".">4.37</td><td align="char" char=".">0.95</td><td align="char" char=".">4.47</td><td align="char" char=".">0.88</td><td align="char" char=".">4.32</td><td align="char" char=".">1.03</td><td align="char" char=".">4.37</td><td align="char" char=".">1.07</td><td align="char" char=".">4.37</td><td align="char" char=".">0.86</td></tr><tr><td align="left"> Poise and confidence</td><td align="char" char=".">4.45</td><td align="char" char=".">0.98</td><td align="char" char=".">4.28</td><td align="char" char=".">1.15</td><td align="char" char=".">4.46</td><td align="char" char=".">1.02</td><td align="char" char=".">4.58</td><td align="char" char=".">0.71</td><td align="char" char=".">4.43</td><td align="char" char=".">0.97</td><td align="char" char=".">4.47</td><td align="char" char=".">0.94</td><td align="char" char=".">4.46</td><td align="char" char=".">0.98</td></tr><tr><td align="left"> Rapport/communication with students</td><td align="char" char=".">4.40</td><td align="char" char=".">1.00</td><td align="char" char=".">4.21</td><td align="char" char=".">1.12</td><td align="char" char=".">4.4</td><td align="char" char=".">1.04</td><td align="char" char=".">4.56</td><td align="char" char=".">0.80</td><td align="char" char=".">4.38</td><td align="char" char=".">1.06</td><td align="char" char=".">4.44</td><td align="char" char=".">0.90</td><td align="char" char=".">4.41</td><td align="char" char=".">0.89</td></tr><tr><td align="left"> Classroom management</td><td align="char" char=".">4.23</td><td align="char" char=".">1.18</td><td align="char" char=".">4.08</td><td align="char" char=".">1.32</td><td align="char" char=".">4.22</td><td align="char" char=".">1.17</td><td align="char" char=".">4.38</td><td align="char" char=".">1.03</td><td align="char" char=".">4.22</td><td align="char" char=".">1.2</td><td align="char" char=".">4.26</td><td align="char" char=".">0.79</td><td align="char" char=".">4.21</td><td align="char" char=".">1.05</td></tr><tr><td align="left"> Effectively addressed question/concerns</td><td align="char" char=".">4.26</td><td align="char" char=".">1.53</td><td align="char" char=".">4.06</td><td align="char" char=".">1.65</td><td align="char" char=".">4.32</td><td align="char" char=".">1.34</td><td align="char" char=".">4.38</td><td align="char" char=".">1.59</td><td align="char" char=".">4.23</td><td align="char" char=".">1.57</td><td align="char" char=".">4.32</td><td align="char" char=".">0.86</td><td align="char" char=".">4.32</td><td align="char" char=".">1.50</td></tr><tr><td align="left"> Overall quality of the program session</td><td align="char" char=".">4.18</td><td align="char" char=".">1.12</td><td align="char" char=".">3.96</td><td align="char" char=".">1.23</td><td align="char" char=".">4.22</td><td align="char" char=".">1.11</td><td align="char" char=".">4.34</td><td align="char" char=".">0.98</td><td align="char" char=".">4.17</td><td align="char" char=".">1.1</td><td align="char" char=".">4.23</td><td align="char" char=".">0.86</td><td align="char" char=".">4.18</td><td align="char" char=".">1.33</td></tr><tr><td align="left" colspan="15">Participant responsiveness</td></tr><tr><td align="left"> Rate how well students responded to the session</td><td align="char" char=".">4.29</td><td align="char" char=".">0.88</td><td align="char" char=".">4.11</td><td align="char" char=".">0.90</td><td align="char" char=".">4.3</td><td align="char" char=".">0.83</td><td align="char" char=".">4.45</td><td align="char" char=".">0.88</td><td align="char" char=".">4.31</td><td align="char" char=".">0.83</td><td align="char" char=".">4.27</td><td align="char" char=".">0.98</td><td align="char" char=".">4.23</td><td align="char" char=".">1.00</td></tr><tr><td align="left"> Rate how actively students participated in discussion and activities</td><td align="char" char=".">4.29</td><td align="char" char=".">0.88</td><td align="char" char=".">4.1</td><td align="char" char=".">0.97</td><td align="char" char=".">4.29</td><td align="char" char=".">0.85</td><td align="char" char=".">4.45</td><td align="char" char=".">0.80</td><td align="char" char=".">4.31</td><td align="char" char=".">0.85</td><td align="char" char=".">4.25</td><td align="char" char=".">0.89</td><td align="char" char=".">4.23</td><td align="char" char=".">1.01</td></tr><tr><td align="left"> Rate how well students understood the lesson</td><td align="char" char=".">4.32</td><td align="char" char=".">0.93</td><td align="char" char=".">4.09</td><td align="char" char=".">1.02</td><td align="char" char=".">4.33</td><td align="char" char=".">0.93</td><td align="char" char=".">4.51</td><td align="char" char=".">0.80</td><td align="char" char=".">4.30</td><td align="char" char=".">0.98</td><td align="char" char=".">4.36</td><td align="char" char=".">1.05</td><td align="char" char=".">4.33</td><td align="char" char=".">0.97</td></tr></tbody></table> </ephtml> </p> <ulist> <item>7 Notes. The minimum and maximum for all items (for full sample and the six subsamples described here) were 1 and 5. All item scores were statistically significant between SES groups (<emph>p</emph> <.05), except for the scores on knowledge between low and med SES. Related to LST Levels, no item scores were statistically significant except for knowledge between Levels 1 and 2.</item> <item>8 <sups>a</sups>LST Level 1 primarily reflects 6<sups>th</sups> grade, Level 2 primarily 7<sups>th</sups> grade, and Level 3 primarily 8<sups>th</sups> grade with few systems implementing LST in 7<sups>th</sups>, 8<sups>th</sups>, and 9<sups>th</sups> grades, respectively. For the present study, 90% of schools delivered LST in grades 6–8 and 10% in grades 7–9.</item> <item>9 <sups>b</sups>All item scores were statistically significantly different between SES groups (<emph>p </emph><.05), except for the scores on knowledge between low and medium SES schools. Related to LST levels, no item scores were statistically significant except for knowledge between Level 1 and 2.</item> </ulist> <hd id="AN0183370751-20">Discussion</hd> <p>Drawing from the validity argument framework proposed by [<reflink idref="bib54" id="ref139">54</reflink>] and used to evaluate classroom observational protocols in assessing teacher quality for high-stakes personnel decisions ([<reflink idref="bib2" id="ref140">2</reflink>]) and the quality of teachers' enactment of mathematics instruction ([<reflink idref="bib48" id="ref141">48</reflink>]), this study aimed to build evidence of validity for a classroom observational protocol measuring FOI of the LST middle school program, which has been shown in several well-conducted and well-implemented randomized control trials to reduce risky behaviors with results maintained over time ([<reflink idref="bib8" id="ref142">8</reflink>], [<reflink idref="bib10" id="ref143">10</reflink>]). Observation items measuring fidelity to process of LST have never been validated (<reflink idref="bib1" id="ref144">1</reflink>) despite wide dissemination of the program across the United States ([<reflink idref="bib34" id="ref145">34</reflink>]) and worldwide ([<reflink idref="bib52" id="ref146">52</reflink>]; [<reflink idref="bib53" id="ref147">53</reflink>]; [<reflink idref="bib58" id="ref148">58</reflink>]; [<reflink idref="bib84" id="ref149">84</reflink>]), and (<reflink idref="bib2" id="ref150">2</reflink>) even though these items have been used to report FOI to the LST middle school model for more than a decade and are still actively reported in LST studies ([<reflink idref="bib24" id="ref151">24</reflink>], [<reflink idref="bib23" id="ref152">23</reflink>], [<reflink idref="bib25" id="ref153">25</reflink>]; [<reflink idref="bib32" id="ref154">32</reflink>]; [<reflink idref="bib67" id="ref155">67</reflink>]; [<reflink idref="bib85" id="ref156">85</reflink>]).</p> <p>In adapting [<reflink idref="bib54" id="ref157">54</reflink>] validity argument and assumptions framework, [<reflink idref="bib48" id="ref158">48</reflink>] identified four areas requiring investigation to build validity evidence: scoring, generalizability, extrapolation, and decision. This paper focused on satisfying the extrapolation assumption because limitations of our dataset restricted us from examining other assumptions. In addition, the two-factor FOI structure of "quality of delivery" and "student responsiveness" on the SRF is (<reflink idref="bib1" id="ref159">1</reflink>) used in the present dissemination project to report to the funder and participating schools implementation outcomes, or the extent to which instructors delivered LST fully and according to program guidelines, and (<reflink idref="bib2" id="ref160">2</reflink>) commonly cited in the fidelity literature to inform scale-up and sustainability of classroom-based drug prevention programs and/or moderate treatment effects ([<reflink idref="bib17" id="ref161">17</reflink>]; [<reflink idref="bib27" id="ref162">27</reflink>]; [<reflink idref="bib31" id="ref163">31</reflink>]; [<reflink idref="bib68" id="ref164">68</reflink>]; [<reflink idref="bib72" id="ref165">72</reflink>]). Our CFA results demonstrated that four of the five fit statistics demonstrated good model fit and standardized factor loadings were uniformly high (above.82), which indicates items correlate within each factor and supports the interpretation that the FOI items on the SRF protocol measures two dimensions of FOI (quality of delivery and participant responsiveness). Meanwhile, FOI items measuring "quality of delivery" and "student responsiveness" on the SRF showed full configural, metric, and scalar invariance across grade levels and diverse economic school environments. In addition, "quality of delivery" and "participant responsiveness" both had strong internal consistency for the sample (Cronbach's alpha =.92 and.86, respectively).</p> <p>However, we found that LST was being consistently implemented with high fidelity across grade levels but more faithfully implemented in higher SES schools relative to lower SES schools (see Table 4). On the other hand, these differences by SES could indicate potential bias in the FOI items within the SRF. Systematic bias that is consistent across all items measuring a particular factor is not detected by a CFA; as such, this finding warrants further investigation, particularly since validation of measures is an iterative task.</p> <p>While CFA primarily assesses the internal structure and factor relationships within a test, it indirectly supports Kane's extrapolation assumption by providing evidence of the stability and consistency of the underlying constructs measured across different contexts (in this case, grade level and school settings). Both CFA and Kane's framework contribute to the broader goal of ensuring that inferences drawn from measurement scores are valid and applicable beyond the specific conditions of the assessment. Thus, our results provide some evidence to support the extrapolation assumption needed to interpret SRF items as measuring quality of delivery and participant responsiveness. However, our application of the argument approach does not provide a conclusive appraisal of the strength of the validity argument for using SRF scores to measure the degree to which LST is implemented as intended, since the strength of the validity argument approach is in its ability to summarize and assess varied evidence collected in different contexts by multiple researchers ([<reflink idref="bib2" id="ref166">2</reflink>]).</p> <p>Our paper evaluates model fit of a two-factor structure (quality and responsiveness) through CFA and assesses consistency of the factor structure across settings (measurement invariance) and items (i.e., internal consistency)—all of which are important pieces of validity evidence supporting the extrapolation assumption. Still, there are additional critical elements of the extrapolation assumption that our analysis does not examine, such as relationships between the SRF and other variables (e.g., whether fidelity predicts student outcomes; [<reflink idref="bib12" id="ref167">12</reflink>]; [<reflink idref="bib38" id="ref168">38</reflink>]) and/or test consistency over time (test-retest reliability) or across researchers (i.e., interrater reliability or agreement; [<reflink idref="bib22" id="ref169">22</reflink>]; [<reflink idref="bib50" id="ref170">50</reflink>]; [<reflink idref="bib77" id="ref171">77</reflink>]). This limits how the validation process described by education measurement scholars such as [<reflink idref="bib54" id="ref172">54</reflink>] and outlined in professional standards ([<reflink idref="bib1" id="ref173">1</reflink>]) applies to fidelity measurement. We believe FOI measures that use direct observational evidence should be vetted using the same rigorous standards found in educational and psychological testing and measurement to support inferences about FOI when EBIs are widely implemented under real-world conditions.</p> <p>Minimal research has evaluated ways to marry effective and efficient fidelity measurement ([<reflink idref="bib78" id="ref174">78</reflink>]). We are aware of only a few EBIs in the intervention research base with studies describing how their FOI measure was scientifically validated. For example, [<reflink idref="bib62" id="ref175">62</reflink>] evaluated model fit through a CFA to validate an observation measure of fidelity to the School-Wide Positive Behavioral Interventions and Supports ([<reflink idref="bib13" id="ref176">13</reflink>]). Similarly, [<reflink idref="bib47" id="ref177">47</reflink>] used CFA to validate therapist fidelity to Multisystemic Therapy (MST; [<reflink idref="bib7" id="ref178">7</reflink>]) in real-world clinical settings. As explained above, evidence that the item scores are correlated and conform to a certain factor structure, however, is only one piece of the overall picture needed to support the claim that scores provide valid indicators of fidelity.</p> <p>A more complete overall depiction of the validity argument (and drawing on the factor structure where applicable) is needed to evaluate FOI measures. [<reflink idref="bib15" id="ref179">15</reflink>] offered such a road map, wherein researchers gathered validity evidence from various sources (i.e., based on test content, rater response processes, internal structure, and relations to other variables) following the guidelines of the Standards for Educational and Psychological Testing (<emph>Standards</emph>: [<reflink idref="bib1" id="ref180">1</reflink>]) and then summarized findings to provide a model for the development and validation of an observation protocol used to measure fidelity to Collaborative Strategic Reading, a multi-component reading comprehension instructional approach for middle school teachers ([<reflink idref="bib83" id="ref181">83</reflink>]). Meanwhile, our study drew from an argument-based approach ([<reflink idref="bib54" id="ref182">54</reflink>]) to demonstrate validity evidence for the FOI observational items within the SRF in satisfying assumptions for extrapolation. [<reflink idref="bib54" id="ref183">54</reflink>] and the <emph>Standards</emph> ([<reflink idref="bib1" id="ref184">1</reflink>]) provide distinct yet equally viable elaboration on the types of evidence and processes relevant to demonstrating validity of a measurement procedure. By describing these frameworks, our study adds to [<reflink idref="bib15" id="ref185">15</reflink>] in providing a guide that researchers can refer to in developing and validating their own FOI observation protocol without having to start from scratch, thereby streamlining the process.</p> <hd id="AN0183370751-21">Limitations</hd> <p>Our study should be interpreted in the context of some limitations. First, CFA tends to overestimate the fit among rater-based data when the same rater completes all items on each occasion and there are few raters per subject ([<reflink idref="bib2" id="ref186">2</reflink>]; [<reflink idref="bib48" id="ref187">48</reflink>]; [<reflink idref="bib63" id="ref188">63</reflink>]). For this study, teachers typically were observed by the same rater on different occasions (though some teachers had different raters over the three project years due to rater turnover). As such, the fit of our CFA may have been inflated because the items, for most teachers, are completed by a single rater. If there are rater-halo effects, this could lead to inflated fit statistics for a CFA model or bias conclusions about dimensionality. Similarly, although the estimates of internal consistency for each SRF sub score were high, they should be interpreted cautiously since a single rater completed the SRF and thus internal consistency estimates could also be inflated by halo effects. In addition, internal consistency estimates cannot quantify error variance due to rater or occasion facets, and thus likely underestimate the reliability of SRF scores across occasions. Lastly, though Cronbach's alpha is widely used to assess internal consistency, it has several limitations that can reduce accuracy, including sensitivity to scale length and sample size, an assumption of homogeneity among items, and a lack of consideration for non-linear relationships. Specifically related to this study, the large sample size may have positively biased the alphas.</p> <p>Second, we do not have interrater agreement or inter-reliability measures, so the degree of error from raters interpreting variables differently is unknown ([<reflink idref="bib64" id="ref189">64</reflink>]). While observer-reported measures are advantageous and considered more objective compared to teacher self-reports or data collected at the end of implementation ([<reflink idref="bib29" id="ref190">29</reflink>]), observational measures are affected by variation in implementation conditions such as the composition of the rater pool (e.g., number of raters or degree of training) and observation procedure (i.e., video or live; [<reflink idref="bib2" id="ref191">2</reflink>]; [<reflink idref="bib48" id="ref192">48</reflink>]; [<reflink idref="bib63" id="ref193">63</reflink>]). Classroom observations also contain multiple sources of error. For example, the score one observer provides can depend on the classroom behaviors they observe on a particular occasion, the observer's general level of leniency in scoring compared with other raters, and their leniencies for a specific lesson and/or teacher. This study controls for some of these potential rating errors by ensuring all raters were thoroughly trained in assessing fidelity of implementation of the LST model (i.e., attended LST workshops and received observer-role training, as well as project coordinator oversight in the form of ongoing feedback and co-observing and co-rating lessons). Despite these steps, the extent to which possible sources of variation described above are present in this study is unknown and future studies could investigate the stability of teacher FOI scores under different rating designs as well as the possible impact of rater effects on conclusions about FOI (as described further below).</p> <p>Finally, the two factors (quality of delivery and student responsiveness) were highly correlated. While the hypothesized (a priori) CFA implemented for this study had good model fit, it is possible that competing CFA models (e.g., unidimensional and bifactor models) offer a more plausible description of the data. The purpose of this study was to test the hypothesized two-factor model given the theoretical foundations of FOI domains and their extensive use in the field (for example, see [<reflink idref="bib17" id="ref194">17</reflink>]; [<reflink idref="bib27" id="ref195">27</reflink>]; [<reflink idref="bib31" id="ref196">31</reflink>]; [<reflink idref="bib68" id="ref197">68</reflink>]; [<reflink idref="bib72" id="ref198">72</reflink>]), as well as to provide an example of developing a validity argument for a FOI measure; thus, model building and testing was outside of the scope of this study.</p> <hd id="AN0183370751-22">Future Research</hd> <p>Key considerations for future research fall into two categories related (<reflink idref="bib1" id="ref199">1</reflink>) specifically to the example presented (i.e., the SRF observation protocol) and (<reflink idref="bib2" id="ref200">2</reflink>) to the field in general. Regarding the former, [<reflink idref="bib54" id="ref201">54</reflink>] and the <emph>Standards</emph> ([<reflink idref="bib1" id="ref202">1</reflink>]) both note that building validity evidence is an iterative process, one in which original interpretations may be revisited and revised considering data collected during the empirical inquiry. In adapting [<reflink idref="bib54" id="ref203">54</reflink>] validity argument and assumptions framework to build evidence of validity of teacher quality, [<reflink idref="bib48" id="ref204">48</reflink>] identified four areas requiring investigation to build validity evidence, and this study provides some empirical evidence for only one of these sources. Future studies should therefore continue to evaluate the interpretation and use of the LST FOI measures for their intended purposes by gathering additional validity evidence.</p> <p>For example, to build evidence supporting the scoring assumption, interrater agreement could help determine whether raters give the "correct" score to a lesson. This could be empirically evaluated by having raters rate videotaped lessons where the "correct" score is already known and investigating whether the raters give the correct score. Alternatively, if an expert rater attends a class observation with a rater and the rater gives the same scores as the expert, such interrater agreement will also provide evidence to support the scoring assumption claim. To build evidence of generalizability, researchers could examine how much a single instructor's FOI scores within the SRF vary from one lesson observation to the next (though this would require attention to whether we believe there are likely to be true changes in FOI for an instructor over time). To further build evidence of validity supporting the extrapolation assumption, researchers could examine whether FOI items within the SRF are positively associated with student behavioral outcomes (e.g., self-reported drug, tobacco or alcohol use), and whether schools (or districts) with higher FOI scores within the SRF have larger treatment effects. And finally, evidence supporting the decision assumption could represent whether feedback and advice given to LST instructors based on SRF scores appropriately reflect key instructor weaknesses and strengths ([<reflink idref="bib48" id="ref205">48</reflink>]). In the context of the FOI items within the SRF, researchers could consider what they hope to accomplish using this measure, and then evaluate whether this occurs in the debriefs with instructors after they are observed.</p> <p>In terms of the second consideration (i.e., future research for the field in general), increasingly, federal, state, and local governments are mandating the use of EBIs to limit unnecessary spending and achieve socially meaningful impact ([<reflink idref="bib16" id="ref206">16</reflink>]; [<reflink idref="bib37" id="ref207">37</reflink>]; [<reflink idref="bib57" id="ref208">57</reflink>]). In addition, states have begun incorporating evidence into budgeting by explicitly basing funding decisions, at least in part, on how rigorous evidence is supporting an intervention's effectiveness ([<reflink idref="bib41" id="ref209">41</reflink>]; [<reflink idref="bib44" id="ref210">44</reflink>]). And, as demand increases for broader dissemination and sustainment of EBIs, so does the expectation that service providers and organizations be held accountable for their outcomes ([<reflink idref="bib78" id="ref211">78</reflink>]). Given the rise in mandates that require evidence to support funding decisions, future research should continue to collect validity evidence from multiple sources because findings produced from valid measures of FOI can improve upon researchers' ability to attribute treatment effects to the intervention and help practitioners feel more confident in implementing the EBI as intended ([<reflink idref="bib15" id="ref212">15</reflink>]).</p> <p>In addition, EBIs serve diverse populations in terms of race, ethnicity, culture, and other sociodemographic factors. In disseminating empirical evidence to practice, questions are continually raised about the extent to which the results of impact trials can be generalized to new populations, geographic locations, and points in time. A key element of decision-makers looking to adopt an EBI is whether a given program will work in their community, which might be different from the population included in the evaluation ([<reflink idref="bib14" id="ref213">14</reflink>]; [<reflink idref="bib61" id="ref214">61</reflink>]; [<reflink idref="bib80" id="ref215">80</reflink>]). To understand how EBIs work across different samples and settings, future research that uses valid FOI observational measures will be valuable to collect data on core components—or the essential intervention activities (e.g., active ingredients and behavioral kernels; [<reflink idref="bib36" id="ref216">36</reflink>]) that are judged necessary to produce desired impacts ([<reflink idref="bib5" id="ref217">5</reflink>])—as well as aspects of both fidelity and adaptation that often co-occur during dissemination ([<reflink idref="bib30" id="ref218">30</reflink>]). Adapting EBIs so they respond to the needs and strengths of the local community and the groups who can most benefit, however, will require co-creating and piloting items measuring FOI with community members possessing relevant historical information and knowledge of cultural norms representing the voice of those who operate and participate in EBIs to both test assumptions, and support continuous improvement, of the adaptation and implementation strategy ([<reflink idref="bib61" id="ref219">61</reflink>]; [<reflink idref="bib65" id="ref220">65</reflink>]).</p> <hd id="AN0183370751-23">Conclusion</hd> <p>Measurement validity refers to the degree to which theory and evidence support the interpretations and uses of surveys and other measurement tools ([<reflink idref="bib1" id="ref221">1</reflink>]). Evidence standards requiring behavioral and/or clinical outcome measures that are reliable and valid in determining the credibility of causal claims of effectiveness is standard practice in the evaluation field ([<reflink idref="bib42" id="ref222">42</reflink>]; [<reflink idref="bib79" id="ref223">79</reflink>]; [<reflink idref="bib87" id="ref224">87</reflink>]). As noted earlier, effectiveness often wanes as EBIs are widely disseminated. Just as is the case with outcome evaluations, reliable and valid FOI measures should be the norm because sound fidelity measurement can facilitate implementation and better linkage to treatment effects, thereby enhancing the successful translation of EBIs into practice. While consensus exists on the importance of FOI ([<reflink idref="bib30" id="ref225">30</reflink>]; [<reflink idref="bib68" id="ref226">68</reflink>]; [<reflink idref="bib72" id="ref227">72</reflink>]; [<reflink idref="bib78" id="ref228">78</reflink>]), lack of time and financial resources to develop and describe sound FOI measurement practices could be a driving force behind the dearth of studies building evidence of validity of FOI measures ([<reflink idref="bib15" id="ref229">15</reflink>]). Despite these challenges, we argue that attention to validation of FOI instruments allows for more complex ways of understanding how EBIs are implemented and may thus help us to learn more from the studies we are able to undertake. The current study adds to previous research by offering an example for how to build evidence of validity for FOI measures. By supporting and elevating issues related to validity of fidelity instruments, we aim to improve the field's understanding of how findings of EBIs can and should be translated into practice—thereby holding the field more accountable to higher standards.</p> <hd id="AN0183370751-24">Acknowledgments</hd> <p>The authors would like to thank Benjamin Shear for paper concepts and his comments and critical read of the manuscript. Process evaluation data utilized in this study were collected through the support of a prevention dissemination grant from the Altria Group; however, this funder had no influence on the hypotheses, analyses, or interpretation and reporting of these results.</p> <hd id="AN0183370751-25">ORCID iD</hd> <p>Pamela R. Buckley https://orcid.org/0000-0002-8268-3524</p> <hd id="AN0183370751-26">Data Availability Statement</hd> <p>Contact the author to inquire about the data availability.</p> <ref id="AN0183370751-27"> <title> References </title> <blist> <bibl id="bib1" idref="ref10" type="bt">1</bibl> <bibtext> American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, NCME). (2014). Standards for educational and psychological testing. AERA, APA, NCME.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref16" type="bt">2</bibl> <bibtext> Bell C. A., Gitomer D. H., McCaffrey D. F., Hamre B. K., Pianta R. C., Qi Y. (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87. https://doi.org/10.1080/10627197.2012.715014</bibtext> </blist> <blist> <bibl id="bib3" idref="ref18" type="bt">3</bibl> <bibtext> Bentler P. M., Liang J. (2003). Two-level mean and covariance structures: Maximum likelihood via EM algorithm. In Reise S. P., Duan N. (Eds.), Multilevel modeling: Methodological advances, issues, and applications (pp. 53–70). Lawrence Erlbaum Associates, Inc.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref54" type="bt">4</bibl> <bibtext> Berkel C., Mauricio A. M., Schoenfelder E., Sandler I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science: The Official Journal of the Society for Prevention Research, 12(1), 23–33. https://doi.org/10.1007/s11121-010-0186-1</bibtext> </blist> <blist> <bibl id="bib5" idref="ref55" type="bt">5</bibl> <bibtext> Blase K., Fixsen D. (2013). Core Intervention components: Identifying and operationalizing what makes programs work. ASPE Research Brief, Issue. Retrieved in January 2023 from: https://aspe.hhs.gov/reports/core-intervention-components-identifying-operationalizing-what-makes-programs-work-0</bibtext> </blist> <blist> <bibl id="bib6" idref="ref47" type="bt">6</bibl> <bibtext> Bond G. R., Evans L., Salyers M. P., Williams J., Kim H. W. (2000). Measurement of measurement of fidelity in psychiatric rehabilitationdelity in psychiatric rehabilitation. Mental Health Services Research, 2(2), 75–87. https://doi.org/10.1023/A:1010153020697</bibtext> </blist> <blist> <bibl id="bib7" idref="ref178" type="bt">7</bibl> <bibtext> Borduin C. M., Mann B. J., Cone L. T., Henggeler S. W., Fucci B. R., Blaske D. M., Williams R. A. (1995). Multisystemic treatment of serious juvenile offenders: Long-term prevention of criminality and violence. Journal of Consulting and Clinical Psychology, 63(4), 569–578. https://doi.org/10.1037//0022-006x.63.4.569</bibtext> </blist> <blist> <bibl id="bib8" idref="ref30" type="bt">8</bibl> <bibtext> Botvin G. J., Baker E., Dusenbury L., Botvin E. M., Diaz T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. JAMA, 273(14), 1106–1112. https://doi.org/10.1001/jama.1995.03520380042033</bibtext> </blist> <blist> <bibl id="bib9" idref="ref94" type="bt">9</bibl> <bibtext> Botvin G. J., Griffin K. W., Botvin C., Murphy M., Acevedo B. (2018). Increasing implementation fidelity for school-based drug abuse prevention: Effectiveness of enhanced training and technical assistance. Journal of the Society for Social Work and Research, 9(4), 599–613. https://doi.org/10.1086/700972</bibtext> </blist> <blist> <bibtext> Botvin G. J., Griffin K. W., Nichols T. D. (2006). Preventing youth violence and delinquency through a universal school-based prevention approach. Prevention Science: The Official Journal of the Society for Prevention Research, 7(4), 403–408. https://doi.org/10.1007/s11121-006-0057-y</bibtext> </blist> <blist> <bibtext> Botvin G. J., Kantor L. W. (2000). Preventing alcohol and tobacco use through life skills training - Theory, methods, and empirical findings. Alcohol Research & Health, 24(4), 250–257.</bibtext> </blist> <blist> <bibtext> Bradshaw C. P., Mitchell M. M., Leaf P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133–148. https://doi.org/10.1177/1098300709334798</bibtext> </blist> <blist> <bibtext> Bradshaw C. P., Waasdorp T. E., Leaf P. J. (2012). Effects of school-wide positive behavioral interventions and supports on child behavior problems. Pediatrics, 130(5), e1136–e1145. https://doi.org/10.1542/peds.2012-0243</bibtext> </blist> <blist> <bibtext> Buckley P. R., Murry V. M., Gust C. J., Ladika A., Pampel F. C. (2023). Racial and ethnic representation in preventive intervention research: A methodological study. Prevention Science: The Official Journal of the Society for Prevention Research, 24(7), 1261–1274. https://doi.org/10.1007/s11121-023-01564-8</bibtext> </blist> <blist> <bibtext> Buckley P. R., Moore B., Boardman A. G., Arya D. J., Maul A. (2017). Validating a fidelity scale to understand intervention effects in classroom-based studies. American Educational Research Journal, 54(6), 1378–1413. https://doi.org/10.3102/0002831217726522</bibtext> </blist> <blist> <bibtext> Burkhardt J. T., Schroter D. C., Magura S., Means S. N., Coryn C. L. (2015). An overview of evidence-based program registers (EBPRs) for behavioral health. Evaluation and Program Planning, 48, 92–99. https://doi.org/10.1016/j.evalprogplan.2014.09.006</bibtext> </blist> <blist> <bibtext> Carroll C., Patterson M., Wood S., Booth A., Rick J., Balain S. (2007). A conceptual framework for implementation fidelity. Implementation Science: IS, 2(1), 40. https://doi.org/10.1186/1748-5908-2-40</bibtext> </blist> <blist> <bibtext> Chambers D. A., Glasgow R. E., Stange K. C. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science: IS, 8(1), 117. https://doi.org/10.1186/1748-5908-8-117</bibtext> </blist> <blist> <bibtext> Chen F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834</bibtext> </blist> <blist> <bibtext> Chen H. (1990). Theory-driven evaluations. Sage.</bibtext> </blist> <blist> <bibtext> Cheung G. W., Rensvold R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/s15328007sem0902_5</bibtext> </blist> <blist> <bibtext> Cohen R., Kincaid D., Childs K. E. (2007). Measuring school-wide positive behavior support implementation: Development and validation of the benchmarks of quality. Journal of Positive Behavior Interventions, 9(4), 203–213. https://doi.org/10.1177/10983007070090040301</bibtext> </blist> <blist> <bibtext> Combs K. M., Buckley P. R., Lain M. A., Drewelow K. M., Urano G., Kerns S. (2022). Influence of classroom-level factors on implementation fidelity during scale-up of evidence-based interventions. Prevention Science: The Official Journal of the Society for Prevention Research, 23(6), 969–981. https://doi.org/10.1007/s11121-022-01375-3</bibtext> </blist> <blist> <bibtext> Combs K. M., Drewelow K. M., Håbesland M. S., Lain M. A., Buckley P. R. (2021). Does training modality predict fidelity of an evidence-based intervention delivered in schools? Prevention Science: The Official Journal of the Society for Prevention Research, 22(7), 928–938. https://doi.org/10.1007/s11121-021-01227-6</bibtext> </blist> <blist> <bibtext> Combs K. M., Drewelow K. M., Lain M. A., Håbesland M., Ippolito A., Finigan-Carr N. (2023). Sustainment of an evidence-based, behavioral health curriculum in schools. Prevention Science: The Official Journal of the Society for Prevention Research, 24(3), 541–551. https://doi.org/10.1007/s11121-022-01454-5</bibtext> </blist> <blist> <bibtext> Cross W. F., West J. C. (2011). Examining implementer fidelity: Conceptualizing and measuring adherence and competence. Journal of Children's Services, 6(1), 18–33. https://doi.org/10.5042/jcs.2011.0123</bibtext> </blist> <blist> <bibtext> Dane A. V., Schneider B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45. https://doi.org/10.1016/s0272-7358(97)00043-3</bibtext> </blist> <blist> <bibtext> Drake R. E., Goldman H. H., Leff H. S., Lehman A. F., Dixon L., Mueser K. T., Torrey W. C. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services, 52(2), 179–182. https://doi.org/10.1176/appi.ps.52.2.179</bibtext> </blist> <blist> <bibtext> Durlak J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science: The Official Journal of the Society for Prevention Research, 16(8), 1123–1127. https://doi.org/10.1007/s11121-015-0606-3</bibtext> </blist> <blist> <bibtext> Durlak J. A., DuPre E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. https://doi.org/10.1007/s10464-008-9165-0</bibtext> </blist> <blist> <bibtext> Dusenbury L., Brannigan R., Falco M., Hansen W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237–256. https://doi.org/10.1093/her/18.2.237</bibtext> </blist> <blist> <bibtext> Dusenbury L., Brannigan R., Hansen W. B., Walsh J., Falco M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20(3), 308–313. https://doi.org/10.1093/her/cyg134</bibtext> </blist> <blist> <bibtext> Elliott D. S. (2022). Reflections on six decades of research. Annual Review of Criminology, 5(1), 1–19. https://doi.org/10.1146/annurev-criminol-030920-122540</bibtext> </blist> <blist> <bibtext> Elliott D. S., Buckley P. R., Gottfredson D. C., Hawkins J. D., Tolan P. H. (2020). Evidence‐based juvenile justice programs and practices: A critical review. Criminology & Public Policy, 19(4), 1305–1328. https://doi.org/10.1111/1745-9133.12520</bibtext> </blist> <blist> <bibtext> Elliott D. S., Mihalic S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science: The Official Journal of the Society for Prevention Research, 5(1), 47–53. https://doi.org/10.1023/b:prev.0000013981.28071.52</bibtext> </blist> <blist> <bibtext> Embry D. D. (2004). Community‐based prevention using simple, low‐cost, evidence‐based kernels and behavior vaccines. Journal of Community Psychology, 32(5), 575–591. https://doi.org/10.1002/jcop.20020</bibtext> </blist> <blist> <bibtext> Fagan A. A., Bumbarger B. K., Barth R. P., Bradshaw C. P., Cooper B. R., Supplee L. H., Walker D. K. (2019). Scaling up evidence-based interventions in US public systems to prevent behavioral health problems: Challenges and opportunities. Prevention Science: The Official Journal of the Society for Prevention Research, 20(8), 1147–1168. https://doi.org/10.1007/s11121-019-01048-8</bibtext> </blist> <blist> <bibtext> Freeman J., Simonsen B., McCoach D. B., Sugai G., Lombardi A., Horner R. (2016). Relationship between school-wide positive behavior interventions and supports and academic, attendance, and behavior outcomes in high schools. Journal of Positive Behavior Interventions, 18(1), 41–51. https://doi.org/10.1177/1098300715580992</bibtext> </blist> <blist> <bibtext> French B. F., Finch W. H. (2008). Multigroup confirmatory factor analysis: Locating the invariant referent sets. Structural Equation Modeling: A Multidisciplinary Journal, 15(1), 96–113. https://doi.org/10.1080/10705510701758349</bibtext> </blist> <blist> <bibtext> Gearing R. E., El-Bassel N., Ghesquiere A., Baldwin S., Gillies J., Ngeow E. (2011). Major ingredients of fidelity: A review and scientific guide to improving quality of intervention research implementation. Clinical Psychology Review, 31(1), 79–88. https://doi.org/10.1016/j.cpr.2010.09.007</bibtext> </blist> <blist> <bibtext> Goodwin K. (2020). The ABCs of evidence-informed policymaking. Retrieved in November 2022 from: https://<ulink href="http://www.ncsl.org/research/labor-and-employment/the-abcs-of-evidence-informed-policymaking.aspx">www.ncsl.org/research/labor-and-employment/the-abcs-of-evidence-informed-policymaking.aspx</ulink></bibtext> </blist> <blist> <bibtext> Gottfredson D. C., Cook T. D., Gardner F. E., Gorman-Smith D., Howe G. W., Sandler I. N., Zafft K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science: The Official Journal of the Society for Prevention Research, 16(7), 893–926. https://doi.org/10.1007/s11121-015-0555-x</bibtext> </blist> <blist> <bibtext> Gottfredson D. C., Gottfredson G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39(1), 3–35. https://doi.org/10.1177/002242780203900101</bibtext> </blist> <blist> <bibtext> Hahn R. (2019). Building upon foundations for evidence-based policy. Science, 364(6440), 534–535. https://doi.org/10.1126/science.aaw9446</bibtext> </blist> <blist> <bibtext> Hawkins J. D., Jenson J. M., Catalano R., Fraser M. W., Botvin G. J., Shapiro V., Brown C. H., Beardslee W., Brent D., Leslie L. K. (2016). Unleashing the power of prevention. American Journal of Medical Research, 3(1), 39.</bibtext> </blist> <blist> <bibtext> Heck S., Stiegelbauer S. M., Hall G. E., Loucks S. F. (1981). Measuring innovation configurations: Procedures and applications. Report No. 3108, 147.</bibtext> </blist> <blist> <bibtext> Henggeler S. W., Schoenwald S. K., Liao J. G., Letourneau E. J., Edwards D. L. (2002). Transporting efficacious treatments to field settings: The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 31(2), 155–167. https://doi.org/10.1207/S15374424JCCP3102_02</bibtext> </blist> <blist> <bibtext> Hill H. C., Charalambous C. Y., Blazar D., McGinn D., Kraft M. A., Beisiegel M., Humez A., Litke E., Lynch K. (2012). Validating arguments for observational instruments: Attending to multiple sources of variation. Educational Assessment, 17(2–3), 88–106. https://doi.org/10.1080/10627197.2012.715019</bibtext> </blist> <blist> <bibtext> Hohmann A. A., Shear M. K. (2002). Community-based intervention research: Coping with the "noise" of real life in study design. American Journal of Psychiatry, 159(2), 201–207. https://doi.org/10.1176/appi.ajp.159.2.201</bibtext> </blist> <blist> <bibtext> Horner R. H., Todd A. W., Lewis-Palmer T., Irvin L. K., Sugai G., Boland J. B. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1), 3–12. https://doi.org/10.1177/10983007040060010201</bibtext> </blist> <blist> <bibtext> Hu L. T., Bentler P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118</bibtext> </blist> <blist> <bibtext> Huang C. C., Chen Y. F., Greene L., Cheung S., Wei Y. H. (2019). Resilience and emotional and behavioral problems of adolescents in China: Effects of a short-term and intensive mindfulness and life skills training. Children and Youth Services Review, 100, 291–297. https://doi.org/10.1016/j.childyouth.2019.03.015</bibtext> </blist> <blist> <bibtext> Jamali S., Sabokdast S., Sharif Nia H., Goudarzian A. H., Beik S., Allen K. A. (2016). The effect of life skills training on mental health of Iranian middle school students: A preliminary study. Iranian Journal of Psychiatry, 11(4), 269–272. https://<ulink href="http://www.ncbi.nlm.nih.gov/pubmed/28050189">www.ncbi.nlm.nih.gov/pubmed/28050189</ulink></bibtext> </blist> <blist> <bibtext> Kane M. T. (2006). Validation (4th ed.). Greenwood Publishing Group.</bibtext> </blist> <blist> <bibtext> Kerns S. E. U., Perrine C. M., Sedlar G., Peterson R., Monroe-DeVita M. (2021). Keeping the faith while keeping it real: Practical, empirical approaches to evaluating treatment fidelity. Global Implementation Research and Applications, 1(2), 90–101. https://doi.org/10.1007/s43477-021-00012-5</bibtext> </blist> <blist> <bibtext> Kline R. B. (2015). Principles and practice of structural equation modeling. Guilford publications.</bibtext> </blist> <blist> <bibtext> Lee M. J., Maranda M. J., Magura S., Greenman G. (2022). References to Evidence-based Program Registry (EBPR) websites for behavioral health in U.S. state government statutes and regulations. Journal of Applied Social Science, 16(2), 442–458. https://doi.org/10.1177/19367244221078278</bibtext> </blist> <blist> <bibtext> Lee M. J., Wu W. C., Chang H. C., Chen H. J., Lin W. S., Feng J. Y., Lee T. S. H. (2020). Effectiveness of a school-based life skills program on emotional regulation and depression among elementary school students: A randomized study. Children and Youth Services Review, 118, 105464. https://doi.org/10.1016/j.childyouth.2020.105464</bibtext> </blist> <blist> <bibtext> Li C. H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7</bibtext> </blist> <blist> <bibtext> Liang J., Bentler P. M. (2004). An EM algorithm for fitting two-level structural equation models. Psychometrika, 69(1), 101–122. https://doi.org/10.1007/BF02295842</bibtext> </blist> <blist> <bibtext> Loper A., Woo B., Metz A. (2021). Equity is fundamental to implementation science. Stanford Social Innovation Review, 19(3), 3–5.</bibtext> </blist> <blist> <bibtext> Massar M. M., McIntosh K., Mercer S. H. (2019). Factor validation of a fidelity of implementation measure for social behavior systems. Remedial and Special Education, 40(1), 16–24. https://doi.org/10.1177/0741932517736515</bibtext> </blist> <blist> <bibtext> McCaffrey D. F., Yuan K., Savitsky T. D., Lockwood J. R., Edelen M. O. (2015). Uncovering multivariate structure in classroom observations in the presence of rater errors. Educational Measurement: Issues and Practice, 34(2), 34–46. https://doi.org/10.1111/emip.12061</bibtext> </blist> <blist> <bibtext> McHugh M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://hrcak.srce.hr/89395</bibtext> </blist> <blist> <bibtext> Metz A., Woo B., Loper A. (2021). Equitable implementation at work. Stanford Social Innovation Review, 19(3), 29–31.</bibtext> </blist> <blist> <bibtext> Mihalic S. F., Elliott D. S. (2015). Evidence-based programs registry: Blueprints for healthy youth development. Evaluation and Program Planning, 48, 124–131. https://doi.org/10.1016/j.evalprogplan.2014.08.004</bibtext> </blist> <blist> <bibtext> Mihalic S. F., Fagan A. A., Argamaso S. (2008). Implementing the LifeSkills training drug prevention program: Factors related to implementation fidelity. Implementation Science: IS, 3(5), 5. https://doi.org/10.1186/1748-5908-3-5</bibtext> </blist> <blist> <bibtext> Mowbray C. T., Holter M. C., Teague G. B., Bybee D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 315–340. https://doi.org/10.1177/109821400302400303</bibtext> </blist> <blist> <bibtext> Muthén B. O. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54(4), 557–585. https://doi.org/10.1007/bf02296397</bibtext> </blist> <blist> <bibtext> Muthén L. K., Muthén B. O. (2017). Mplus: Statistical analysis with latent variables: User's guide (version 8). Authors. For 8.1, see: https://<ulink href="http://www.statmodel.com/verhistory.shtml">www.statmodel.com/verhistory.shtml</ulink></bibtext> </blist> <blist> <bibtext> National Health Promotion Associates, NHPA. (2017). LST planning workbook. NHPA. Retrieved in January 2023 from: https://<ulink href="http://www.lifeskillstraining.com/planning-workbook/">www.lifeskillstraining.com/planning-workbook/</ulink></bibtext> </blist> <blist> <bibtext> O'Donnell C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78(1), 33–84. https://doi.org/10.3102/0034654307313793</bibtext> </blist> <blist> <bibtext> Proctor E., Silmere H., Raghavan R., Hovmand P., Aarons G., Bunger A., Griffey R., Hensley M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7</bibtext> </blist> <blist> <bibtext> Putnick D. L., Bornstein M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review: DR, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004</bibtext> </blist> <blist> <bibtext> Ringwalt C., Hanley S., Vincus A. A., Ennett S. T., Rohrbach L. A., Bowling J. M. (2008). The prevalence of effective substance use prevention curricula in the nation's high schools. The Journal of Primary Prevention, 29(6), 479–488. https://doi.org/10.1007/s10935-008-0158-4</bibtext> </blist> <blist> <bibtext> Ryu E. (2014). Model fit evaluation in multilevel structural equation models. Frontiers in Psychology, 5(81), 1–9. https://doi.org/10.3389/fpsyg.2014.00081</bibtext> </blist> <blist> <bibtext> Sawada D., Piburn M. D., Judson E., Turley J., Falconer K., Benford R., Bloom I. (2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics, 102(6), 245–253. https://doi.org/10.1111/j.1949-8594.2002.tb17883.x</bibtext> </blist> <blist> <bibtext> Schoenwald S. K., Garland A. F., Chapman J. E., Frazier S. L., Sheidow A. J., Southam-Gerow M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0</bibtext> </blist> <blist> <bibtext> Steeger C. M., Buckley P. R., Pampel F. C., Gust C. J., Hill K. G. (2021). Common methodological problems in randomized controlled trials of preventive interventions. Prevention Science: The Official Journal of the Society for Prevention Research, 22(8), 1159–1172. https://doi.org/10.1007/s11121-021-01263-2</bibtext> </blist> <blist> <bibtext> Supplee L. H., Meyer A. L. (2015). The intersection between prevention science and evidence-based policy: How the SPR evidence standards support human services prevention programs. Prevention Science: The Official Journal of the Society for Prevention Research, 16(7), 938–942. https://doi.org/10.1007/s11121-015-0590-7</bibtext> </blist> <blist> <bibtext> Teague G. B., Drake R. E., Ackerson T. H. (1995). Evaluating use of continuous treatment teams for persons with mental-illness and substance-abuse. Psychiatric Services, 46(7), 689–695. https://doi.org/10.1176/ps.46.7.689</bibtext> </blist> <blist> <bibtext> Tobler N. S., Stratton H. H. (1997). Effectiveness of school-based drug prevention programs: A meta-analysis of the research. The Journal of Primary Prevention, 18(1), 71–128. https://doi.org/10.1023/A:1024630205999</bibtext> </blist> <blist> <bibtext> Vaughn S., Klingner J. K., Swanson E. A., Boardman A. G., Roberts G., Mohammed S. S., Stillman-Spisak S. J. (2011). Efficacy of collaborative strategic reading with middle school students. American Educational Research Journal, 48(4), 938–964. https://doi.org/10.3102/0002831211410305</bibtext> </blist> <blist> <bibtext> Velasco V., Griffin K. W., Botvin G. J., Corrado C., Gruppo L. S. T. L. (2017). Preventing adolescent substance use through an evidence-based program: Effects of the Italian adaptation of Life Skills Training. Prevention Science: The Official Journal of the Society for Prevention Research, 18(4), 394–405. https://doi.org/10.1007/s11121-017-0776-2</bibtext> </blist> <blist> <bibtext> Vroom E. B., Massey O. T., Yampolskaya S., Levin B. L. (2020). The impact of implementation fidelity on student outcomes in the Life Skills Training program. School Mental Health, 12(1), 113–123. https://doi.org/10.1007/s12310-019-09333-1</bibtext> </blist> <blist> <bibtext> Washington Institute of Public Policy, WSIPP. (2019). Benefit-cost techincal documentation. Retrieved in January 2023 from: https://<ulink href="http://www.wsipp.wa.gov/BenefitCost">www.wsipp.wa.gov/BenefitCost</ulink></bibtext> </blist> <blist> <bibtext> What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0. This report is available on the What Works Clearinghouse website at: https://ies.ed.gov/ncee/wwc/Handbooks</bibtext> </blist> <blist> <bibtext> Widaman K. F., Reise S. P. (1997). Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In Bryant K. J., Windle M., West S. G. (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research (pp. 281–324). American Psychological Association. https://doi.org/10.1037/10222-009</bibtext> </blist> <blist> <bibtext> Wilson M. (2004). Constructing measures: An item response modeling approach. Routledge.</bibtext> </blist> </ref> <ref id="AN0183370751-28"> <title> Footnotes </title> <blist> <bibtext> The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibtext> The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Altria Group.</bibtext> </blist> </ref> <aug> <p>By Pamela R. Buckley; Katie Massey Combs; Karen M. Drewelow; Brittany L. Hubler and Marion Amanda Lain</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib37" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib45" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib18" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib32" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib43" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib75" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib40" firstref="ref7"></nolink> <nolink nlid="nl8" bibid="bib68" firstref="ref8"></nolink> <nolink nlid="nl9" bibid="bib78" firstref="ref9"></nolink> <nolink nlid="nl10" bibid="bib15" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib20" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib49" firstref="ref13"></nolink> <nolink nlid="nl13" bibid="bib66" firstref="ref14"></nolink> <nolink nlid="nl14" bibid="bib81" firstref="ref17"></nolink> <nolink nlid="nl15" bibid="bib72" firstref="ref23"></nolink> <nolink nlid="nl16" bibid="bib47" firstref="ref26"></nolink> <nolink nlid="nl17" bibid="bib62" firstref="ref27"></nolink> <nolink nlid="nl18" bibid="bib10" firstref="ref31"></nolink> <nolink nlid="nl19" bibid="bib86" firstref="ref32"></nolink> <nolink nlid="nl20" bibid="bib84" firstref="ref33"></nolink> <nolink nlid="nl21" bibid="bib52" firstref="ref34"></nolink> <nolink nlid="nl22" bibid="bib58" firstref="ref35"></nolink> <nolink nlid="nl23" bibid="bib53" firstref="ref36"></nolink> <nolink nlid="nl24" bibid="bib48" firstref="ref38"></nolink> <nolink nlid="nl25" bibid="bib54" firstref="ref39"></nolink> <nolink nlid="nl26" bibid="bib27" firstref="ref44"></nolink> <nolink nlid="nl27" bibid="bib28" firstref="ref48"></nolink> <nolink nlid="nl28" bibid="bib26" firstref="ref49"></nolink> <nolink nlid="nl29" bibid="bib73" firstref="ref50"></nolink> <nolink nlid="nl30" bibid="bib17" firstref="ref56"></nolink> <nolink nlid="nl31" bibid="bib31" firstref="ref58"></nolink> <nolink nlid="nl32" bibid="bib23" firstref="ref64"></nolink> <nolink nlid="nl33" bibid="bib25" firstref="ref65"></nolink> <nolink nlid="nl34" bibid="bib55" firstref="ref66"></nolink> <nolink nlid="nl35" bibid="bib46" firstref="ref67"></nolink> <nolink nlid="nl36" bibid="bib24" firstref="ref71"></nolink> <nolink nlid="nl37" bibid="bib12" firstref="ref74"></nolink> <nolink nlid="nl38" bibid="bib38" firstref="ref75"></nolink> <nolink nlid="nl39" bibid="bib82" firstref="ref76"></nolink> <nolink nlid="nl40" bibid="bib11" firstref="ref78"></nolink> <nolink nlid="nl41" bibid="bib33" firstref="ref79"></nolink> <nolink nlid="nl42" bibid="bib35" firstref="ref80"></nolink> <nolink nlid="nl43" bibid="bib30" firstref="ref89"></nolink> <nolink nlid="nl44" bibid="bib67" firstref="ref95"></nolink> <nolink nlid="nl45" bibid="bib71" firstref="ref97"></nolink> <nolink nlid="nl46" bibid="bib89" firstref="ref102"></nolink> <nolink nlid="nl47" bibid="bib70" firstref="ref123"></nolink> <nolink nlid="nl48" bibid="bib59" firstref="ref124"></nolink> <nolink nlid="nl49" bibid="bib60" firstref="ref126"></nolink> <nolink nlid="nl50" bibid="bib76" firstref="ref127"></nolink> <nolink nlid="nl51" bibid="bib56" firstref="ref128"></nolink> <nolink nlid="nl52" bibid="bib19" firstref="ref129"></nolink> <nolink nlid="nl53" bibid="bib21" firstref="ref130"></nolink> <nolink nlid="nl54" bibid="bib51" firstref="ref131"></nolink> <nolink nlid="nl55" bibid="bib39" firstref="ref132"></nolink> <nolink nlid="nl56" bibid="bib69" firstref="ref133"></nolink> <nolink nlid="nl57" bibid="bib74" firstref="ref134"></nolink> <nolink nlid="nl58" bibid="bib88" firstref="ref135"></nolink> <nolink nlid="nl59" bibid="bib34" firstref="ref145"></nolink> <nolink nlid="nl60" bibid="bib85" firstref="ref156"></nolink> <nolink nlid="nl61" bibid="bib22" firstref="ref169"></nolink> <nolink nlid="nl62" bibid="bib50" firstref="ref170"></nolink> <nolink nlid="nl63" bibid="bib77" firstref="ref171"></nolink> <nolink nlid="nl64" bibid="bib13" firstref="ref176"></nolink> <nolink nlid="nl65" bibid="bib83" firstref="ref181"></nolink> <nolink nlid="nl66" bibid="bib63" firstref="ref188"></nolink> <nolink nlid="nl67" bibid="bib64" firstref="ref189"></nolink> <nolink nlid="nl68" bibid="bib29" firstref="ref190"></nolink> <nolink nlid="nl69" bibid="bib16" firstref="ref206"></nolink> <nolink nlid="nl70" bibid="bib57" firstref="ref208"></nolink> <nolink nlid="nl71" bibid="bib41" firstref="ref209"></nolink> <nolink nlid="nl72" bibid="bib44" firstref="ref210"></nolink> <nolink nlid="nl73" bibid="bib14" firstref="ref213"></nolink> <nolink nlid="nl74" bibid="bib61" firstref="ref214"></nolink> <nolink nlid="nl75" bibid="bib80" firstref="ref215"></nolink> <nolink nlid="nl76" bibid="bib36" firstref="ref216"></nolink> <nolink nlid="nl77" bibid="bib65" firstref="ref220"></nolink> <nolink nlid="nl78" bibid="bib42" firstref="ref222"></nolink> <nolink nlid="nl79" bibid="bib79" firstref="ref223"></nolink> <nolink nlid="nl80" bibid="bib87" firstref="ref224"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1466339
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Pamela+R%2E+Buckley%22">Pamela R. Buckley</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-8268-3524">0000-0002-8268-3524</externalLink>)<br /><searchLink fieldCode="AR" term="%22Katie+Massey+Combs%22">Katie Massey Combs</searchLink><br /><searchLink fieldCode="AR" term="%22Karen+M%2E+Drewelow%22">Karen M. Drewelow</searchLink><br /><searchLink fieldCode="AR" term="%22Brittany+L%2E+Hubler%22">Brittany L. Hubler</searchLink><br /><searchLink fieldCode="AR" term="%22Marion+Amanda+Lain%22">Marion Amanda Lain</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Evaluation+Review%22"><i>Evaluation Review</i></searchLink>. 2025 49(2):237-269.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 33
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Teachers%22">Middle School Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence+Based+Practice%22">Evidence Based Practice</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Development%22">Program Development</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Implementation%22">Program Implementation</searchLink><br /><searchLink fieldCode="DE" term="%22Prevention%22">Prevention</searchLink><br /><searchLink fieldCode="DE" term="%22Drug+Abuse%22">Drug Abuse</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Validation%22">Program Validation</searchLink><br /><searchLink fieldCode="DE" term="%22Generalization%22">Generalization</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Use%22">Test Use</searchLink><br /><searchLink fieldCode="DE" term="%22Predictive+Validity%22">Predictive Validity</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/0193841X241248864
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0193-841X<br />1552-3926
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers' ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane's (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1466339
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1466339
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/0193841X241248864
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 33
        StartPage: 237
    Subjects:
      – SubjectFull: Middle School Students
        Type: general
      – SubjectFull: Middle School Teachers
        Type: general
      – SubjectFull: Evidence Based Practice
        Type: general
      – SubjectFull: Program Development
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
      – SubjectFull: Program Implementation
        Type: general
      – SubjectFull: Prevention
        Type: general
      – SubjectFull: Drug Abuse
        Type: general
      – SubjectFull: Fidelity
        Type: general
      – SubjectFull: Program Validation
        Type: general
      – SubjectFull: Generalization
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Test Use
        Type: general
      – SubjectFull: Predictive Validity
        Type: general
    Titles:
      – TitleFull: Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Pamela R. Buckley
      – PersonEntity:
          Name:
            NameFull: Katie Massey Combs
      – PersonEntity:
          Name:
            NameFull: Karen M. Drewelow
      – PersonEntity:
          Name:
            NameFull: Brittany L. Hubler
      – PersonEntity:
          Name:
            NameFull: Marion Amanda Lain
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0193-841X
            – Type: issn-electronic
              Value: 1552-3926
          Numbering:
            – Type: volume
              Value: 49
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Evaluation Review
              Type: main
ResultId 1