View in EDS HTML Full Text PDF Full Text

Setting Standards for SDLMI Fidelity: Promoting Data-Driven Decision Making to Advance Self-Determination Instruction

Saved in:
Bibliographic Details
Title: Setting Standards for SDLMI Fidelity: Promoting Data-Driven Decision Making to Advance Self-Determination Instruction
Language: English
Authors: Karrie A. Shogren (ORCID 0000-0001-7925-1299), Jesse R. Pace (ORCID 0000-0003-3961-5718), Tyler A. Hicks, Sheida K. Raley (ORCID 0000-0001-8422-1916), Kathleen Lynne Lane (ORCID 0000-0001-6364-838X)
Source: Psychology in the Schools. 2024 61(2):532-552.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 21
Publication Date: 2024
Sponsoring Agency: Institute of Education Sciences (ED)
Document Type: Journal Articles
Reports - Research
Education Level: Secondary Education
Descriptors: Cutting Scores, Fidelity, Program Implementation, Evidence Based Practice, Intervention, Self Determination, Instruction, Goal Orientation, Secondary School Students, Students with Disabilities, Inclusion, Standard Setting, Models
DOI: 10.1002/pits.23068
ISSN: 0033-3085
1520-6807
Abstract: This study used the standard setting to establish cutscores for the fidelity of implementation of an evidence-based intervention, the Self-Determined Learning Model of Instruction (SDLMI) designed to enhance goal-directed actions in secondary students with and without disabilities. Cutscores were then applied to fidelity data from a large, randomized trial of the SDLMI with teacher implementers. Findings suggest teachers demonstrate a range of fidelity outcomes over time across three dimensions adherence, quality of delivery, and student responsiveness. Almost all teachers (93%) immediately meet cutscores for the adherence dimension after training, but smaller numbers meet cutscores for quality of delivery (64%) and student responsiveness (69%). However, the quality of delivery and student responsiveness showed growth over time with implementation experience and there was a small effect of intensifying implementation support.
Abstractor: As Provided
Notes: https://osf.io/v7azb/?view_only=038b12cff8ba4a4982b94212e559b2ab
IES Funded: Yes
Entry Date: 2024
Accession Number: EJ1406601
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwG7ve07d24Y1p0nbm6SUPseAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDFa98pdNlwbfz7DOzAIBEICBmpQLhLj5eWxtb0okR-RrdKSlbmkLmrWHSMNxTKUT9IqS89LObJ4sX3JT18BgRa29CRaJ9a5GKHUZ7fjFM8K3yx3ADtZw0i4SPBTDTYNMOVcildliKHTvwONOKQvJ0sDhHBkrw9iKnNqWyn7nn9EF3fen7oGIbyn2Bq3x-Cb1p0uV9pfJUd8JSy0I9tyBr3uf_2m2CefPh2ET_-o=
Text:
  Availability: 1
  Value: <anid>AN0174763706;pis01feb.24;2024Jan16.05:30;v2.2.500</anid> <title id="AN0174763706-1">Setting standards for SDLMI fidelity: Promoting Data‐driven decision making to advance Self‐determination instruction </title> <p>This study used the standard setting to establish cutscores for the fidelity of implementation of an evidence‐based intervention, the Self‐Determined Learning Model of Instruction (SDLMI) designed to enhance goal‐directed actions in secondary students with and without disabilities. Cutscores were then applied to fidelity data from a large, randomized trial of the SDLMI with teacher implementers. Findings suggest teachers demonstrate a range of fidelity outcomes over time across three dimensions adherence, quality of delivery, and student responsiveness. Almost all teachers (93%) immediately meet cutscores for the adherence dimension after training, but smaller numbers meet cutscores for quality of delivery (64%) and student responsiveness (69%). However, the quality of delivery and student responsiveness showed growth over time with implementation experience and there was a small effect of intensifying implementation support.</p> <p>Practitioner points: The Self‐Determined Learning Model of Instruction (SDLMI) is implemented by teachers to support students in setting goals and implementing action plans, problem‐solving, and self‐regulating their learning as they revise their goals and action plans.It is important to understand how teachers implement the SDLMI in inclusive classrooms.Teachers can implement the SDLMI with their students, and their quality of delivery and student responsiveness to the SDLMI grows over time as teachers get coaching and implementation support.</p> <p>Keywords: fidelity of implementation; inclusive education; self‐determination; standards setting</p> <hd id="AN0174763706-2">INTRODUCTION</hd> <p>The Self‐Determined Learning Model of Instruction (SDLMI; Shogren et al., [<reflink idref="bib33" id="ref1">33</reflink>]; Wehmeyer et al., [<reflink idref="bib37" id="ref2">37</reflink>]) is an evidence‐based practice, designed to be used flexibly by trained educators across curricular and classroom contexts to promote student self‐determination, goal attainment, and other valued educational outcomes (Hagiwara et al., [<reflink idref="bib15" id="ref3">15</reflink>]; Rowe et al., [<reflink idref="bib27" id="ref4">27</reflink>]). The SDLMI can be flexibly implemented, but it has three core components (Student Questions, Teacher Objectives, and Educational Supports) that must be delivered during instruction for it to be implemented as intended (Shogren et al., [<reflink idref="bib33" id="ref5">33</reflink>]) or with implementation fidelity (Carroll et al., [<reflink idref="bib6" id="ref6">6</reflink>]). Implementation fidelity is recognized as essential to drawing inferences regarding intervention outcomes and is a core feature of all treatment‐outcome studies. Examining implementation fidelity is necessary to determine if the outcomes of research‐based educational interventions are due to the intervention itself or other variables in in the natural contexts where interventions take place (Fixsen et al., [<reflink idref="bib9" id="ref7">9</reflink>]). There has been a growing focus on better‐documenting implementation fidelity in intervention research to ensure outcomes can be attributed to interventions being implemented as intended (Bellg et al., [<reflink idref="bib2" id="ref8">2</reflink>]; Gersten et al., [<reflink idref="bib12" id="ref9">12</reflink>]; Toste et al., [<reflink idref="bib38" id="ref10">38</reflink>]; What Works Clearinghouse [WWC], [<reflink idref="bib39" id="ref11">39</reflink>]). There has also been an increased focus on how, in practice, to effectively support implementation fidelity, aligned with the tenets of implementation science (Fixsen et al., [<reflink idref="bib10" id="ref12">10</reflink>], [<reflink idref="bib9" id="ref13">9</reflink>]).</p> <p>Whereas, on the surface, documenting an intervention implemented as intended can seem to be a simple and straightforward concept, there has been little agreement on how to define implementation fidelity and the factors that influence it (Buckman et al., [<reflink idref="bib4" id="ref14">4</reflink>]). Multiple frameworks for documenting implementation fidelity exist in education and related fields (Harn et al., [<reflink idref="bib16" id="ref15">16</reflink>]). Complex interventions, like the SDLMI, introduce additional challenges to implementation fidelity. Complex interventions in education are those that require problem‐solving and flexibility on the part of the implementers to customize the implementation of the core components to local contexts and curricular areas, as well as to individualize supports based on student needs and cultural identities (Graham et al., [<reflink idref="bib13" id="ref16">13</reflink>]; Harn et al., [<reflink idref="bib16" id="ref17">16</reflink>]). Understanding implementation fidelity related to the delivery of core components of complex interventions is also critical for cost and cost‐effectiveness research to inform implementation supports, such as educator coaching.</p> <p>The lack of consensus on how to define, measure, and report fidelity of implementation of complex interventions has led to inconsistent measurement and reporting on fidelity in intervention research, including in SDLMI intervention research (Kiblen, Shogren, Zimmerman, et al., [<reflink idref="bib20" id="ref18">20</reflink>]). This has limited analyses of the relationship between fidelity and student outcomes and tailoring of supports for implementers based on fidelity data. For these reasons, Shogren et al. ([<reflink idref="bib35" id="ref19">35</reflink>]) synthesized the literature on the fidelity of complex interventions to create the SDLMI fidelity framework. The SDLMI fidelity framework includes six dimensions, including three dimensions that can be systematically measured through direct fidelity observations (i.e., adherence, quality of delivery, student responsiveness) and three assumed to influence fidelity (i.e., study design, training providers, receipt of treatment, program differentiation). Given the centrality of directly observing implementation to document fidelity to the SDLMI core components and quantifying that delivery occurred as intended, Shogren et al. ([<reflink idref="bib35" id="ref20">35</reflink>]) also provided information about the development of the <emph>SDLMI Fidelity Measure</emph> (Shogren & Raley, [<reflink idref="bib31" id="ref21">31</reflink>]) designed to assess adherence, quality of delivery, and student responsiveness to the core components of the SDLMI.</p> <p>The <emph>SDLMI Fidelity Measure</emph> was designed to have different versions and uses, targeting different contexts for implementation (e.g., inclusive general education classrooms, transition planning for students with disabilities) and different purposes (e.g., direct observations for research purposes, direct observation for coaching purposes, self‐report by implementers). Across versions, the measure includes three sections. Section A collects background information about the SDLMI lesson that will be observed to explicate the Targeted Student Question(s) and Teacher Objective(s). Section B—SDLMI lesson observation is a direct observation of an SDLMI lesson delivered by a trained implementer; this reflects the primary documentation of fidelity to the SDLMI core components that can be reported on for research purposes. Section C—content instruction observation is an observation of other content instruction (e.g., general education curriculum content instruction, transition instruction) by the implementer, to examine if SDLMI concepts are infused into instruction outside of SDLMI lessons (consistent with training and implementation protocols), providing supplemental information on the quality of program delivery and student responsiveness. In section B, adherence is measured with four items that assess the degree to which the teacher implementer demonstrates core components of the SDLMI during the lesson, including posing targeted Student Question(s), meeting Teacher Objective(s), and using Educational Support(s). Quality of program delivery includes five items that provide information on the teacher's facilitation of student‐directed learning during the lesson. The student responsiveness dimension is comprised of three items that provide data on the extent to which students are engaged and involved during the SDLMI lesson. Different rating scales are used across items. Items related to adherence are rated on a 3‐point, Likert‐type scale (<emph>No, Partially, Yes</emph>), and items in the quality of program delivery and student responsiveness dimensions across both sections are rated on a 5‐point, Likert‐type scale (in which 1 = <emph>Not at all</emph>, 2 = <emph>A little</emph>, 3 = <emph>Somewhat</emph>, 4 = <emph>Mostly</emph>, and 5 = <emph>Definitely</emph>).</p> <p>Shogren et al. ([<reflink idref="bib35" id="ref22">35</reflink>]) provided details on item refinement and reliability of the <emph>SDLMI Fidelity Measure</emph> and the range of scores observed across the three dimensions in sections B and C in the context of a longitudinal trial of the efficacy of the SDLMI with varying intensities of implementation supports for teachers in inclusive, general education settings. Variability in fidelity "scores" across dimensions during a semester of implementation with three direct observations were reported, suggesting variability in fidelity over time. However, Shogren et al. ([<reflink idref="bib35" id="ref23">35</reflink>]) did not identify how scores on the measure could be interpreted to reflect "sufficient" levels of fidelity. This was identified as a future research direction. While a criterion of 80% has been frequently used in the literature, often referring to the adherence dimension (e.g., Borrelli et al., [<reflink idref="bib3" id="ref24">3</reflink>]), there has been limited explication of why 80%, nor recognition that, for complex interventions, more dimensional and nuanced approaches to "scoring" fidelity might be warranted (Kiblen, Shogren, Zimmerman, et al., [<reflink idref="bib20" id="ref25">20</reflink>]). For example, items on the <emph>SDLMI Fidelity Measure</emph>, are not simple yes/no or delivered/not delivered items, but instead reflect judgments of adherence, quality of delivery, and student responsiveness. As such, scoring of a measure of fidelity of a complex intervention reflects more nuanced ratings, training, scoring, and interpretation of scores.</p> <p>The purpose of this paper is to report on an innovative use of standards setting applied to SDLMI fidelity assessment to provide preliminary guidance on the parameters of "sufficient" fidelity across three dimensions: adherence, quality of delivery, and student responsiveness. Such work has the potential to inform not only the measurement of SDLMI implementation fidelity but also other complex interventions such as integrated tiered systems installed in preK‐12 across the United States and beyond (e.g., Ci3T; Buckman et al., [<reflink idref="bib5" id="ref26">5</reflink>]; Gandhi et al., [<reflink idref="bib11" id="ref27">11</reflink>]). It also can be used to guide ongoing research on the relationship between fidelity and student outcomes and effective support to improve and sustain fidelity. In the first section of this paper, we describe standards setting and how this approach was applied to SDLMI fidelity assessment. Next, we report on the implications of cutscores developed through the standards‐setting process using real data collected using the <emph>SDLMI Fidelity Measure</emph> within a large‐scale, longitudinal efficacy trial of different implementation supports for the SDLMI, the same trial used to inform measure development in Shogren et al. ([<reflink idref="bib35" id="ref28">35</reflink>]). Finally, we use these findings to describe implications for ongoing research on the relationship between fidelity and student outcomes and the development of approaches to enable data‐based decision making to inform coaching and implementation supports to ensure sufficient fidelity levels in research and practice.</p> <hd id="AN0174763706-3">STANDARDS SETTING</hd> <p>There are systematic approaches to establishing standards for the demonstration of knowledge or skills that have been applied to educational and occupational tests (Zieky et al., [<reflink idref="bib40" id="ref29">40</reflink>]). This approach, called standard setting, follows a set of procedures to systematically define expected levels of performance, develop indicators for this performance, and then determine a <emph>cutscore</emph> for performance measuring using standardized assessments. Although not as frequently applied to fidelity for educational interventions, this approach has been suggested as a feasible way to define levels of fidelity of educational interventions (Center on IDEA Early Childhood Data Systems and Early Childhood TA Center, [<reflink idref="bib7" id="ref30">7</reflink>]). We chose to apply this method to establish cutscores for SDLMI implementation fidelity based on the assumption that enacting skills learned through training and coaching on the SDLMI, as measured through observations of SDLMI implementation, provides data that could be evaluated to determine if sufficient implementation occurred.</p> <p>We followed the standard‐setting process outlined by Zieky et al. ([<reflink idref="bib40" id="ref31">40</reflink>]). The standard setting process involves three steps and the application of those steps to the SDLMI are described below. The first step is a discussion with <emph>policy makers</emph> to determine the number of performance levels. The next step is developing <emph>policy definitions</emph> of sufficient implementation and <emph>performance level descriptors</emph> (PLDs) for each of the fidelity dimensions (adherence, quality, and responsiveness). The third step is to implement a systematic method to develop cutscores and the fourth step is to finalize the scores and apply them to real data.</p> <hd id="AN0174763706-4">Applying standard setting to the SDLMI</hd> <p>As described in Section 1, significant work has focused on developing a fidelity framework for the SDLMI, rooted in the literature on implementation science and fidelity of complex interventions (Shogren et al., [<reflink idref="bib35" id="ref32">35</reflink>]). Informed by this work, the <emph>SDLMI Fidelity Measure</emph> was developed to standardize the assessment of three domains of the fidelity framework (adherence, quality of delivery, and student responsiveness), and versions of this tool have been used in multiple SDLMI intervention studies (Kiblen, Shogren, Zimmerman, et al., [<reflink idref="bib20" id="ref33">20</reflink>]). While preliminary psychometrics of the <emph>SDLMI Fidelity Measure</emph>, including variability in scores during implementation and associations with student outcomes, have been explored (Shogren, Burke, et al., [<reflink idref="bib29" id="ref34">29</reflink>]; Shogren et al., [<reflink idref="bib35" id="ref35">35</reflink>]), there has not yet been a clear criterion for sufficient fidelity.</p> <hd id="AN0174763706-5">Step 1: Determining performance levels</hd> <p>To implement the first step of the standards‐setting process and establish a framework for performance levels on the <emph>SDLMI Fidelity Measure</emph>, a meeting was held with <emph>policy makers</emph> to determine performance levels. Consistent with standards‐setting procedures, policy makers were defined as those who have the authority to set the cutscores for the SDLMI assessment (Zieky et al., [<reflink idref="bib40" id="ref36">40</reflink>]). A core group of three SDLMI <emph>policy makers</emph>, who were also current leaders in SDLMI development and self‐determination research, came together with an expert in standard setting, who guided the overall process. This core group reviewed the literature on SDLMI fidelity with a particular focus on the conceptual framework developed by Shogren et al. ([<reflink idref="bib35" id="ref37">35</reflink>]) as well as applications of fidelity assessment procedures in SDLMI intervention research (see Kiblen, Shogren, Zimmerman, et al., [<reflink idref="bib20" id="ref38">20</reflink>]). As measuring fidelity and factors that influence it is multifaceted and can inform multiple outcomes (e.g., informing research implementation, coaching, and intervention enhancements), the policy‐making group narrowed the focus of standards setting focus to the three domains of fidelity directly assessed by section B of the <emph>SDLMI Fidelity Measure</emph> as these are the three domains viewed as essential to documenting fidelity. The group also determined that, per the established SDLMI fidelity framework (Shogren et al., [<reflink idref="bib35" id="ref39">35</reflink>]), the three domains are each considered necessary, at sufficient levels, for the SDLMI to be implemented with fidelity. Further, it was also determined that each of the three dimensions is a binary category: a given domain can independently be said to have been implemented with sufficient fidelity or not have been. Thus, the overall purpose of the fidelity tool is to delineate at what point an implementer's implementation crosses the line into sufficient fidelity, defined as <emph>just good enough</emph> implementation. While the specific pattern of findings within and across domains can be informative in practice, establishing this <emph>sufficient</emph> fidelity is important to determine when additional support might be needed.</p> <hd id="AN0174763706-6">Step 2: Developing policy definitions and PLDs</hd> <p>To begin to determine what defined this sufficient level of implementation, a broader group of experts in self‐determination research, the SDLMI, and its implementation were identified and brought together in virtual meetings to (a) review the work of the policy‐making group summarizing the existing literature on SDLMI fidelity, including the current <emph>SDLMI Fidelity Measure</emph> and its use and (b) review established definitions of the three fidelity domains and refine PLDs of each of the three dimensions. This process followed the recommendations of Zieky et al. ([<reflink idref="bib40" id="ref40">40</reflink>]). The policy‐making group and Shogren et al. ([<reflink idref="bib35" id="ref41">35</reflink>]) provided preliminary PLDs, without calling them such, and this served as the starting point.</p> <p>The goal of PLDs is to express what knowledge and skills are required in each of the three fidelity dimensions to ensure that implementation is <emph>just good enough</emph> or to define the minimally sufficient level of implementation assumed to impact student outcomes. Higher levels of fidelity can occur, and may further enhance outcomes, but <emph>perfect</emph> performance is not expected or warranted given the complexities of SDLMI implementation in high school settings, as has been noted by SDLMI developers in describing the complex nature of SDLMI implementation (Raley et al., [<reflink idref="bib24" id="ref42">24</reflink>]; Shogren, Hicks, et al., [<reflink idref="bib30" id="ref43">30</reflink>]). Two, 1‐h meetings were held with SDLMI experts to discuss, revise, and enhance the PLDs. SDLMI experts included 11 stakeholders who had published on the SDLMI and/or implemented it in practice. Before the first meeting, introductory materials to the standard‐setting process were shared, as were drafts of the PLDs developed by Shogren et al. ([<reflink idref="bib35" id="ref44">35</reflink>]). All 11 members attended the first meeting, 9 attended the second due to scheduling conflicts. In the first meeting, the larger group was broken into smaller groups, then returned to the larger group, to facilitate discussion. In the second meeting, due to the reduced participant numbers, all discussions were conducted with the full group. During these meetings, the PLD panel discussed the draft PLDs and potential edits and additions. In the end, the draft PLD for the adherence domain was accepted without revision. The quality of delivery and student responsiveness PLDs were accepted with minor revisions. Revisions mainly focused on clarifying terms, like adding "multiple means of response" to reflect how delivery must be individualized to student and class needs (see Table 1 for final PLDs).</p> <p>1 Table SDLMI Fidelity Dimensions PLDs and Recommended Cutscores.</p> <p> <ephtml> <table><thead valign="bottom"><tr valign="bottom"><th>Fidelity dimension</th><th>PLD</th><th>Cutscore</th></tr></thead><tbody valign="top"><tr><td>Adherence</td><td>Defined by (a) teachers delivering SDLMI instruction consistent with their individualized SDLMI Implementation schedule and (b) teachers, during instructional sessions, addressing the three SDLMI core components, namely presenting the SDLMI Student Question that is the focus of the lesson, meeting a majority of the aligned Teacher Objectives, and using aligned Educational Supports during instruction.</td><td>7 (out of 12 points)</td></tr><tr><td>Quality of delivery</td><td>Defined by the teacher individualizing SDLMI instruction to the needs and learning goals of the class, reflected in the teacher breaking down instruction into manageable units, linking instruction to the content goals of the class, providing support for students based on identified needs, and providing multiple means and opportunities for response</td><td>14 (out of 25 points)</td></tr><tr><td>Participant responsiveness</td><td>Defined by students engaging in SDLMI instruction, providing responses to Student Questions when presented by the teacher, completing instructional materials and activities, showing self‐direction and engagement in the learning process, and taking advantage of opportunities to respond during instruction.</td><td>9 (out of 15 points)</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph> : Reprinted with permission from the Kansas University Center on Developmental Disabilities.</p> <p>2 Abbreviations: PLD, performance level descriptors; SDLMI, Self‐Determined Learning Model of Instruction.</p> <p>The PLD panel also agreed on the parameters of SDLMI fidelity (e.g., it was multidimensional, it could be represented by a cutscore) as well as discussed terminology. For example, there was a discussion on terminology that would be used to describe the cutscore. The PLD panel recommended <emph>sufficient</emph>, although there was discussion about other terms including <emph>acceptable</emph> or <emph>high</emph>. Research and policy implications of various terms were discussed, and <emph>sufficient</emph> was determined to be the least value‐laden and most descriptive.</p> <hd id="AN0174763706-7">Step 3: Establishing SDLMI cutscores</hd> <p>Following PLD finalization, the next step was to implement a process for establishing cutscores. Using processes described by Zieky et al. ([<reflink idref="bib40" id="ref45">40</reflink>]), we convened a group that included some of the members of the PLD committee, representing researchers and practitioners but was also expanded to include additional stakeholders. The goal was to have two representatives of the following groups: self‐determination researchers, implementers of the SDLMI, SDLMI coaches, trained SDLMI fidelity observers, and SDLMI trainers. In accordance with recommendations from standards‐setting experts, advanced materials were sent approximately 2 weeks in advance to standard‐setting panelists to orient them to the process, review the PLDs, provide the <emph>SDLMI Fidelity Measure</emph>, and provide premeeting videos highlighting the key elements of the SDLMI fidelity framework and the standard setting process. In addition, a glossary was sent which contained the keywords from both the measure and the standard‐setting training. Per standard setting recommendations, the lead SDLMI developer and researcher were not included in these meetings so as not to impact the discussion and to allow for final decision making about cutscores.</p> <p>Ten individuals were invited to the 4‐h standard‐setting meeting. Three were unable to attend due to last‐minute scheduling conflicts; thus, the standard setting meeting was held with seven experts in attendance, representing each of the stakeholder groups. The meeting began with an hour of training on the <emph>SDLMI Fidelity Measure</emph> and the standard‐setting method, followed by practice using the method. The method chosen to set cutscores was the extended Angoff (see Plake & Cizek, [<reflink idref="bib22" id="ref46">22</reflink>]). Using this method, judges are presented with items and asked to rate how they believe the "minimally proficient implementer" would perform on that item. The methods detailed in Plake and Cizek ([<reflink idref="bib22" id="ref47">22</reflink>]) highlight two versions by which Angoff can be applied to polytomous items (and all items on the <emph>SDLMI Fidelity Measure</emph> are such items): one is to have judges estimate the mean score they would expect for a "minimally proficient implementer," and the other is to have judges estimate the score a "minimally proficient implementer" would likely achieve on each item. Using technology supports, we used the method of having judges estimate the score they would expect for a "minimally proficient implementer," or the number of minimally proficient implementers (out of 100 total) that they predict would score a 1, 2, and so forth.</p> <p>As recommended by Zieky et al. ([<reflink idref="bib40" id="ref48">40</reflink>]), standard setting was conducted in rounds. In the first round, all judges rated each item on the <emph>SDLMI Fidelity Measure</emph> separately. After rating each individual item, judges were provided with reality feedback (distributions on that item from existing SDLMI Fidelity Measure data) as well as normative feedback (information on how other judges rated the item). In cases of large differences, judges at the high and low ranges were encouraged to defend their positions to stimulate discussion. Once judges had discussed, a second round was undertaken where scores could be adjusted based on the discussion. This process was done for all items within each dimension assessed by the <emph>SDLMI Fidelity Measure</emph> (adherence, quality of delivery, student responsiveness). Then, once a second round was completed, the cutscore recommendation for that domain (based on the expected value provided for each item averaged across all judges) was shared with the panel. Using real data from the SDLMI implementation described previously, judges were provided with the percentage of first‐time implementers of the SDLMI that would have achieved sufficient fidelity based on the cutscore set by the panel. Judges were then asked if they would like to modify their entries based on the feedback data. For all domains, the judges declined.</p> <hd id="AN0174763706-8">Step 4: Finalizing cutscores</hd> <p>The cutscores developed in step 3 were then taken to the lead SDLMI developer and researcher who was not involved in Step 3 for final decision making, consistent with standard‐setting recommendations. Cutscore recommendations from step 3 were noninteger, which means that a decision needed to be made about whether to round scores up or down based on a determination of the types of error (false positive or false negative) that were most problematic because the goal was to have cutscore reflecting the scoring on the tool (i.e., whole numbers). For any given item, it can be assumed that the actual score is between the two integers. Given the complexity of the SDLMI, the difficulties of understanding all aspects of implementation during short observations, and the available data, the lead SDLMI developer decided to minimize false negatives (e.g., deciding implementation was not sufficient when it was) by adopting the more generous lower bound for each item. The final column in Table 1 provides the outcomes of this process, the final cutscores recommended by the expert panel and reviewed and accepted by the lead policy maker, which were then tested with real data, as described next.</p> <hd id="AN0174763706-9">IMPLICATIONS OF THE SDLMI CUTSCORES</hd> <p>Next, we explore the potential implications of these cutscores using data collected on the <emph>SDLMI Fidelity Measure</emph> in a 3‐‐year longitudinal efficacy trial of different intensities of support for teachers implementing the SDLMI with students in inclusive general education classrooms (Shogren et al., [<reflink idref="bib34" id="ref49">34</reflink>]), the same trial used to inform measure development and refinement in Shogren et al. ([<reflink idref="bib35" id="ref50">35</reflink>]). The data from this cluster randomized controlled trial (C‐RCT) comparing online supports only versus online supports + coaching for teachers implementing the SDLMI is highly relevant for an initial examination of the application of cutscores in practice. First, data were collected using the <emph>SDLMI Fidelity Measure</emph> across all 3 years of the trial, with up to six times points per academic year per teacher in the trial. Thus, it is possible to look at changes in teacher fidelity over time and the percentage of teachers that were implementing at sufficient levels at different points in their implementation. For example, we hypothesize that fidelity scores would increase over time, particularly on the quality of delivery and student responsiveness dimensions, given the complexity of implementation of the SDLMI and the documented impact of opportunities to repeatedly deliver the intervention (for teachers) and engage with the intervention (for students). Second, the two trial conditions in this C‐RCT involved different intensities of support for teachers making it possible to explore if changes in fidelity and aligned cutscores were sensitive to different types of supports. For example, we hypothesize that the more intensive online + coaching condition may lead to greater growth in teacher fidelity scores over time and greater implementation at sufficient levels.</p> <hd id="AN0174763706-10">Overview of the project</hd> <p>The 3‐year trial was implemented in high schools across two Mid‐Atlantic states (Shogren et al., [<reflink idref="bib34" id="ref51">34</reflink>]). The overall purpose of the trial was to evaluate the impact of intensifying teacher support for SDLMI implementation (online support or online support + in‐person coaching) on student outcomes in inclusive general education classes. As described at the onset of this article, the SDLMI (Shogren et al., [<reflink idref="bib33" id="ref52">33</reflink>]; Wehmeyer et al., [<reflink idref="bib37" id="ref53">37</reflink>]) is an evidence‐based intervention designed to be implemented by trained facilitators (e.g., general and special education teachers). It has three core components: Student Questions, Teacher Objectives, and Educational Supports. Trained SDLMI facilitators deliver targeted instruction aligned with the Teacher Objectives to teach students to use a series of 12 total questions (Student Questions) to guide themselves through a self‐regulated, problem‐solving process for setting goals, building action plans, and evaluating their progress toward goal attainment. Trained facilitators use Educational Supports that are individualized to students and class needs to enable all students to respond to the Student Questions and meet the Teacher Objectives. The SDLMI is delivered over an academic semester, with instruction organized around 12 Student Questions divided into three phases (phase 1: set a goal, phase 2: take action, and phase 3: adjust goal or plan). Instruction is repeated across semesters as students target new goals to build abilities and skills associated with self‐determination (see Raley et al., [<reflink idref="bib25" id="ref54">25</reflink>] for additional details on implementation).</p> <p>Implementing teachers in this project participated in a 2‐day standardized training during the summer before each implementation year. The training involved overall group instruction in the SDLMI and its implementation as well as school‐level planning for implementation. General education teachers, and special education teachers when schools used collaborative or co‐teaching models in inclusive classrooms, were trained to follow specific protocols for SDLMI whole‐class, core content implementation (Raley et al., [<reflink idref="bib25" id="ref55">25</reflink>]; Shogren et al., [<reflink idref="bib32" id="ref56">32</reflink>]). Implementing teachers delivered 2 weekly SDLMI minilessons (i.e., 15‐min instructional sessions) aligned with their curriculum, and infused students' goals and action plans throughout core content instruction (e.g., English Language Arts [ELA], mathematics, science).</p> <p>Based on random assignment, teachers received one of two types of implementation support after training: (a) online modules disseminated every 2 weeks via email (online only) or (b) online modules and in‐person coaching provided monthly by trained SDLMI coaches (online + coaching). The 15 online modules aligned with the three SDLMI phases and provided additional resources and support for teachers to meet targeted Teacher Objectives and use Educational Supports (e.g., self‐scheduling instruction, self‐evaluation instruction). Modules provided information to support implementation, including knowledge checks, but there was no direct interaction or communication with the research/coaching team. Teacher participants assigned to the online + coaching group received the online modules plus monthly, in‐person coaching from trained SDLMI coaches. Coaches had previous experience as teachers, administrators, and/or coaches and completed a standardized 2‐day training to learn how to implement the SDLMI coaching model (Hagiwara et al., [<reflink idref="bib14" id="ref57">14</reflink>]). Coaches conducted six individual coaching sessions with each teacher over the academic year (one coaching session per SDLMI phase each semester). Coaching sessions included involved a 30‐min observation using the <emph>SDLMI Fidelity Measure</emph>, a 30‐min conversation reflecting feedback session, and goal setting and action planning for the teacher's implementation before the next coaching session.</p> <hd id="AN0174763706-11">School and teacher participants</hd> <p>Schools were recruited to the project and randomly assigned to the online or online + coaching condition. During the first year of implementation (2018–2019), ninth‐grade teachers selected by school administrators were trained to implement the SDLMI in their inclusive core content (ELA, mathematics, science) classes with students with and without disabilities. In the second year of implementation, 9th‐grade teachers continued implementing with 10th‐grade teachers in the same schools being added along with 9th‐ and 10th‐grade teachers in newly implementing schools. COVID‐19 disrupted the project during Spring 2020 (second year). Implementation continued, as feasible, with all materials and supports including coaching and fidelity observations transitioning to virtual formats. Yet, no new schools were recruited, and significant attrition occurred.</p> <p>The sample for the present analysis is eight high schools and the teachers within them who participated for at least 1 of the 3 years of the project. Four schools (50%) were randomly assigned to the online only and four (50%) to the online + coaching support condition. Additional details on the overall study design, recruitment, attrition, and retention can be found in other sources (Shogren et al., [<reflink idref="bib34" id="ref58">34</reflink>]). Across the eight schools, there was variability in implementation duration. Six schools began implementation in year 1 (2018–2019) and two began in year 2 (2019–2020). After the first year, three schools withdrew as teachers did not feel that they had administrative support to continue implementation. In Spring 2020, with the onset of the COVID‐19 pandemic, implementation continued virtually with support from the research team, but two schools were unable to sustain implementation in 2020–2021 and two additional schools suspended implementation for Fall 2020 but reengaged in Spring 2021. See Table 2 for an overview of the eight schools, the semesters they participated, and the number of implementing teachers trained/retrained at each school, each year.</p> <p>2 Table School participation and general education teacher numbers.</p> <p> <ephtml> <table><thead valign="bottom"><tr valign="bottom"><th>ID</th><th>SY</th><th>Implementation support group</th><th>Setting</th><th>Title I</th><th>Project participation</th><th align="left">Number of teachers trained</th></tr><tr valign="bottom"><th>Year 1</th><th>Year 2</th><th>Year 3</th><th>Year 1</th><th>Year 2</th><th>Year 3</th></tr><tr valign="bottom"><th>Fall</th><th>Spring</th><th>Fall</th><th>Spring0003</th><th>Fall0003</th><th>Spring0003</th></tr></thead><tbody valign="top"><tr><td>1</td><td>18–19</td><td>Coaching</td><td>Urban</td><td>Yes</td><td>D</td><td>D</td><td>I</td><td>I</td><td>I</td><td>I</td><td /><td>4</td><td>4</td></tr><tr><td>2</td><td /><td>Coaching</td><td>Suburban</td><td>Yes</td><td>I</td><td>I</td><td>I</td><td>I</td><td>D</td><td>I</td><td>2</td><td>2</td><td>4</td></tr><tr><td>3</td><td /><td>Coaching</td><td>Urban</td><td>Yes</td><td>I</td><td>I</td><td>W</td><td /><td /><td /><td>1</td><td /><td /></tr><tr><td>4</td><td /><td>Online</td><td>Suburban</td><td>No</td><td>I</td><td>I</td><td>W</td><td /><td /><td /><td>1</td><td /><td /></tr><tr><td>5</td><td /><td>Online</td><td>Urban</td><td>No</td><td>I</td><td>I</td><td>I</td><td>I</td><td>W</td><td /><td>4</td><td>7</td><td /></tr><tr><td>6</td><td /><td>Online</td><td>Urban</td><td>No</td><td>I</td><td>I</td><td>W</td><td /><td /><td /><td>1</td><td /><td /></tr><tr><td>7</td><td /><td>Online</td><td>Rural</td><td>Yes</td><td>I</td><td>I</td><td>I</td><td>I</td><td>D</td><td>I</td><td>3</td><td>2</td><td>3</td></tr><tr><td>8</td><td>19–20</td><td>Coaching</td><td>Suburban</td><td>No</td><td /><td /><td>I</td><td>I</td><td>W</td><td /><td /><td>4</td><td /></tr></tbody></table> </ephtml> </p> <ulist> <item>3 <emph>Note</emph> : Reprinted with permission from the Kansas University Center on Developmental Disabilities.</item> <item>4 Abbreviations: D, deferred Implementation; I, implemented Self‐Determined Learning Model of Instruction; SY, school year entered project; W, withdrew from project.</item> <item>5 a COVID‐19 impact; continuing teachers in a school were retrained each year, and continuing teachers are represented in year 2 and 3 training totals.</item> </ulist> <p>Across the eight implementing schools, we collected fidelity data with 28 general education teachers implementing the SDLMI in inclusive, general education classrooms. Six special educators also participated in training and implementation in inclusive general education classrooms. However, there were varying levels of collaboration between general and special education teachers in implementing the SDLMI, ranging from no collaboration to joint participation in training with co‐teaching or consultative models, within and across sites making it difficult to examine links between the role of special education teachers—or, more generally, collaborative approaches in inclusive classrooms—to fidelity outcomes in the present analyses. As shown in Table 3, general and special education teachers taught an array of subject core content subject areas and had a range of teaching experiences. The general education teachers predominately identified as female and as White/European American. The small number of special education teachers identified as either male or female and 66.7% identified as Black/African American. Throughout the rest of the paper, <emph>teacher</emph> will primarily refer to general education teachers as general educators consistently implemented across schools and the <emph>SDLMI Fidelity Measure</emph> was completed based on their implementation. Future work is needed to further explore the impact of different teaching arrangements on SDLMI implementation and fidelity, as the wide variability in this study did not allow us to examine different approaches.</p> <p>3 Table Implementing general education teacher demographic information.</p> <p> <ephtml> <table><thead valign="bottom"><tr valign="bottom"><th>Teacher characteristics</th><th align="left">Implementers</th></tr><tr valign="bottom"><th>General education</th><th align="left">Special education</th></tr><tr valign="bottom"><th><italic>N</italic></th><th>%</th><th><italic>N</italic></th><th>%</th></tr></thead><tbody valign="top"><tr><td>Total sample</td><td align="char" char=".">28</td><td align="char" char=".">100.00</td><td align="char" char=".">6</td><td align="char" char=".">100.00</td></tr><tr><td>Gender</td><td /><td /><td /><td /></tr><tr><td>Female</td><td align="char" char=".">22</td><td align="char" char=".">78.57</td><td align="char" char=".">3</td><td align="char" char=".">50.00</td></tr><tr><td>Male</td><td align="char" char=".">6</td><td align="char" char=".">21.43</td><td align="char" char=".">3</td><td align="char" char=".">50.00</td></tr><tr><td>Nonbinary</td><td align="char" char=".">0</td><td align="char" char=".">0.00</td><td align="char" char=".">0</td><td align="char" char=".">0.00</td></tr><tr><td>Race</td><td /><td /><td /><td /></tr><tr><td>White/European American</td><td align="char" char=".">24</td><td align="char" char=".">85.71</td><td align="char" char=".">2</td><td align="char" char=".">33.33</td></tr><tr><td>Black/African American</td><td align="char" char=".">4</td><td align="char" char=".">14.29</td><td align="char" char=".">4</td><td align="char" char=".">66.67</td></tr><tr><td>Ethnicity</td><td /><td /><td /><td /></tr><tr><td>Non‐Hispanic/Non‐Latinx</td><td align="char" char=".">25</td><td align="char" char=".">89.29%</td><td align="char" char=".">5</td><td align="char" char=".">83.33%</td></tr><tr><td>Hispanic/Latinx</td><td align="char" char=".">2</td><td align="char" char=".">7.14</td><td align="char" char=".">1</td><td align="char" char=".">16.67</td></tr><tr><td>Missing</td><td align="char" char=".">1</td><td align="char" char=".">3.57</td><td align="char" char=".">0</td><td align="char" char=".">0.00</td></tr><tr><td>Highest degree earned</td><td /><td /><td /><td /></tr><tr><td>Bachelor's degree</td><td align="char" char=".">13</td><td align="char" char=".">46.43</td><td align="char" char=".">4</td><td align="char" char=".">66.67</td></tr><tr><td>Master's degree</td><td align="char" char=".">10</td><td align="char" char=".">35.71</td><td align="char" char=".">1</td><td align="char" char=".">16.67</td></tr><tr><td>Master's degree + credits</td><td align="char" char=".">5</td><td align="char" char=".">17.86</td><td align="char" char=".">1</td><td align="char" char=".">16.67</td></tr><tr><td>Subject taught0002</td><td /><td /><td /><td /></tr><tr><td>English</td><td align="char" char=".">12</td><td align="char" char=".">42.86</td><td align="char" char=".">3</td><td align="char" char=".">0.50%</td></tr><tr><td>Science</td><td align="char" char=".">12</td><td align="char" char=".">42.86</td><td align="char" char=".">0</td><td align="char" char=".">0.00%</td></tr><tr><td>Math</td><td align="char" char=".">4</td><td align="char" char=".">14.29</td><td align="char" char=".">3</td><td align="char" char=".">0.50%</td></tr><tr><td>Electives (e.g., music, art)</td><td align="char" char=".">1</td><td align="char" char=".">3.57</td><td align="char" char=".">3</td><td align="char" char=".">0.50%</td></tr><tr><td>Grade taught0003</td><td /><td /><td /><td /></tr><tr><td>9th</td><td align="char" char=".">22</td><td align="char" char=".">78.57</td><td align="char" char=".">5</td><td align="char" char=".">83.33</td></tr><tr><td>10th</td><td align="char" char=".">14</td><td align="char" char=".">50.00</td><td align="char" char=".">3</td><td align="char" char=".">50.00</td></tr><tr><td>11th</td><td align="char" char=".">9</td><td align="char" char=".">32.14</td><td align="char" char=".">1</td><td align="char" char=".">16.67</td></tr><tr><td>12th</td><td align="char" char=".">8</td><td align="char" char=".">28.57</td><td align="char" char=".">1</td><td align="char" char=".">16.67</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr><th /><th align="left"><italic>M</italic></th><th align="left">SD</th><th align="left"><italic>M</italic></th><th align="left">SD</th></tr></thead><tbody valign="top"><tr><td>Teaching experience</td><td>10.43</td><td>7.44</td><td>10.18</td><td>6.84</td></tr><tr><td>Age</td><td>37.18</td><td>9.07</td><td>35</td><td>8.6</td></tr></tbody></table> </ephtml> </p> <ulist> <item>6 <emph>Note</emph>: Reprinted with permission from the Kansas University Center on Developmental Disabilities.</item> <item>7 a Teachers could select more than one subject taught.</item> <item>8 b Teachers could select more than one grade taught.</item> </ulist> <p>Although student outcomes were not a focus of the present analysis, it is worth noting there were a total of 2903 students (1392 students whose teachers were in the online condition; 1511 students whose teachers were in online + coaching) that contributed student‐level outcome data to the project and were in the targeted general education classrooms of the 28 general and 6 special education teachers. The average number of students per school was 402 (SD = 298), ranging from 78 to 967. In the student sample, 16.5% had a documented disability; 43.9% identified as Black/African American, 35.9% identified as White/European American, and 11.6% identified as Hispanic/Latinx.</p> <hd id="AN0174763706-12">Fidelity data collection</hd> <p>For both trial conditions, fidelity observations were conducted in each classroom during one of the class periods that teachers were implementing the SDLMI using the <emph>SDLMI Fidelity Measure</emph> (described previously). Fidelity observations occurred three times each semester (six times each year of implementation) aligned with the three phases of the SDLMI (i.e., one classroom observation occurred while the teacher was implementing phase 1, one in phase 2, one in phase 3). Fidelity observations were completed by external, trained observers unaware of the trial condition to which participants were assigned. Fidelity observations were separate from coaching observations described previously. Fidelity observers included retired school administrators with numerous years of experience conducting classroom observations in inclusive, general education classrooms as well as former educators who served as professional learning facilitators. All fidelity observers attended a 2‐day training organized by the research team before the start of each school year that provided a comprehensive overview of self‐determination and the SDLMI as well as an in‐depth training on applying the <emph>SDLMI Fidelity Measure</emph> to example classroom videos of SDLMI instruction. All observers were required to reach 90% reliability across all items on the measure before completing fidelity observations for this study. We examined the interrater reliability of 30% of observations by having a secondary observer also complete the <emph>SDLMI Fidelity Measure</emph> simultaneously in the same classroom. Kappa ranged from 0.62 to 0.95, and using the criteria put forward by WWC ([<reflink idref="bib39" id="ref59">39</reflink>]), where 0.50 or greater is deemed acceptable. During the first year and a half of the project, observations occurred in person in the classroom. After the onset of the COVID‐19 pandemic, all observations shifted to virtual. During virtual instruction, teachers uploaded implementation videos per SDLMI phase via a secure video website platform, Torsh, using their smartphone's camera or recorded Zoom class sessions. Fidelity and reliability observers were able to view videos asynchronously and complete the <emph>SDLMI Fidelity Measure</emph>.</p> <p>If teachers participated for the entire 3 years of the project, there could have been 18 fidelity checks. However, as teachers and schools moved in and out of the project (see Table 2), the number of completed fidelity checks per teacher was lower, ranging from 1 to 12 (<emph>m</emph> = 5.75; SD = 3.18), the average reflecting approximately a year of fidelity observations. In total, across teachers, there were 161 unique fidelity observations. There was a small amount of missing data within fidelity observations which was handled using full information estimation.</p> <hd id="AN0174763706-13">Analysis plan</hd> <p>To explore teacher fidelity and the impact of the established cutscores, we first descriptively summarized teachers' average rating for each fidelity dimension over time. Then, we examined change over time comparing data from the first fidelity check for each teacher to a midpoint fidelity check and the endpoint fidelity check. We explored both the mid‐ and endpoint fidelity checks to explore change over the course of teacher SDLMI implementation as well as to address the reduced sample size at the end‐point because of attrition in teachers and schools due to the COVID‐19 public health emergency. We used Cohen's <emph>d</emph> as our effect size measure. Traditionally, Cohen ([<reflink idref="bib8" id="ref60">8</reflink>]) proposed that the borders of small, medium, and large for this effect size be 0.2, 0.5, and 0.8, respectively. Second, we inspected data to examine our exploratory hypothesis that as teachers gained implementation experience and received coaching, the probability that their fidelity increased to sufficient levels increased. Our analytic strategy for statistically evaluating this hypothesis was Bayesian model averaging (BMA; Hoeting, et al., [<reflink idref="bib17" id="ref61">17</reflink>]). The logic of BMA analysis differs from null hypothesis significance testing (NHST). Traditionally, NHST aims to reject the null ( <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0001" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>0</mn></msub></mrow></math> </ephtml> ) in one zero/sum decision, which is challenging with small sample sizes. In contrast, BMA analysis only evaluates the probability of candidate models, without necessarily rejecting or accepting them, which obviates an assumed need in NHST to categorically falsify the null when only small sample sizes are accessible (Howson & Urbach, [<reflink idref="bib18" id="ref62">18</reflink>]). In addition to avoiding the need to categorically reject the null, a benefit of BMA analyses is that they allow predictions about the same event coming from a diverse group of models to be combined (or averaged) into one unified, robust prediction.</p> <p>We implemented this BMA analysis in multiple steps using the Markov Chain Monte (MCMC) simulation procedure in SAS 9.4 software (PROC MCMC; SAS Institute Inc., [<reflink idref="bib28" id="ref63">28</reflink>]). All scripts and data needed to replicate these analyses can be found on Open Science Foundation at <https://osf.io/v7azb/?view%5fonly=038b12cff8ba4a4982b94212e559b2ab>. First, separately for each domain (adherence, quality of delivery, and student responsiveness), we fit and tested different versions of a three‐level logistic regression model (occasions nested in teachers; teachers nested in schools): <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0002" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>0</mn></msub></mrow></math> </ephtml> (null); <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0003" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>1</mn></msub></mrow></math> </ephtml> (time effect); <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0004" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>2</mn></msub></mrow></math> </ephtml> (group effect); and <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0005" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>3</mn></msub></mrow></math> </ephtml> (group‐by‐time effect). The dependent variable was fidelity status (coded: sufficient vs. nonsufficient). We also implemented Bayesian estimation with diffuse priors, such that parameter estimates would coincide with full information maximum likelihood to seamlessly manage imbalanced numbers of fidelity data across sampled educators. Second, the relative fit of each estimated model was evaluated using the Deviance information criterion (DIC; Ando, [<reflink idref="bib1" id="ref64">1</reflink>]). In DIC analysis, model fit is quantified with the DIC value (smaller is better), which balances the virtues of model simplicity and explanatory power. Third, DIC values were translated into model weights, which denote the probability that a model has the most explanatory power (McElreath, [<reflink idref="bib21" id="ref65">21</reflink>]). Fourth, taking advantage of BMA posterior predictive modeling, the separate predictions coming from each candidate model ( <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>0</mn></msub><mo>,</mo><msub><mi>M</mi><mn>1</mn></msub><mo>,</mo><msub><mi>M</mi><mn>2</mn></msub><mo>,</mo><msub><mi>M</mi><mn>3</mn></msub></mrow></math> </ephtml> ) were averaged into one unified prediction for a more comprehensive picture of the model‐implied percentage of implementers in the population who would obtain sufficient implementation over time disaggregated by group across domains to better evaluate the practical significance of model output. A strength of these models is that they estimate growth trajectories based on all available data, including the more robust data available from the first half of the project before the onset of the COVID‐19 public health emergency.</p> <hd id="AN0174763706-14">Results</hd> <p>Figure 1 provides a visualization of data on obtained fidelity outcomes across measurement points, with the cutscore line provided. This visualization suggests there was variability around the cutscores in terms of actual fidelity outcomes, but large proportions of teachers were meeting cutscores, particularly for adherence. Moving beyond the raw data, Table 4 provides a summary of the results of BMA analysis and Figure 2 presents the results of posterior predictive modeling using BMA analysis, which synthesizes the predictions of separate models into one robust prediction, disaggregated by group. Key findings across the three dimensions of the <emph>SDLMI Fidelity Measure</emph> (adherence, quality of delivery, and student responsiveness) are described below.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/PIS/01feb24/pits23068-fig-0001.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="pits23068-fig-0001.jpg" title="1 Fidelity outcomes across time by domain. The horizontal dashed line denotes the respective cutscore. The solid line denotes a regression curve on the raw ratings using time as a predictor. Mean, average rating; N, sample size; Pct, proportion of implementer sample with sufficient implementation; STD, standard deviation of ratings." /> </p> <p></p> <p>4 Table Results of BMA analysis.</p> <p> <ephtml> <table><thead valign="bottom"><tr valign="bottom"><th /><th><p><math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>M</mi><mn>0</mn></msub></mrow></math></p></th><th><p><math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>M</mi><mn>1</mn></msub></mrow></math></p></th><th><p><math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>M</mi><mn>2</mn></msub></mrow></math></p></th><th><p><math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>M</mi><mn>3</mn></msub></mrow></math></p></th></tr><tr valign="bottom"><th>Domain</th><th>Model parameters</th><th>Estimate</th><th>SE</th><th>Estimate</th><th>SE</th><th>Estimate</th><th>SE</th><th>Estimate</th><th>SE</th></tr></thead><tbody valign="top"><tr><td>Adherence</td><td>Fixed effects</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>Intercept</td><td>3.106</td><td>0.665</td><td>2.795</td><td>0.786</td><td>2.807</td><td>0.931</td><td>2.615</td><td>1.058</td></tr><tr><td /><td>Time</td><td /><td /><td>0.079</td><td>0.113</td><td /><td /><td>0.070</td><td>0.131</td></tr><tr><td /><td>Group</td><td /><td /><td /><td /><td>0.675</td><td>1.269</td><td>0.540</td><td>1.622</td></tr><tr><td /><td>Time × group</td><td /><td /><td /><td /><td /><td /><td>0.072</td><td>0.268</td></tr><tr><td /><td>Random effects</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>Teacher variance</td><td>1.198</td><td>0.929</td><td>1.126</td><td>0.735</td><td>1.170</td><td>0.803</td><td>1.255</td><td>0.817</td></tr><tr><td /><td>School variance</td><td>1.281</td><td>1.046</td><td>1.380</td><td>1.179</td><td>1.471</td><td>1.406</td><td>1.577</td><td>1.630</td></tr><tr><td /><td>BMA analysis</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>Dbar (posterior mean of deviance)</td><td>90.103</td><td /><td>90.623</td><td /><td>90.427</td><td /><td>91.345</td><td /></tr><tr><td /><td>Dmean (deviance evaluated at posterior mean)</td><td>80.726</td><td /><td>80.486</td><td /><td>80.843</td><td /><td>79.885</td><td /></tr><tr><td /><td>pD (effective number of parameters)</td><td>9.377</td><td /><td>10.137</td><td /><td>9.584</td><td /><td>11.460</td><td /></tr><tr><td /><td>DIC (smaller is better)</td><td>99.480</td><td /><td>100.760</td><td /><td>100.010</td><td /><td>102.805</td><td /></tr><tr><td /><td>Model weight</td><td>0.403</td><td /><td>0.212</td><td /><td>0.309</td><td /><td>0.076</td><td /></tr><tr><td>Quality of delivery</td><td>Fixed effects</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>Intercept</td><td>1.678</td><td>0.701</td><td>0.728</td><td>0.763</td><td>1.887</td><td>1.039</td><td>0.875</td><td>1.241</td></tr><tr><td /><td>Time</td><td /><td /><td>0.282</td><td>0.098</td><td /><td /><td>0.268</td><td>0.118</td></tr><tr><td /><td>Group</td><td /><td /><td /><td /><td>−0.298</td><td>1.308</td><td>−0.313</td><td>1.672</td></tr><tr><td /><td>Time × group</td><td /><td /><td /><td /><td /><td /><td>0.082</td><td>0.229</td></tr><tr><td /><td>Random effects</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>Teacher variance</td><td>1.733</td><td>1.198</td><td>1.710</td><td>1.157</td><td>1.707</td><td>1.074</td><td>1.831</td><td>1.291</td></tr><tr><td /><td>School variance</td><td>1.970</td><td>1.638</td><td>2.210</td><td>2.027</td><td>2.326</td><td>2.084</td><td>2.637</td><td>2.322</td></tr><tr><td /><td>BMA analysis</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>Dbar (posterior mean of deviance)</td><td>136.208</td><td /><td>128.832</td><td /><td>136.081</td><td /><td>129.336</td><td /></tr><tr><td /><td>Dmean (deviance evaluated at posterior mean)</td><td>120.241</td><td /><td>111.961</td><td /><td>119.811</td><td /><td>110.938</td><td /></tr><tr><td /><td>pD (effective number of parameters)</td><td>15.966</td><td /><td>16.871</td><td /><td>16.269</td><td /><td>18.397</td><td /></tr><tr><td /><td>DIC (smaller is better)</td><td>152.174</td><td /><td>145.702</td><td /><td>152.350</td><td /><td>147.733</td><td /></tr><tr><td /><td>Model weight</td><td>0.027</td><td /><td>0.696</td><td /><td>0.025</td><td /><td>0.252</td><td /></tr><tr><td>Student responsiveness</td><td>Fixed effects</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>Intercept</td><td>2.370</td><td>0.818</td><td>1.293</td><td>0.956</td><td>2.570</td><td>1.385</td><td>1.477</td><td>1.438</td></tr><tr><td /><td>Time</td><td /><td /><td>0.337</td><td>0.116</td><td /><td /><td>0.298</td><td>0.132</td></tr><tr><td /><td>Group</td><td /><td /><td /><td /><td>−0.083</td><td>1.912</td><td>−0.407</td><td>2.010</td></tr><tr><td /><td>Time × group</td><td /><td /><td /><td /><td /><td /><td>0.201</td><td>0.298</td></tr><tr><td /><td>Random effects</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>Teacher Var</td><td>2.347</td><td>1.811</td><td>2.338</td><td>1.699</td><td>2.395</td><td>1.763</td><td>2.615</td><td>2.226</td></tr><tr><td /><td>School Var</td><td>2.463</td><td>2.444</td><td>3.129</td><td>3.199</td><td>3.071</td><td>3.028</td><td>3.918</td><td>5.510</td></tr><tr><td /><td>BMA analysis</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>Dbar (posterior mean of deviance)</td><td>107.146</td><td /><td>97.858</td><td /><td>106.655</td><td /><td>97.855</td><td /></tr><tr><td /><td>Dmean (deviance evaluated at posterior mean)</td><td>92.807</td><td /><td>82.386</td><td /><td>92.157</td><td /><td>81.179</td><td /></tr><tr><td /><td>pD (effective number of parameters)</td><td>14.339</td><td /><td>15.472</td><td /><td>14.498</td><td /><td>16.676</td><td /></tr><tr><td /><td>DIC (smaller is better)</td><td>121.485</td><td /><td>113.330</td><td /><td>121.153</td><td /><td>114.530</td><td /></tr><tr><td /><td>Model weight (on probability scale)</td><td>0.011</td><td /><td>0.631</td><td /><td>0.013</td><td /><td>0.346</td><td /></tr></tbody></table> </ephtml> </p> <ulist> <item>9 <emph>Note</emph> : This table shows the results of BMA analysis by domain (adherence, quality, and responsiveness). The four rows under "fixed effects" denote the estimated effect (with standard error) for model parameters on the log‐odds scale. The two rows under "Random Effects" denote the variance of random effects for teachers and schools. In the five rows under BMA analysis, "DIC" denotes the deviation information criterion value and "Model Weight" the corresponding model probability.</item> <item>10 Abbreviation: BMA, Bayesian model averaging.</item> </ulist> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/PIS/01feb24/pits23068-fig-0002.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="pits23068-fig-0002.jpg" title="2 BMA posterior predictive modeling. BMA analysis synthesized the predictions of all models (M0,M1,M2,M3) into one unified, optimally weighted prediction of the percentage of a population of implementers with practices that will be rated as sufficient in the respective domain given equal group sizes. Inspection of these predictions indicates the model‐implied change over time in the probability of obtaining sufficient implementation. BMA, Bayesian model averaging." /> </p> <p></p> <hd id="AN0174763706-17">Adherence</hd> <p>At the first fidelity check, a large majority of the 28 educators (93%) obtained sufficient adherence ratings, with an average rating of 9.96 (SD = 2.03). This was maintained over time with fidelity scores between 92% and 100% of teachers meeting cutscores at each observation. Given the ceiling effects and attrition, it is more informative to look at the totality of the data over time in the BMA analysis. Examining different models for obtaining sufficient adherence, which adjusts for complexities, such as imbalanced sample sizes across time, BMA analysis found that <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>0</mn></msub></mrow></math> </ephtml> (null) actually had a plurality of probability of being the most explanatory model, <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mrow><mo>(</mo><msub><mi>M</mi><mn>0</mn></msub><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>0</mn><mo>.</mo><mn>403</mn></mrow></math> </ephtml> , which makes sense given that at the first fidelity check the overwhelming majority of teachers were already well above what constitutes sufficient adherence. Interpreted, <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>0</mn></msub></mrow></math> </ephtml> simply predicts that, irrespective of time or the intensification of implementation supports, 95.7% of the implementers will obtain sufficient adherence after initial training. Although confidence in other models was lower, their model probabilities still warrant consideration [ <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mrow><mo>(</mo><msub><mi>M</mi><mn>1</mn></msub><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>0</mn><mo>.</mo><mn>212</mn></mrow></math> </ephtml> , <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mrow><mo>(</mo><msub><mi>M</mi><mn>2</mn></msub><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>0</mn><mo>.</mo><mn>309</mn></mrow></math> </ephtml> , <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mrow><mo>(</mo><msub><mi>M</mi><mn>3</mn></msub><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>0</mn><mo>.</mo><mn>076</mn></mrow></math> </ephtml> ], particularly with larger data sets in future research. Given this, we pooled the separate predictions of each model into one overall prediction, BMA predicts that experience and intensifying implementation supports (i.e., providing coaching) generate very subtle, but noticeable increases in the probability of sufficient adherence (e.g., BMA predicts that intensifying supports leads to a 1% improvement in the probability of sufficient adherence; see Figure 2).</p> <hd id="AN0174763706-18">Quality of delivery</hd> <p>At first check, 18 (64%) out of 28 educators obtained sufficient quality of delivery ratings, with an average rating of 14.64 (SD = 3.72). Thus, teachers had lower rates of sufficient quality of delivery, after training. Over time, between 59% and 100% of educators met the cutscore. In fact, by the eighth fidelity check, seven out of eight remaining educators obtained sufficient quality of delivery, with an average rating of 18.57. This change from the initial data collection point constitutes a large effect size (<emph>d</emph> = 1.13). By the 12th (or last) fidelity check, two out of two remaining educators obtained sufficient quality of delivery, with an average rating of 16.50. This change from the initial data collection point constitutes a moderate effect size (<emph>d</emph> = 0.50). Controlling for variability in the sample over time, BMA analysis found that, among models, <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0018" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>1</mn></msub></mrow></math> </ephtml> (time effects) has the greatest probability of being the most explanatory model, <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0019" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>P</mi><mo>(</mo><mi>M</mi></mrow><mn>1</mn></msub><mrow><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>0</mn><mo>.</mo><mn>696</mn></mrow></math> </ephtml> , but that <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0020" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>3</mn></msub></mrow></math> </ephtml> (time‐by‐group effects) also had a decent probability <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0021" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>P</mi><mo>(</mo><mi>M</mi></mrow><mn>3</mn></msub><mrow><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>0</mn><mo>.</mo><mn>252</mn></mrow></math> </ephtml> . These results suggest a 94.8% confidence that experience implementing over time increases the probability of sufficient quality of delivery and a 25.2% confidence that intensification of supports (i.e., adding coaching) further amplifies the positive impact of time. These findings align with our hypotheses that practice implementation over time will lead to a higher percentage of educators obtaining sufficient quality of delivery ratings. Synthesizing across models, BMA predicts—based on growth trajectories in all available longitudinal data—that, irrespective of group, less than 75% of implementers will have sufficient quality of delivery at the first check but that this percentage will steadily grow to over 95% by the 12th check (or equivalent to 2 full years of implementation experience) and that online versus online + coaching support has less probability of impacting quality outcomes (see Figure 2 for a visual representation, where the growth curves for the online and online + coaching group are almost indistinguishable).</p> <hd id="AN0174763706-19">Student responsiveness</hd> <p>At first check, 18 (69%) out of 26 educators with usable data were rated as demonstrating sufficient student responsiveness with an average rating of 9.23 (SD = 2.57). By the eighth fidelity check, seven out of the eight remaining educators obtained sufficient levels of student responsiveness, with an average rating of 11.42. This change constitutes a borderline large effect size (<emph>d</emph> = 0.86). By the last check, two out of the two remaining educators obtained sufficient student responsiveness, with an average rating of 11.50 with a borderline large effect size (<emph>d</emph> = 0.80) although interpretation must be tempered given the small sample size. These large effect sizes at the mid‐ and endtime point are further supported by BMA analysis that found that—accounting for all data over time—out of the models, <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0022" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>1</mn></msub></mrow></math> </ephtml> (time effects) has the greatest probability, <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0023" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>P</mi><mo>(</mo><mi>M</mi></mrow><mn>1</mn></msub><mrow><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>0</mn><mo>.</mo><mn>631</mn></mrow></math> </ephtml> , but that <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0024" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>3</mn></msub></mrow></math> </ephtml> (time‐by‐group effects) also has a high probability, <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0025" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>P</mi><mo>(</mo><mi>M</mi></mrow><mn>3</mn></msub><mrow><mo>|</mo><mi>D</mi><mo>)</mo></mrow><mo>=</mo><mn>346</mn></mrow></math> </ephtml> . This suggests 97.7% confidence of a time effect, with a 34.6% confidence that intensifying implementation supports accelerates the time effect, which conforms to the research hypothesis. However, more data needs to be collected to select between the simpler <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0026" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>1</mn></msub></mrow></math> </ephtml> and the more complex <ephtml> <math altimg="urn:x-wiley:00333085:media:pits23068:pits23068-math-0027" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>M</mi><mn>3</mn></msub></mrow></math> </ephtml> . Synthesizing across models, BMA predicts (using growth trajectories based on all available longitudinal data) that, within a population of implementers, less than 80% will have sufficient student responsiveness at the first check but that this percentage will steady grow to over 95% by the 12th check (or equivalent to 2 full years of implementation experience). In addition, the intensification of implementation supports will accelerate the positive impact of experience on the probability of achieving sufficient student responsiveness, although subtly (see Figure 2 for a visual).</p> <hd id="AN0174763706-20">IMPLICATIONS FOR ONGOING RESEARCH AND PRACTICE</hd> <p>This paper reports on the application of a standards‐setting process (Zieky et al., [<reflink idref="bib40" id="ref66">40</reflink>]) to establishing cutscores for the fidelity of implementation for an evidence‐based intervention, the SDLMI, designed to enhance goal‐directed actions in secondary students with and without disabilities. While there have been other applications of this process (Center on IDEA Early Childhood Data Systems and Early Childhood TA Center, [<reflink idref="bib7" id="ref67">7</reflink>]), this use of standards setting in the context of fidelity assessment remains novel in inclusive education research. However, applying this process to a multidimensional approach to defining fidelity of a complex intervention like the SDLMI shows the promise of the approach for establishing clear standards for sufficient fidelity in practice as well as for understanding the impacts of intensifying supports on different dimensions of fidelity. Our findings not only impact future SDLMI inquiry but also have implications for future research and practice directed at understanding the personalized support teachers need to implement complex interventions with students with and without disabilities as well as the relationship between dimensions of fidelity and student outcomes. With the shift toward integrated tiered systems of support, away from reactive approaches to meeting student's academic, behavioral, and social and emotional well‐being needs, this work on measuring fidelity holds promise on a large scale.</p> <p>The present work applying standards setting builds on the established SDLMI fidelity framework that defined three key fidelity dimensions that are observable and measurable (adherence, quality of delivery, and student responsiveness) using the <emph>SDLMI Fidelity Measure</emph> (Shogren et al., [<reflink idref="bib35" id="ref68">35</reflink>]). However, this is the first attempt to define what <emph>sufficient</emph> fidelity means for each of these dimensions. Such work has the power to guide the development of future support for in‐service as well as preservice teachers, particularly to inform decision making about when teachers may need additional or more intensive support. Just as with educating students (Raley et al., [<reflink idref="bib23" id="ref69">23</reflink>]), teachers can benefit from tiered supports that are individualized to their needs based on data collected about implementation (Kiblen, Shogren, Kurth, et al., [<reflink idref="bib19" id="ref70">19</reflink>]). Yet, there has not yet been a clear means to determine how to use data to define teachers' support needs for implementation. Findings from this study suggest teachers demonstrate a range of fidelity outcomes across the three dimensions of adherence, quality of delivery, and student responsiveness over time. Almost all teachers (93%) of teachers in this sample immediately meet cutscores for the adherence dimension after training, but smaller numbers meet cutscores for quality of delivery (64%) and student responsiveness (69%). However, the quality of delivery and student responsiveness show growth over time with implementation experience. There was also an impact‐ albeit smaller than desired—of intensifying implementation supports. Figure 2 highlights this growth and shows the subtle effect of coaching. Interestingly as shown in Figure 2, there may have been a greater impact of more intense support on adherence, but it was maintained over time, and impacted by ceiling effects.</p> <p>It is important to note, however, that as shown in Figure 1, while most teachers showed sufficient adherence initially and quality of delivery and student responsiveness during their first year of implementation, some teachers continue to struggle to meet cutscores over time. This suggests a need to focus on identifying teachers who struggle with fidelity to inform more intensive support. It may be these supports are most impactful early in implementation (e.g., during the first semester or year). This information is important not only to determine needs but also to consider corresponding implications in terms of personnel time and associated cost demands. Ongoing work is needed to determine how to effectively create a system that can further develop an understanding of when, where, and how to intensify support for teachers to enhance their instruction with the goal of enhancing student outcomes, aligning with work in other areas (Snyder et al., [<reflink idref="bib36" id="ref71">36</reflink>]). A key aspect of this work must be analyzing factors influencing teacher's fidelity and determining if, when, and what types of support teachers need using data to inform decision making, support, and coaching. Ongoing research must seek to understand how to better understand what influences the support teachers need at different points in time to inform effective resource utilization in schools as coaching is a cost‐intensive element of SDLMI implementation (Rifenbark et al., [<reflink idref="bib26" id="ref72">26</reflink>]). The SDLMI coaching model is currently being expanded within a tiered system to examine how tiered approaches could be applied to supporting all teachers to have and access needed support, with more intensive support provided as needed, based on data (Kiblen, Shogren, Kurth, et al., [<reflink idref="bib19" id="ref73">19</reflink>]).</p> <p>This work can be informed by these findings as it confirms that conceptualizing fidelity as a multidimensional construct allows for the differentiation of aspects of implementation that may respond to different supports. An important next step will be defining the supports needed to enhance fidelity to these different dimensions. For example, adherence (common focus in current research and practice) had the highest levels of teachers' meeting sufficient fidelity standards at the start and throughout implementation. For this reason, adherence may not be a primary target of implementation support after training. Yet, coaching approaches may need to consider how to ensure coaching sessions support movement beyond basic adherence to quality of delivery and promoting student responsiveness. These domains may need to be targeted through implementation support after initial training. Further, promoting student responsiveness had the greatest probably of being impacted by time and with more intensive support, suggesting a need to determine why and how to best support teachers with this aspect of fidelity and if there are unique aspects of engaging students in SDLMI instruction. Overall, there are a range of questions to be examined on the most effective type, timing, and intensity of in‐person (or virtual) coaching and how teacher's initial fidelity could inform data‐based decision making on the most cost‐effective ways to deliver and individualize implementation supports.</p> <p>Additionally, examining the relationship between the three dimensions of fidelity, at sufficient levels, and student outcomes is a critical next step in this line of research as this is the ultimate outcome of supporting fidelity of implementation at sufficient levels. The SDLMI has been found to have a positive effect on student self‐determination and goal attainment outcomes across multiple studies (Hagiwara et al., [<reflink idref="bib15" id="ref74">15</reflink>]), but there is still substantial variability in outcomes. Research drawing on teacher's self‐reports of their fidelity, suggested a relationship between fidelity and student outcomes that is influenced both by teacher support as well as student characteristics (Shogren, Burke, et al., [<reflink idref="bib29" id="ref75">29</reflink>]). Further, given that the policy makers involved in step 1 of the standards‐setting process all had expertise in the SDLMI and self‐determination research, ongoing validation of these PLDs and cutscores must occur, ensuring consideration of the range of factors that impact implementation in secondary schools.</p> <p>Future research must also address the additional limitations of this study, including the impacts of the pandemic on sustained implementation, specifically the small number of teachers that had 2 full years of fidelity observations. To address these issues, we calculated effect sizes for changes in fidelity outcomes for quality of delivery and student responsiveness at a mid‐ and endtime point as attrition was less at the midtime point to ensure we were not overestimating effects. Further, our BMA analytic approach also used all available data and was not dependent on the number of teachers available as the last observation. However, future research is needed with larger samples, sustained over time, to replicate and extend these findings. There is also a need to consider how other implementation drivers (e.g., administrative support) influence sustained implementation and fidelity (Raley et al., [<reflink idref="bib24" id="ref76">24</reflink>]). For example, we had three schools where teachers reported they struggled to find the time to plan and implement after the first year as there was not an administrative "champion" at the building. This suggests the importance of administrative support for implementation, generally, and fidelity of implementation, in particular. Finally, we were unable to fully explore the impact of using collaborative teaching models that involved general and special education in delivery. Despite the project being focused on implementation in inclusive, general education classrooms and 16.5% of the student sample having a documented disability, there were not consistent collaborative or co‐teaching models being implemented within or across schools. Most school leaders were unable to identify general and special education teachers to be co‐trained and of those that did, there were no consistent models being implemented to enable general and special educators to collaboratively deliver instruction to benefit all students. Ongoing work is needed to examine these broader, systemic issues in schools and how to effectively support all teachers and all students, using evidence‐based practices that are rooted in inclusive and culturally responsive approaches to instruction. Even with these considerations, we view the lessons learned and reported in this manuscript as paving the path for future inquiry exploring issues related to assessing implementation fidelity of complex interventions—including the intersections of implementation fidelity, customized professional learning and supports, and student outcomes.</p> <hd id="AN0174763706-21">ACKNOWLEDGMENTS</hd> <p>The research reported here was supported by the Institute of Education Sciences, US Department of Education, through Grant R324A170008 to the University of Kansas. The opinions expressed are those of the authors and do not represent the views of the Institute or the US Department of Education.</p> <hd id="AN0174763706-22">DATA AVAILABILITY STATEMENT</hd> <p>The data that support the findings of this study are openly available in Open Science Framework at https://osf.io/v7azb/?view_only=038b12cff8ba4a4982b94212e559b2ab.</p> <ref id="AN0174763706-23"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref64" type="bt">1</bibl> <bibtext> Ando, T. (2007). Bayesian model selection and statistical models. CPC Press.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref8" type="bt">2</bibl> <bibtext> Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., Ogedegbe, G., Orwig, D., Ernst, D., & Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology, 23 (5), 443 – 451. https://doi.org/10.1037/0278-6133.23.5.443</bibtext> </blist> <blist> <bibl id="bib3" idref="ref24" type="bt">3</bibl> <bibtext> Borrelli, B., Sepinwall, D., Ernst, D., Bellg, A. J., Czajkowski, S., Breger, R., DeFrancesco, C., Levesque, C., Sharp, D. L., Ogedegbe, G., Resnick, B., & Orwig, D. (2005). A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology, 73 (5), 852 – 860. https://doi.org/10.1037/0022-006X.73.5.852</bibtext> </blist> <blist> <bibl id="bib4" idref="ref14" type="bt">4</bibl> <bibtext> Buckman, M. M., Lane, K. L., Common, E. A., Royer, D. J., Oakes, W. P., Allen, G. E., Lane, K. S., & Brunsting, N. C. (2021). Treatment integrity of primary (tier 1) prevention efforts in tiered systems: Mapping the literature. Education and Treatment of Children, 44, 145 – 168. https://doi.org/10.1007/s43494-021-00044-4</bibtext> </blist> <blist> <bibl id="bib5" idref="ref26" type="bt">5</bibl> <bibtext> Buckman, M. M., Lane, K. L., & Oakes, W. P. (2022). Treatment integrity and social validity in tiered systems: Using data to inform implementation efforts. In C. J. Lemons, S. R. Powell, K. L. Lane, & T. C. Aceves (Eds.), Handbook of special education research: Research based practices and intervention innovations (pp. 19 – 30). Routledge.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref6" type="bt">6</bibl> <bibtext> Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2 (1), 40. https://doi.org/10.1186/1748-5908-2-40</bibtext> </blist> <blist> <bibl id="bib7" idref="ref30" type="bt">7</bibl> <bibtext> Center on IDEA Early Childhood Data Systems and Early Childhood TA Center. (2019). Evaluating implementation of evidence‐based practices—Tip sheet series. Tip Sheet 3: Establishing a Fidelity Threshold. https://dasycenter.org/evaluating-the-implementation-of-evidence-based-practices-tip-sheet-series/tip-sheet-3-establishing-a-fidelity-threshold/</bibtext> </blist> <blist> <bibl id="bib8" idref="ref60" type="bt">8</bibl> <bibtext> Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203771587</bibtext> </blist> <blist> <bibl id="bib9" idref="ref7" type="bt">9</bibl> <bibtext> Fixsen, D. L., Blase, K. A., Duda, M. A., Naoom, S. F., & Van Dyke, M. (2010a). Implementation of evidence‐based treatments for children and adolescents: Research findings and their implications for the future, In J. R. Weisz & A. E. Kazdin (Eds.), Evidence‐based psychotherapies for children and adolescents (pp. 435 – 450). The Guilford Press.</bibtext> </blist> <blist> <bibtext> Fixsen, D. L., Blase, K. A., Duda, M. A., Naoom, S. F., & Van Dyke, M. (2010b). Sustainability of evidence‐based programs in education. Journal of Evidence‐Based Practices for Schools, 11, 30 – 46.</bibtext> </blist> <blist> <bibtext> Gandhi, A. G., Clemens, N., Coyne, M., Goodman, S. L.L. K., Lembke, E., & Simonsen, B. (in press). Integrated multi‐tiered systems of support (I‐MTSS): New directions for supporting students with or at risk for learning disabilities, Handbook of learning disabilities (3rd ed.). Guildford Press.</bibtext> </blist> <blist> <bibtext> Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi‐experimental research in special education. Exceptional Children, 71 (2), 149 – 164. https://doi.org/10.1177/001440290507100202</bibtext> </blist> <blist> <bibtext> Graham, L., Wright, J., Walwyn, R., Russell, A. M., Bryant, L., Farrin, A., & House, A. (2016). Measurement of adherence in a randomised controlled trial of a complex intervention: supported self‐management for adults with learning disability and type 2 diabetes. BMC Medical Research Methodology, 16 (1), 132. https://doi.org/10.1186/s12874-016-0236-x</bibtext> </blist> <blist> <bibtext> Hagiwara, M., Shogren, K. A., Lane, K. L., Raley, S. K., & Smith, S. A. (2020). Development of the self‐determined learning model of instruction coaching model: implications for research and practice. Education and Training in Autism and Developmental Disabilities, 55, 17 – 27.</bibtext> </blist> <blist> <bibtext> Hagiwara, M., Shogren, K., & Leko, M. (2017). Reviewing research on the self‐determined learning model of instruction: Mapping the terrain and charting a course to promote adoption and use. Advances in Neurodevelopmental Disorders, 1, 3 – 13. https://doi.org/10.1007/s41252-017-0007-7</bibtext> </blist> <blist> <bibtext> Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools? Exceptional Children, 79 (3), 181 – 193. https://doi.org/10.1177/001440291307900204</bibtext> </blist> <blist> <bibtext> Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999). Bayesian model averaging: A tutorial. Statistical Science, 14 (4), 382 – 417. https://<ulink href="http://www.jstor.org/stable/2676803">www.jstor.org/stable/2676803</ulink></bibtext> </blist> <blist> <bibtext> Howson, C., & Urbach, P. (2006). Scientific reasoning: The Bayesian approach (3rd ed.). Open Court.</bibtext> </blist> <blist> <bibtext> Kiblen, J. C., Shogren, K. A., Kurth, J. A., & Raley, S. K. (2023). Tiered educational coaching to promote self‐determination in the classroom. Manuscript submitted for publication.</bibtext> </blist> <blist> <bibtext> Kiblen, J. C., Shogren, K. A., Raley, S. K., Zimmerman, K. N., Mansouri, M. C., & Alsaeed, A. (in press). Review of teacher fidelity measurement of the self‐determined learning model of instruction. Education and Training in Autism and Developmental Disabilities.</bibtext> </blist> <blist> <bibtext> McElreath, R. (2016). Statistical rethinking: A Bayesian course with examples in R and Stan. CRC Press.</bibtext> </blist> <blist> <bibtext> Plake, B., & Cizek, G. J. (2012). Variations on a theme: The modified Angoff, extended Angoff, and yes/no standard setting methods. In G. J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed.). Routledge.</bibtext> </blist> <blist> <bibtext> Raley, S. K., Hagiwara, M., Burke, K. M., Kiblen, J. C., & Shogren, K. A. (in press). Supporting all students to be self‐determined: Using the self‐determined learning model of instruction within multi‐tiered systems of supports. Inclusive Practices. https://doi.org/10.1177/27324745221132478</bibtext> </blist> <blist> <bibtext> Raley, S. K., Shogren, K. A., Matusevich, H., Quirk, C., Smith, S., & O'Rourke, S. (2023). Building blocks for systems change: Using implementation science to enhance using the self‐determined learning model of instruction as a universal support. Manuscript submitted for publication.</bibtext> </blist> <blist> <bibtext> Raley, S. K., Shogren, K. A., & McDonald, A. (2018). How to implement the Self‐Determined learning model of instruction in inclusive general education classrooms. Teaching Exceptional Children, 51 (1), 62 – 71. https://doi.org/10.1177/0040059918790236</bibtext> </blist> <blist> <bibtext> Rifenbark, G. G., Hicks, T. A., Burke, K. M., Swinburne Romine, R. E., Fleming, K. K., & Shogren, K. A. (2023). Evaluation of program costs for implementing a self‐determination intervention using the ingredients method. Career Development and Transition for Exceptional Individuals, 46 (3), 133 – 141. https://doi.org/10.1177/21651434221130295</bibtext> </blist> <blist> <bibtext> Rowe, D. A., Mazzotti, V. L., Fowler, C. H., Test, D. W., Mitchell, V. J., Clark, K. A., Holzberg, D., Owens, T. L., Rusher, D., Seaman‐Tullis, R. L., Gushanas, C. M., Castle, H., Chang, W.‐H., Voggt, A., Kwiatek, S., & Dean, C. (2021). Updating the secondary transition research base: Evidence‐ and research‐based practices in functional skills. Career Development and Transition for Exceptional Individuals, 44 (1), 28 – 46. https://doi.org/10.1177/2165143420958674</bibtext> </blist> <blist> <bibtext> SAS Institute Inc. (2015). SAS/STAT® 14.1 user's guide. Author.</bibtext> </blist> <blist> <bibtext> Shogren, K. A., Burke, K. M., Anderson, M. H., Antosh, A., LaPlante, T., & Hicks, T. (2020). Examining the relationship between teacher perceptions of implementation of the SDLMI and student self‐determination outcomes. Career Development and Transition for Exceptional Individuals, 43 (1), 53 – 63. https://doi.org/10.1177/2165143419887855</bibtext> </blist> <blist> <bibtext> Shogren, K. A., Hicks, T. A., Burke, K. M., Antosh, A., LaPlante, T., & Anderson, M. H. (2020). Examining the impact of the SDLMI and whose future is it? Over a two‐year period with students with intellectual disability. American Journal on Intellectual and Developmental Disabilities, 125 (3), 217 – 229. https://doi.org/10.1352/1944-7558-125.3.217</bibtext> </blist> <blist> <bibtext> Shogren, K. A., & Raley, S. K. (2018). Self‐determined learning model of instruction (SDLMI) fidelity measure. Kansas University Center on Developmental Disabilities.</bibtext> </blist> <blist> <bibtext> Shogren, K. A., Raley, S. K., & Burke, K. M. (2019). SDLMI teacher's guide supplement: Implementing the SDLMI with the whole class. Kansas University Center on Developmental Disabilities.</bibtext> </blist> <blist> <bibtext> Shogren, K. A., Raley, S. K., Burke, K. M., & Wehmeyer, M. L. (2018). The self‐determined learning model of instruction: Teacher's guide. Kansas University Center on Developmental Disabilities.</bibtext> </blist> <blist> <bibtext> Shogren, K. A., Raley, S. K., Hicks, T. A., Lane, K. L., Gerasimova, D., & Quirk, C. (2023). Implementing the self‐determined learning model of instruction in inclusive classrooms: Impacts of teacher implementation supports on student self‐determination and goal attainment outcomes. Manuscript submitted for publication.</bibtext> </blist> <blist> <bibtext> Shogren, K. A., Raley, S. K., Rifenbark, G. G., Lane, K. L., Bojanek, E. K., Karpur, A., & Quirk, C. (2021). The self‐determined learning model of instruction: promoting implementation fidelity. Inclusion, 9 (1), 46 – 62. https://doi.org/10.1352/2326-6988-9.1.46</bibtext> </blist> <blist> <bibtext> Snyder, P. A., Hemmeter, M. L., & Fox, L. (2015). Supporting implementation of evidence‐based practices through practice‐based coaching. Topics in Early Childhood Special Education, 35 (3), 133 – 143. https://doi.org/10.1177/0271121415594925</bibtext> </blist> <blist> <bibtext> Wehmeyer, M. L., Palmer, S. B., Agran, M., Mithaug, D. E., & Martin, J. E. (2000). Promoting causal agency: The self‐determined learning model of instruction. Exceptional Children, 66 (4), 439 – 453. https://doi.org/10.1177/001440290006600401</bibtext> </blist> <blist> <bibtext> Toste, J. R., Logan, J. L., Shogren, K. A., & Boyd, B. (2023). The next generation of quality indicators for group design research in special education. Exceptional Children, 89 (4), 359 – 378. https://doi.org/10.1177/00144029221150801</bibtext> </blist> <blist> <bibtext> What Works Clearinghouse (WWC). (2017). Standards handbook (version 4.0). Institute of Education Sciences.</bibtext> </blist> <blist> <bibtext> Zieky, M. J., Perie, M., & Livingston, S. A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Services.</bibtext> </blist> </ref> <aug> <p>By Karrie A. Shogren; Jesse R. Pace; Tyler A. Hicks; Sheida K. Raley and Kathleen Lynne Lane</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib33" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib37" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib15" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib27" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib12" firstref="ref9"></nolink> <nolink nlid="nl6" bibid="bib38" firstref="ref10"></nolink> <nolink nlid="nl7" bibid="bib39" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib10" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib16" firstref="ref15"></nolink> <nolink nlid="nl10" bibid="bib13" firstref="ref16"></nolink> <nolink nlid="nl11" bibid="bib20" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib35" firstref="ref19"></nolink> <nolink nlid="nl13" bibid="bib31" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib11" firstref="ref27"></nolink> <nolink nlid="nl15" bibid="bib40" firstref="ref29"></nolink> <nolink nlid="nl16" bibid="bib29" firstref="ref34"></nolink> <nolink nlid="nl17" bibid="bib24" firstref="ref42"></nolink> <nolink nlid="nl18" bibid="bib30" firstref="ref43"></nolink> <nolink nlid="nl19" bibid="bib22" firstref="ref46"></nolink> <nolink nlid="nl20" bibid="bib34" firstref="ref49"></nolink> <nolink nlid="nl21" bibid="bib25" firstref="ref54"></nolink> <nolink nlid="nl22" bibid="bib32" firstref="ref56"></nolink> <nolink nlid="nl23" bibid="bib14" firstref="ref57"></nolink> <nolink nlid="nl24" bibid="bib17" firstref="ref61"></nolink> <nolink nlid="nl25" bibid="bib18" firstref="ref62"></nolink> <nolink nlid="nl26" bibid="bib28" firstref="ref63"></nolink> <nolink nlid="nl27" bibid="bib21" firstref="ref65"></nolink> <nolink nlid="nl28" bibid="bib23" firstref="ref69"></nolink> <nolink nlid="nl29" bibid="bib19" firstref="ref70"></nolink> <nolink nlid="nl30" bibid="bib36" firstref="ref71"></nolink> <nolink nlid="nl31" bibid="bib26" firstref="ref72"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1406601
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Setting Standards for SDLMI Fidelity: Promoting Data-Driven Decision Making to Advance Self-Determination Instruction
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Karrie+A%2E+Shogren%22">Karrie A. Shogren</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-7925-1299">0000-0001-7925-1299</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jesse+R%2E+Pace%22">Jesse R. Pace</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-3961-5718">0000-0003-3961-5718</externalLink>)<br /><searchLink fieldCode="AR" term="%22Tyler+A%2E+Hicks%22">Tyler A. Hicks</searchLink><br /><searchLink fieldCode="AR" term="%22Sheida+K%2E+Raley%22">Sheida K. Raley</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-8422-1916">0000-0001-8422-1916</externalLink>)<br /><searchLink fieldCode="AR" term="%22Kathleen+Lynne+Lane%22">Kathleen Lynne Lane</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-6364-838X">0000-0001-6364-838X</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Psychology+in+the+Schools%22"><i>Psychology in the Schools</i></searchLink>. 2024 61(2):532-552.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 21
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Cutting+Scores%22">Cutting Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Implementation%22">Program Implementation</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence+Based+Practice%22">Evidence Based Practice</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Self+Determination%22">Self Determination</searchLink><br /><searchLink fieldCode="DE" term="%22Instruction%22">Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Goal+Orientation%22">Goal Orientation</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Students+with+Disabilities%22">Students with Disabilities</searchLink><br /><searchLink fieldCode="DE" term="%22Inclusion%22">Inclusion</searchLink><br /><searchLink fieldCode="DE" term="%22Standard+Setting%22">Standard Setting</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/pits.23068
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0033-3085<br />1520-6807
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study used the standard setting to establish cutscores for the fidelity of implementation of an evidence-based intervention, the Self-Determined Learning Model of Instruction (SDLMI) designed to enhance goal-directed actions in secondary students with and without disabilities. Cutscores were then applied to fidelity data from a large, randomized trial of the SDLMI with teacher implementers. Findings suggest teachers demonstrate a range of fidelity outcomes over time across three dimensions adherence, quality of delivery, and student responsiveness. Almost all teachers (93%) immediately meet cutscores for the adherence dimension after training, but smaller numbers meet cutscores for quality of delivery (64%) and student responsiveness (69%). However, the quality of delivery and student responsiveness showed growth over time with implementation experience and there was a small effect of intensifying implementation support.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/v7azb/?view_only=038b12cff8ba4a4982b94212e559b2ab
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1406601
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1406601
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/pits.23068
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 21
        StartPage: 532
    Subjects:
      – SubjectFull: Cutting Scores
        Type: general
      – SubjectFull: Fidelity
        Type: general
      – SubjectFull: Program Implementation
        Type: general
      – SubjectFull: Evidence Based Practice
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Self Determination
        Type: general
      – SubjectFull: Instruction
        Type: general
      – SubjectFull: Goal Orientation
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Students with Disabilities
        Type: general
      – SubjectFull: Inclusion
        Type: general
      – SubjectFull: Standard Setting
        Type: general
      – SubjectFull: Models
        Type: general
    Titles:
      – TitleFull: Setting Standards for SDLMI Fidelity: Promoting Data-Driven Decision Making to Advance Self-Determination Instruction
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Karrie A. Shogren
      – PersonEntity:
          Name:
            NameFull: Jesse R. Pace
      – PersonEntity:
          Name:
            NameFull: Tyler A. Hicks
      – PersonEntity:
          Name:
            NameFull: Sheida K. Raley
      – PersonEntity:
          Name:
            NameFull: Kathleen Lynne Lane
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0033-3085
            – Type: issn-electronic
              Value: 1520-6807
          Numbering:
            – Type: volume
              Value: 61
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Psychology in the Schools
              Type: main
ResultId 1