Using an Embedded Mixed Methods Design to Assess and Improve Intervention Acceptability of an Equity-Focused Intervention: A Methodological Demonstration
Saved in:
| Title: | Using an Embedded Mixed Methods Design to Assess and Improve Intervention Acceptability of an Equity-Focused Intervention: A Methodological Demonstration |
|---|---|
| Language: | English |
| Authors: | Bastable, Eoin, Meng, Paul (ORCID |
| Source: | Behavioral Disorders. May 2023 48(3):201-211. |
| Availability: | SAGE Publications and Hammill Institute on Disabilities. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 11 |
| Publication Date: | 2023 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R324A180027 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Mixed Methods Research, Intervention, Equal Education, Discipline, Racism, Program Evaluation, Program Implementation, Faculty Development, Teacher Attitudes, Rating Scales, Fidelity, Positive Behavior Supports, Student Behavior, Barriers, Adoption (Ideas) |
| DOI: | 10.1177/0198742919880486 |
| ISSN: | 0198-7429 2163-5307 |
| Abstract: | Educators have shown reluctance to implement interventions aimed at improving racial equity in school disciplinary practice. Mixed methods were applied to assess and improve the acceptability of a new intervention designed to reduce racial disparities in school discipline. A descriptive concurrent parallel design was used to assess U.S. educators' perceptions of the acceptability of the intervention. Quantitative findings from professional development workshops introducing the intervention to 118 educators were corroborated with qualitative findings from a separate sample of 4 teachers who implemented it in their classrooms. Quantitative findings indicated that the intervention was acceptable to a broad range of potential implementers, and qualitative findings were used to modify the intervention to further improve its utility. The strengths, limitations, and implications of embedding mixed methods approaches to assess school-based interventions are also discussed. [For the corresponding grantee submission, see ED603401.] |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2023 |
| Accession Number: | EJ1376214 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHEwCvmv5WUXv7pKIJm1OgbAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDACNyfwcB1eeeA9zZwIBEICBm0nPC4PnjG9uYJXx4ExFLfDmQRNTkeYJ3TEzJICA5tIToFvf-4YlQgwS4mpgHhIR0Lx81gsW3xVZrzAV86MgVNR3BbL2oGNFGDmcJSHUvX3u9QOeGjBElbVfLArsgQDJjn0wLZYjWNRBgyptWPJH1hX3gCcfY0Oc0JgTAxN3Fh52yfNOJAm8mibhwmQuIjnLifBtGGsGSaw4O9tl Text: Availability: 1 Value: <anid>AN0163453184;bhd01may.23;2023May04.06:41;v2.2.500</anid> <title id="AN0163453184-1">Using an Embedded Mixed Methods Design to Assess and Improve Intervention Acceptability of an Equity-Focused Intervention: A Methodological Demonstration </title> <p>Educators have shown reluctance to implement interventions aimed at improving racial equity in school disciplinary practice. Mixed methods were applied to assess and improve the acceptability of a new intervention designed to reduce racial disparities in school discipline. A descriptive concurrent parallel design was used to assess U.S. educators' perceptions of the acceptability of the intervention. Quantitative findings from professional development workshops introducing the intervention to 118 educators were corroborated with qualitative findings from a separate sample of 4 teachers who implemented it in their classrooms. Quantitative findings indicated that the intervention was acceptable to a broad range of potential implementers, and qualitative findings were used to modify the intervention to further improve its utility. The strengths, limitations, and implications of embedding mixed methods approaches to assess school-based interventions are also discussed.</p> <p>Keywords: mixed methods research; school-based interventions; equity; discipline disproportionality</p> <p>A central challenge in education is not only developing potential solutions (e.g., interventions) to important problems but also assessing the extent to which potential solutions are acceptable to school personnel. Widespread evidence of racial discipline disparities in U.S. schools has underscored an urgent need for interventions that educators can implement to improve racial equity in student outcomes ([<reflink idref="bib10" id="ref1">10</reflink>]; [<reflink idref="bib37" id="ref2">37</reflink>]). However, individuals in U.S. society generally remain reluctant to address issues related to race or ethnicity ([<reflink idref="bib13" id="ref3">13</reflink>]; [<reflink idref="bib17" id="ref4">17</reflink>]). Educators, like the rest of U.S. society, are reported to be ambivalent or avoidant toward examining race and equity in schools ([<reflink idref="bib4" id="ref5">4</reflink>]; [<reflink idref="bib35" id="ref6">35</reflink>]; [<reflink idref="bib38" id="ref7">38</reflink>]). Therefore, it is necessary to assess and improve how educators perceive the effectiveness and feasibility of equity-focused school interventions. We embedded mixed methods within the study design to assess and improve the acceptability of an intervention called ReACT, developed to reduce racial disproportionality in school discipline ([<reflink idref="bib26" id="ref8">26</reflink>]).</p> <p>ReACT is a school-based professional development intervention designed to help educators improve racial equity in school discipline practice. The ReACT intervention includes the following three elements: (a) training educators to assess discipline data to detect patterns of disproportionality (e.g., assigning more discipline referrals to Black students for defiance compared with other racial/ethnic groups), (b) assisting classroom teachers to adapt their school and classroom behavior support systems to be more culturally responsive (e.g., aligning school and home behavioral expectations), and (c) training educators on implicit bias and using strategies to neutralize implicit bias in school discipline decision-making. The intervention also includes ongoing coaching provided to school teams or individual educators.</p> <p>Elements of ReACT were evaluated in two previous studies. A school case study documented reductions in discipline referrals assigned to Black students compared with White students in a K–Grade 8 setting ([<reflink idref="bib26" id="ref9">26</reflink>]). A single-case design study showed a functional relation between use of the intervention (implemented across four teachers) and increased equity in student–teacher interactions for Black students ([<reflink idref="bib16" id="ref10">16</reflink>]).</p> <hd id="AN0163453184-2">Methods of Assessing Intervention Acceptability</hd> <p>Instead of criticizing educators for implementing interventions with poor fidelity, it may be more helpful to examine an intervention's perceived acceptability by school personnel. Acceptability refers to whether potential implementers of an intervention, based on their knowledge or direct experience with the intervention, perceive it as agreeable or satisfactory ([<reflink idref="bib31" id="ref11">31</reflink>]). In school settings, acceptability is often assessed after interventions are implemented with a rating scale, instead of beforehand with an eye to improving specific aspects of a new practice.</p> <p>Acceptability is a multi-dimensional construct typically analyzed using data collected from different sources: surveys, key informant interviews, and focus groups. Mixed methods research offers a promising avenue for improving the understanding of intervention acceptability by assessing interventions from different vantage points (e.g., quantitative surveys of large groups and qualitative interviews with individual implementers). Increasingly, mixed methods are used to assess intervention acceptability by helping identify barriers and facilitators to implementation, or as a tool to refine interventions ([<reflink idref="bib30" id="ref12">30</reflink>]). For example, [<reflink idref="bib1" id="ref13">1</reflink>] compared findings from analyses of qualitative and quantitative data to evaluate the acceptability of a leadership intervention to help staff implement evidence-based practices in mental health service agencies.</p> <p>Due to the complex nature of improving racial equity in school discipline practice, scholars have called for more integrated methodological approaches to identify potential solutions ([<reflink idref="bib10" id="ref14">10</reflink>]; [<reflink idref="bib22" id="ref15">22</reflink>]). Specifically, analytic approaches that utilize both statistical analyses of quantitative data and in-depth qualitative interviews have been proposed ([<reflink idref="bib36" id="ref16">36</reflink>]). Increasingly, methodological approaches from the field of Implementation Science (e.g., hybrid, step-wedge designs) have been considered to assess the effectiveness and utility of interventions developed for schools ([<reflink idref="bib24" id="ref17">24</reflink>]; [<reflink idref="bib25" id="ref18">25</reflink>]). More flexible methodological approaches are needed to enhance the effectiveness and acceptability of school-based interventions aimed at reducing discipline disparities.</p> <p>It is important to gain a more robust understanding of educators' willingness to adopt equity-focused approaches that may cause discomfort or raise defensiveness among school personnel (e.g., challenging educators' pre-conceived notions of fairness, equity, or neutrality). Understanding the perspectives of school personnel throughout the design, implementation, and evaluation stages of intervention development may contribute to the generalizability and usability equity-focused practices ([<reflink idref="bib36" id="ref19">36</reflink>]). Given the benefits of using mixed methods to contribute to a broader line of research on discipline disproportionality, it is surprising that few studies have applied this integrative methodology ([<reflink idref="bib14" id="ref20">14</reflink>]; [<reflink idref="bib20" id="ref21">20</reflink>]).</p> <hd id="AN0163453184-3">Purpose</hd> <p>The purpose of this article is to demonstrate the use of mixed methods to assess and enhance the acceptability of a school-wide professional development intervention to reduce discipline disproportionality. Specifically, we integrated quantitative and qualitative data to evaluate an equity-focused intervention (i.e., ReACT). We delivered an overview of the intervention during a series of full-day workshops, then used a validated intervention-acceptability measure to identify overall acceptability among a sample of educators and tested for any differences in acceptability by race/ethnicity, gender, and U.S. geographic region of workshop attendees. In addition to the quantitative analyses, we used with a pragmatic interview approach to obtain rich information from teachers who actually implemented the intervention in their classrooms. Next, we used a mixed methods (i.e., concurrent parallel design) to analyze the extent to which results obtained regarding acceptability were consistent across workshop participants who were just learning about the intervention, which we corroborated with reports of teachers who had implemented ReACT in a school setting (i.e., classrooms). Specifically, we asked the following research questions:</p> <p></p> <ulist> <item> RQ1. To what extent do educators rate the intervention as acceptable and feasible, and do ratings vary by (a) educator characteristics and (b) experience actually implementing it (quantitative data)?</item> <p></p> <item> RQ2. What variables do classroom teachers identify as enablers and barriers to implementation of the intervention (qualitative data)?</item> </ulist> <hd id="AN0163453184-4">Method</hd> <p></p> <hd id="AN0163453184-5">Mixed Methods Approach</hd> <p>We used a descriptive concurrent parallel design ([<reflink idref="bib11" id="ref22">11</reflink>]) to analyze the quantitative and qualitative data collected for this study. Utilizing this approach, we determined the study elements at the outset, then collected quantitative and qualitative data in a parallel manner, but analyzed them independently. Next, quantitative and qualitative results were integrated to support an overall interpretation of how school personnel perceived the acceptability of the ReACT intervention.</p> <p>There are compelling reasons for using mixed methods during intervention development. First, analyzing both quantitative and qualitative results independently may provide a more robust understanding of acceptability. Second, qualitative data were viewed as helping to corroborate quantitative results by providing additional information on the research topic based on participants' own words and experiences implementing the intervention. Third, mixing methods allowed for comparing quantitative and qualitative data to increase the legitimacy (i.e., validity) of the overall study's findings ([<reflink idref="bib39" id="ref23">39</reflink>]).</p> <hd id="AN0163453184-6">Settings and Participants</hd> <p></p> <hd id="AN0163453184-7">Quantitative strand: professional development workshop attendees</hd> <p>Participants for the first research question were a convenience sample of educators and administrators from three U.S. states (one in the Midwest, one in the Northeast, and one in the South) who elected to participate in 1-day professional development workshops focusing on ReACT and its elements, delivered in 2017. Sites were selected as a deliberate sample to provide geographic diversity in attendees to assess group differences in acceptability by U.S. region. Of the 181 attendees across all three sites, 118 (65%) consented to participate in the study. According to self-report, participants were working in or supporting schools that were implementing school-wide positive behavioral interventions and supports (SWPBIS) with high fidelity (26%), some fidelity (42%), or not at all (10%). The remaining 21% either did not respond or indicated that this item did not apply to them. See Table 1 for descriptive statistics.</p> <p>Graph</p> <p>Table 1. Descriptive Statistics for Workshop Participants (n = 118).</p> <p> <ephtml> &lt;table&gt;&lt;colgroup&gt;&lt;col align="left" /&gt;&lt;col align="char" char="." /&gt;&lt;col align="char" char="." /&gt;&lt;/colgroup&gt;&lt;thead&gt;&lt;tr&gt;&lt;th align="center"&gt;Participant characteristics&lt;/th&gt;&lt;th align="center"&gt;&lt;italic&gt;n&lt;/italic&gt; (%)&lt;/th&gt;&lt;th align="center"&gt;Acceptability &lt;italic&gt;M&lt;/italic&gt; (&lt;italic&gt;SD&lt;/italic&gt;)&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td colspan="3"&gt;Gender&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Male&lt;/td&gt;&lt;td&gt;24 (20)&lt;/td&gt;&lt;td&gt;5.36 (0.54)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Female&lt;/td&gt;&lt;td&gt;87 (74)&lt;/td&gt;&lt;td&gt;5.19 (0.47)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td colspan="3"&gt;Race/ethnicity&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Black&lt;/td&gt;&lt;td&gt;26 (22)&lt;/td&gt;&lt;td&gt;5.29 (0.47)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; White&lt;/td&gt;&lt;td&gt;79 (67)&lt;/td&gt;&lt;td&gt;5.42 (0.49)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Others&lt;/td&gt;&lt;td&gt;13 (11)&lt;/td&gt;&lt;td&gt;5.18 (0.40)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td colspan="3"&gt;Region&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Midwest&lt;/td&gt;&lt;td&gt;30 (25)&lt;/td&gt;&lt;td&gt;5.15 (0.50)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; South&lt;/td&gt;&lt;td&gt;56 (47)&lt;/td&gt;&lt;td&gt;5.22 (0.48)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Northeast&lt;/td&gt;&lt;td&gt;32 (27)&lt;/td&gt;&lt;td&gt;5.27 (0.48)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td colspan="3"&gt;Role&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Administrator&lt;/td&gt;&lt;td&gt;23 (19)&lt;/td&gt;&lt;td&gt;5.20 (0.50)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Teacher&lt;/td&gt;&lt;td&gt;15 (13)&lt;/td&gt;&lt;td&gt;5.17 (0.56)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Student support&lt;/td&gt;&lt;td&gt;30 (25)&lt;/td&gt;&lt;td&gt;5.22 (0.42)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Other (e.g., coach)&lt;/td&gt;&lt;td&gt;25 (21)&lt;/td&gt;&lt;td&gt;5.33 (0.52)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; No response&lt;/td&gt;&lt;td&gt;17 (14)&lt;/td&gt;&lt;td&gt;5.11 (0.46)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td colspan="3"&gt;Years in education&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; 0&amp;#8211;5&lt;/td&gt;&lt;td&gt;14 (12)&lt;/td&gt;&lt;td&gt;5.19 (0.37)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; 6&amp;#8211;10&lt;/td&gt;&lt;td&gt;15 (13)&lt;/td&gt;&lt;td&gt;5.17 (0.53)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; 11&amp;#8211;20&lt;/td&gt;&lt;td&gt;49 (42)&lt;/td&gt;&lt;td&gt;5.27 (0.46)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; &amp;#62;20&lt;/td&gt;&lt;td&gt;33 (28)&lt;/td&gt;&lt;td&gt;5.20 (0.56)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td colspan="3"&gt;Fidelity of SWPBIS (self-report)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Fidelity&lt;/td&gt;&lt;td&gt;31 (26)&lt;/td&gt;&lt;td&gt;5.25 (0.53)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; Partial&lt;/td&gt;&lt;td&gt;48 (41)&lt;/td&gt;&lt;td&gt;5.15 (0.47)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt; No SWPBIS&lt;/td&gt;&lt;td&gt;12 (10)&lt;/td&gt;&lt;td&gt;5.09 (0.45)&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>1 <emph>Note.</emph> Acceptability was measured using the Primary Intervention Rating Scale ([<reflink idref="bib23" id="ref24">23</reflink>]). SWPBIS = school-wide positive behavioral interventions and supports.</p> <hd id="AN0163453184-8">Qualitative strand: classroom teachers</hd> <p>We deliberately conducted qualitative interviews with classroom teachers who had first-hand experiences implementing the intervention. We used the data collected from the qualitative interviews as an independent credibility check for the quantitative survey data. The sample was obtained by asking administrators in two schools in a large urban school district to identify individual teachers who required additional support in equitable classroom behavior support. Although the teachers agreed to participate in the study, they did not seek out the training as the workshop sample did. In addition, they actually implemented the intervention (with adequate fidelity) before rating its acceptability, instead of simply learning about it. These participants were four general education teachers who were coached and implemented the ReACT intervention in their classrooms in an elementary and a K-8 school in the Pacific Northwest in 2018. The teachers were interviewed about their experiences after approximately 1 month of implementing the ReACT intervention. Teachers identified as White, Non-Hispanic (<emph>n</emph> = 2), White, Hispanic (<emph>n</emph> = 1), and Asian/Pacific Islander (<emph>n</emph> = 1); and 75% were female (<emph>n</emph> = 3). Teachers reported working in the field of education for an average of 6.5 years (from 1 to 17 years).</p> <hd id="AN0163453184-9">Measures</hd> <p></p> <hd id="AN0163453184-10">Acceptability</hd> <p>To assess social validity for the workshop attendees and four classroom teachers who implemented the intervention, we used the <emph>Primary Intervention Rating Scale</emph> (PIRS; [<reflink idref="bib23" id="ref25">23</reflink>]). The PIRS is a 17-item measure of overall intervention acceptability for school-wide behavior support interventions. The PIRS has been validated in the context of implementation research in school settings (α =.97 for elementary level). PIRS survey items are rated on a 6-point Likert-type scale ranging from 1 (<emph>strongly disagree</emph>) to 6 (<emph>strongly agree</emph>). PIRS survey items assess different aspects of acceptability of school-based interventions (e.g., item 7: "<emph>I would be willing to use this intervention in the school setting</emph>").</p> <hd id="AN0163453184-11">Teacher interview protocol</hd> <p>To obtain rich descriptions of the four classroom teachers' implementation experiences, we used a semi-structured interview protocol and format (available from the first author) based on the Critical Incident Technique (CIT; [<reflink idref="bib9" id="ref26">9</reflink>]; [<reflink idref="bib15" id="ref27">15</reflink>]). CIT was used to identify categories that either enabled or hindered implementation of the ReACT intervention by teachers in their classrooms. The CIT method has proven to be especially useful for interpreting how incidents or experiences described by practitioners can inform how to improve current practices or policies ([<reflink idref="bib8" id="ref28">8</reflink>]; [<reflink idref="bib15" id="ref29">15</reflink>]). We used this protocol to elicit responses from each participant during one-on-one phone interviews to collect specific, observable, and replicable incidents (critical incidents [CIs]; [<reflink idref="bib15" id="ref30">15</reflink>]) to address the second research question. [<reflink idref="bib15" id="ref31">15</reflink>] stated that interviews should continue until exhaustiveness or redundancy in data occurs (i.e., the point at which participants mention no new CIs and no new categories are needed to describe incidents).</p> <p>We asked participants to discuss what helped and hindered their implementation of the intervention in their classrooms adapting a common CIT interview format ([<reflink idref="bib9" id="ref32">9</reflink>]). Specifically, we analyzed responses to the following two questions: (a) <emph>What were the important events (i.e., specific behaviors, examples, or observable happenings) that helped you to implement the ReACT intervention in your classroom or school?</emph> (b) <emph>What were the important events (i.e., specific behaviors, examples, or observable happenings) that hindered the use of the ReACT intervention in your classroom or school?</emph></p> <hd id="AN0163453184-12">Procedure</hd> <p>All study documents received approval from the University of Oregon Human Subjects Institutional Review Board. Recruitment for the workshop attendee participants took place at the start of each 1-day (i.e., 6 hr) professional development workshop provided by the fourth author. These workshops described ReACT and provided training and practice on the three intervention components (i.e., analyzing disaggregated discipline data, culturally adapting behavior support practices, and training on understanding and neutralizing the effects of implicit bias on disciplinary decisions). We offered attendees the opportunity to participate at the start of the workshop with provision of the surveys in a hard copy packet and a description of the study. Participants completed the acceptability survey at the end of the workshop. The classroom teacher interviewees were recruited as part of their participation in a small-scale trial of the intervention ([<reflink idref="bib16" id="ref33">16</reflink>]). Each teacher implemented the intervention with fidelity, as measured through direct observation. All four teachers implementing the intervention participated in the interviews. PIRS administration and interviews began 1 week after the intervention trial concluded. We recorded and transcribed the four participants' responses to ensure that data were collected verbatim, as recommended from prior CIT studies ([<reflink idref="bib3" id="ref34">3</reflink>]; [<reflink idref="bib9" id="ref35">9</reflink>]). Interviews lasted 45 to 75 minutes and were conducted over the phone with teachers after school hours.</p> <hd id="AN0163453184-13">Analytic Plan</hd> <p></p> <hd id="AN0163453184-14">Quantitative analyses</hd> <p>We first computed mean scores for PIRS rating scores to determine the overall ratings of acceptability and willingness across all workshop participants. We then analyzed survey responses by participant characteristics to determine the extent to which acceptability was consistent across demographic characteristics, including (a) gender, (b) race/ethnicity, (c) U.S. region, (d) educational role, (e) years in education (median split), and (f) perceived fidelity of implementation of SWPBIS (high, moderate, or none). For each characteristic, we conducted a separate one-way analysis of variance (ANOVA) to assess differences in acceptability. We used a Bonferroni-corrected α level for significance testing (α =.01) to account for family-wise error ([<reflink idref="bib21" id="ref36">21</reflink>]). We checked the data for the standard ANOVA assumptions and found no violations.</p> <hd id="AN0163453184-15">Qualitative analyses</hd> <p>We adhered to steps described by [<reflink idref="bib9" id="ref37">9</reflink>] using CIT procedures to analyze participant interviews. First, we extracted CIs from four interview transcripts (conducted with four educators) that were associated with the "frame of reference" (i.e., what helped or hindered implementation of the ReACT intervention) for the study ([<reflink idref="bib15" id="ref38">15</reflink>]). Next, we identified patterns, themes, and differences among CIs to formulate categories with headings to summarize major themes. We then reviewed and coded the interview transcripts to determine the fit of the additional CIs into categories. Butterfield ([<reflink idref="bib9" id="ref39">9</reflink>]) recommended 25% as a minimum participation rate needed to form a viable category (i.e., a category should be noted by at least 25% of the participants). The threshold participant rate used for this study was 50% (i.e., two participants). If the threshold of 50% was not met for a proposed category, we considered combining smaller categories with those already formed ([<reflink idref="bib9" id="ref40">9</reflink>]).</p> <hd id="AN0163453184-16">Credibility checks</hd> <p>When all CIs were reviewed, coded, and placed in operationally defined categories, we initiated a series of credibility checks to determine trustworthiness, as used in other CIT studies ([<reflink idref="bib4" id="ref41">4</reflink>]; [<reflink idref="bib27" id="ref42">27</reflink>]). Our credibility checks served as important quality indicators of this type of qualitative approach ([<reflink idref="bib7" id="ref43">7</reflink>]). Credibility checks included: (a) recording and transcribing all interviews for accuracy, (b) submitting one interview for independent review to ensure that the protocol was followed, (c) establishing intercoder reliability in the extraction of CIs and categories formed, (d) submitting categories to expert review (e.g., <emph>did you find the categories to be useful?</emph>), and (e) evaluating the categories for theoretical agreement.</p> <hd id="AN0163453184-17">Critical incidents extraction check</hd> <p>We recruited and trained an independent reviewer with a doctorate in Special Education to extract CIs from one randomly selected interview transcript. The independent CI extraction was compared with the extraction conducted by a member of our research team. Intercoder agreement (ICA) was calculated by dividing the total number of CIs extracted by the total number of unique CIs identified across both extractions. The percentage of agreement was 100% for CIs extracted.</p> <hd id="AN0163453184-18">Category coding check</hd> <p>We chose at random 25% of the CIs and asked a member of our research team (who did not conduct the interviews) to review category headings and operational definitions. For this credibility check, the reviewer was asked to match headings with operational definitions provided in an electronic PowerPoint file. We used [<reflink idref="bib2" id="ref44">2</reflink>] recommended criterion of 80% agreement or higher as benchmark for reliability. Initial ICA was 85%. With feedback from the reviewer, one category title was modified (Consistent Use of Praise was changed to Inconsistent Use of Praise) and one category definition augmented (we added coaching on alternative classroom behaviors to Coaching on Positive Behavioral Strategies) to improve the overall fit of category title and category descriptions. After these modifications, ICA was raised to 100% for the categories formed.</p> <hd id="AN0163453184-19">Expert check</hd> <p>We recruited two experts from the field of education, scholars versed in the study's topic area and aware of current practices used to address racial equity in school settings. We asked the experts to review the final category titles and definitions and respond to a set of questions about whether they found the categories appropriate, surprising, or useful ([<reflink idref="bib15" id="ref45">15</reflink>]). The experts were asked the following questions: (a) <emph>Do you find the categories to be useful?</emph> (b) <emph>Are you surprised by any of the categories?</emph> (c) <emph>Do you think there is anything missing based on your experience?</emph> The experts agreed that all the categories generated were useful and relevant. One expert suggested some slight wording changes for category definitions. Overall, the experts did not report that anything was missing or surprising from a review of the category headings and definitions.</p> <hd id="AN0163453184-20">Mixed methods data analysis</hd> <p>Following completion of the quantitative and qualitative analyses, a descriptive concurrent parallel design was used to assess the extent to which the quantitative survey results corroborated the findings of a qualitative study using structured interviews with classroom teachers who actually implemented the intervention. The analytic approach was concurrent because all measures and methods were determined before both survey and interview data collection took place. Furthermore, integration only occurred at the conclusion of the study ([<reflink idref="bib39" id="ref46">39</reflink>]). Findings (i.e., categories) that emerged from the qualitative study were interpreted based on results from the quantitative study.</p> <hd id="AN0163453184-21">Findings</hd> <p></p> <hd id="AN0163453184-22">Quantitative Results</hd> <p>Of the 118 respondents, 105 (89%) completed some portion of the PIRS rating scale. Complete PIRS rating data were available for 95 (81%) of the workshop participants and the four implementing teachers. Cronbach's alpha for PIRS from the sample of workshop attendees (α =.92) was excellent. Due to the constraints presented by the sample size, we used mean substitution to generate PIRS scores for analyses. Each participant's PIRS average was interpreted as an index of overall intervention acceptability.</p> <p>Overall, workshop attendees provided consistently high ratings for intervention acceptability. The mean rating for the PIRS scale was 5.23 on a scale of 1 to 6, between "Agree" and "Strongly Agree." These results indicated that the intervention and its components were regarded as both acceptable and feasible to the workshop participants surveyed. As seen in the right columns of Table 1, mean values by participant characteristics for acceptability were similar across participant characteristics, with all subgroup mean values above 5.0, and there were no statistically significant differences by participant group (all <emph>p</emph>-values above the family-wise α value of.01). In other words, the intervention was rated as highly acceptable to workshop attendees, regardless of the individual characteristics we assessed.</p> <p>To corroborate these findings and present a check against error introduced by ratings of acceptability without actually implementing the intervention, we compared these scores with the PIRS ratings from the four classroom teachers who implemented the intervention with fidelity. This group also provided consistently high ratings, with a mean PIRS score of 5.32 and no responses for any items below "slightly agree." These results were congruent across samples and support a finding of high ratings of intervention acceptability across individual characteristics and experience implementing the intervention.</p> <hd id="AN0163453184-23">Qualitative Findings</hd> <p>The results of the CIT analysis identified four Helping and four Hindering categories. We used an iterative process that included multiple revisions to operationalize definitions and titles (i.e., adding, dropping, and modifying categories titles and definitions to fit CIs into categories). The final count included eight categories, encompassing 34 CIs. Table 2 displays the final categories sorted by Helping and Hindering CIs. The table includes the category title, total number of CIs, and the representation rate (percentage of participants that endorsed categories/total number of participants). The table is ordered hierarchically, largest to smallest, by participant representation.</p> <p>Graph</p> <p>Table 2. Categories Reported by Classroom Teachers (n = 4) Implementing the ReACT Intervention.</p> <p> <ephtml> &lt;table&gt;&lt;colgroup&gt;&lt;col align="left" /&gt;&lt;col align="char" char="." /&gt;&lt;col align="char" char="." /&gt;&lt;/colgroup&gt;&lt;thead&gt;&lt;tr&gt;&lt;th align="center"&gt;Type of critical incident&lt;/th&gt;&lt;th align="center"&gt;Categories&lt;/th&gt;&lt;th align="center"&gt;Number of CIs (% of teachers reporting)&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Helping&lt;/td&gt;&lt;td&gt;1. Receiving Feedback on Use of Praise and Corrections2. Coaching on Positive Behavioral Strategies3. Defining and Offering Examples of Praise4. Conducting a Student Preference Assessment&lt;/td&gt;&lt;td&gt;7 (75)5 (75)3 (50)2 (50)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Hindering&lt;/td&gt;&lt;td&gt;1. Inconsistent Use of Praise2. Lacking Personal Capacity to Implement3. Lacking Alignment with Existing Classroom Practices or Teaching Philosophy4. Competing School Priorities/Tasks&lt;/td&gt;&lt;td&gt;8 (100)5 (75)3 (50)3 (50)&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>2 <emph>Note.</emph> A 50% participation rate (&gt; two participants) was the minimal level acceptable for category formation. CI = critical incident.</p> <hd id="AN0163453184-24">Helping incidents</hd> <p>CIs that participants described as enabling implementation were coded into four helping categories: Receiving Feedback on Use of Praise and Corrections, Coaching on Positive Behavioral Strategies, Defining and Offering Examples of Praise, and Conducting a Student Preference Assessment.</p> <hd id="AN0163453184-25">Receiving feedback on use of praise and corrections</hd> <p>Participants reported receiving feedback helped to increase praise and decrease corrections delivered to students during classroom lessons. A coach provided verbal feedback and data reports (e.g., graphed data, counts, rates, and ratio) on the type and amount of praise or corrections observed by race of the students. Participants discussed the benefits of receiving immediate feedback on observed use of praise and corrections during classroom instruction. Participant 4 reported:</p> <p>It was good to have [the coach] observing me and then sending me, you know, my little feedback every night, to kind of read through and see... because you're teaching, and you don't always totally know what you're doing or saying because you're just doing it.</p> <p>Activities in this category included monitoring rates of praise and corrections delivered, examining trends on use of praise/corrections by race of student, and educators adjusting use of praise based on the coach's feedback. Participants described valuing simple and interpretable data to track their classroom performance.</p> <hd id="AN0163453184-26">Coaching on positive behavioral strategies</hd> <p>This category refers to receiving coaching to support use of positive behavioral strategies in classrooms. Activities included meeting with a mentor outside of class to share ideas about teaching practices and receiving guidance to manage student classroom behaviors (e.g., reminders to restate school rules in positive, concise language). Participant 2 appreciated receiving coaching on implementing positive behavioral strategies: "I thought the check in after about a week was also beneficial to just kind of talk things through and how things were going with the [classroom] strategies. So that was just me and the coach."</p> <hd id="AN0163453184-27">Defining and offering examples of praise</hd> <p>This category refers to describing and providing examples of what use of praise can look and sound like in classroom settings. Activities in this category include providing specific examples of praise statements, operationalizing use of praise, and providing rationales for delivering praise under different classroom conditions (individual vs. whole-class instruction). For example, Participant 1 reported on the benefits of differentiating how she used praise with her students:</p> <p>Well, I think at the beginning I was kind of confused about like, what kind of praise I was supposed to be giving, so I asked [the coach] when we sat down together... he said, there are different kinds of praise, [one type] which is good for relational rapport type praise, but he was more focusing on, you know, the behavior and performance praise. So, I thought that was useful.</p> <hd id="AN0163453184-28">Conducting a student preference assessment</hd> <p>This category refers to a strategy used in ReACT to assess what types of acknowledgment (e.g., public verbal praise and acknowledgment ticket) students prefer or not prefer to receive for positive behaviors displayed in the classroom. Participants reported on the benefits of understanding what acknowledgment their students desired:</p> <p>I thought they were all going say the goldfish or even the "wow" ticket. Many of them wanted just like, verbal praise and that's not something that I think that I would have guessed, you know, so that was super helpful. (Participant 3)</p> <p>Activities in this category include using paper questionnaires, group discussions, and individual interviews to collect information on students' prior classroom experiences with praise and specific preferences for being recognized by educators for meeting classroom expectations.</p> <hd id="AN0163453184-29">Hindering incidents</hd> <p>Participants described four hindering categories that impeded implementation of the ReACT intervention to fidelity: Inconsistent Use of Praise, Lacking Personal Capacity to Implement, Lacking Alignment with Existing Classroom Practices or Teaching Philosophy, and Competing School Priorities or Tasks.</p> <hd id="AN0163453184-30">Inconsistent use of praise</hd> <p>This category refers to participants reporting their irregular use of praise as hindering implementation of the ReACT intervention to fidelity. Behaviors included using praise sparsely (even when coached to offer more praise), attending more to negative behaviors of students (instead of intentionally ignoring), experiencing pressure or fatigue when asked to increase rates of praise, and expressing doubt whether increasing rates of praise would positively influence student behaviors.</p> <p>Participant 1 commented on the challenge of delivering more praise to students during classroom instruction: "I feel like I had to bounce around to get the numbers of praise in and to sit with some who struggle with, you know, some of the writing assignments to get them started." Participant 3 reported, "I've been struggling with correcting students versus like, focusing on the positive or like what [the coach] had said about restating the rule. And so that was really challenging."</p> <hd id="AN0163453184-31">Lacking personal capacity to implement to fidelity</hd> <p>This category refers to participants' lacking confidence or the ability to implement the intervention to fidelity. Within this category participants questioned whether they could accurately self-monitor their rates of praise and corrections or self-assess how equitably they were delivering classroom supports to students without external coaching (e.g., a colleague observing). Participant 1 related obstacles experienced while implementing the intervention representative of this category:</p> <p>I didn't have time to manage in my head how many times I've called on this White person once and this Black person, you know? So, I guess maybe the feedback was helpful, but it was hard like, I can't say I was managing it in my head and trying to make it all come out even.</p> <hd id="AN0163453184-32">Lack of alignment with existing classroom practices or teaching philosophy</hd> <p>This category refers to not implementing the intervention to fidelity due to perceived lack of fit with existing classroom activities or participants' classroom management systems. Activities in this category include discounting requests to increase praise based on personal reasons (e.g., "I'm a positive person"), raising concerns about how to taper/reduce high rates of praise to address unwanted behaviors, and difficulty providing praise due to pedagogical approaches. For example, Participant 1 remarked, "some of the lessons or activities that I had planned didn't allow for so much praise."</p> <p>Participants also raised concerns that implementing the intervention did not always feel authentic or aligned with their personal teaching approach. Participant 2 described this hindrance, "I feel like I'm being, unreal with, you know, like it's too much positivity, and... sometimes I feel like it's almost too much and for me... I give my students praise [that] is very individualized." Participant 4 also shared concerns about adapting and sustaining use of the intervention in her classroom:</p> <p>I didn't want to get stuck on, you know, praising students for being on task for weeks and weeks and weeks. It's something that I wanted to move on from, um, but I don't know how that this strategy would have allowed me to do that.</p> <hd id="AN0163453184-33">Competing school priorities or tasks</hd> <p>This category refers to school events and classroom duties that interfered with implementing the intervention to fidelity (e.g., class activities, testing). Activities include meeting with the coach outside of class time and struggling to use the intervention within the class schedule. Participant 4 described how her classroom schedule was viewed as a barrier to implementing the intervention:</p> <p>It was like, during our math time and it was kindergarten... with any grade level there's so many things to do that sometimes when he [the coach] was coming [to observe the class] I'm like, I'm so sorry, but today we have this special lesson coming... we're not doing [the intervention] that today, you know?</p> <hd id="AN0163453184-34">Integration of Results</hd> <p>We next integrated the quantitative and qualitative stands to further assess intervention acceptability. Survey data (PIRS ratings) were corroborated with interview data collected from the four teachers who actually implemented the ReACT intervention with fidelity in their classrooms. Ratings of acceptability were uniformly high across all participant demographic groups. Those who actually implemented the intervention also rated it highly and identified specific incidents that helped or hindered their implementation.</p> <p>The qualitative findings provided an additional set of rich information indicating which components of the ReACT were perceived by classroom teachers as helping them to implement the intervention to fidelity (e.g., defining and teaching expectations, personalizing classroom acknowledgements, viewing disaggregated data, and classroom coaching). Qualitative findings also helped to identify barriers not described in the PIRS data, which showed uniformly high ratings across all items (including all four teachers strongly agreeing that they "would be willing to use this intervention in the school setting"). The four teachers were asked to describe barriers perceived as obstacles to implementing the intervention to fidelity. These barriers included having to provide high rates of praise to students, lacking personal capacity or confidence to implement the intervention to fidelity (without external support), and balancing competing school priorities (other duties assigned as teachers).</p> <hd id="AN0163453184-35">Discussion</hd> <p>Discipline disproportionality remains a vexing and costly issue affecting schools and students, without clear evidence-based practices to solve it. In addition, educators' reluctance to acknowledge and address racial school discipline disparities presents another obstacle to implementing viable solutions. Hence, it can be valuable to assess the acceptability and feasibility of equity interventions through a range of methodological approaches. We used mixed methods to obtain data from multiple participant groups and perspectives. Findings indicated high ratings of acceptability across all participant demographic groups, including a diverse sample of workshop participants and teachers who actually implemented the intervention. Moreover, implementing teachers identified specific incidents that helped or hindered their implementation, which we used in our efforts to improve the intervention.</p> <hd id="AN0163453184-36">Interpretation and Application of Primary Findings</hd> <p></p> <hd id="AN0163453184-37">Intervention acceptability</hd> <p>Embedding mixed methods into the study design allowed for a more robust analysis of how school personnel perceived the acceptability of the ReACT intervention shared in workshops or implemented in classrooms. It was encouraging to see that intervention acceptability, as measured by a validated quantitative measure, was strong (mean values above 5 on a scale of 1 to 6) for all workshop groups. There were no significant differences in ratings by gender, race, U.S. region, role, or years in education. Given the reluctance of some educators to implement equity interventions ([<reflink idref="bib4" id="ref47">4</reflink>]; [<reflink idref="bib13" id="ref48">13</reflink>]) and regional variations in perspectives regarding racial disproportionality ([<reflink idref="bib34" id="ref49">34</reflink>]), the strong acceptability indicates promise for ReACT. Moreover, the four teachers who implemented the intervention (and completed the PIRS after implementation) had mean scores slightly higher than those who only heard about it. This congruence across samples allows for stronger trustworthiness of the findings, albeit with the possibility of inflated scores in all samples due to social desirability bias.</p> <hd id="AN0163453184-38">Implementation enablers and barriers</hd> <p>The qualitative strand of the study yielded thick descriptions from the experiences from teachers implementing the intervention, beyond simple ratings of social validity. Qualitative data added a level of detail that provided useful information for improving components of the intervention. For example, individual teachers found the classroom coaching and feedback to be indispensable for supporting implementation of the intervention. This finding aligns with the existing research on the utility of individual coaching and performance feedback delivered to classroom teachers ([<reflink idref="bib6" id="ref50">6</reflink>]; [<reflink idref="bib19" id="ref51">19</reflink>]; [<reflink idref="bib32" id="ref52">32</reflink>]). Such results are heartening because they point to individual coaching as a key enabler. However, the findings are also somewhat discouraging because individual classroom coaching is costly (in terms of resources required) and thus is rarely provided by coaches in practice ([<reflink idref="bib5" id="ref53">5</reflink>]).</p> <p>Integration of qualitative data provided more detailed information on perceived enablers and barriers that could be used to enhance overall acceptability of the ReACT intervention. Enabling factors included implementation supports that are common to many school interventions (e.g., coaching, direct teaching with examples, performance feedback; [<reflink idref="bib33" id="ref54">33</reflink>]). Likewise, barriers such as competing initiatives or lack of resources are common concerns for school personnel ([<reflink idref="bib28" id="ref55">28</reflink>]). Interestingly, participants identified consistent use of behavior-specific praise across the school day as a challenge to implement ReACT to fidelity. Participants described skepticism (e.g., more praise may not improve behaviors) and identified barriers (e.g., finding enough opportunities to deliver praise during lessons) that indicated additional strategies or coaching may be needed to ensure that praise is delivered equitably by educators implementing ReACT in classrooms.</p> <hd id="AN0163453184-39">Intervention improvement</hd> <p>Although the quantitative data indicated that the intervention was acceptable to a diverse group of implementers, we found the interview results to be valuable in refining the intervention in an effort to make it more likely to be implemented with fidelity. For example, relying only on changing teacher practices without attending to the systems and contexts that encourage those practices places all of the responsibility for success on the individual teacher. Instead, given the participants' positive experience with coaching (e.g., Coaching on Positive Behavioral Strategies), we will emphasize coaching support (e.g., dedicating resources to classroom coaching) to capitalize on this helping variable ([<reflink idref="bib27" id="ref56">27</reflink>]). In addition, although increasing behavior-specific praise is an effective and acceptable approach (e.g., Defining and Offering Examples of Praise) for creating a positive classroom environment, we have emphasized additional, less-intensive strategies to build positive student–teacher relationships to complement the focus on increasing praise rates (e.g., greeting students at the door, micro-affirmations, student strengths, and interests surveys).</p> <hd id="AN0163453184-40">Limitations and Strengths</hd> <p>This study provided an opportunity to evaluate the benefits and limitations of embedding a mixed methods design within a larger intervention development project. Although there are benefits to applying mixed methods approaches to explore this topic area, there are also limitations. In fact, mixed methods projects are often subject to a larger set of limitations because they may be judged against quality standards of multiple research methodology traditions.</p> <p>The high acceptability ratings in the survey component of the study may have been affected by social desirability bias, in which respondents felt compelled to provide higher ratings than they might otherwise, or they may provide high ratings because providing one in the context of a workshop is easier than actually implementing with fidelity. This limitation was mitigated to some extent in that the classroom teachers provided similar acceptability ratings and implemented the intervention with high fidelity.</p> <p>The samples used for this study were non-random (i.e., convenience, purposeful) and therefore did not represent typical school personnel. The workshop participants were likely already supportive of equity-focused approaches based on their attendance at the workshops (and willingness to complete a survey). Consequently, inferences and generalizability of the results of this study are limited only to the educators sampled. However, because workshop participants and classroom teachers both viewed components of the ReACT intervention favorably, the meta-inference quality was likely higher than if the results has been contradictory ([<reflink idref="bib29" id="ref57">29</reflink>]). There was also a wide discrepancy in the two sample sizes used in the quantitative and qualitative strands of the study (118 vs. 4). [<reflink idref="bib29" id="ref58">29</reflink>] noted that sampling differences can make integration in mixed methods studies challenging and can threaten the validity or credibility of results generated. Sampling discrepancies can also affect the quality of meta-inferences drawn from the data gathered.</p> <p>These limitations also provide an opportunity to reflect on elements of the study that we believe could be improved if we were to replicate it in another project. One key issue is that the sample, although racially and ethnically diverse, was limited to educators and administrators. We could widen the sample to include a broader range of stakeholders, including students and family members. Specifically, students could be recruited to describe specific intervention strategies that helped or hindered the development of positive student–teacher relationships. Likewise, family members could describe experiences related to family–school partnerships.</p> <p>To address threats to internal validity in mixed methods designs, the use of CIT may be advantageous. A feature of CIT is reaching exhaustiveness when analyzing interview data gathered from study participants. Exhaustiveness is defined as the point at which participants mention no new incidents, or no new categories emerged or are needed to describe CIs ([<reflink idref="bib9" id="ref59">9</reflink>], p. 270). Exhaustiveness has been described as a useful criterion to improve the quality of inferences and strength the validity of mixed methods studies in the absence of statistical sampling methods ([<reflink idref="bib15" id="ref60">15</reflink>]; [<reflink idref="bib39" id="ref61">39</reflink>]). CIT may offer an approach to increase internal validity, even with discrepant sample sizes, by collecting data until exhaustiveness is achieved.</p> <p>CIT may also strengthen inside–outside legitimacy as described by [<reflink idref="bib12" id="ref62">12</reflink>]. Legitimation refers to ensuring findings or inferences based on results are credible, trustworthy, dependable, transferable, and confirmable. [<reflink idref="bib29" id="ref63">29</reflink>] described inside–outside legitimacy as the degree to which a researcher accurately represents an insider's perspective (e.g., teachers directly implementing the intervention in their classrooms) and an outsider's perspective (e.g., research team). A distinctive feature of CIT is interviewing "insiders" (e.g., educators) to understand turning points or to gain insights for improving existing practices or policies ([<reflink idref="bib15" id="ref64">15</reflink>]). Using CIT as a methodological approach may be viewed as a tool for strengthening inside-outside legtimacy in mixed methods studies.</p> <p>To assess credibility of findings, this study included five credibility checks, some conducted by trained reviewers (i.e., outsiders) to assess the content validity of data and categories formed during a study ([<reflink idref="bib9" id="ref65">9</reflink>]). The credibility checks built into CIT studies could help to address threats to validity (i.e., legitimacy) related to sampling or recruitment procedures. Furthermore, as a qualitative approach, CIT may improve the strength of meta-inferences from data generated using different methods.</p> <hd id="AN0163453184-41">Implications for Research</hd> <p>Use of mixed methods designs to improve school interventions shows promise as an approach to enhance our current understanding of discipline disproportionality and feasible remedies to address this issue in schools. Mixed methods approaches are well suited for assessing intervention acceptability across a broad range of stakeholders in schools or other settings. Although mixed methods research requires substantial effort, such effort is warranted when considering the effort wasted in developing a potentially efficacious intervention that is not acceptable to school administrators or classroom teachers. The evidence demonstrating the effectiveness of the ReACT intervention has to date been limited to a few studies ([<reflink idref="bib16" id="ref66">16</reflink>]; [<reflink idref="bib26" id="ref67">26</reflink>]). It is currently too early to recommend widespread use of ReACT. However, there appear to be elements of this intervention that are acceptable to a diverse group of educators (e.g., across race, regions, and roles) that may make it appealing to other educators.</p> <p>It is possible that the acceptability of the ReACT intervention could be improved by aligning the intervention to fit within existing school-wide frameworks rather than as a stand-alone intervention ([<reflink idref="bib18" id="ref68">18</reflink>]). For example, classroom teachers described potential threats to implementing ReACT to fidelity that included a lack of alignment between the intervention and their teaching approach and implementing new strategies alongside competing school tasks or priorities. Although these types of hindrances may be common in school settings, such obstacles could be mitigated by helping teachers to adapt the intervention to fit their contexts and by ensuring school leaders prioritize disciplinary equity as school-wide goal. Overall, use of mixed methods not only advanced our current knowledge of this important topic area, but also helped us better understand what enabled or hindered key school stakeholders from implementing the ReACT intervention to fidelity.</p> <p>Based on our experiences applying this intergrative approach, we plan to continue to embed mixed methods into our development project. For example, we will add a CIT interview study to complement our randomized controlled trial to assess implementation of the full, school-wide ReACT intervention. Use of a mixed methods design will allow us to assess modifications to the intervention and better understand how educators implement ReACT in a school-wide context. Such research could also reveal how to more effectively help educators to address school discipline disparities that are not captured by measures typically used in randomized controlled trials.</p> <p>Despite the advantages of using mixed methods to study a topic like discipline disproportionality, there may be reasons why this approach is not used more frequently. Funding structures used to support education research (e.g., Institute of Education Sciences) generally prioritize quantitative approaches. Furthermore, researchers typically seek to publish separate studies (i.e., quantitative and qualitative) rather than combining methods in a single study. Although addressing such concerns is beyond the scope of this article, there is clearly a need to consider how to promote broader use of this methodological approach to advance educational research.</p> <ref id="AN0163453184-42"> <title> References </title> <blist> <bibl id="bib1" idref="ref13" type="bt">1</bibl> <bibtext> Aarons G. A., Ehrhart M. G., Farahnak L. R., Hurlburt M. S. (2015). Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10, Article 11. doi:10.1186/s13012-014-0192-y</bibtext> </blist> <blist> <bibl id="bib2" idref="ref44" type="bt">2</bibl> <bibtext> Andersson B.-E., Nilsson S.-G. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology, 48, 398–403.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref34" type="bt">3</bibl> <bibtext> Andreou T. E., McIntosh K., Ross S. W., Kahn J. D. (2015). Critical incidents in sustaining school-wide positive behavioral interventions and supports. The Journal of Special Education, 49, 157–167. doi:10.1177/0022466914554298</bibtext> </blist> <blist> <bibl id="bib4" idref="ref5" type="bt">4</bibl> <bibtext> Bastable E., McIntosh K. (2019). Exploring educators' commitment to racial equity: A qualitative study of critical incidents. Manuscript submitted for publication.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref53" type="bt">5</bibl> <bibtext> Bastable E., Massar M., McIntosh K. (2019). A survey of team members' perceptions of coaching activities related to tier 1 SWPBIS implementation. Journal of Positive Behavior Supports. Advance online publication. doi:10.1177/1098300719861566</bibtext> </blist> <blist> <bibl id="bib6" idref="ref50" type="bt">6</bibl> <bibtext> Bradshaw C. P., Pas E. T., Bottiani J. H., Debnam K. J., Reinke W. M., Herman K. C., Rosenberg M. S. (2018). Promoting cultural responsivity and student engagement through double check coaching of classroom teachers: An efficacy study. School Psychology Review, 47, 118–134.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref43" type="bt">7</bibl> <bibtext> Brantlinger E., Jimenez R., Klingner J., Pugach M., Richardson V. (2005). Qualitative studies in special education. Exceptional Children, 71, 195–207. doi:10.1177/001440290507100205</bibtext> </blist> <blist> <bibl id="bib8" idref="ref28" type="bt">8</bibl> <bibtext> Butterfield L. D., Borgen W. A., Amundson N. E., Maglio A.-S. T. (2005). Fifty years of the critical incident technique: 1954-2004 and beyond. Qualitative Research, 5, 475–497.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref26" type="bt">9</bibl> <bibtext> Butterfield L. D., Borgen W. A., Maglio A.-S. T., Amundson N. E. (2009). Using the enhanced critical incident technique in counselling psychology research. Canadian Journal of Counselling, 43, 265–282.</bibtext> </blist> <blist> <bibtext> Carter P. L., Skiba R., Arredondo M. I., Pollock M. (2017). You can't fix what you don't look at: Acknowledging race in addressing racial discipline disparities. Urban Education, 52, 207–235.</bibtext> </blist> <blist> <bibtext> Creswell J. W., Clark V. L. P. (2011). Designing and conducting mixed methods research. SAGE.</bibtext> </blist> <blist> <bibtext> Currall S. C., Towler A. J. (2003). Research methods in management and organizational research: Toward integration of qualitative and quantitative techniques. In Tashakkori A., Teddlie C. (Eds.), Handbook of mixed methods in social and behavioral sciences (pp. 512–526). SAGE.</bibtext> </blist> <blist> <bibtext> DiAngelo R. (2011). White fragility. The International Journal of Critical Pedagogy, 3, 54–70.</bibtext> </blist> <blist> <bibtext> Fenning P., Pigott T., Engler E., Bradshaw K., Gamboney E., Grunewald S.,.. Kato M. M. (2011, August 4–7). A mixed methods approach examining disproportionality in school discipline. Paper presented at the Annual Convention of the American Psychological Association, Washington, DC. Retrieved from https://escholarship.org/uc/item/46c8z6z2</bibtext> </blist> <blist> <bibtext> Flanagan J. C. (1954). The critical incident technique. Psycho-logical Bulletin, 51, 327–358.</bibtext> </blist> <blist> <bibtext> Gion C., McIntosh K., Falcon F. S. (2019). Effects of a multifaceted classroom intervention on racial disproportionality. Manuscript submitted for publication.</bibtext> </blist> <blist> <bibtext> Goff P. A., Jackson M. C., Nichols A. H., Di Leone B. A. L. (2013). Anything but race: Avoiding racial discourse to avoid hurting you or me. Psychology, 4, 335–339. doi:10.4236/psych.2013.43A048</bibtext> </blist> <blist> <bibtext> Good C., McIntosh K., Gietz C. (2011). Integrating bullying prevention into school-wide positive behavior support. Teaching Exceptional Children, 44, 48–56. doi:10.1177/004005991104400106</bibtext> </blist> <blist> <bibtext> Gregory A., Allen J. P., Mikami A. Y., Hafen C. A., Pianta R. C. (2015). The promise of a teacher professional development program in reducing racial disparity in classroom exclusionary discipline. In Losen D. J. (Ed.), Closing the school discipline gap: Research for policymakers (pp. 166–179). Teachers College Press.</bibtext> </blist> <blist> <bibtext> Haight W., Gibson P. A., Kayama M., Marshall J. M., Wilson R. (2014). An ecological-systems inquiry into racial disproportionalities in out-of-school suspensions from youth, caregiver and educator perspectives. Children Youth Services Review, 46, 128–138.</bibtext> </blist> <blist> <bibtext> Huberty C. J., Morris J. D. (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302–308.</bibtext> </blist> <blist> <bibtext> Klingner J. K., Boardman A. G. (2011). Addressing the "research gap" in special education through mixed methods. Learning Disability Quarterly, 34, 208–218.</bibtext> </blist> <blist> <bibtext> Lane K. L., Kalberg J. R., Bruhn A. L., Driscoll S. A., Wehby J. H., Elliott S. N. (2009). Assessing social validity of school-wide positive behavior support plans: Evidence for the reliability and structure of the Primary Intervention Rating Scale. School Psychology Review, 38, 135–144.</bibtext> </blist> <blist> <bibtext> Leeman J., Wiecha J. L., Vu M., Blitstein J. L., Allgood S., Lee S., Merlo C. (2018). School health implementation tools: A mixed methods evaluation of factors influencing their use. Implementation Science, 13, Article 48. doi:10.1186/s13012-018-0738-5</bibtext> </blist> <blist> <bibtext> Lyon A. R., Bruns E. J. (2019). From evidence to impact: Joining our best school mental health practices with our best implementation strategies. School Mental Health, 11, 106–114.</bibtext> </blist> <blist> <bibtext> McIntosh K., Ellwood K., McCall L., Girvan E. J. (2018). Using discipline data within a PBIS framework to enhance equity in school discipline. Intervention in School and Clinic, 53, 146–152.</bibtext> </blist> <blist> <bibtext> McIntosh K., Kelm J. L., Canizal Delabra A. (2016). In search of how principals change: A qualitative study of events that help and hinder administrator support for school-wide PBIS. Journal of Positive Behavior Interventions, 18, 100–110. doi:10.1177/1098300715599960</bibtext> </blist> <blist> <bibtext> McIntosh K., Predy L. K., Upreti G., Hume A. E., Turri M. G., Mathews S. (2014). Perceptions of contextual features related to implementation and sustainability of school-wide positive behavior support. Journal of Positive Behavior Interventions, 16, 29–41. doi:10.1177/1098300712470723</bibtext> </blist> <blist> <bibtext> Onwuegbuzie A. J., Johnson R. B. (2006). Special issue on "new directions in mixed methods research.". Research in the Schools, 13, 48–63.</bibtext> </blist> <blist> <bibtext> Palinkas L. A., Cooper B. R. (2017). Mixed methods evaluation in dissemination and implementation science. In Brownson R. C., Colditz G. A., Proctor E. K. (Eds.), Dissemination and implementation research in health: Translating science to practice (2nd ed., pp. 335–353). Oxford University Press.</bibtext> </blist> <blist> <bibtext> Proctor E., Silmere H., Raghavan R., Hovmand P., Aarons G., Bunger A.,.. Hensley M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38, 65–76.</bibtext> </blist> <blist> <bibtext> Reinke W. M., Lewis-Palmer T. L., Merrell K. W. (2008). The classroom check-up: A classwide teacher consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37, 315–332.</bibtext> </blist> <blist> <bibtext> Sanetti L. M. H., Collier-Meek M. A., Long A. C., Byron J., Kratochwill T. R. (2015). Increasing teacher treatment integrity of behavior support plans through consultation and implementation planning. Journal of School Psychology, 53, 209–229. doi:10.1016/j.jsp.2015.03.002</bibtext> </blist> <blist> <bibtext> Shaw S. R., Braden J. P. (1990). Race and gender bias in the administration of corporal punishment. School Psychology Review, 19, 378–383.</bibtext> </blist> <blist> <bibtext> Singleton G. E. (2015). Courageous conversations about race: A field guide for achieving equity in schools. Corwin Press.</bibtext> </blist> <blist> <bibtext> Skiba R. J., Arredondo M. I., Rausch M. K. (2014). New and developing research on disparities in discipline (The Equity Project). Bloomington: Indiana University. Retrieved from <ulink href="http://www.njjn.org/uploads/digital-library/OSF%5fDiscipline-Disparities%5fDisparity%5fNewResearch%5f3.18.14.pdf">http://www.njjn.org/uploads/digital-library/OSF%5fDiscipline-Disparities%5fDisparity%5fNewResearch%5f3.18.14.pdf</ulink></bibtext> </blist> <blist> <bibtext> Skiba R. J., Losen D. J. (2016). From reaction to prevention: Turning the page on school discipline. American Educator, 39, 4–44.</bibtext> </blist> <blist> <bibtext> Tatum B. D. (2017). Why are all the Black kids sitting together in the cafeteria? And other conversations about race. Basic Books.</bibtext> </blist> <blist> <bibtext> Teddlie C., Tashakkori A. (2003). Handbook of mixed methods in social behavioral research. SAGE.</bibtext> </blist> </ref> <ref id="AN0163453184-43"> <title> Footnotes </title> <blist> <bibtext> The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibtext> The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A1800027 to the University of Oregon. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.</bibtext> </blist> <blist> <bibtext> Paul Meng</bibtext> </blist> <blist> <bibtext>Graph</bibtext> </blist> <blist> <bibtext>https://orcid.org/0000-0002-0278-2996 Kent McIntosh</bibtext> </blist> <blist> <bibtext>Graph https://orcid.org/0000-0002-7765-2993</bibtext> </blist> </ref> <aug> <p>By Eoin Bastable; Paul Meng; Sarah Fairbanks Falcon and Kent McIntosh</p> <p>Reported by Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib37" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib13" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib17" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib35" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib38" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib26" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib16" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib31" firstref="ref11"></nolink> <nolink nlid="nl10" bibid="bib30" firstref="ref12"></nolink> <nolink nlid="nl11" bibid="bib22" firstref="ref15"></nolink> <nolink nlid="nl12" bibid="bib36" firstref="ref16"></nolink> <nolink nlid="nl13" bibid="bib24" firstref="ref17"></nolink> <nolink nlid="nl14" bibid="bib25" firstref="ref18"></nolink> <nolink nlid="nl15" bibid="bib14" firstref="ref20"></nolink> <nolink nlid="nl16" bibid="bib20" firstref="ref21"></nolink> <nolink nlid="nl17" bibid="bib11" firstref="ref22"></nolink> <nolink nlid="nl18" bibid="bib39" firstref="ref23"></nolink> <nolink nlid="nl19" bibid="bib23" firstref="ref24"></nolink> <nolink nlid="nl20" bibid="bib15" firstref="ref27"></nolink> <nolink nlid="nl21" bibid="bib21" firstref="ref36"></nolink> <nolink nlid="nl22" bibid="bib27" firstref="ref42"></nolink> <nolink nlid="nl23" bibid="bib34" firstref="ref49"></nolink> <nolink nlid="nl24" bibid="bib19" firstref="ref51"></nolink> <nolink nlid="nl25" bibid="bib32" firstref="ref52"></nolink> <nolink nlid="nl26" bibid="bib33" firstref="ref54"></nolink> <nolink nlid="nl27" bibid="bib28" firstref="ref55"></nolink> <nolink nlid="nl28" bibid="bib29" firstref="ref57"></nolink> <nolink nlid="nl29" bibid="bib12" firstref="ref62"></nolink> <nolink nlid="nl30" bibid="bib18" firstref="ref68"></nolink> |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1376214 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Using an Embedded Mixed Methods Design to Assess and Improve Intervention Acceptability of an Equity-Focused Intervention: A Methodological Demonstration – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Bastable%2C+Eoin%22">Bastable, Eoin</searchLink><br /><searchLink fieldCode="AR" term="%22Meng%2C+Paul%22">Meng, Paul</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-0278-2996">0000-0002-0278-2996</externalLink>)<br /><searchLink fieldCode="AR" term="%22Falcon%2C+Sarah+Fairbanks%22">Falcon, Sarah Fairbanks</searchLink><br /><searchLink fieldCode="AR" term="%22McIntosh%2C+Kent%22">McIntosh, Kent</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7765-2993">0000-0002-7765-2993</externalLink>) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Behavioral+Disorders%22"><i>Behavioral Disorders</i></searchLink>. May 2023 48(3):201-211. – Name: Avail Label: Availability Group: Avail Data: SAGE Publications and Hammill Institute on Disabilities. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 11 – Name: DatePubCY Label: Publication Date Group: Date Data: 2023 – Name: SourceSuprt Label: Sponsoring Agency Group: SrcSuprt Data: Institute of Education Sciences (ED) – Name: NumberContract Label: Contract Number Group: NumCntrct Data: R324A180027 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Research – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Mixed+Methods+Research%22">Mixed Methods Research</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Equal+Education%22">Equal Education</searchLink><br /><searchLink fieldCode="DE" term="%22Discipline%22">Discipline</searchLink><br /><searchLink fieldCode="DE" term="%22Racism%22">Racism</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Implementation%22">Program Implementation</searchLink><br /><searchLink fieldCode="DE" term="%22Faculty+Development%22">Faculty Development</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Attitudes%22">Teacher Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Rating+Scales%22">Rating Scales</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Positive+Behavior+Supports%22">Positive Behavior Supports</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Behavior%22">Student Behavior</searchLink><br /><searchLink fieldCode="DE" term="%22Barriers%22">Barriers</searchLink><br /><searchLink fieldCode="DE" term="%22Adoption+%28Ideas%29%22">Adoption (Ideas)</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1177/0198742919880486 – Name: ISSN Label: ISSN Group: ISSN Data: 0198-7429<br />2163-5307 – Name: Abstract Label: Abstract Group: Ab Data: Educators have shown reluctance to implement interventions aimed at improving racial equity in school disciplinary practice. Mixed methods were applied to assess and improve the acceptability of a new intervention designed to reduce racial disparities in school discipline. A descriptive concurrent parallel design was used to assess U.S. educators' perceptions of the acceptability of the intervention. Quantitative findings from professional development workshops introducing the intervention to 118 educators were corroborated with qualitative findings from a separate sample of 4 teachers who implemented it in their classrooms. Quantitative findings indicated that the intervention was acceptable to a broad range of potential implementers, and qualitative findings were used to modify the intervention to further improve its utility. The strengths, limitations, and implications of embedding mixed methods approaches to assess school-based interventions are also discussed. [For the corresponding grantee submission, see ED603401.] – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: CodeSource Label: IES Funded Group: SrcInfo Data: Yes – Name: DateEntry Label: Entry Date Group: Date Data: 2023 – Name: AN Label: Accession Number Group: ID Data: EJ1376214 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1376214 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1177/0198742919880486 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 11 StartPage: 201 Subjects: – SubjectFull: Mixed Methods Research Type: general – SubjectFull: Intervention Type: general – SubjectFull: Equal Education Type: general – SubjectFull: Discipline Type: general – SubjectFull: Racism Type: general – SubjectFull: Program Evaluation Type: general – SubjectFull: Program Implementation Type: general – SubjectFull: Faculty Development Type: general – SubjectFull: Teacher Attitudes Type: general – SubjectFull: Rating Scales Type: general – SubjectFull: Fidelity Type: general – SubjectFull: Positive Behavior Supports Type: general – SubjectFull: Student Behavior Type: general – SubjectFull: Barriers Type: general – SubjectFull: Adoption (Ideas) Type: general Titles: – TitleFull: Using an Embedded Mixed Methods Design to Assess and Improve Intervention Acceptability of an Equity-Focused Intervention: A Methodological Demonstration Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Bastable, Eoin – PersonEntity: Name: NameFull: Meng, Paul – PersonEntity: Name: NameFull: Falcon, Sarah Fairbanks – PersonEntity: Name: NameFull: McIntosh, Kent IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Type: published Y: 2023 Identifiers: – Type: issn-print Value: 0198-7429 – Type: issn-electronic Value: 2163-5307 Numbering: – Type: volume Value: 48 – Type: issue Value: 3 Titles: – TitleFull: Behavioral Disorders Type: main |
| ResultId | 1 |
