View in EDS HTML Full Text PDF Full Text

Utilizing Meta-Analyses to Guide Practice: A Primer

Saved in:
Bibliographic Details
Title: Utilizing Meta-Analyses to Guide Practice: A Primer
Language: English
Authors: Therrien, William J. (ORCID 0000-0003-0594-5129), Cook, Bryan G. (ORCID 0000-0001-9294-0873), Cook, Lysandra
Source: Learning Disabilities Research & Practice. Aug 2020 35(3):111-117.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 7
Publication Date: 2020
Document Type: Journal Articles
Information Analyses
Descriptors: Meta Analysis, Guidance, Instructional Effectiveness, Students with Disabilities, Learning Disabilities, Literature Reviews, Evidence Based Practice
DOI: 10.1111/ldrp.12230
ISSN: 0938-8982
Abstract: Meta-analysis is one approach for synthesizing research studies to identify generally effective instructional practices for students with learning disabilities (LD). In this article, we define core components of meta-analytic literature reviews, discuss how to interpret findings from meta-analyses, and provide guidelines for how research consumers can utilize meta-analyses to guide instructional practices for students with LD. We conclude that meta-analyses provide valuable information for determining evidence-based practices for students with LD.
Abstractor: As Provided
Entry Date: 2020
Accession Number: EJ1263027
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwE4Lta2wYktYNoGnDQhZJNsAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDPpt0ZpzBlm6e04o2wIBEICBm7O691tcD5ZQQvXzP3lItWY-y_rxQcwgxrIyDEzac7jGpdchv-Q5AmI7ZqUHFj7rOWt88hjF6ykAZo7dSzRFOo4wpMrX1P1Vd8AkTjvQYmut1OP4UyONd3BDpKzoeF89eVIJ4Vj7YOw1PTaw8g4vTJIruClohxvSFr8396KJb0vRHCFLaSHtpMqmAlIewmnAtqpM4QXmapMiPVUZ
Text:
  Availability: 1
  Value: <anid>AN0145042708;7mj01aug.20;2020Aug12.11:27;v2.2.500</anid> <title id="AN0145042708-1">Utilizing Meta‐Analyses to Guide Practice: A Primer </title> <p>Meta‐analysis is one approach for synthesizing research studies to identify generally effective instructional practices for students with learning disabilities (LD). In this article, we define core components of meta‐analytic literature reviews, discuss how to interpret findings from meta‐analyses, and provide guidelines for how research consumers can utilize meta‐analyses to guide instructional practices for students with LD. We conclude that meta‐analyses provide valuable information for determining evidence‐based practices for students with LD.</p> <p> <emph>Ms. Jackson was a first‐year special education teacher at Mountain View Middle School, where she provided supports for many students with learning disabilities (LD) in general education classes. She noticed that many of her students with LD were scoring very low on their multiple‐choice chapter quizzes in social studies; those quizzes make up a large part of their final grade for that class. She remembered graphic organizers being discussed in her licensure coursework, and decided to see if she could find research supporting their use for middle school students with LD in social studies. She was excited to find a study by</emph> DiCecco and Gleason (2002) <emph>that examined the effects of graphic organizers on social studies outcomes for 24 middle school students with LD. The study showed that, although graphic organizers had a positive effect on students' essays, they did not result in improved scores on multiple‐choice knowledge tests for participating students. Ms. Jackson mentioned this to Ms. Stoffers, the veteran special education teacher at her school. Ms. Stoffers said that didn't make sense to her, and that she was sure she remembered reading research studies claiming that graphic organizers were effective for middle‐school students with LD. After school that day, Ms. Jackson received a text from Ms. Stoffers, asking Ms. Jackson to come to her room and see what she had found. Ms. Stoffers showed her an article by</emph> Dexter, Park, and Hughes (2011) <emph>reporting a meta‐analysis of findings from 16 studies examining the effects of graphic organizers for students with LD. The meta‐analysis indicated that graphic organizers had large effects for students with LD in general, with large results specifically in social studies and for near‐transfer measures like factual, multiple‐choice quizzes. Ms. Jackson was perplexed. How could the results of a study and a meta‐analysis contradict each other? Which finding should she trust?</emph></p> <p>Science, and experimental research in particular, is the most trustworthy approach for determining the effectiveness of instructional practices (Cook & Cook, 2016). Meta‐analyses are a powerful approach for synthesizing findings across studies. It is important for educators to consider findings across an entire research base, rather than focusing on individual studies, when making practical decisions (Santangelo, Novosel, Cook, & Gapsis, 2015). The reason is that individual studies likely contain some error, and results may be unique to the students and interventionists who participated in the study. Indeed, studies can be "outliers" and may report that a generally effective practice did not work, or that a generally ineffective practice was highly effective. If, however, one examines the results of all the studies examining the effects of an instructional practice, it is possible to identify the typical or average effect that an intervention has across those studies—an approach that provides a more robust estimate than those provided by individual studies. To make an analogy to sports, it is more accurate to evaluate how good a sports team is after watching them play once than never having seen them play; but—because teams can have particularly good or bad games—it is even more accurate to evaluate them after watching a whole season. Thus, although informing instructional decisions on the results of one research study is better than acting based on a hunch, synthesizing findings from an entire research base is the most reliable approach. In Ms. Jackson's case, for example, reviewing only one study on graphic organizers might have discouraged her from using them with her students, while aggregated results from a meta‐analysis indicated the approach is likely to be effective for improving her students' achievement on multiple‐choice social studies tests.</p> <p>In this article, we examine one popular approach for synthesizing findings across research studies: meta‐analysis. We examine core elements of meta‐analysis, consider how to interpret the results of meta‐analyses, describe different types of meta‐analyses, and provide guidelines for interpreting and applying meta‐analyses, with examples from recent meta‐analyses for students with LD. Our take‐home message is that meta‐analyses statistically synthesize effects across research studies to provide robust estimates of an intervention's effectiveness and other research outcomes.</p> <hd id="AN0145042708-2">LITERATURE REVIEWS</hd> <p>Reviews of the literature are important sources for determining the effectiveness of instructional practices for a population of learners, such as students with LD. Authors of reviews do the hard work for research consumers, integrating findings from many studies so others do not have to locate, read through, and integrate findings from each and every relevant study to ascertain the effectiveness of instructional approaches. There are many types of literature reviews, including traditional narrative literature reviews, box‐score analyses, evidence‐based reviews, and meta‐analyses. In traditional narrative literature reviews, authors qualitatively synthesize study findings in order to reach their conclusions. Although these reviews can provide important information, it is often unclear what decision‐making rules authors used to reach their conclusions, such as whether a practice is shown to be effective. Without reporting‐procedures transparency (i.e., specifics on how studies were identified to review, and how findings from those studies were analyzed in order to allow others to replicate the review and its findings), it is uncertain whether others would reach the same conclusions had they conducted their own review.</p> <p>Box‐score analyses add some transparency to the review process by using set criteria for determining whether the studies reviewed support an intervention's effectiveness, and "keep score" of a practice's effectiveness by counting how many studies indicate a practice is effective, and how many do not. For example, a box‐score analysis might report how many studies had statistically significant findings supporting the effectiveness of an instructional practice, and how many studies reported nonsignificant findings (Therrien, Zaman, & Banda, 2011b). Although the objectivity with which studies are categorized in box‐score analyses is an improvement over traditional literature reviews, these reviews may leave out critical information that can improve one's understanding of the effectiveness of a practice (Kavale & Glass, 1981). Box‐score analyses do not, for example, take into account the magnitude of the effects in each study, or weight studies with greater precision more than studies with less precision. Without accounting for this information, authors of box‐score analyses might draw invalid conclusions.</p> <p>Evidence‐based reviews apply a set of standards to a research base in order to determine whether a sufficient number of high‐quality experimental studies support a practice as effective so that the practice can be identified as evidence‐based (Cook, Collins, Cook, & Cook, 2020). Although evidence‐based reviews are transparent and useful for informing practice, they may not consider all research conducted on a topic (evidence‐based reviews only consider studies that meet rigorous indicators of high‐quality research); they do not give greater weight to more precise studies; and they typically categorize practices as evidence‐based or not, failing to distinguish between, for example, evidence‐based practices with moderate as opposed to very large effects.</p> <hd id="AN0145042708-3">META‐ANALYSES</hd> <p>Meta‐analysis techniques were developed to address weaknesses found in other types of reviews (Glass, 1977). Meta‐analysts use transparent procedures to quantitatively synthesize bodies of literature. Meta‐analytic techniques enable review authors to harness all relevant data reported in a research base. This process in turn permits the meta‐analyst to provide nuanced information on an intervention and, because of transparency in how study findings are analyzed, allows research consumers to be highly confident in the conclusions reported. See Table 1 for explanations of key terms that readers are likely to encounter in meta‐analyses.</p> <p>1 TABLEKey Terms Related to Meta‐Analysis</p> <p> <ephtml> <table><thead><tr><th>Term</th><th align="center">Definition</th><th align="center">Example</th></tr></thead><tbody><tr><td>95 percent confidence interval</td><td>A range of outcomes within which it is 95 percent likely that the true outcome falls. If the 95 percent confidence interval for an effect size does not include 0, it is considered statistically significant.</td><td>The meta‐analysis reported an overall, weighted effect size of 0.88 for the intervention, with a 95 percent confidence interval of 0.68 to 1.08.</td></tr><tr><td>Coding</td><td>Extracting information from studies reviewed in a meta‐analysis so the effects of studies with similar characteristics can be compared to the effects of other studies that do not share that same characteristic.</td><td>The meta‐analyst coded studies into age groups (elementary, middle, and high) to explore whether the program was more or less effective for younger or older students.</td></tr><tr><td>Effect size</td><td>A measure in quantitative research indicating the size or magnitude of the effect of an intervention or strength of the relation between variables.</td><td>The meta‐analysis reported an overall, weighted effect size of 0.88 for the intervention.</td></tr><tr><td>Meta‐analysis</td><td>A type of research review in which researchers statistically combine results across multiple studies.</td><td>Meta‐analyses can be conducted to determine the average effect of an intervention across studies.</td></tr><tr><td>Moderator and subgroup analyses</td><td>Moderator and subgroup analyses examine whether and how effect sizes differ, or are moderated by, participant, outcome, intervention, and other study elements.</td><td>Moderator analyses showed that effect sizes were significantly larger for elementary than secondary students.</td></tr><tr><td>Standardized mean difference</td><td>The most common type of effect size reported in group experimental research studies and meta‐analyses.</td><td>The meta‐analysis reported a standardized mean difference effect size of 0.88.</td></tr><tr><td>Statistically significant</td><td>Results are traditionally considered statistically significant when the probability that they occurred without an actual effect existing in the population is less than 5 percent (see Travers, Cook, & Cook, 2017).</td><td>The meta‐analysis reported that difference in achievement for younger students was higher than for older students, and that the difference was statistically significant.</td></tr><tr><td>Weighted effect size</td><td>Statistically adjusting effects sizes in meta‐analyses so that more precise effects (e.g., effects with less variance, effects from studies with larger number of participants) have more sway or influence when estimating the overall effect size.</td><td>The meta‐analysis reported an overall, weighted effect size of 0.88 for the intervention.</td></tr></tbody></table> </ephtml> </p> <hd id="AN0145042708-4">Core Components of Meta‐Analyses</hd> <p>Core components of meta‐analyses include (a) a clear and transparent methodology, (b) study effect sizes (ESs), (c) study effect weights, (d) codes of study characteristics, and (e) use of statistical analyses to explore review questions (Borenstein, Hedges, Higgins, & Rothstein, 2011). In this section, we provide a nontechnical description of these components.</p> <p>Method transparency is a core component of meta‐analyses not always found in other types of reviews. In the Method section of a meta‐analysis, authors detail exactly how studies were located and selected, how studies were coded, what outcomes were used to calculate effects, and how outcomes were combined and evaluated for significance. As a result, the literature searches and reviews on which meta‐analyses are based are called systematic, because they are carried out objectively and transparently. To foster systematic and objective reviews, many meta‐analysts follow standard guidelines when conducting and reporting their reviews. A well‐executed meta‐analysis that follows standardized guidelines can be replicated by others. This transparency increases one's confidence that the findings reached are valid.</p> <p>Effect sizes (ESs) that are calculated from the studies reviewed are the main ingredients of meta‐analyses. As discussed with regard to the work of Cook, Cook, and Therrien (2018) and Maggin, Cook, and Cook (2019), ESs represent the strength of the relationship or effect examined in a research study. A variety of ESs are available for different types of research designs (e.g., group or single‐case design experiments) and for different relationships (e.g., correlations, mean differences, risk ratios) found in studies. Meta‐analysts select an appropriate ES metric to calculate based on the questions driving their investigation and the nature of the studies being reviewed. For example, if a meta‐analyst is exploring the relative effectiveness of an intervention, and most of the studies utilized single‐case research designs (e.g., Swanson & Sachse‐Lee, 2000), then a single‐case ES (e.g., Percentage of All Non‐Overlapping Data; Parker, Hagan‐Burke, & Vannest, 2007) would be selected. Conversely, if a meta‐analyst was exploring the relative usefulness of a curriculum‐based measure to predict achievement on standardized tests (e.g., Romig, Therrien, & Lloyd, 2017), a correlation (<emph>r</emph>; Borenstein et al., 2011) would be used as the ES.</p> <p>In this article, we focus on meta‐analyses that use the most common ES in education: the standardized mean difference. This ES, typically reported as Cohen's <emph>d</emph> or Hedges <emph>g</emph>, is appropriate for group experiments, and represents the standardized (i.e., measures with different scales put on the same metric) difference between two groups—typically a treatment and a control group. In other words, this ES indicates the strength or magnitude of the intervention effect under investigation. For a primer on standardized mean difference effect sizes, see Cook et al. (2018). Although our focus is on standardized mean differences here, the concepts discussed in this article apply to meta‐analyses that utilize different ESs, because basic meta‐analytic procedures do not change based on the ES used.</p> <p>Once ESs are calculated, meta‐analysts often utilize statistical procedures to determine how much weight each study's findings should contribute to the results of the review. Instead of assuming that each study should count the same, meta‐analysts award more sway or weight to studies with more precise findings (Borenstein et al., 2011). The exact formula used depends on certain assumptions made by the meta‐analyst, but in general, the more participants are involved in the study, and the less variability there is in participants' outcomes, the more weight a study is given. Accounting for this precision ensures that a study with 1,000 students with very consistent intervention effects on a student outcome counts more in the overall analyses than a study with only 16 students with highly variable outcomes across students.</p> <p>Along with calculating ESs and weighting studies, meta‐analysts also extract and code important study characteristics. These codes represent meaningful details about each of the studies involved in the review. For example, in a meta‐analysis examining the effects of a particular intervention, meta‐analysts often code for student characteristics (e.g., age/grade level, ethnicity, disability type), intervention components, and outcomes measured, among other study elements. These codes, combined with the ESs and study weights, can be utilized to examine questions such as who the intervention is effective for, what the essential components of the intervention are, and what effects teachers should expect to see if they implement the practice with their students.</p> <p>Once meta‐analysts calculate ESs, weight studies, and code studies for important characteristics, these data are used to answer the review's research questions. Unlike other types of reviews, meta‐analyses use statistical procedures to examine research questions by calculating weighted average ESs across studies reviewed. As with statistical procedures used in original research studies, meta‐analysts are able to explore the statistical significance of their results. For example, they can determine whether the overall ES across studies reviewed is significantly different from zero, and they can provide a range of confidence around that ES. For example, Jitendra et al. (2018) reported an overall, weighted ES of 0.37 for mathematics interventions for students with LD across the 19 studies reviewed, with a 95 percent confidence interval between 0.18 and 0.56. Because the 95 percent confidence interval did not include 0, the overall effect of mathematics interventions was statistically significant. In addition, if the review includes enough studies, meta‐analysts can conduct moderator or subgroup analyses to examine additional questions that are important for practitioners, such as what conditions are associated with program effectiveness, whether an intervention is more or less effective for students in different grade levels or for different types of outcomes, or whether effects varied depending on how the intervention was applied.</p> <hd id="AN0145042708-5">Meta‐Analysis Results</hd> <p>Now that we have covered the core components, let's turn to how results are reported in meta‐analyses examining intervention effectiveness. The overall effect is typically reported first. The overall ES represents the aggregate weighted mean effect from all the studies reviewed. This overall ES is often highlighted in the abstract, and gives research consumers an estimate of the general effectiveness of the intervention(s) under review. The magnitudes of these ESs are interpreted in the same way as they are for individual studies, with the caveat that meta‐analysis ESs are based on an entire research base instead of one study, and thus are generally considered more credible. See Cook et al. (2018) and Maggin et al. (2019) for primers on interpreting ESs commonly used in group experimental and single‐case design research.</p> <p>Although the overall ES often gets the headlines, meta‐analysts also conduct subanalyses when there is a statistically significant degree of variability in the ESs of the studies reviewed. Here, meta‐analysts explore potential reasons for why the intervention yielded different effects across studies. To explore potential reasons for this variation, meta‐analysts use the information coded from the studies to categorize studies into different groups, and examine whether ESs varied meaningfully across those groups. For example, studies might be categorized as involving students younger and older than a certain age. The meta‐analyst could then examine whether the weighted mean effect sizes for younger and older students varied significantly. For example, Rohrbeck, Ginsburg‐Block, Fantuzzo, and Miller (2003) conducted a meta‐analysis to examine the effects of peer assisted learning for elementary students, and found that the intervention was more effective for younger than for older elementary students. When a large number of studies are reviewed, the meta‐analyst can explore these subquestions statistically and determine which study characteristics account for differences in ESs across studies reviewed in the meta‐analysis. When there are not enough studies to explore these subquestions statistically, differences may be reported descriptively to shed light on reasons why an intervention's effectiveness might vary across studies.</p> <hd id="AN0145042708-6">THE CONTINUUM OF META‐ANALYSES: BROAD TO SPECIFIC</hd> <p>Before the providing of guidelines for interpreting and applying meta‐analytic results, it is important to consider the different types of meta‐analysis in the education research base. Although the previously described core components are common across meta‐analysis, the focus of meta‐analyses ranges from broad to specific (Therrien et al., 2011b). In the following sections, we discuss three points on this continuum, with examples from the literature for each type of meta‐analysis.</p> <hd id="AN0145042708-7">Meta‐Analyses with a Broad Focus</hd> <p>On one end of the continuum are meta‐analyses that examine the effects of different practices within broad areas of study. The strength of broad focused meta‐analyses is their comprehensiveness. Many studies are included in these reviews across a wide range of instructional disciplines (e.g., social skills, reading, mathematics), so the findings are built on a large and broad research base. Yet because the primary studies evaluated include disparate instructional programs and used many different outcome measures, it may be difficult to glean specific instructional recommendations from them. In other words, these meta‐analyses do not always examine the effectiveness of specific interventions for specific groups of learners for specific outcomes (e.g., which social skills program is most effective for improving social functioning for adolescents with LD; Kavale & Mostert, 2004). Instead, broad focused meta‐analyses provide evidence of what general instructional components tend to be effective for populations of learners (e.g., students with LD) across interventions and outcomes.</p> <p>Perhaps the most well‐known series of large‐scale meta‐analyses was conducted by Hattie (2009). In his book <emph>Visible Learning</emph>, Hattie synthesized the results of prior meta‐analyses to explore what general practices have the largest effects on student achievement. Hattie found that instructional components such as scaffolding, deliberate practice, and feedback have large positive effects on student achievement, whereas multiage classrooms, charter schools, and student grade retention have small positive and even potentially negative effects on student achievement. Swanson and colleagues (Swanson, 1999; Swanson & Hoskyn, 1998, 2001; Swanson & Sachse‐Lee, 2000; Swanson, Carson, & Saches‐Lee, 1996) conducted a series of broad focused meta‐analyses examining effective instructional practices across domains specifically for students with LD. They reported that instructional approaches that combine components of strategy and direct instruction tend to be most effective for students with LD. Specifically, they highlighted the effectiveness of practices similar to those reported by Hattie (2009), such as breaking tasks down into smaller units, providing practice opportunities, and providing students with strategy cues (Swanson, 1999). Overall, broad focused meta‐analyses provide us with valuable information on generally effective practices for all students, including students with LD.</p> <hd id="AN0145042708-8">Meta‐Analyses with a Focus on an Instructional Domain</hd> <p>Falling in the middle of the continuum of specificity, many meta‐analyses examine the effectiveness of multiple different instructional approaches, but focus on a particular instructional domain (e.g., reading, mathematics, social skills), often for a particular population of learners, such as students with LD. In addition to examining average effectiveness across interventions, these meta‐analyses often investigate which interventions were more or less effective. Thus, these meta‐analyses allow educators to glean information regarding the effects of one or more instructional practices in an instructional domain, and to determine whether and to what degree those effects might apply to students in one's school or classroom.</p> <p>In the area of reading, Flynn, Zheng, and Swanson (2012) examined the effectiveness of reading interventions for upper elementary and middle school students with reading disabilities. They reported an overall ES of 0.41, with no significant differences in treatment effects based on student characteristics, reading skills targeted (e.g., phonics, fluency), or programs used. In mathematics, Jitendra et al. (2018) conducted a meta‐analysis exploring effective instructional practices for secondary students with LD. Overall, mathematics interventions had an overall ES of 0.37 on students' achievement, with instructional programs being more effective when they lasted more than 10 hours. Use of visual models (e.g., diagrams, manipulatives) were effective on their own, but effects were as strong or stronger (ES = 0.52) when combined with other instructional components such as computer instruction or use of real‐life problem scenarios. Gillespie and Graham (2014) examined effective writing interventions for students with LD in first through 12th grade. Overall, writing interventions had a positive effect on student achievement (ES = 0.74) with strategy instruction, dictation, goal‐setting, and teaching the writing process resulting in the largest improvements in writing performance.</p> <p>Ciullo et al. (2020) conducted an extensive meta‐analysis examining instructional interventions in social studies for students with LD in kindergarten through 12th grade. They reported an overall ES of 0.76, finding that the interventions with the largest effects included graphic organizers, mnemonics, and use of technology. Therrien, Taylor, Hosp, Kaldenberg, and Gorsh (2011a) examined science instruction for students with LD in 3rd through 12th grade. They found an overall ES of 0.78 across all the studies they examined. When analyzing effects for specific interventions, mnemonics was particularly effective for acquisition and retention of science facts, with structured inquiry (hands‐on science activities combined with explicit instruction) more effective than inquiry without the inclusion of explicit instruction.</p> <hd id="AN0145042708-9">Meta‐Analyses with a Focus on Specific Instructional Practices</hd> <p>The narrowest types of meta‐analyses are those that examine the effects of a specific intervention, often for a specific population of learners, such as students with LD, and sometimes for a specific outcome. These types of meta‐analysis typically address research questions such as "How effective is repeated reading for students with LD?" or "What version of repeated reading is most effective for students with LD?"</p> <p>Lee and Yoon (2017) reported that repeated reading had a large effect on reading fluency (ES = 1.41) for students with reading disabilities across 34 studies. Moderator analyses showed that, among other findings, effects were significantly larger for elementary (ES = 1.63) in contrast to secondary (ES = 0.86) students; and that ESs were significantly larger for studies in which researchers provided a listening preview of the passage (ES = 1.94) than for studies in which they did not (ES = 0.94). Jung, McMaster, Kunkel, Shin, and Stecker (2018) meta‐analyzed the effects of data‐based individualization (e.g., applying data‐based decision rules to progress‐monitoring data to determine instructional changes) for students with intensive learning needs (including students with LD) across 14 studies. The overall ES was small, but statistically significant (ES = 0.37). Moderator analyses indicated that effects were largest when progress monitoring occurred two times per week (ES = 0.47) and when teachers were supported through individual consultation (ES = 0.46).</p> <hd id="AN0145042708-10">GUIDELINES FOR INTERPRETING AND APPLYING META‐ANALYSES</hd> <p>So, how can educators interpret and apply findings from meta‐analyses to guide practice for students, including those with LD? Most generally, we recommend going beyond the aggregate overall ESs reported across all studies reviewed in meta‐analyses. The aggregated overall ESs do not always provide the information that educators need to select instructional practices for their students. Instead, we encourage educators to explore how the meta‐analyst analyzed their findings to examine practical implications, such as who the intervention works for and under what conditions it is most effective. For example, if a teacher wishes to improve reading comprehension for her secondary students with LD, and reads in an abstract from a meta‐analysis that an intervention has a large overall ES for improving reading outcomes for students with LD, they may be tempted to believe the intervention is a great fit for their class. A full reading of the meta‐analysis, however, may reveal that effects vary by student age (with small effects for secondary students) or type of reading outcome (with small effects for reading comprehension). In this section, we describe guidelines, as summarized in Table 2, to help educators appropriately interpret and apply findings from meta‐analyses.</p> <p>2 TABLESummary of Guidelines for Interpreting and Applying Results from Meta‐Analyses</p> <p> <ephtml> <table><tbody><tr><td><list list-type="Bullet"><list-item><p>Examine the relevance of the meta‐analysis to one's population of interest.</p></list-item><list-item><p>Pay close attention to how the authors define core terms, how studies were included and excluded from the review, and which outcomes from the studies were coded for meta‐analysis.</p></list-item><list-item><p>Go beyond the aggregate ES and closely examine the more nuanced moderator or subgroup analyses.</p></list-item><list-item><p>Access meta‐analyses published by reputable sources.</p></list-item><list-item><p>Use meta‐analyses to help select instructional practices, but then monitor student progress on important outcome variables and make instructional changes as needed.</p></list-item></list></td></tr></tbody></table> </ephtml> </p> <p>First, ascertain the potential relevance of the review to one's population of interest. Students with LD, for example, have unique learning needs, so the results of meta‐analyses that focus on students without disabilities, or on students with autism, may not apply to students with LD. Similarly, the results of a meta‐analysis that reviews only studies conducted with preschool students may not apply to high school populations. In addition to age/grade level and disability status, consumers of meta‐analyses might also examine the alignment between the participants in the studies reviewed in a meta‐analysis and their own students with regard to achievement level, ethnicity, English‐language status, and socioeconomic status. The key to remember here is that the less similar the participants involved in the studies reviewed in a meta‐analysis are to one's own students, the less likely it is that the effects reported in the meta‐analysis will apply to one's students (Cook & Cook, 2017).</p> <p>Second, we recommend paying close attention to how the authors defined key terms, how studies were included and excluded from the review, and which outcomes from the studies were coded for meta‐analysis. For example, it is possible that the authors defined an intervention narrowly, and excluded many studies from the review that did not apply the intervention in the defined way. In this case, the results of the meta‐analysis only apply to the specific version of the intervention defined in the study. Similarly, if a meta‐analysis shows that a practice is associated with large effects for an outcome, like reading fluency, one should not assume that the intervention will positively affect other outcomes of potential interest, like reading comprehension.</p> <p>Third, to fully understand the findings of a meta‐analysis, we recommend closely examining the more nuanced moderator or subgroup analyses. These analyses flesh out which versions of the intervention, which subgroups of students, and which outcomes are associated with smaller or larger effects. For example, moderator analyses in Jitendra et al. (2018) meta‐analysis of math interventions for secondary students with LD and math difficulties indicated that, although the overall effect of interventions was small (ES = 0.37), the duration of the intervention is a critical factor for teachers to consider. Interventions implemented for 10 or fewer hours had an ES of 0.11, whereas those applied for >10 hours had an ES of 0.58. In contrast, moderator analyses showed that group size had little impact on intervention effects.</p> <p>Fourth, we recommend using meta‐analyses published by reputable sources, such as peer‐reviewed journals, not‐for‐profit organizations such as the Campbell Collaboration (https://campbellcollaboration.org/), and governmental sources such as the Institute of Education Sciences (https://ies.ed.gov/). Be wary of sources that do not fall into these categories, because it could mean that the authors have an agenda that could skew or bias the results, or that the review may not have been peer‐reviewed for rigor and transparency.</p> <p>Finally, although meta‐analyses are excellent resources for selecting instructional practices for students with LD, it is important to remember that no instructional practice, even one shown to have large effects in a meta‐analysis, is universally effective. Therefore, while we encourage educators to use meta‐analyses to select instruction practices for their students, educators should conduct regular progress monitoring to evaluate the effects of any intervention on student performance. If the intervention is not positively affecting student outcomes, then instructional changes should be made.</p> <hd id="AN0145042708-11">CONCLUSIONS</hd> <p>Meta‐analyses have been become a popular approach for synthesizing study findings across entire research bases in education and special education research. Our take‐home message is that meta‐analyses statistically synthesize effects across research studies to provide robust estimates of an intervention's effectiveness and other research outcomes. Because all studies contain some error, meta‐analyzing findings from all relevant studies on a topic is generally more accurate than relying on findings from a single study, or from a nonsystematic review of the literature. Therefore, we recommend that practitioners and other special education stakeholders look to systematic research reviews, including meta‐analyses, when available, rather than looking to individual studies to inform instructional decisions (Santangelo et al., 2015).</p> <ref id="AN0145042708-12"> <title> REFERENCES </title> <blist> <bibl id="bib1" type="bt">1</bibl> <bibtext> Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to meta‐analysis. West Sussex, United Kingdom : John Wiley & Sons.</bibtext> </blist> <blist> <bibl id="bib2" type="bt">2</bibl> <bibtext> Ciullo, S., Collins, A., Wissinger, D. R., McKenna, J. W., Lo, Y.‐L., & Osman, D. (2020). Students with learning disabilities in the social studies: A meta‐analysis of intervention research. Exceptional Children. Advance online publication. https://doi.org/10.1177/0014402919893932</bibtext> </blist> <blist> <bibl id="bib3" type="bt">3</bibl> <bibtext> Cook, B. G., Collins, L. W., Cook, S. C., & Cook, L. (2020). Evidence‐based reviews: How evidence‐based practices are systematically identified. Learning Disabilities Research & Practice, 35, 6 – 13. https://doi.org/10.1111/ldrp.12213</bibtext> </blist> <blist> <bibl id="bib4" type="bt">4</bibl> <bibtext> Cook, B. G., & Cook, L. (2016). Research designs and special education research: Different designs address different questions. Learning Disabilities Research & Practice, 31, 190 – 198. https://doi.org/10.1111/ldrp.12110</bibtext> </blist> <blist> <bibl id="bib5" type="bt">5</bibl> <bibtext> Cook, B. G., & Cook, L. (2017). Sampling and special education research: Examining whether and how study results apply to you. Learning Disabilities Research & Practice, 32, 78 – 84. https://doi.org/10.1111/ldrp.12132</bibtext> </blist> <blist> <bibl id="bib6" type="bt">6</bibl> <bibtext> Cook, B. G., Cook, L., & Therrien, W. J. (2018). Group‐difference effect sizes: Gauging the practical importance of findings from group‐experimental research. Learning Disabilities Research & Practice, 33, 56 – 63. https://doi.org/10.1111/ldrp.12167</bibtext> </blist> <blist> <bibl id="bib7" type="bt">7</bibl> <bibtext> Dexter, D. D., Park, Y. J., & Hughes, C. A. (2011). A meta‐analytic review of graphic organizers and science instruction for adolescents with learning disabilities: Implications for the intermediate and secondary science classroom. Learning Disabilities Research & Practice, 26, 204 – 213. https://doi.org/10.1111/j.1540-5826.2011.00341.x</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> DiCecco, V. M., & Gleason, M. M. (2002). Using graphic organizers to attain relational knowledge from expository text. Journal of Learning Disabilities, 35, 306 – 320. https://doi.org/10.1177/00222194020350040201</bibtext> </blist> <blist> <bibl id="bib9" type="bt">9</bibl> <bibtext> Flynn, L. J., Zheng, X., & Swanson, H. L. (2012). Instructing struggling older readers: A selective meta‐analysis of intervention research. Learning Disabilities Research & Practice, 27, 21 – 32. https://doi.org/10.1111/j.1540-5826.2011.00347.x</bibtext> </blist> <blist> <bibtext> Gillespie, A., & Graham, S. (2014). A meta‐analysis of writing interventions for students with learning disabilities. Exceptional Children, 80, 454 – 473. https://doi.org/10.1177/0014402914527238</bibtext> </blist> <blist> <bibtext> Glass, G. V. (1977). Integrating findings: The meta‐analysis of research. Review of Research in Education, 5, 351 – 371. https://doi.org/10.2307/1167179</bibtext> </blist> <blist> <bibtext> Hattie, J. (2009). Visible learning: A synthesis of over 800 meta‐analyses relating to achievement. New York, NY : Routledge.</bibtext> </blist> <blist> <bibtext> Jitendra, A. K., Lein, A. E., Im, S., Alghamdi, A. A., Hefte, S. B., & Mouanoutoua, J. (2018). Mathematical interventions for secondary students with learning disabilities and mathematics difficulties: A meta‐analysis. Exceptional Children, 84, 177 – 196. https://doi.org/10.1177/0014402917737467</bibtext> </blist> <blist> <bibtext> Jung, P. G., McMaster, K. L., Kunkel, A. K., Shin, J., & Stecker, P. M. (2018). Effects of data‐based individualization for students with intensive learning needs: A meta‐analysis. Learning Disabilities Research & Practice, 33, 144 – 155. https://doi.org/10.1111/ldrp.12172</bibtext> </blist> <blist> <bibtext> Kavale, K. A., & Glass, G. V. (1981). Meta‐analysis and the integration of research in special education. The Journal of Learning Disabilities, 14, 531 – 538. https://doi.org/10.1177/002221948101400909</bibtext> </blist> <blist> <bibtext> Kavale, K. A., & Mostert, M. P. (2004). Social skills interventions for individuals with learning disabilities. Learning Disability Quarterly, 27, 31 – 43. https://doi.org/10.2307/1593630</bibtext> </blist> <blist> <bibtext> Lee, J., & Yoon, S. Y. (2017). The effects of repeated reading on reading fluency for students with reading disabilities: A meta‐analysis. Journal of Learning Disabilities, 50, 213 – 224. https://doi.org/10.1177/0022219415605194</bibtext> </blist> <blist> <bibtext> Maggin, D. M., Cook, B. G., & Cook, L. (2019). Making sense of single‐case design effect sizes. Learning Disabilities Research & Practice, 34, 124 – 132. https://doi.org/10.1111/ldrp.12204</bibtext> </blist> <blist> <bibtext> Parker, R. I., Hagan‐Burke, S., & Vannest, K. (2007). Percentage of all non‐overlapping data (PAND) an alternative to PND. The Journal of Special Education, 40, 194 – 204. https://doi.org/10.1177/00224669070400040101</bibtext> </blist> <blist> <bibtext> Rohrbeck, C. A., Ginsburg‐Block, M. D., Fantuzzo, J. W., & Miller, T. R. (2003). Peer‐assisted learning interventions with elementary school students: A meta‐analytic review. Journal of Educational Psychology, 95, 240 – 257. https://doi.org/10.1037/0022-0663.95.2.240</bibtext> </blist> <blist> <bibtext> Romig, J. E., Therrien, W. J., & Lloyd, J. W. (2017). Meta‐analysis of criterion validity for curriculum‐based measurement in written language. The Journal of Special Education, 51, 72 – 82. https://doi.org/10.1177/0022466916670637</bibtext> </blist> <blist> <bibtext> Santangelo, T., Novosel, L., Cook, B. G., & Gapsis, M. (2015). Using the 6S Pyramid to identify research‐based instructional practices for students with learning disabilities. Learning Disabilities Research and Practice, 30, 91 – 101. https://doi.org/10.1111/ldrp.12055</bibtext> </blist> <blist> <bibtext> Swanson, H. L. (1999). Instructional components that predict treatment outcomes for students with learning disabilities: Support for a combined strategy and direct instruction model. Learning Disabilities Research & Practice, 14, 129 – 140.</bibtext> </blist> <blist> <bibtext> Swanson, H. L., & Hoskyn, M. (1998). Experimental intervention research on students with learning disabilities: A meta‐analysis of treatment outcomes. Review of Educational Research, 68, 277 – 321. https://doi.org/10.3102/00346543068003277</bibtext> </blist> <blist> <bibtext> Swanson, H. L., & Hoskyn, M. (2001). Instructing adolescents with learning disabilities: A component and composite analysis. Learning Disabilities Research and Practice, 16, 109 – 119. https://doi.org/10.1111/0938-8982.00012</bibtext> </blist> <blist> <bibtext> Swanson, L., & Sachse‐Lee, C. (2000). A meta‐analysis of single‐subject‐design intervention research for students with LD. Journal of Learning Disabilities, 33, 114 – 136. https://doi.org/10.1177/002221940003300201</bibtext> </blist> <blist> <bibtext> Swanson, H. L., Carson, C., & Saches‐Lee, M. C. (1996). A selective synthesis of intervention research for students with learning disabilities. School Psychology Review, 25, 370 – 391.</bibtext> </blist> <blist> <bibtext> Therrien, W. J., Taylor, J. C., Hosp, J. L., Kaldenberg, E. R., & Gorsh, J. (2011a). Science instruction for students with learning disabilities: A meta‐analysis. Learning Disabilities Research & Practice, 26, 188 – 203. https://doi.org/10.1111/j.1540-5826.2011.00340.x</bibtext> </blist> <blist> <bibtext> Therrien, W. J., Zaman, M., & Banda, D. R. (2011b). How can meta‐analyses guide practice? A review of the learning disability research base. Remedial and Special Education, 32, 206 – 218. https://doi.org/10.1177/0741932510361266</bibtext> </blist> <blist> <bibtext> Travers, J. C., Cook, B. G., & Cook, L. (2017). Null hypothesis significance testing and p‐values. Learning Disabilities Research & Practice, 32, 208 – 215. https://doi.org/10.1111/ldrp.12147</bibtext> </blist> </ref> <aug> <p>By William J. Therrien; Bryan G. Cook and Lysandra Cook</p> <p>Reported by Author; Author; Author</p> <p></p> <p>William J. Therrien holds the Thomas G. Jewell Professorship of Education at the University of Virginia. His research interests include academic instruction for individuals with cognitive disabilities, including students with learning disabilities and implementation of open science practices in special education research.</p> <p>Bryan G. Cook is a professor in the Special Education Program at the Curry School of Education and Human Development, and received his PhD in special education at the University of California at Santa Barbara. His primary lines of inquiry include open science, conducting meta‐research on the special education research base, and evidence‐based practices.</p> <p>Lysandra Cook is an associate professor of special education at the University of Virginia, earned her PhD from Kent State University. Her scholarly interests include evidence‐based practices, teacher preparation, bridging the research‐to‐practice gap, and effective coteaching at the post‐secondary level.</p> </aug>
Header DbId: eric
DbLabel: ERIC
An: EJ1263027
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Utilizing Meta-Analyses to Guide Practice: A Primer
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Therrien%2C+William+J%2E%22">Therrien, William J.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0594-5129">0000-0003-0594-5129</externalLink>)<br /><searchLink fieldCode="AR" term="%22Cook%2C+Bryan+G%2E%22">Cook, Bryan G.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9294-0873">0000-0001-9294-0873</externalLink>)<br /><searchLink fieldCode="AR" term="%22Cook%2C+Lysandra%22">Cook, Lysandra</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Learning+Disabilities+Research+%26+Practice%22"><i>Learning Disabilities Research & Practice</i></searchLink>. Aug 2020 35(3):111-117.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 7
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2020
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Information Analyses
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Meta+Analysis%22">Meta Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Guidance%22">Guidance</searchLink><br /><searchLink fieldCode="DE" term="%22Instructional+Effectiveness%22">Instructional Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Students+with+Disabilities%22">Students with Disabilities</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Disabilities%22">Learning Disabilities</searchLink><br /><searchLink fieldCode="DE" term="%22Literature+Reviews%22">Literature Reviews</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence+Based+Practice%22">Evidence Based Practice</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/ldrp.12230
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0938-8982
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Meta-analysis is one approach for synthesizing research studies to identify generally effective instructional practices for students with learning disabilities (LD). In this article, we define core components of meta-analytic literature reviews, discuss how to interpret findings from meta-analyses, and provide guidelines for how research consumers can utilize meta-analyses to guide instructional practices for students with LD. We conclude that meta-analyses provide valuable information for determining evidence-based practices for students with LD.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1263027
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1263027
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/ldrp.12230
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 7
        StartPage: 111
    Subjects:
      – SubjectFull: Meta Analysis
        Type: general
      – SubjectFull: Guidance
        Type: general
      – SubjectFull: Instructional Effectiveness
        Type: general
      – SubjectFull: Students with Disabilities
        Type: general
      – SubjectFull: Learning Disabilities
        Type: general
      – SubjectFull: Literature Reviews
        Type: general
      – SubjectFull: Evidence Based Practice
        Type: general
    Titles:
      – TitleFull: Utilizing Meta-Analyses to Guide Practice: A Primer
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Therrien, William J.
      – PersonEntity:
          Name:
            NameFull: Cook, Bryan G.
      – PersonEntity:
          Name:
            NameFull: Cook, Lysandra
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2020
          Identifiers:
            – Type: issn-print
              Value: 0938-8982
          Numbering:
            – Type: volume
              Value: 35
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Learning Disabilities Research & Practice
              Type: main
ResultId 1