View in EDS HTML Full Text PDF Full Text

Curriculum Reform in the Common Core Era: Evaluating Elementary Math Textbooks across Six U.S. States

Saved in:
Bibliographic Details
Title: Curriculum Reform in the Common Core Era: Evaluating Elementary Math Textbooks across Six U.S. States
Language: English
Authors: Blazar, David, Heller, Blake, Kane, Thomas J., Polikoff, Morgon, Staiger, Douglas O., Carrell, Scott, Goldhaber, Dan, Harris, Douglas N., Hitch, Rachel, Holden, Kristian L., Kurlaender, Michal
Source: Grantee Submission. 2020 39(4).
Peer Reviewed: Y
Page Count: 54
Publication Date: 2020
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305B150010
R305E150006
Document Type: Journal Articles
Reports - Evaluative
Education Level: Elementary Education
Descriptors: Curriculum Development, Common Core State Standards, Mathematics Materials, Achievement Gains, Textbook Selection, Elementary School Mathematics
DOI: 10.1002/pam.22257
Abstract: Can a school or district improve student achievement simply by switching to a higher quality textbook or curriculum? We conducted the first multi-textbook, multi-state effort to estimate textbook efficacy following widespread adoption of the Common Core State Standards (CCSS) and associated changes in the textbook market. Pooling textbook adoption and student test score data across six geographically and demographically diverse U.S. states, we found little evidence of differences in average achievement gains for schools using different math textbooks. We found some evidence of greater variation in achievement gains among schools using pre-CCSS editions, which may have been more varied in their content than post-CCSS editions because they were written for a broader set of standards. We also found greater variation among schools that had more exposure to a given text. However, these differences were small. Despite considerable interest and attention to textbooks as a low-cost, "silver bullet" intervention for improving student outcomes, we conclude that the adoption of a new textbook or set of curriculum materials, on its own, is unlikely to achieve this goal. [This article was published in "Journal of Policy Analysis and Management" (EJ1268634).]
Abstractor: As Provided
What Works Clearinghouse (WWC) Reviewed: Does Not Meet Evidence Standards
WWC Study Page: https://ies.ed.gov/ncee/wwc/Study/90550
IES Funded: Yes
Entry Date: 2021
Accession Number: ED614241
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHauwX6kl9Kj52_ckwbWk3rAAAA6DCB5QYJKoZIhvcNAQcGoIHXMIHUAgEAMIHOBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDK-ri6x4_E9PNfg2sQIBEICBoOtn53BjYEMwnAICoBvs5EmuJeJeWgCXRuiC0Xqy37Vqmw_M8-2O4l9k0Qw-Erxa7YOssyqlKNJQKLrZTAkd0a8XrK9I3beTth-C7_Sb9psyw_pzuGf1c9e4F35ZUBk5uYh5ocdB-Vw3Eucl0G2VBQaDrYXVXvgCut_5aNqFfU6fFt2rxny5O84Wos4S1DKcbRcf7ct_a_EK9zZ1KHcpOPU=
Text:
  Availability: 1
  Value: <anid>AN0146104504;jpa01sep.20;2020Sep29.05:53;v2.2.500</anid> <title id="AN0146104504-1">Curriculum Reform in The Common Core Era: Evaluating Elementary Math Textbooks Across Six U.S. States </title> <p>Can a school or district improve student achievement simply by switching to a higher‐quality textbook or curriculum? We conducted the first multi‐textbook, multi‐state effort to estimate textbook efficacy following widespread adoption of the Common Core State Standards (CCSS) and associated changes in the textbook market. Pooling textbook adoption and student test score data across six geographically and demographically diverse U.S. states, we found little evidence of differences in average achievement gains for schools using different math textbooks. We found some evidence of greater variation in achievement gains among schools using pre‐CCSS editions, which may have been more varied in their content than post‐CCSS editions because they were written for a broader set of standards. We also found greater variation among schools that had more exposure to a given text. However, these differences were small. Despite considerable interest and attention to textbooks as a low‐cost, "silver bullet" intervention for improving student outcomes, we conclude that the adoption of a new textbook or set of curriculum materials, on its own, is unlikely to achieve this goal.</p> <p>Keywords: textbooks, curriculum, Common Core State Standards; JEL: H75, I21, I24, I28</p> <hd id="AN0146104504-2">INTRODUCTION</hd> <p>The choice of textbook or curriculum is an enticing lever for improving student outcomes. Few central office decisions have such far‐ranging implications for the work that students and teachers do together in classrooms every day. In our own survey, we found that teachers in 94 percent of elementary schools in six geographically and demographically diverse U.S. states reported using the official district‐adopted textbook or curriculum in more than half of their lessons. Given such widespread usage, helping schools and districts switch from less to more effective materials offers a large potential return on investment (Kirst, 1982; Whitehurst, 2009). As Chingos and Whitehurst (2012) point out, "...whereas improving teacher quality...is challenging, expensive, and time consuming, making better choices among available instructional materials should be relatively easy, inexpensive, and quick" (p. 1).</p> <p>Textbook choice has been especially salient and has gained national policy attention in recent years after many states adopted the Common Core State Standards (CCSS), which generally are considered to be more rigorous than prior state standards (Friedberg et al., 2018). Curriculum reform is one of the primary mechanisms by which policymakers, practitioners, and researchers hypothesized that the introduction of the CCSS could improve student outcomes at scale (Carmichael et al., 2010; Porter et al., 2011). In the years since CCSS adoption, large publishing houses (e.g., Houghton Mifflin Harcourt, McGraw Hill, Pearson) have invested heavily in adapting existing textbooks and curriculum materials to new standards, and in writing new materials from scratch. New York State spent over $35 million dollars to develop a set of curriculum materials, <emph>Engage NY</emph>, which are now widely used across the country under this title and <emph>Eureka</emph> (Cavanaugh, 2015). Once new textbooks are written, the marginal cost to schools and districts of switching from one textbook to another is quite small. On average, elementary math textbooks cost roughly $35 per student, which represents less than 1 percent of per‐pupil expenditures (Boser, Chingos, & Straus, 2015). As of 2017, over 80 percent of the schools in our sample had adopted a CCSS‐edition textbook in elementary math.</p> <p>Despite the potential value to districts and schools, the research literature on the efficacy of alternative textbooks or curricula is sparse. We are aware of one multi‐textbook randomized trial (Agodini et al., 2010), two randomized trials assessing the effectiveness of a single textbook (Eddy et al., 2014; Jaciw et al., 2016), and a handful of non‐experimental studies that rely on matching techniques to estimate textbook effects (Bhatt & Koedel, 2012; Bhatt, Koedel, & Lehmann, 2013; Koedel et al., 2017). However, most of the textbook editions or curricula materials in common use today have never been subjected to a rigorous test of efficacy. Further, no studies have examined the sensitivity of textbook effects across time or across states. Although some textbook editions are written for local markets (e.g., California, Texas), logic suggests that a high‐quality curriculum or textbook should be effective across settings, especially when the materials are written to align with a common or similar set of standards. Yet, to our knowledge, no studies have assessed this claim empirically. Of the studies cited above, most are analyses of single districts or states. Two studies (Agodini et al., 2010; Eddy et al., 2014) recruited participants across states; but, schools and districts volunteered and so are not representative of those settings, let alone of U.S. states more broadly.</p> <p>One reason for the weakness of the evidence base is the historic diversity in state standards and assessments. When each state had its own standards and assessments, single‐state studies were relevant only for schools in a given state, and few states were sufficiently large to justify the cost of such an analysis. A second, more practical barrier has been the omission of textbook adoptions from state data collection efforts (Polikoff, 2018). As useful as textbook adoption data would be for estimating efficacy, states have concentrated their data collection efforts on fulfilling federal accountability requirements that focus on student test score performance rather than informing district purchasing decisions. States typically have stayed away from collecting data on curricula adoptions in deference to local authorities (Hutt & Polikoff, 2018). We are aware of only a handful of states that regularly collect information on the textbooks used by schools. As a result, it has been difficult to bring to bear states' longitudinal data on student achievement to compare the achievement gains of similar schools using different curricula.</p> <p>We designed our study as a field test of a replicable, low‐cost approach to measuring curriculum efficacy. Although experiments may be the most convincing way to estimate the causal effect of textbooks, the estimated impacts might not generalize beyond the small subset of schools that are willing to have their textbooks randomly assigned. Instead, by combining publicly available administrative data on textbooks in two states (California and New Mexico) with a survey administered to a random sample of schools in four additional states (Louisiana, Maryland, New Jersey, and Washington state), we ensured that we had state‐representative samples of schools and were studying the textbook editions that schools are using in the current CCSS era. By pairing textbook adoption data with student test‐score data on state‐administered CCSS‐aligned assessments, we also eliminated the need to collect our own assessments. Further, coordination amongst a large set of researchers across states—with different teams estimating the same "value‐added" model with student‐level data and then sharing only aggregated data—reduced the need to share sensitive data across state lines. In short, our methodology could be used to update results as textbook editions come and go.</p> <p>Although our value‐added methodology was the only way to examine textbook efficacy at scale, it comes with a trade‐off with regard to internal validity. We estimate the association between textbook adoption and aggregate student achievement gains, and so cannot account for all factors that led schools and districts to choose different textbooks (e.g., characteristics of school leadership). However, we were able to examine a model of textbook selection based on observable school and district characteristics. These models indicate that selection varies substantially across textbooks and states, and so is unlikely to lead to systematic bias. Intuition also suggests that it is highly unlikely that high‐ or low‐growth schools are systematically choosing the most effective textbooks when they (and we) do not know which textbooks those are. Empirically, we found that results were robust to models with different sets of school‐ and district‐level controls, as well as models that replaced observable characteristics with school fixed effects that account for fixed differences across schools.</p> <p>Unlike the prior literature, we found little evidence of substantial differences in average math achievement gains in schools using different textbooks. Our null findings are similar for schools with varied student populations related to English Language Learner status, free or reduced‐price lunch eligibility, and high‐ or low‐baseline achievement. Neither did we find textbook effects in the subset of schools in which teachers reported above‐ or below‐median levels of textbook usage, or where teachers received above‐ or below‐median levels of support to implement the textbook. We found some evidence of greater variation in achievement gains among schools using pre‐CCSS editions, which may have been more varied in their content than post‐CCSS editions because they were written for a broader set of standards. We also found greater variation among schools that had more exposure to a given text. In both of these latter subgroups, though, effects were smaller than average effects identified in prior studies.</p> <p>In our conclusion, we attempt to reconcile our findings with the prior literature, which did find meaningful differences in efficacy across textbooks. We discuss the role of possible biases in our value‐added methodology, the limited generalizability of the randomized controlled trials relative to our multi‐state analysis, the role of implementation, and the possible greater uniformity in textbook content following the CCSS. Ultimately, though, we conclude that the adoption of a new textbook or set of curricula materials, on its own, is unlikely to improve student outcomes at scale and in the way that some policymakers, practitioners, and researchers have suggested and hoped.</p> <hd id="AN0146104504-3">MOTIVATION</hd> <p></p> <hd id="AN0146104504-4">Textbooks as a Lever for Efficiency</hd> <p>Textbooks and curriculum materials comprise a roughly $10 billion‐per‐year industry (Boser, Chingos, & Straus, 2015; McFarland et al., 2017), driven largely by their widespread use across K‐12 public schools. In a nationally representative survey of schools focused on teachers' instructional decisions in the CCSS era, Opfer, Kaufman, and Thompson (2016) found that 98 percent of elementary math teachers reported using instructional materials selected or developed by district leadership. Roughly 85 percent of teachers reported that their districts required (57 percent) or recommended (27 percent) that they use specific textbooks to teach mathematics. Given such widespread adoption across classrooms and the small marginal difference in cost between curricula (Boser, Chingos, & Straus, 2015), textbook selection has been described as a uniquely powerful lever for reform (Chingos & Whitehurst, 2012; Kirst, 1982; Whitehurst, 2009).</p> <p>Textbook selection matters for many reasons, but perhaps most important is the extent to which the choice between one text versus another is likely to influence student learning. A broad theoretical literature from multiple disciplinary perspectives theorize critical links between curricula and desired student outcomes (e.g., Altonji, Blom, & Meghir, 2012; Schmidt et al., 2001; Stein, Remillard, & Smith, 2007), highlighting in particular the role of content (<emph>what</emph> material is presented) and structure (<emph>how</emph> that material is presented). For example, in mathematics, which is the focus of the current study, the "math wars" of the last several decades resulted in new learning standards (i.e., National Council of Teachers of Mathematics, 1989, 1991, 2000) focused on conceptual rather than procedural instruction (Schoenfeld, 2004), and new textbooks and curriculum materials that align with this approach (Reys, 2001). Reform‐oriented materials that focus on higher‐order thinking typically have competed with more conventional resources, focused on teaching standard algorithms in sequence and using these procedures to solve basic problems. Debates regarding the design of textbooks typically have occurred in the educator practice community, but align with a broader push for schools to help students build the sorts of non‐routine problem‐solving skills that increasingly are valued in the labor market (Autor, Levy, & Murnane, 2003).</p> <p>Current state‐adopted math content standards aligned to the CCSS and associated textbooks are the most recent iteration of longstanding efforts to increase the academic rigor of instruction in U.S. classrooms and, in turn, student outcomes. One desired consequence of the CCSS, as a national policy initiative, is that widespread adoption of the standards across U.S. states would drive the textbook market to be uniformly more rigorous in terms of the curricula available to schools, teachers, and students (Heck, Weiss, & Pasley, 2011; Porter et al., 2011). The evidence to date is mixed. A recent study (Polikoff, 2015) used a prominent alignment methodology, the Surveys of Enacted Curriculum (SEC), to examine the content and standards alignment of several pre‐ and post‐CCSS mathematics textbooks, all of which show up in our sample. That study found moderate overall levels of alignment of textbooks to state standards or to the CCSS. While the textbooks were more similar to each other (including pre‐ and post‐CCSS editions of the same series) than they were to standards, they varied in terms of their relative emphasis on different domains of math content, and on levels of rigor and cognitive demand. That study is the only recent research evidence on the alignment of textbooks to each other or to standards. SEC research provides detailed alignment evidence but is limited by the research burden of analyzing entire textbooks from cover to cover.</p> <p>Because reviewing, analyzing, and comparing the entire content of each textbook in our sample was beyond the scope of this project, we instead relied on publicly available reviews conducted by the nonprofit organization EdReports (see https://<ulink href="http://www.edreports.org/">www.edreports.org/</ulink>). EdReports engages with educators to conduct analyses of textbook series that are meant to align to the CCSS regarding their focus and coherence, rigor and mathematical practices, and usability. As we illustrate in our results below, EdReports identifies variation in these metrics across the most commonly used CCSS‐edition math textbooks in our six partner states. These differences suggest that there may also be differences in student outcomes across schools using different textbooks.</p> <hd id="AN0146104504-5">Prior Evidence on Textbook Efficacy</hd> <p>Despite a strong theoretical basis linking textbooks to student learning, limited evidence exists on this topic. Of the 38 textbooks we observe in our sample, only six have been evaluated in a manner meeting the highest evidence standards (i.e., experiments, regression discontinuity designs) of the What Works Clearinghouse (WWC), a federal repository for education research. Only four of these are among the top 15 most commonly used textbooks in our sample, and only two were textbook editions meant to align with the CCSS. Thus, in the vast majority of cases, districts must select curricula in the absence of evidence of efficacy, relying instead on the judgments of central office staff, selection committees, and the choices of neighboring districts (Polikoff et al., 2020).</p> <p>We discuss the past research on math textbook efficacy in two broad categories: randomized trials and non‐experimental studies.</p> <hd id="AN0146104504-6">Randomized Trials</hd> <p>To our knowledge, only one study has used a randomized design to compare the impact of multiple elementary math textbooks on student achievement. Agodini et al. (2010) randomly assigned one of four curricula to 1st‐ and 2nd‐grade classrooms in 111 schools in 12 districts across 10 U.S. states. The study began in the 2006/2007 school year, prior to the development of the CCSS, and so none of the texts were designed to align with these standards. However, all texts have since been updated and now are marketed as aligned to the CCSS. These texts include: <emph>Investigations in Mathematics</emph> and <emph>Math Expressions</emph>, which generally are considered to be reform‐oriented with a focus on building conceptual understanding; and <emph>Saxon Math</emph> and <emph>enVision Math</emph>, which are described as more conventional and focused on teacher‐directed instruction of procedures (Remillard, Harris, & Agodini, 2014; Stein, Remillard, & Smith, 2007). Teachers received two to three days of training on the assigned textbook over the course of the study. Second‐grade classrooms using <emph>Math Expressions</emph> or <emph>Saxon Math</emph> outperformed those using <emph>enVision</emph> by 0.12 standard deviations (SD) and 0.17 SD, respectively. These effect sizes are large relative to the majority of educational interventions (Fryer, 2017). They would be larger than the effect of having an experienced versus a novice teacher (roughly 0.08 SD) and roughly equivalent to a 1 SD increase in teacher efficacy (Rockoff, 2004).</p> <p>Two other studies used randomized designs to study the effect of individual curricula, both meant to align to the CCSS. Jaciw et al. (2016) evaluated the effectiveness of the post‐CCSS edition of <emph>Math in Focus</emph> (there also is a pre‐CCSS edition) by randomly assigning 22 clusters of 3rd‐ through 5th‐grade teachers in 12 schools in Clark County School District (Las Vegas) in Nevada during the 2011/2012 school year. This text is modeled on a Singaporean math curriculum, emphasizing student‐directed problem solving. Teachers attended a short training session (1.5 to 3 hours) during the summer and four half‐ or full‐day sessions throughout the school year. The authors found that, after the first year of usage, students in grade‐level teams randomly assigned to adopt <emph>Math in Focus</emph> outperformed students in the control group, who used the math curricula already in place in their school, by 0.11 to 0.15 SD on the Stanford Achievement Test. However, the study team found no impact of <emph>Math in Focus</emph> on the criterion‐referenced test required by the state of Nevada.</p> <p>Eddy et al. (2014) randomly assigned <emph>Go Math</emph> to 1st‐ through 3rd‐grade classrooms in nine schools across seven states during the 2012/2013 school year. <emph>Go Math</emph> was written for the CCSS and aims to balance conceptual and procedural understanding. After one year, the authors did not find statistically significant differences in average student achievement in classrooms using <emph>Go Math</emph>, as compared to control classrooms using the mathematics program already in place in their schools. However, the study was underpowered to detect effects smaller than 0.2 SD.</p> <hd id="AN0146104504-7">Non‐Experimental Studies</hd> <p>A handful of non‐experimental studies have identified moderate effects of textbooks on student achievement gains (upwards of 0.14 SD). Koedel and co‐authors used matching methods and school‐level aggregate achievement to measure textbook efficacy in three states: California, Florida, and Indiana (Bhatt & Koedel, 2012; Bhatt, Koedel & Lehmann, 2013; Koedel et al., 2017). Given the large number of texts observed in California and Florida, the analysts used a two‐step process in these states. They first identified a differentially effective text based on an initial exploratory analysis, and then compared that text against a composite comparison group. Although this process helps to narrow the focus of inquiry, the danger is that the initial exploration may identify the "winning" text due to chance differences in achievement. The authors were careful to conduct a number of validity tests in the second step—e.g., verifying that the timing of any achievement increase aligned with the textbook adoption and that achievement did not grow in English Language Arts (ELA). Yet, such tests would not necessarily reveal a within‐sample anomaly.</p> <p>Two textbooks appear in both the set of non‐experimental studies and the randomized trials, yet with different conclusions about their relative efficacy. While <emph>Saxon Math</emph> outperformed <emph>enVision</emph> in the multi‐textbook randomized trial (Agodini et al., 2010), the pattern was switched in one non‐experimental matching study (Bhatt & Koedel, 2012). These differences in ranking might be due to the methodology used to estimate textbook efficacy, or to differences in generalizability. As Bhatt and Koedel (2012) discuss, <emph>Saxon Math</emph> is a highly scripted curriculum and was designed for implementation in schools where the teachers have weak math backgrounds, the very type of schools that were willing to have their textbook randomized.</p> <hd id="AN0146104504-8">DATA COLLECTION</hd> <p>Ours is the first multi‐state, multi‐textbook study in the CCSS era, and aims to provide generalizable knowledge on the efficacy of math textbook editions in common use today. To achieve this goal, we assembled data from over 6,000 schools across six states: California, Louisiana, Maryland, New Jersey, New Mexico, and Washington. Our collaboration with these six states was strategic. At the time of our study, all six had adopted the CCSS; they also had ready access to administrative data on textbook adoptions or agreed to take part in our survey. Nonetheless, the states are geographically and demographically diverse, capturing variation across the U.S. As shown in Appendix Table A1, three states (Maryland, New Jersey, and Washington) scored above the national average on the National Assessment of Educational Progress (NAEP), while the other three states (California, Louisiana, and New Mexico) scored below average. Two states (Maryland and New Jersey) also had above‐average instructional expenditures and household income, while other states (Louisiana and New Mexico) fell below the national average on both measures. All six states were similar with regard to the percent of students eligible to receive special education services. California and New Mexico had an above‐average share of Hispanic students and English Language Learners, while Louisiana and Maryland had an above‐average share of African‐American students.</p> <p>We used the Common Core of Data (CCD) to construct state‐specific sampling frames of public schools enrolling both 4th‐ and 5th‐grade students between the 2014/2015 school year (the first year of testing using new CCSS‐aligned assessments) and the 2016/2017 school year (the year we began data collection). Focusing on upper‐elementary grades ensured that students across all six partner states had prior‐year test scores. We also hypothesized that we would be more likely to see effects of textbooks in elementary grades rather than middle or high school, given that students, on average, tend to make larger academic gains in earlier rather than later grades (Hill et al., 2008). We included public charter schools but excluded private schools. We also limited the sampling frame to schools with test score data for more than 10 students, as many of our data use agreements made this a requirement for security and confidentiality purposes.</p> <p>Within this sampling frame, we assembled three types of data. First is textbook adoption data, which came both from administrative records covering close to full populations in two states (California and New Mexico) and a project‐administered survey to a representative sample of schools in the other four states (Louisiana, Maryland, New Jersey, and Washington). Second is a survey administered to teachers in a representative sample of schools using one of the seven most frequently used CCSS‐edition curricula, which asked teachers about their use of the textbook. In Table 1, we provide a summary of the state sampling frames and subsequent samples generated for each of these two sources of data. The third data source is state‐collected information on students (excluded from Table 1, as the data were not sampled), including demographics and test scores on CCSS‐aligned assessments. We describe each of the three sources of data in turn below.</p> <p>1 TableSample of schools and teachers</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th /><th align="center">Textbook Adoption Data</th><th align="center">Teacher Survey Data</th></tr><tr valign="bottom"><th>States (by Source of Textbook Data)</th><th align="center"># Schools in Sampling Frame</th><th align="center"># Sampled Schools</th><th align="center"># Schools with Textbook Data</th><th align="center"># Originally Sampled Schools (Eligible Teachers)</th><th align="center"># Sampled Schools (Eligible Teachers) after Replacement</th><th align="center"># Schools (Teachers) with Survey Data</th></tr></thead><tbody><tr><td>A. Administrative Data States</td></tr><tr><td>California</td><td>5,841</td><td align="center">N/A</td><td>5,107</td><td>95 (479)</td><td>150 (727)</td><td>100 (324)</td></tr><tr><td>New Mexico</td><td>439</td><td align="center">N/A</td><td>297</td><td>24 (94)</td><td>32 (116)</td><td>24 (67)</td></tr><tr><td>B. Sampled States</td></tr><tr><td>Louisiana</td><td>668</td><td>192</td><td>161</td><td>42 (150)</td><td>58 (218)</td><td>38 (79)</td></tr><tr><td>Maryland</td><td>853</td><td>139</td><td>121</td><td>23 (126)</td><td>34 (190)</td><td>15 (71)</td></tr><tr><td>New Jersey</td><td>1,146</td><td>427</td><td>322</td><td>116 (661)</td><td>134 (774)</td><td>107 (434)</td></tr><tr><td>Washington</td><td>1,054</td><td>316</td><td>247</td><td>68 (292)</td><td>78 (345)</td><td>61 (220)</td></tr><tr><td>Full Sample of Schools (Teachers)</td><td>10,001</td><td>1,074</td><td>6,255</td><td>368 (1,802)</td><td>486 (2,370)</td><td>345 (1,195)</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Notes</emph>: The sampling frame includes public schools that enrolled 4th‐ and 5th‐grade students between the 2014/2015 and 2016/2017 school years, and had achievement data for 10 or more students. Each school is counted once, even though most were observed in more than one school year; we leverage school‐year observations in subsequent analyses. In California and New Mexico, textbook adoption data come from administrative records (school report cards in California, and purchasing data in New Mexico) and so cover close to full populations. "N/A" indicates that schools were not sampled in these states to collect textbook adoption data. In the remaining four states, textbook adoption data come from a survey administered to a representative sample of schools within the sampling frame. Schools were randomly selected using a two stage process in which we first sampled districts within states (all school districts with two or more schools in our sampling frame and half of the districts with just one school in our sampling frame). Then, we sampled schools within districts with the total number of schools proportional to district size. For the teacher survey, in all six states we randomly sampled a subset of schools for whom we had textbook adoption data and that were identified as using one of the top seven CCSS‐edition textbooks by market share (see Table 3). The final sample includes 50 to 60 schools using each of these seven texts. The target number of schools using a given textbook per state was proportional to the distribution of that textbook across states, with the California sample downweighted by a factor of three. We randomly selected schools within strata based on textbook, state, and a median split of the proportion of students eligible for free or reduced‐price lunch. We allowed for sampling with replacement, and so show the orignially sampled set of schools and the full sample after replacement. The final teacher survey sample records all valid reponses that successfully merged to state test‐score records.</p> <hd id="AN0146104504-9">Textbook Adoptions</hd> <p>In two of our partner states, textbook adoption data came from administrative records. In California, state law requires that schools report on textbook adoptions each year as part of school accountability report cards (Hutt & Polikoff, 2018). Our raw data came from reports hosted on the California Department of Education website, in which schools reported the title of textbooks they adopted by subject and grade (see Koedel et al., 2017, for additional information). Those data allowed us to identify the math textbook adopted for 87 percent of elementary schools in our sampling frame (<emph>n</emph> = 5,107 of 5,841 schools; see Table 1). Given the state mandate, very few schools were missing these reports. Instead, our inability to identify the math textbook for 13 percent of schools in our California sampling frame was due to the quality of information provided (e.g., reporting the name of a publisher rather than the textbook title). Given the lag in school‐level reporting and state‐level release of these data, we captured textbook adoptions in 2014/2015 and 2015/2016 but not in 2016/2017.</p> <p>In New Mexico, the second administrative data state, we relied on purchasing data to capture textbook adoptions. Schools received a discount when they purchased textbooks from a centralized warehouse. Likely as a result, we observed textbook orders for a large share (68 percent) of public elementary schools in our New Mexico sampling frame (<emph>n</emph> = 297 of 439 schools; see Table 1). Presumably, schools not found in the purchasing records did not use a regular textbook or accessed textbooks or curricula from other sources, including open educational resources available online that would not be found in a physical warehouse. The purchasing data, available for 2010 through 2017, included ISBNs, textbook titles, quantity purchased, and school addresses. To distinguish between one‐off purchases and official textbook adoptions, we limited our analysis to instances where the number of texts purchased was at least 50 percent of 4th‐ and 5th‐grade enrollment in a given school and year. If we observed this sort of qualifying textbook purchase between 2010 and 2017, we imputed that data to future years, until/unless we observed a subsequent qualifying purchase. Of the 67 teachers from New Mexico whose schools also were randomly selected to participate in our teacher survey (see below), 91 percent indicated that the textbook identified for their school from the purchasing data was correct.</p> <p>In the four remaining states (Louisiana, Maryland, New Jersey, and Washington), we surveyed schools in the winter of the 2016/2017 school year in order to identify the textbooks they adopted that year and the two years prior. We took a multi‐step process to select schools in order to account for the fact that schools within most districts share a single text (see Koedel et al., 2017, and our own data). If we took a one‐step process to sample schools proportional to their enrollments, we would have ended up with a large number of urban schools using a small number of texts. Instead, we first selected a random sample of districts within each state, stratified by the number of elementary schools (i.e., districts with just one school that fit our sampling frame criteria, two to three schools, four to seven schools, eight or more schools) and the percentage of students receiving free or reduced‐price lunches (above and below the state median). By state, we randomly selected half of the one‐school districts as part of our sample; we selected all districts with two or more schools. We then sampled one school from our sampling frame from districts with two to three schools, and two schools from districts with four to seven schools. In districts with eight or more elementary schools, each school had a 20 percent chance of being selected. Under this schema, we arrived at a total sample of 1,074 schools in our sample across the four survey states (see Table 1).</p> <p>To recruit these randomly selected schools to participate in our study, we sent an electronic version of the textbook adoption survey to the principal or curriculum coordinator in each school, also providing a description of the project and a promise of a $50 gift card for successful completion of the survey. We followed up with an e‐mail and called the school every 10 days until we received a response. We also contacted district personnel to help with outreach. After multiple rounds of follow‐up over five months, we achieved response rates of 79 percent overall and between 75 and 87 percent for each state (derived from numbers in Table 1). We attribute these high response rates to our multi‐pronged approach, and to support for our study from state and district leaders who helped ensure that school personnel saw and responded to our e‐mails or phone calls.</p> <p>In Table 2, we compare observable school and district characteristics for schools with and without textbook data. In California and New Mexico, where we relied on administrative records, we see some differences between schools with and without textbook data. In California, the schools without textbook data had higher percentages of special education students, higher per‐pupil expenditures, and lower baseline math achievement. However, overall coverage of schools is high (87 percent). The large sample of schools in California also makes us more likely to detect statistically significant differences even when the magnitude of those differences is small. In New Mexico, the schools without textbook data had somewhat lower percentages of students eligible for free or reduced‐price lunch, but other differences were small or not statistically significant. In the states where we conducted our own survey, the non‐respondents differed from respondents on only one out of 17 characteristics. Though not shown in Table 2, our randomly selected sample of schools in these four states looks similar to the larger sampling frame, as we expected given the randomized sampling process. On observable measures, the schools in our analyses generally are representative of their states.</p> <p>2 TableComparing schools with and without textbook adoption data</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">Administrative Data States</th><th align="center">Sampled States</th></tr><tr valign="bottom"><th /><th align="center">California</th><th align="center">New Mexico</th><th align="center">Louisiana, Maryland, New Jersey, and Washington</th></tr><tr valign="bottom"><th>School or District Characteristics</th><th align="center">Schools with Known Textbook</th><th align="center">Difference for Schools with Unknown Textbook</th><th align="center">Schools with Known Textbook</th><th align="center">Difference for Schools with Unknown Textbook</th><th align="center">Respondents</th><th align="center">Difference for Non‐Respondents</th></tr></thead><tbody><tr><td>Free or Reduced‐Price Lunch (%)</td><td>62.0</td><td>−3.90001</td><td>84.3</td><td>−10.60001</td><td>50.5</td><td>3.5</td></tr><tr><td>Special Education (%)</td><td>12.6</td><td>5.60001</td><td>17.6</td><td>−0.9</td><td>14.5</td><td>0.0</td></tr><tr><td>English Language Learner (%)</td><td>24.2</td><td>−5.90001</td><td>16.1</td><td>−4.00001</td><td>605</td><td>1.5</td></tr><tr><td>Male (%)</td><td>51.4</td><td>2.50001</td><td>50.7</td><td>0.2</td><td>51.3</td><td>−0.2</td></tr><tr><td>African American (%)</td><td>5.8</td><td>0.5</td><td>1.6</td><td>0.0</td><td>22.7</td><td>1.6</td></tr><tr><td>Asian (%)</td><td>10.5</td><td>−3.40001</td><td>0.9</td><td>0.1</td><td>6.4</td><td>−2.10001</td></tr><tr><td>Hispanic (%)</td><td>52.4</td><td>−9.40001</td><td>63.0</td><td>−5.7</td><td>17.4</td><td>5.1</td></tr><tr><td>Native American (%)</td><td>0.9</td><td>0.40001</td><td>11.8</td><td>−1.9</td><td>0.6</td><td>0.5</td></tr><tr><td>Mixed Race or Other (%)</td><td>3.5</td><td>0.40001</td><td>1.9</td><td>0.2</td><td>3.1</td><td>0.0</td></tr><tr><td>Prior Math Test Score (Standardized)</td><td>−0.026</td><td>−0.0890001</td><td>−0.023</td><td>0.027</td><td>−0.032</td><td>−0.09</td></tr><tr><td>Prior Reading Test Score (Standardized)</td><td>−0.021</td><td>−0.049</td><td>−0.017</td><td>0.042</td><td>−0.024</td><td>−0.059</td></tr><tr><td>Per‐pupil Instructional Expenditure ($)</td><td align="center">5,859</td><td align="center">1,1790001</td><td align="center">5,505</td><td align="center">−108</td><td align="center">7,956</td><td align="center">400001</td></tr><tr><td>Parents Married (%)</td><td>65.3</td><td>0.4</td><td>55.4</td><td>2.2</td><td>62.2</td><td>−1.0</td></tr><tr><td>Speaks Language Other than English (%)</td><td>45.2</td><td>−4.60001</td><td>33.8</td><td>−3.9</td><td>20.0</td><td>1.2</td></tr><tr><td>Median Household Income ($)</td><td align="center">62,380</td><td align="center">1,197</td><td align="center">46,444</td><td align="center">4,8330001</td><td align="center">70,898</td><td align="center">−142</td></tr><tr><td>Parent Attended College, no BA (%)</td><td>29.1</td><td>1.0</td><td>33.0</td><td>2.90001</td><td>30.2</td><td>1.10001</td></tr><tr><td>Parent Holds BA Degree + (%)</td><td>23.7</td><td>1.2</td><td>19.9</td><td>−1.3</td><td>30.1</td><td>−2.10001</td></tr><tr><td>P‐Value from Test of Joint Significance</td><td /><td>0.000</td><td /><td>0.000</td><td /><td>0.097</td></tr><tr><td>School Observations</td><td align="center">5,107</td><td align="center">734</td><td align="center">297</td><td align="center">142</td><td align="center">851</td><td align="center">223</td></tr></tbody></table> </ephtml> </p> <ulist> <item>2 <emph>Notes</emph>: For states where schools were randomly sampled, estimates are weighted by the inverse of a school's sampling probability. Expenditure data and family characteristics come from the 2010 to 2014 American Community Survey, captured at the district level. Other characteristics are captured at the school level.</item> <item>3 <sups>∼</sups>p <.10; *p <.05; **p <.01; ***p <.001.</item> </ulist> <p>We briefly describe the distribution of textbooks across schools in each state, which motivates our sampling strategy for a subsequent teacher survey. In Table 3, we show the percent of schools in each state using specific texts. Because some schools switched textbooks across years, we calculated means in a school‐year dataset. However, we weighted estimates by the inverse of the probability that an individual school was selected for the sample. In this and subsequent tables, we use "CC" to refer to textbooks that were meant to align to the CCSS. Some titles (e.g., <emph>enVision</emph>) are listed twice—with and without "CC"—because they originally were written prior to the rollout of the CCSS and then adapted in some way to align to the new standards.</p> <p>3 TableTextbook market share by state</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Textbook (Sorted by Share in Pooled Sample)</th><th align="center">Pooled 6 States</th><th align="center">California</th><th align="center">Louisiana</th><th align="center">Maryland</th><th align="center">New Jersey</th><th align="center">New Mexico</th><th align="center">Washington</th></tr></thead><tbody><tr><td>enVision CC (% share)</td><td>14.8</td><td>11.5</td><td>15.1</td><td>14.7</td><td>27.7</td><td>28.4</td><td>7.9</td></tr><tr><td>Engage NY/Eureka CC</td><td>12.7</td><td>8.8</td><td>58.8</td><td>16.6</td><td>1.4</td><td>0.0</td><td>19.3</td></tr><tr><td>Go Math CC</td><td>12.5</td><td>17.3</td><td>10.2</td><td>1.5</td><td>19.1</td><td>2.6</td><td>0.6</td></tr><tr><td>My Math CC</td><td>12.0</td><td>17.1</td><td>8.3</td><td>2.1</td><td>6.0</td><td>29.4</td><td>3.3</td></tr><tr><td>enVision</td><td>7.8</td><td>13.7</td><td>0.9</td><td>4.9</td><td>1.2</td><td>0.0</td><td>1.6</td></tr><tr><td>Math Expressions CC</td><td>7.3</td><td>9.7</td><td>0.4</td><td>0.0</td><td>2.1</td><td>0.0</td><td>16.0</td></tr><tr><td>Everyday Mathematics CC</td><td>4.6</td><td>3.6</td><td>0.0</td><td>0.5</td><td>15.7</td><td>5.5</td><td>1.1</td></tr><tr><td>Everyday Mathematics</td><td>2.7</td><td>3.5</td><td>0.0</td><td>3.4</td><td>3.1</td><td>0.2</td><td>1.2</td></tr><tr><td>Houghton Mifflin Math</td><td>2.3</td><td>4.5</td><td>0.0</td><td>0.0</td><td>0.2</td><td>0.0</td><td>0.0</td></tr><tr><td>Math in Focus CC</td><td>2.2</td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.8</td><td>1.4</td><td>4.0</td></tr><tr><td>Bridges in Mathematics CC</td><td>2.2</td><td>1.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>9.5</td></tr><tr><td>Stepping Stones CC</td><td>2.0</td><td>0.3</td><td>0.0</td><td>0.9</td><td>0.0</td><td>31.1</td><td>2.1</td></tr><tr><td>Ready Common Core CC</td><td>1.7</td><td>0.1</td><td>1.6</td><td>14.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Math Connects</td><td>1.2</td><td>0.0</td><td>0.0</td><td>2.1</td><td>1.2</td><td>0.0</td><td>5.8</td></tr><tr><td>Math Expressions</td><td>1.0</td><td>0.2</td><td>0.0</td><td>0.0</td><td>0.2</td><td>0.0</td><td>7.1</td></tr><tr><td>Other Known Textbooks</td><td>6.4</td><td>7.9</td><td>2.0</td><td>0.9</td><td>3.6</td><td>1.2</td><td>12.4</td></tr><tr><td>District Developed</td><td>4.1</td><td align="center">N/A</td><td>0.4</td><td>31.0</td><td>2.3</td><td align="center">N/A</td><td>2.4</td></tr><tr><td>No Consistent Textbook</td><td>2.5</td><td align="center">N/A</td><td>2.3</td><td>7.3</td><td>5.5</td><td align="center">N/A</td><td>5.7</td></tr><tr><td>School*Year Observations (All Categories)</td><td align="center">11,816</td><td align="center">8,711</td><td align="center">310</td><td align="center">352</td><td align="center">944</td><td align="center">832</td><td align="center">667</td></tr><tr><td>School*Year Observations (All Known Textbooks)</td><td align="center">11,516</td><td align="center">8,711</td><td align="center">298</td><td align="center">195</td><td align="center">874</td><td align="center">832</td><td align="center">606</td></tr><tr><td>School*Year Observations (Top 15 Textbooks)</td><td align="center">10,797</td><td align="center">8,121</td><td align="center">292</td><td align="center">191</td><td align="center">840</td><td align="center">822</td><td align="center">531</td></tr></tbody></table> </ephtml> </p> <p>4 <emph>Notes</emph>: For states where schools were randomly sampled (Louisiana, Maryland, New Jersey, and Washington), estimates are weighted by the inverse of a school's sampling probability. When pooling data across states, schools in California and New Mexico, which were not sampled, are given a sampling probaility of 1. While our sampling approach selected individual schools (see Table 1), here we use a school‐year dataset given that some schools switched textbooks across years. The school‐year sample sizes listed at the bottom of the table are those used in later analyses. In California and Louisiana, we have at most two years of data on each school due to availability either of textbook adotion data (California) or state test‐score data (Louisiana). In the other four states, we have at most three years of data on each school. "CC" indicates that a given textbook was written for or adapted to the CCSS standards. Some series titles are listed twice because they were written prior to the CCSS and then updated in some way after the rollout of the policy. The "other known textbooks" category includes 23 texts, each used by a small number of schools (in some instances, by just one school). We include all district‐developed curricula in a single category, despite variation in materials across districts. "N/A" indicates that the nature of the administrative records in California and New Mexico could not identify the categories listed in these rows (i.e., district‐developed materials, no consistent textbook).</p> <p>Despite the large number of options available (38 in our sample), we find that the elementary math textbook market remains dominated by a small number of texts. Over 70 percent of elementary schools in the six states used one of seven texts, and roughly 90 percent used one of 15 texts. Nevertheless, the market share for a particular curriculum varied by state. For instance, in New Mexico, the market was fairly split between three textbook series, all written for or adapted to the CCSS. Comparatively, in Louisiana, almost 60 percent of schools used just one open‐source curriculum (<emph>Engage NY</emph>/<emph>Eureka Math</emph>) that was written specifically for the CCSS. In California, a sizeable share of school‐year observations were attached to a non‐CCSS edition textbook, driven largely by the Los Angeles Unified School District, where schools generally updated to a CCSS‐edition text during the time frame of our study. In Maryland, over 30 percent of schools reported using materials developed by districts, rather than materials developed by outside publishers.</p> <hd id="AN0146104504-10">Teacher Survey</hd> <p>To gain insight into teacher use of adopted textbooks, we conducted a teacher survey asking about their frequency of textbook use for different activities (e.g., preparation of lessons, classroom assignments), use of supplementary materials (e.g., those found online, developed by the teacher), reasons for supplementation, and access to professional development. We administered the teacher survey in the fall of 2017 but asked teachers about activities and practices in the prior academic year (i.e., the year in which we administered the school survey).</p> <p>Given our interest in the CCSS textbook market, we focused teacher survey data collection on a subset of schools that reported using one of the seven most commonly adopted CCSS‐edition texts (see Table 3). Four of these—<emph>enVision</emph>, <emph>Everyday Mathematics</emph>, <emph>Math Expressions</emph>, and <emph>Math in Focus</emph>—were in press prior to the rollout of the CCSS and then were adapted to align to the new standards. The other three—<emph>Engage NY/Eureka, Go Math</emph>, and <emph>My Math</emph>—were written specifically for the CCSS. We excluded three other CCSS‐edition textbooks that were part of the top 15 textbooks by market share because their use was dominated by schools in just one rather than multiple states. Our target was to receive surveys from 50 to 60 schools using each of these seven texts, for a total of 350 to 420 schools.</p> <p>To begin the sampling process, we identified a target number of schools per textbook such that the distribution of textbooks across states in the teacher survey sample was proportional to the distribution in our larger sample. For example, roughly 25 percent of schools in our full sample that reported using <emph>enVision</emph> (CCSS edition) came from New Jersey, and so roughly 25 percent of the schools using this textbook sampled for the teacher survey came from this state. In making these calculations, we downweighed the California sample by a factor of three, given that this state was larger than all others combined and would have led to a teacher survey sample that was dominated by this state. In analyses, we adjusted our estimates for this sampling design.</p> <p>Next, we randomly selected that number of schools, stratifying on textbook, state, and an indicator of whether they were above or below the median percentage of students receiving free or reduced‐price lunch within each state. We allowed for sampling with replacement in instances where districts or schools turned down our request to survey teachers, where we were unable to identify the roster of 4th‐ and 5th‐grade teachers in a particular school, or where no teachers in the school responded within a three‐month period. When a school needed to be replaced, we picked the next school on the randomly sorted list that reported using the same textbook, was in the same income bracket, and (where possible) was in the same state. If we selected a school for participation, we searched the school's website or called the school office for contact information on all 4th‐ and 5th‐grade teachers from the 2016/2017 school year. We provided a survey link by e‐mail and a $30 gift card for each respondent. We stopped replacing schools once we met our target of 50 to 60 schools per textbook that had at least one teacher complete our survey. We began with an original randomly selected sample of 368 schools (1,802 eligible teachers within these schools), and added 118 schools as part of our replacement strategy (see Table 1). Of the 486 total schools and 2,370 eligible teachers we contacted for this portion of the project, 1,195 teachers from 345 schools completed the survey and were successfully linked to administrative records (see Table 1). These numbers represent a school‐level response rate of 71 percent and a teacher‐level response rate of 50 percent.</p> <p>While the teacher‐level response rate is comparable to other recent surveys with a similar focus (e.g., Opfer, Kaufman, & Thompson, 2016), it is lower than desired to draw generalizable conclusions about individual teachers. However, we remind readers that, given our sampling design, the primary unit of analysis is the school, where coverage was substantially higher. Low within‐school response rates could lead to measurement error (and possible bias) about practices in those schools. As such, we view our analyses of the teacher survey data as exploratory. At the same time, patterns of textbook use were similar when we analyzed the teacher‐level data versus school‐level data that averaged teacher responses to the school level, indicating that variation in within‐school response rates is unlikely to be a first‐order concern. Further, in analyses where we used the teacher survey data to examine textbook effects across subgroups of schools (e.g., in schools with above‐ versus below‐median levels of use), we found that results were similar when limiting the sample to schools with high teacher response rates.</p> <hd id="AN0146104504-11">Student Achievement and Demographic Data</hd> <p>All six partner states administered end‐of‐year mathematics assessments created by one of the two CCSS assessment consortia, the Partnership for Assessment of Readiness for College and Career (PARCC) or the Smarter Balanced Assessment Consortium (SBAC). At the time of our study, Louisiana, Maryland, New Jersey, and New Mexico were part of PARCC, and California and Washington were part of SBAC. Although each consortium constructed its own test, both were designed to align to the same CCSS standards, and analyses of test items suggest that they cover similar content and with a similar level of rigor (Herman & Linn, 2014). Test scores were collected by state agencies, alongside a set of student demographic information including gender, race/ethnicity, eligibility for free or reduced‐price lunch, eligibility to receive special education services, English Language Learner status, and grade level in school.</p> <p>The primary research team worked with student achievement and demographic data for three states—Maryland, New Jersey, and New Mexico—with which we signed data use agreements. In the three remaining states—California, Louisiana, and Washington—the primary team coordinated with additional researchers who had access to the student‐level data through their own agreements with state agencies. In these states, the partner researchers implemented similar statistical specifications to those estimated by the primary team and sent either data aggregated to the school level (in Louisiana and Washington) or parameter estimates (in California), both of which could be pooled with data/estimates from the remaining states. In Louisiana, test score data were not yet available for the 2016/2017 school year by the time of our analyses.</p> <hd id="AN0146104504-12">EMPIRICAL METHODOLOGY</hd> <p></p> <hd id="AN0146104504-13">Value‐Added Model</hd> <p>To examine differences in textbook efficacy, we used a two‐step process where we first estimated school‐level "value‐added" and then used these scores as the outcome measure in a second‐stage equation in which textbooks were the key predictors of interest. This two‐step process allowed some partner states to share aggregated rather than student‐level data.</p> <p>First, we estimated school‐level differences in achievement growth using the following model for student <emph>i</emph> in school <emph>j</emph> and grade <emph>g</emph> in year <emph>t</emph>, estimated separately for each state:</p> <olist> <item> <ephtml> <math display="block" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0001" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>S</mi><mrow><mi>i</mi><mi>j</mi><mi>g</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mi>β</mi><mn>0</mn></msub><mo>+</mo><msub><mi>β</mi><mrow><mn>1</mn><mi>g</mi></mrow></msub><mi>f</mi><mfenced separators="" open="(" close=")"><msup><msub><mi>S</mi><mrow><mi>i</mi><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mrow><mi>M</mi><mi>a</mi><mi>t</mi><mi>h</mi></mrow></msup></mfenced><mo>+</mo><msub><mi>β</mi><mrow><mn>2</mn><mi>g</mi></mrow></msub><mi>f</mi><mfenced separators="" open="(" close=")"><msup><msub><mi>S</mi><mrow><mi>i</mi><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mrow><mi>E</mi><mi>L</mi><mi>A</mi></mrow></msup></mfenced><mo>+</mo><msub><mi>β</mi><mn>3</mn></msub><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi><mi>g</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mi>θ</mi><mrow><mi>g</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mi>δ</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mi>ε</mi><mrow><mi>i</mi><mi>j</mi><mi>g</mi><mi>t</mi></mrow></msub><mo>.</mo></mrow></math> </ephtml> </item> </olist> <p>We modeled students' current‐year math achievement score (<emph>S</emph>) as a cubic function of prior achievement in math and English Language Arts (ELA). We interacted these with grade fixed effects, allowing for different relationships between prior and current test scores across grades. In addition, we included grade‐by‐year fixed effects, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0002" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>θ</mi><mrow><mi>g</mi><mi>t</mi></mrow></msub></math> </ephtml> , to account for differences in scaling of tests at this level. We controlled for a vector of student characteristics, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0003" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi><mi>g</mi><mi>t</mi></mrow></msub></math> </ephtml> , including all demographic information provided by states and an indicator for students that repeated the current grade (at time <emph>t</emph>). Finally, we included school‐by‐year fixed effects, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0004" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>δ</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub></math> </ephtml> , which were our parameters of interest from equation (<reflink idref="bib1" id="ref1">1</reflink>). The estimated school‐by‐year effects, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0005" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><msub><mi>δ</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>̂</mo></mover></math> </ephtml> , are commonly referred to as "value‐added" estimates because they measure the degree to which students in a given school outperform or underperform other students with similar characteristics (Angrist et al., 2017). In equation (<reflink idref="bib1" id="ref2">1</reflink>), we conditioned on students' prior achievement scores to measure a student's achievement growth in the current year. Thus, even if the students used the same textbook in the prior year, we are measuring the effect of that textbook on a student's current year achievement growth. In a set of robustness tests, we also estimated textbook effects using school‐by‐grade‐by‐year value‐added as the outcome of interest—estimated with one‐ versus two‐year lagged achievement—to examine the possibility that textbooks may be a cross‐grade intervention.</p> <p>Next, we used the estimated school‐by‐year effects in a second stage to estimate a vector of textbook effects, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>μ</mi><mi>k</mi></msub></math> </ephtml> , controlling for mean school characteristics, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><mover><msub><mi>X</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>¯</mo></mover></math> </ephtml> , and characteristics of public school parents in the school district, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Z</mi><mi>d</mi></msub></math> </ephtml> , from the 2010 to 2014 American Community Survey (ACS):</p> <p>2 <ephtml> <math display="block" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><msub><mi>δ</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>̂</mo></mover><mo>=</mo><msub><mi>γ</mi><mn>0</mn></msub><mo>+</mo><munder><mo>∑</mo><mi>k</mi></munder><msub><mi>μ</mi><mi>k</mi></msub><mspace width="0.33em" /><mi>t</mi><mi>e</mi><mi>x</mi><mi>t</mi><mi>b</mi><mi>o</mi><mi>o</mi><mi>k</mi><mo>_</mo><msub><mi>k</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>+</mo><mi>γ</mi><mover><msub><mi>X</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>¯</mo></mover><mo>+</mo><mi>λ</mi><msub><mi>Z</mi><mi>d</mi></msub><mo>+</mo><msub><mi>ω</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>.</mo></mrow></math> </ephtml></p> <p>We used <emph>enVision</emph> (CCSS edition) as the left‐out category, since it was one of the few textbooks in high use in all states and years (see Table 3). Our parameter estimates, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><msub><mi>μ</mi><mi>k</mi></msub><mo>̂</mo></mover></math> </ephtml> , thus estimate differences in student achievement growth for a given textbook relative to schools using this left‐out text. Coefficients for specific textbooks would differ if we changed the left‐out category, but statistics from the test of whether textbook effects are jointly equal to zero—our primary estimand of interest—would not. When pooling data across years or states, we included state‐by‐year fixed effects.</p> <p>Because textbooks typically vary at the district level, we needed to account for the fact that the school‐by‐year error term, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0011" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>ω</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub></math> </ephtml> , is not independent for schools in the same district. We also needed to account for correlated errors for the same school over time. As a result, we separated the error term into three components: a district component, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>ϕ</mi><mi>d</mi></msub></math> </ephtml> , a school component, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>χ</mi><mi>j</mi></msub></math> </ephtml> , and an independent school‐by‐year error, <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>ξ</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub></math> </ephtml> :</p> <p> <ephtml> <math display="block" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>ω</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mi>ϕ</mi><mi>d</mi></msub><mo>+</mo><msub><mi>χ</mi><mi>j</mi></msub><mo>+</mo><msub><mi>ξ</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mo>.</mo></mrow></math> </ephtml> </p> <p>We estimated equation (<reflink idref="bib2" id="ref3">2</reflink>) using a hierarchical (random effects) model to account for the error terms at the district and school levels.</p> <p>When estimating the efficacy of individual textbooks using equation (<reflink idref="bib2" id="ref4">2</reflink>), we treated <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>μ</mi><mi>k</mi></msub></math> </ephtml> as a vector of fixed effects. While each should provide an unbiased estimate of the effect of an individual textbook, the variance across these parameter estimates overstates the underlying heterogeneity in textbook effects, since collectively they include sampling error at the state, district, and school levels. As a result, we also specified <ephtml> <math display="inline" altimg="urn:x-wiley:02768739:media:pam22257:pam22257-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>μ</mi><mi>k</mi></msub></math> </ephtml> as a set of textbook random effects, with state‐by‐year fixed effects (in pooled analyses), and state, district, and school random effects nested within each textbook. Because the textbook random effect variance estimate is adjusted for the school‐, district‐, and state‐level errors, we interpret it as an estimate of the "true" underlying variance in textbook efficacy. This specification allows for variation in value‐added within textbook across states, districts, and schools, and estimates the component of variation in student achievement gains attributable to textbook effects that are common across states and over time. While fixed effects specification allows the textbook effects to be correlated with covariates, the random effects specification assumes no correlation. Given use of observational data, this may be too strong of an assumption. However, below we provide evidence to suggest that this assumption does not lead us to different answers across the two methods used to estimate textbook effects.</p> <hd id="AN0146104504-14">Identifying Assumptions</hd> <p>The key identifying assumption of our value‐added approach is that textbook selection is uncorrelated with characteristics of districts, schools, and students, above and beyond those included in our model. However, since textbooks can vary in instructional approach, level of rigor, marketing materials, etc., we might also expect some differences in the baseline characteristics of schools adopting different texts (Bianchini & Kelly, 2003; Polikoff, 2018; Seeley, 2003; Tulley, 1985). In Table 4, we report the baseline characteristics of schools and districts adopting different textbooks, pooling across all six states. There are statistically significant differences in the observed characteristics of schools adopting different textbooks. (All tests of significance control for state fixed effects.) For example, the schools using <emph>Stepping Stones</emph> (CCSS edition), <emph>My Math</emph> (written for the CCSS), <emph>enVision</emph> (either pre‐ or post‐CCSS edition), and <emph>Engage NY/Eureka</emph> (written for the CCSS) tended to be somewhat more disadvantaged. They had higher percentages of students receiving free or reduced‐price lunches, had lower expenditures per student, and lower levels of parental education. In contrast, <emph>Everyday Mathematics</emph> (CCSS edition), <emph>Bridges in Mathematics</emph> (CCSS edition), and <emph>Ready Common Core</emph> (written for the CCSS) seemed to be used in somewhat more affluent schools, with lower percentages of students receiving federal subsidized lunches and higher levels of parental education. As a result, we included statistical controls for a variety of school‐by‐year demographic and test score measures and for district characteristics.</p> <p>4 TableDifferences in school and district characteristics adopting different textbooks (top 15)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">School Characteristics</th><th align="center">District Characteristics</th></tr><tr valign="bottom"><th>Textbooks (Sorted by EdReports Ratings for Alignment to CCSS)</th><th align="center">Free or Reduced‐Price Lunch (%)</th><th align="center">Special Education (%)</th><th align="center">English Language Learner (%)</th><th align="center">African American (%)</th><th align="center">Hispanic (%)</th><th align="center">Prior Math Test Score (Standardized)</th><th align="center">Per‐pupil Instructional Expenditure ($)</th><th align="center">Parent Holds BA Degree or Higher (%)</th></tr></thead><tbody><tr><td>A. Minimum EdReports CCSS Alignment Rating: Meets Expectations</td></tr><tr><td>Bridges in Mathematics CC</td><td>44.0</td><td>15.5</td><td>13.3</td><td>8.9</td><td>29.2</td><td>0.038</td><td>7,916</td><td>34.1</td></tr><tr><td>Engage NY/Eureka CC</td><td>58.7</td><td>13.2</td><td>20.4</td><td>10.3</td><td>45.5</td><td>−0.036</td><td>6,509</td><td>24.0</td></tr><tr><td>My Math CC</td><td>70.7</td><td>13.5</td><td>23.0</td><td>9.2</td><td>52.7</td><td>−0.136</td><td>6,187</td><td>19.6</td></tr><tr><td>Ready Common Core CC</td><td>43.4</td><td>13.2</td><td>14.7</td><td>3.1</td><td>27.0</td><td>0.102</td><td>6,072</td><td>34.4</td></tr><tr><td>B. Minimum EdReports CCSS Alignment Rating: Partially Meets Expectations</td></tr><tr><td>Go Math CC</td><td>57.7</td><td>14.9</td><td>14.5</td><td>14.0</td><td>35.6</td><td>0.002</td><td>7,199</td><td>28.9</td></tr><tr><td>Math Expressions CC</td><td>54.7</td><td>13.6</td><td>23.5</td><td>7.2</td><td>38.3</td><td>0.022</td><td>5,991</td><td>29.3</td></tr><tr><td>C. Other CCSS Editions</td></tr><tr><td>enVision CC</td><td>64.6</td><td>15.0</td><td>33.3</td><td>9.3</td><td>51.4</td><td>−0.047</td><td>6,093</td><td>23.7</td></tr><tr><td>Everyday Mathematics CC</td><td>47.3</td><td>13.7</td><td>19.5</td><td>7.9</td><td>20.3</td><td>0.006</td><td>5,982</td><td>36.9</td></tr><tr><td>Math in Focus CC</td><td>47.6</td><td>12.9</td><td>25.9</td><td>6.2</td><td>31.3</td><td>0.176</td><td>6,841</td><td>35.9</td></tr><tr><td>Stepping Stones CC</td><td>74.7</td><td>18.9</td><td>16.6</td><td>5.3</td><td>56.8</td><td>−0.042</td><td>5,424</td><td>25.8</td></tr><tr><td>D. Non‐CCSS Editions</td></tr><tr><td>enVision</td><td>65.3</td><td>12.8</td><td>13.9</td><td>27.5</td><td>26.8</td><td>−0.123</td><td>6,606</td><td>19.6</td></tr><tr><td>Everyday Mathematics</td><td>50.8</td><td>11.8</td><td>29.4</td><td>3.6</td><td>46.8</td><td>0.172</td><td>5,478</td><td>31.9</td></tr><tr><td>Houghton Mifflin Math</td><td>49.9</td><td>17.2</td><td>13.7</td><td>12.5</td><td>31.4</td><td>0.018</td><td>7,241</td><td>33.0</td></tr><tr><td>Math Connects</td><td>49.1</td><td>14.3</td><td>4.6</td><td>5.0</td><td>17.4</td><td>−0.024</td><td>6,802</td><td>25.7</td></tr><tr><td>Math Expressions</td><td>44.5</td><td>10.6</td><td>3.0</td><td>33.5</td><td>12.2</td><td>0.133</td><td>8,037</td><td>33.2</td></tr><tr><td>P‐Value from Test of Joint Significance</td><td align="center">0.000</td><td align="center">0.000</td><td align="center">0.000</td><td align="center">0.000</td><td align="center">0.000</td><td align="center">0.000</td><td align="center">0.000</td><td align="center">0.000</td></tr><tr><td>School*Year Observations</td><td align="center">10,797</td><td align="center">10,797</td><td align="center">10,797</td><td align="center">10,797</td><td align="center">10,797</td><td align="center">10,797</td><td>10,797</td><td align="center">10,797</td></tr></tbody></table> </ephtml> </p> <p>5 <emph>Notes</emph>: Estimates are weighted by the inverse of a school's sampling probability. The sample is restricted to school‐by‐year observations known to have adopted one of the top 15 textbooks by market share. Joint tests of significance are estimated from models that include state‐by‐year fixed effects and cluster standard errors at the district level.</p> <p>At the same time, two additional sources of information suggest that nonrandom selection may not be as concerning when estimating textbook efficacy. First, patterns of selection do not appear to correlate with information published by EdReports. Recall that, with a lack of evidence on efficacy for the majority of CCSS‐edition textbooks, publicly available reviews such as those provided by EdReports are likely to be a primary source of information for textbook adoption decisions (Polikoff et al., 2020). In Table 4, we sort textbooks by EdReports' ratings for alignment to the CCSS and find no clear pattern related to the observable characteristics of schools and districts using different texts. For example, of the four textbooks in our sample that received the highest rating by EdReports, two were used by schools that had well‐below average prior math achievement; the two other texts from our sample in this high EdReports' rating category were used by schools that had well‐above average prior math achievement.</p> <p>Similarly, we observe that patterns of nonrandom selection are inconsistent across states. In Table A2, we disaggregate mean characteristics of schools using different textbooks across states, focusing on two characteristics that may be most likely to bias our textbook efficacy estimates: average prior math test scores (our outcome of interest), and percent of students eligible for free or reduced‐price lunch (where we see some of the largest variation across textbooks; SD = 10 percentage points). In Table A3, we correlate these characteristics across states, finding weak negative to moderate positive relationships. For the three states with the broadest coverage of textbooks (i.e., California, New Jersey, Washington), cross‐state correlations for school‐average prior achievement associated with different textbooks range from −0.28 to 0.57. For average percentage of students eligible for free or reduced‐price lunch in schools using different textbooks, correlations across these states are no higher than 0.49. Looking across all six states, several correlations are statistically significant and negative in magnitude. These findings are consistent with intuition suggesting that it is highly unlikely that certain types of schools are systematically choosing the most effective textbooks when school and district leaders (and we) do not know which textbooks those are.</p> <p>Although we can control for observed school and district characteristics, it is possible that there are unmeasured determinants of student performance related to textbook adoption. Therefore, we also specified models that included school fixed effects. These models implicitly control for fixed, time‐invariant differences between schools using different texts by focusing on changes in student achievement gains associated with changes in textbook adoptions. We also continued to control for time‐varying, observable school characteristics. For our switcher models, our preferred sample comes from California, where we not only observed a large number of schools switching textbooks between the two years of textbook data available (<emph>n</emph> = 1,060) but also found switchers and non‐switchers to look similar in terms of observed school and district characteristics (<emph>p</emph> = 0.857 on joint test of significance; see Table A4). Thus, estimates from the California switcher models are likely to generalize across elementary schools in this state. In the other five states, only 161 schools switched textbooks during the time frame of our study, and these schools differed from non‐switchers (<emph>p</emph> = 0.053 on joint test of significance; see Table 4). Nonetheless, we show results across both samples with the caveat that school fixed effect estimates from our five‐state sample excluding California may have limited generalizability.</p> <hd id="AN0146104504-15">RESULTS</hd> <p></p> <hd id="AN0146104504-16">Teacher‐Reported Use of Textbooks</hd> <p>In Tables 5 and 6, we report estimates of teachers' use of textbooks in schools. These analyses include the sample of schools that adopted one of the top seven CCSS‐edition textbooks by market share, which were the focus of the teacher survey sampling scheme. Given our sampling approach—which randomly selected schools (not teachers) for participation—we aggregated teacher‐level responses to the school level, and then weighted these data by the inverse of the probability that a given school was included in the teacher survey (i.e., the product of the probability that a school was included in the textbook adoption data and the probability that the same school also was selected for the teacher survey). This approach aims to ensure that results generalize across our six partner states, at least for schools using one of these top seven CCSS‐edition textbooks.</p> <p>5 TableTeacher‐reported use of textbooks (top 7 CCSS‐editions) or other materials</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>In what percentage of your lessons did you use textbook/materials for this purpose?</th><th align="center">0% of lessons</th><th align="center">1%–25%</th><th align="center">26%–50%</th><th align="center">51%–75%</th><th align="center">More than 75%</th></tr></thead><tbody><tr><td>Used textbook to:</td></tr><tr><td>Choose objective (% of teachers)</td><td>2.3</td><td>5.7</td><td>7.5</td><td>20.4</td><td>64.1</td></tr><tr><td>Refresh content knowledge (%)</td><td>4.5</td><td>16.6</td><td>16.4</td><td>23.1</td><td>39.4</td></tr><tr><td>Create tasks and activities (%)</td><td>3.3</td><td>12.4</td><td>13.9</td><td>27.8</td><td>42.6</td></tr><tr><td>Select examples (%)</td><td>1.7</td><td>7.9</td><td>12.2</td><td>31.3</td><td>46.9</td></tr><tr><td>Assign independent classwork (%)</td><td>0.7</td><td>5.2</td><td>9.8</td><td>26.6</td><td>57.7</td></tr><tr><td>Choose homework problems (%)</td><td>4.4</td><td>9.6</td><td>8.9</td><td>23.2</td><td>54.0</td></tr><tr><td>Build assessments (%)</td><td>4.4</td><td>9.5</td><td>10.2</td><td>21.5</td><td>54.4</td></tr><tr><td>Any of the above (%)</td><td>0.3</td><td>2.8</td><td>3.0</td><td>16.1</td><td>77.8</td></tr><tr><td>Used other materials:</td></tr><tr><td>State, district, or charter‐produced materials (%)</td><td>17.4</td><td>25.3</td><td>10.6</td><td>17.9</td><td>28.9</td></tr><tr><td>Repositories on the web (%)</td><td>7.2</td><td>44.3</td><td>23.1</td><td>19.6</td><td>5.9</td></tr><tr><td>Materials created by teacher or with colleagues (%)</td><td>5.0</td><td>48.9</td><td>21.9</td><td>16.3</td><td>7.9</td></tr><tr><td>Materials created by other teachers in the school (%)</td><td>33.7</td><td>45.3</td><td>11.4</td><td>6.5</td><td>3.1</td></tr><tr><td>Materials from personal library (%)</td><td>19.6</td><td>46.6</td><td>17.8</td><td>10.8</td><td>5.2</td></tr><tr><td>Released test items (%)</td><td>13.5</td><td>48.5</td><td>15.8</td><td>14.5</td><td>7.7</td></tr><tr><td>Test‐preparation books purchased by school/district (%)</td><td>53.4</td><td>26.9</td><td>8.4</td><td>6.2</td><td>5.1</td></tr><tr><td>Online content videos (%)</td><td>15.2</td><td>42.5</td><td>19.5</td><td>14.2</td><td>8.6</td></tr><tr><td>Online software (%)</td><td>35.2</td><td>27.7</td><td>14.9</td><td>12.5</td><td>9.7</td></tr></tbody></table> </ephtml> </p> <ulist> <item>6 <emph>Notes</emph>: Sample includes 1,194 teachers in 345 schools; one additional teacher with a valid survey skipped all questions reported in this table. Means were estimated by averaging teacher responses to the school level, and then applying a teacher survey sampling weight derived by dividing the school sampling weights by the probability that a given school was also selected to participate in the teacher survey. The percentages for "any of the above" reflect the maximum for the seven uses for each teacher. The sample is restricted to schools using one of the top seven CCSS‐edition textbooks by market share, which were the focus of the teacher survey.</item> <item>6 TableTeacher‐reported supplementation or substitution, and amount of professional development by textbook (top 7 CCSS‐editions)</item> </ulist> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">All 7 Textbooks</th><th align="center"><italic>Engage NY/Eureka</italic> CC</th><th align="center"><italic>enVision</italic> CC</th><th align="center"><italic>Everyday Mathematics</italic> CC</th><th align="center"><italic>Go Math</italic> CC</th><th align="center"><italic>Math Expressions</italic> CC</th><th align="center"><italic>Math in Focus</italic> CC</th><th align="center"><italic>My Math</italic> CC</th></tr></thead><tbody><tr><td>Reasons teachers used materials other than main textbook (not mutually exclusive):</td></tr><tr><td>School or district requires use of other materials (%)</td><td>9.2</td><td>7.4</td><td>14.0</td><td>12.7</td><td>5.8</td><td>8.2</td><td>11.4</td><td>7.7</td></tr><tr><td>Textbook is too easy (%)</td><td>15.8</td><td>7.60002</td><td>20.3</td><td>8.30002</td><td>8.40002</td><td>8.50002</td><td>5.30002</td><td>31.50002</td></tr><tr><td>Textbook is too hard (%)</td><td>28.8</td><td>41.2</td><td>19.9</td><td>26.9</td><td>39.3</td><td>26.6</td><td>40.8</td><td>18.40002</td></tr><tr><td>Textbook does not cover all of the standards (%)</td><td>18.0</td><td>9.40002</td><td>23.8</td><td>9.00002</td><td>14.2</td><td>11.50002</td><td>35.80002</td><td>22.7</td></tr><tr><td>Textbook is not user friendly (%)</td><td>13.5</td><td>20.1</td><td>8.1</td><td>10.7</td><td>18.3</td><td>13.7</td><td>12.2</td><td>10.9</td></tr><tr><td>Examples in textbook are not sufficiently engaging for students (%)</td><td>51.7</td><td>40.30002</td><td>45.9</td><td>36.60002</td><td>51.4</td><td>48.5</td><td>49.9</td><td>72.40002</td></tr><tr><td>Access to materials used in the past (%)</td><td>41.7</td><td>32.0</td><td>38.3</td><td>48.0</td><td>52.50002</td><td>51.0</td><td>39.4</td><td>38.3</td></tr><tr><td>N/A – textbook used exclusively (%)</td><td>8.0</td><td>17.70002</td><td>8.9</td><td>5.2</td><td>3.60002</td><td>3.90002</td><td>3.40002</td><td>6.9</td></tr><tr><td>Access to professional development (PD):</td></tr><tr><td>PD received this year (days)</td><td>5.4</td><td>5.4</td><td>5.6</td><td>5.0</td><td>4.7</td><td>5.3</td><td>4.9</td><td>6.1</td></tr><tr><td>PD specific to math received this year (days)</td><td>1.9</td><td>2.60002</td><td>1.5</td><td>1.9</td><td>2.0</td><td>1.6</td><td>2.3</td><td>1.8</td></tr><tr><td>PD specific to math textbook received this year (days)</td><td>1.1</td><td>1.80002</td><td>0.80002</td><td>1.2</td><td>1.0</td><td>0.80002</td><td>1.4</td><td>0.9</td></tr><tr><td>PD specific to math textbook received over entire career (days)</td><td>3.4</td><td>4.60002</td><td>2.70002</td><td>4.3</td><td>2.8</td><td>4.0</td><td>5.50002</td><td>2.70002</td></tr><tr><td>Provided with math coach this year (%)</td><td>37.1</td><td>51.0</td><td>31.7</td><td>39.3</td><td>33.6</td><td>48.9</td><td>41.2</td><td>27.3</td></tr><tr><td>Met with math coach (count)</td><td>1.3</td><td>1.90002</td><td>1.0</td><td>1.8</td><td>1.1</td><td>1.2</td><td>1.5</td><td>1.0</td></tr><tr><td>Observed teaching this year (count)</td><td>1.9</td><td>3.00002</td><td>1.8</td><td>2.1</td><td>1.8</td><td>1.30002</td><td>1.7</td><td>1.8</td></tr><tr><td>Received feedback about teaching this year (count)</td><td>14.6</td><td>16.6</td><td>18.20002</td><td>15.6</td><td>12.8</td><td>12.00002</td><td>13.8</td><td>12.00002</td></tr></tbody></table> </ephtml> </p> <ulist> <item>7 <emph>Notes</emph>: Sample includes 1,195 teachers in 345 schools. Means estimated by averaging teacher responses to the school level, and then applying a teacher survey sampling weight derived by dividing the school sampling weights by the probability that a given school was also selected to participate in the teacher survey. The sample is restricted to schools using one of the top seven CCSS‐edition textbooks by market share, which were the focus of the teacher survey.</item> <item>8 <sups>∼</sups>p <.10; *p <.05; **p <.01; ***p <.001, comparing means for one textbook to all others combined.</item> </ulist> <p>In most schools, teachers reported using the adopted textbook frequently in their classes (see Table 5). Teachers in 78 percent of schools reported using their textbook for one of the listed purposes (e.g., creating tasks and activities, selecting examples, and building assessments) in at least 75 percent of lessons; teachers in 94 percent of schools used their textbook for one of these purposes in 50 percent or more of their lessons. Teachers also reported covering an average of 82 percent of chapters over the course of the school year (not shown in Table 5). At the same time, we found that teachers often supplemented or substituted into their lessons content from other sources. For example, teachers in 44 percent of schools used materials provided to them by the state, district, or school in more than half of their lessons. Teachers in about a quarter of schools reported using materials they found online or developed themselves in more than half of their lessons (see Table 5). Overall, teachers in only 8 percent of schools used their textbooks exclusively (see Table 6).</p> <p>While Table 5 reports teachers' usage characteristics pooling across all seven texts that were a focus of the teacher survey, Table 6 examines differences by textbook. We found that teachers were likely to use supplemental materials with some textbooks more than others. In schools that adopted <emph>Engage NY/Eureka</emph>, teachers were more likely to use the textbook exclusively (18 percent of schools) than in schools that adopted other textbooks. For other texts such as <emph>Go Math</emph>, <emph>Math Expressions</emph>, and <emph>Math in Focus</emph>, teachers in fewer than 4 percent of schools used the textbook exclusively. Supplementing or substituting materials often was related to teachers' perception of the level of rigor of that text. For example, teachers in 32 percent of schools that adopted <emph>My Math</emph> indicated that they used other materials because they perceived the textbook to be "too easy," compared to teachers in 5 to 20 percent of schools using other textbooks. In contrast, teachers in roughly 40 percent of schools that adopted <emph>Engage NY/Eureka, Go Math</emph>, or <emph>Math in Focus</emph> reported using other materials because they perceived these textbooks to be "too hard" for their students. Teachers whose schools adopted <emph>Math in Focus</emph> also supplemented their textbook with other materials because they did not feel that it "covered all of the [CCSS] standards" (teachers in 36 percent of schools using this text, compared to teachers in 9 to 24 percent of schools using other textbooks).</p> <p>Access to professional development—particularly math‐specific programming and those aligned to textbook implementation—also differed by textbook. Teachers in schools that adopted <emph>Engage NY/Eureka</emph> reported receiving the most professional development tailored to the curriculum, but it was still modest: 1.8 days, on average, in the most recent school year, compared to 0.8 to 1.4 days, on average, for teachers in schools using other textbooks. Across all schools, teachers reported an average of just 3.4 days of textbook‐aligned professional development over the course of their career. By design, teachers in schools using textbook series that were in print prior to the CCSS (i.e., <emph>enVision</emph>, <emph>Everyday Mathematics, Math Expressions, Math in Focus</emph>) tended to have more days of textbook‐aligned professional developmemt, on average. Yet, for teachers in schools using <emph>Engage NY/Eureka</emph>—which is a newer text written after the rollout of the CCSS—the average amount of textbook‐aligned professional development (4.6 days over the course of their career) was higher than the sample average and higher than the average in schools that adopted two other newer textbooks (i.e., <emph>Go Math, My Math</emph>).</p> <p>Differences in use and support could influence a given textbook's efficacy. However, we did not attempt to "control for" such differences given that these factors almost certainly are endogenous. That is, implementation differences likely are a result of a given textbook's perceived strengths and flaws. Instead, we report below the average achievement gains of the schools using a given textbook <emph>as adopted</emph>. We do not report—because we cannot validly estimate—the magnitude of gain that a given school or district <emph>could</emph> have achieved with a given text <emph>if</emph> implemented in the ideal manner. The estimates presented below also are those most relevant to our policy question of interest: What is the effect of <emph>adopting</emph> a higher‐quality textbook? The adoption <emph>and</emph> high‐quality implementation of new textbook is a different, much more complicated, and much more expensive intervention to assess. We return to this topic in our conclusion.</p> <hd id="AN0146104504-17">Differences in Average Student Achievement Gains Across Textbooks</hd> <p>In Table 7, we report estimates from equation (<reflink idref="bib2" id="ref5">2</reflink>) of the average student achievement gains associated with different textbooks. In all models, estimates are reported in student‐level SD of math achievement in a given state, grade, and year. All analytic samples are limited to school‐years using one of the top 15 textbooks by market share (covering roughly 90 percent of school‐year observations in our sample). In supplementary analyses, we also found similar results when expanding the sample to the full set of 38 known textbooks, as well as further restricting the sample to the top five or 10 textbooks that all had substantial shares of the market (see Table A5).</p> <p>7 TableDifferences in math achievement growth by textbook (top 15)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">Pooled 6 States: All Years</th><th align="center">Pooled 5 States (Excluding California): All Years</th><th align="center">California: All Years</th><th align="center">Pooled 6 States: 2014‐15</th><th align="center">Pooled 6 States: 2015‐16</th><th align="center">Pooled 4 States (Excluding California and Louisiana): 2016‐17</th></tr><tr valign="bottom"><th /><th align="center">(1)</th><th align="center">(2)</th><th align="center">(3)</th><th align="center">(4)</th><th align="center">(5)</th><th align="center">(6)</th></tr></thead><tbody><tr><td>A. Minimum EdReports CCSS Alignment Rating: Meets Expectations</td></tr><tr><td>Bridges in Mathematics CC</td><td>0.010</td><td>0.0690002</td><td>−0.020</td><td>0.016</td><td>0.0510002</td><td>0.041</td></tr><tr><td /><td>(0.018)</td><td>(0.033)</td><td>(0.022)</td><td>(0.033)</td><td>(0.017)</td><td>(0.034)</td></tr><tr><td>Engage NY/Eureka CC</td><td>−0.003</td><td>0.021</td><td>−0.0300002</td><td>−0.005</td><td>0.018</td><td>0.016</td></tr><tr><td /><td>(0.011)</td><td>(0.018)</td><td>(0.014)</td><td>(0.018)</td><td>(0.012)</td><td>(0.023)</td></tr><tr><td>My Math CC</td><td>0.0190002</td><td>0.018</td><td>0.018</td><td>−0.018</td><td>0.003</td><td>−0.001</td></tr><tr><td /><td>(0.010)</td><td>(0.017)</td><td>(0.013)</td><td>(0.017)</td><td>(0.011)</td><td>(0.019)</td></tr><tr><td>Ready Common Core CC</td><td>−0.020</td><td>−0.015</td><td>−0.060</td><td>−0.039</td><td>0.015</td><td>−0.008</td></tr><tr><td /><td>(0.039)</td><td>(0.047)</td><td>(0.075)</td><td>(0.067)</td><td>(0.051)</td><td>(0.041)</td></tr><tr><td>B. Minimum EdReports CCSS Alignment Rating: Partially Meets Expectations</td></tr><tr><td>Go Math CC</td><td>0.001</td><td>0.021</td><td>−0.006</td><td>0.009</td><td>0.013</td><td>−0.014</td></tr><tr><td /><td>(0.010)</td><td>(0.017)</td><td>(0.013)</td><td>(0.017)</td><td>(0.011)</td><td>(0.019)</td></tr><tr><td>Math Expressions CC</td><td>0.0390002</td><td>0.022</td><td>0.0460002</td><td>0.0630002</td><td>0.0340002</td><td>0.0490002</td></tr><tr><td /><td>(0.013)</td><td>(0.025)</td><td>(0.016)</td><td>(0.020)</td><td>(0.013)</td><td>(0.027)</td></tr><tr><td>C. Other CCSS Editions</td><td /><td /><td /><td /><td /><td /></tr><tr><td>Everyday Mathematics CC</td><td>−0.009</td><td>0.015</td><td>−0.018</td><td>−0.012</td><td>0.0590002</td><td>0.021</td></tr><tr><td /><td>(0.012)</td><td>(0.019)</td><td>(0.017)</td><td>(0.021)</td><td>(0.015)</td><td>(0.021)</td></tr><tr><td>Math in Focus CC</td><td>0.002</td><td>0.000</td><td>0.003</td><td>0.039</td><td>0.013</td><td>−0.016</td></tr><tr><td /><td>(0.017)</td><td>(0.022)</td><td>(0.028)</td><td>(0.027)</td><td>(0.019)</td><td>(0.025)</td></tr><tr><td>Stepping Stones CC</td><td>0.010</td><td>−0.019</td><td>0.071</td><td>−0.024</td><td>0.037</td><td>−0.0540002</td></tr><tr><td /><td>(0.031)</td><td>(0.036)</td><td>(0.061)</td><td>(0.060)</td><td>(0.034)</td><td>(0.026)</td></tr><tr><td>D. Non‐CCSS Editions</td><td /><td /><td /><td /><td /><td /></tr><tr><td>enVision</td><td>−0.0290002</td><td>−0.0760002</td><td>−0.0280002</td><td>0.024</td><td>0.017</td><td>0.077</td></tr><tr><td /><td>(0.010)</td><td>(0.034)</td><td>(0.013)</td><td>(0.017)</td><td>(0.017)</td><td>(0.052)</td></tr><tr><td>Everyday Mathematics</td><td>0.0860002</td><td>−0.021</td><td>0.1170002</td><td>0.028</td><td>0.014</td><td>0.030</td></tr><tr><td /><td>(0.015)</td><td>(0.028)</td><td>(0.019)</td><td>(0.024)</td><td>(0.023)</td><td>(0.048)</td></tr><tr><td>Houghton Mifflin Math</td><td>−0.020</td><td>0.046</td><td>−0.027</td><td>−0.0600002</td><td>−0.001</td><td /></tr><tr><td /><td>(0.015)</td><td>(0.090)</td><td>(0.017)</td><td>(0.023)</td><td>(0.022)</td><td /></tr><tr><td>Math Connects</td><td>0.015</td><td>0.017</td><td /><td>0.062</td><td>−0.003</td><td>0.046</td></tr><tr><td /><td>(0.028)</td><td>(0.029)</td><td /><td>(0.048)</td><td>(0.035)</td><td>(0.036)</td></tr><tr><td>Math Expressions</td><td>−0.016</td><td>−0.014</td><td>−0.005</td><td>0.025</td><td>−0.003</td><td>0.000</td></tr><tr><td /><td>(0.034)</td><td>(0.039)</td><td>(0.082)</td><td>(0.052)</td><td>(0.043)</td><td>(0.040)</td></tr><tr><td>P‐Value from Test of Joint Significance</td><td>0.000</td><td>0.323</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.223</td></tr><tr><td>SD of Textbook Fixed Effects</td><td>0.029</td><td>0.035</td><td>0.048</td><td>0.036</td><td>0.020</td><td>0.035</td></tr><tr><td>School*Year Observations</td><td align="center">10,797</td><td align="center">2,676</td><td align="center">8,121</td><td align="center">4,854</td><td align="center">5,100</td><td align="center">843</td></tr></tbody></table> </ephtml> </p> <ulist> <item>9 <emph>Notes</emph>: Estimates in each column come from separate linear regression models of school value‐added. We report coefficients for a set of binary indicators for each textbook. The omitted textbook category is <emph>enVision</emph> CC, which is in the "other CCSS editions" category of EdReports ratings. All models are multi‐level, mixed‐effects models that include random effects for schools nested within districts. We control for school‐by‐year demographic characteristics, 2010 to 2014 district census characteristics, and state‐by‐year fixed effects (restricted to year fixed effects only or state fixed effects only in specifications limited to a single state or year). The sample is restricted to school‐by‐year observations with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school years who are known to have used one of the top 15 textbooks by market share. The 2016/2017‐specific estimates come from Maryland, New Jersey, New Mexico, and Washington, and exclude California (which did not have textbook data in that year) and Louisiana (which did not have value‐added data in that year). Empty cells indicate that no school in a given set of states used that textbook.</item> <item>10 <sups>∼</sups>p <.10; *p <.05; **p <.01; ***p <.001.</item> </ulist> <p>In some models in Table 7, we separate out the California estimates because the sample size is larger than all other states combined and data use agreements meant that the California team shared only these second‐stage estimates. To pool second‐stage estimates from California with those from others states, we used a precision‐weighted average of the estimates from each (that is, weighting each set of estimates by the inverse of the variance‐covariance matrix of the parameters). Estimating models by subsets of states and school years also allows us to examine the sensitivity of textbook effects across states and time. California and Louisiana are excluded for the 2016/2017 estimates due to data constraints (no textbook adoption data in California in this year, and no student achievement data in Louisiana in this year).</p> <p>In the pooled sample across all six states in model (<reflink idref="bib1" id="ref6">1</reflink>), three texts have student achievement gains statistically significantly different from the left‐out category, <emph>enVision</emph> (CCSS edition): two with larger gains (<emph>Everyday Mathematics</emph> pre‐CCSS and <emph>Math Expressions</emph> CCSS edition), and one with smaller gains (<emph>enVision</emph> pre‐CCSS). However, these results are driven by the sample of schools in California. When we exclude California in model (<reflink idref="bib2" id="ref7">2</reflink>), we see two individual coefficients that are statistically different from zero (<emph>enVision</emph> pre‐CCSS and <emph>Bridges in Mathematics</emph> CCSS edition). When we test the stronger hypothesis that <emph>all</emph> differences between textbooks are equal to zero (the <emph>p</emph>‐value reported at the bottom of the table), we cannot reject it (<emph>p</emph> = 0.323). In other words, when comparing such a large number of textbooks, it is possible that one or two are significantly different from each other by chance. However, when looking across the full set of textbooks, there were so few such instances that we could not reject the hypothesis of no difference between textbooks.</p> <p>In model (<reflink idref="bib3" id="ref8">3</reflink>), we report estimates for the California sample on its own. Here, <emph>Math Expressions</emph> (CCSS edition) and <emph>Everyday Mathematics</emph> (pre‐CCSS edition) are associated with better math test‐score gains relative to <emph>enVision</emph> (CCSS edition), while schools using <emph>Engage NY/Eureka</emph> or the pre‐CCSS edition of <emph>enVision</emph> have lower test‐score gains. Compared to the five‐state sample that excludes California, we could reject the null hypothesis that textbook effects were jointly equal to zero (<emph>p</emph> < 0.001). The same is true in the pooled six‐state sample (model 1), driven by the large sample size in California. However, even for textbooks where we do see statistically significant differences, estimates generally are far smaller than those reported in prior evaluations.</p> <p>Notably, no single text stands out as a consistent high or low performer in multiple states, nor in multiple years. In models (<reflink idref="bib4" id="ref9">4</reflink>) through (<reflink idref="bib6" id="ref10">6</reflink>), individual textbooks that appear to be effective in one year (e.g., <emph>Everyday Mathematics</emph> CCSS edition, <emph>Stepping Stones</emph>) often are not identified as effective in another year. In Table A6, we report the correlation between textbook efficacy estimates across states and years. The estimated correlation between the California estimates and those in the other five states, for example, is mildly negative (−0.16). The correlations between textbook efficacy estimates across years range from −0.02 to 0.56 and also point to a lack of stability.</p> <p>We also see little to no relationship between textbook efficacy and alignment to standards. In California, for example, <emph>Engage NY/Eureka</emph> has a negative coefficient relative to <emph>enVision</emph> (CCSS edition), but was rated by EdReports as "meeting expectations" for alignment to the CCSS. The pre‐CCSS edition of <emph>Everyday Mathematics</emph> has the largest observed relationship to student achievement gains, but, by design, was not rated by EdReports because that edition was not meant to align to the CCSS. (The updated CCSS edition of <emph>Everyday Mathematics</emph> did not meet nor partially meet expectations, as rated by EdReports.) Similar discrepancies between efficacy and alignment to standards are observed in other states and samples.</p> <p>Next, we describe results from our textbook switcher models to examine whether our primary estimates are driven by unobserved characteristics at the school level. In models presented in Table 8, we expand the analytic sample to include schools using any known textbook in order to capture instances where a school switched from a non‐CCSS textbook (relatively rare in the top 15 textbooks) to a CCSS edition. However, we continue to show estimates for the top 15 textbooks by market share. Textbook coefficients in Table 8 are identified off of those schools that switched texts. Therefore, we report estimates from models that include all schools (and school fixed effects) as well as from models that limit the sample to switchers only (excluding school fixed effects).</p> <p>8 TableMarginal effects for textbook switchers (all known textbooks)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">Pooled 5 States: With School Fixed Effects</th><th align="center">Pooled 5 States: Switchers Only, No School Fixed Effects</th><th align="center">California: With School Fixed Effects</th><th align="center">California: Switchers Only, No School Fixed Effects</th></tr><tr><th /><th align="center">(1)</th><th align="center">(2)</th><th align="center">(3)</th><th align="center">(4)</th></tr></thead><tbody><tr><td>A. Minimum EdReports CCSS Alignment Rating: Meets Expectations</td></tr><tr><td>Bridges in Mathematics CC</td><td>0.5460002</td><td>0.038</td><td>−0.025</td><td>−0.011</td></tr><tr><td /><td>(0.050)</td><td>(0.079)</td><td>(0.070)</td><td>(0.036)</td></tr><tr><td>Engage NY/Eureka CC</td><td>0.048</td><td>0.004</td><td>−0.009</td><td>−0.0490002</td></tr><tr><td /><td>(0.047)</td><td>(0.037)</td><td>(0.062)</td><td>(0.025)</td></tr><tr><td>My Math CC</td><td>0.070</td><td>−0.022</td><td>0.038</td><td>0.0690002</td></tr><tr><td /><td>(0.067)</td><td>(0.053)</td><td>(0.027)</td><td>(0.021)</td></tr><tr><td>Ready Common Core CC</td><td /><td>0.169</td><td>0.077</td><td>−0.136</td></tr><tr><td /><td /><td>(0.111)</td><td>(0.120)</td><td>(0.110)</td></tr><tr><td>B. Minimum EdReports CCSS Alignment Rating: Partially Meets Expectations</td></tr><tr><td>Go Math CC</td><td>0.062</td><td>0.005</td><td>0.017</td><td>0.006</td></tr><tr><td /><td>(0.048)</td><td>(0.040)</td><td>(0.024)</td><td>(0.022)</td></tr><tr><td>Math Expressions CC</td><td>−0.068</td><td>−0.113</td><td>0.175</td><td>0.0970002</td></tr><tr><td /><td>(0.105)</td><td>(0.072)</td><td>(0.121)</td><td>(0.040)</td></tr><tr><td>C. Other CCSS Editions</td></tr><tr><td>Everyday Mathematics CC</td><td>−0.004</td><td>0.008</td><td>0.026</td><td>−0.009</td></tr><tr><td /><td>(0.049)</td><td>(0.043)</td><td>(0.039)</td><td>(0.025)</td></tr><tr><td>Math in Focus CC</td><td>0.026</td><td>−0.044</td><td>−0.038</td><td>0.010</td></tr><tr><td /><td>(0.040)</td><td>(0.062)</td><td>(0.088)</td><td>(0.059)</td></tr><tr><td>Stepping Stones CC</td><td>0.2620002</td><td>0.007</td><td /><td /></tr><tr><td /><td>(0.052)</td><td>(0.066)</td><td /><td /></tr><tr><td>D. Non‐CCSS Editions</td></tr><tr><td>enVision</td><td>−0.1490002</td><td>−0.2420002</td><td>0.016</td><td>−0.010</td></tr><tr><td /><td>(0.057)</td><td>(0.058)</td><td>(0.024)</td><td>(0.023)</td></tr><tr><td>Everyday Mathematics</td><td>−0.009</td><td>−0.0740002</td><td>0.1340002</td><td>0.1370002</td></tr><tr><td /><td>(0.033)</td><td>(0.045)</td><td>(0.040)</td><td>(0.030)</td></tr><tr><td>Houghton Mifflin Math</td><td>0.087</td><td>0.027</td><td>0.032</td><td>−0.014</td></tr><tr><td /><td>(0.060)</td><td>(0.094)</td><td>(0.038)</td><td>(0.028)</td></tr><tr><td>Math Connects</td><td>0.1470002</td><td>−0.022</td><td /><td /></tr><tr><td /><td>(0.044)</td><td>(0.057)</td><td /><td /></tr><tr><td>Math Expressions</td><td /><td /><td>−0.037</td><td>−0.172</td></tr><tr><td /><td /><td /><td>(0.119)</td><td>(0.170)</td></tr><tr><td>School Fixed Effects</td><td align="center">Yes</td><td align="center">No</td><td align="center">Yes</td><td align="center">No</td></tr><tr><td>P‐Value from Test of Joint Significance</td><td>0.000</td><td>0.002</td><td>0.000</td><td>0.000</td></tr><tr><td>SD of Textbook Fixed Effects</td><td>0.271</td><td>0.144</td><td>0.078</td><td>0.100</td></tr><tr><td>School*Year Observations</td><td align="center">2,805</td><td align="center">379</td><td align="center">8,711</td><td align="center">2,115</td></tr></tbody></table> </ephtml> </p> <ulist> <item>11 <emph>Notes</emph>: Estimates in each column come from separate linear regression models of school value‐added. We report coefficients for a set of binary indicators for each textbook. The omitted textbook category is <emph>enVision</emph> CC, which is in the "other CCSS editions" category of EdReports ratings. Models (<reflink idref="bib1" id="ref11">1</reflink>) and (<reflink idref="bib3" id="ref12">3</reflink>) include school fixed effects, and the sample is restricted to school‐by‐year observations with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year who used a known textbook (excluding the distict‐developed naterials or no known textbook categories from Table 3). We also control for school‐by‐year demographic characeristics. Models (<reflink idref="bib2" id="ref13">2</reflink>) and (<reflink idref="bib4" id="ref14">4</reflink>) limit the sample to schools that are known to have switched textbooks sometime between the 2014/2015 and 2016/2017 school years, but exclude school fixed effects. Empty cells indicate that no school switched to or from that textbook to or from another known textbook. Switcher models cluster standard errors at the district level.</item> <item>12 <sups>∼</sups>p <.10; *p <.05; **p <.01; ***p <.001.</item> </ulist> <p>In California, which is our preferred sample for the school fixed effects analysis given a large number of switchers that look similar to non‐switchers (see Table A4) textbook efficacy estimates are similar with or without conditioning on school fixed effects. The correlation between the estimates in California from models that include or exclude school fixed effects is 0.58 (see Table A6). The correlation between the textbook efficacy estimates from the California switcher model and the pooled six‐state sample that excludes school fixed effects is 0.63. The primary difference is that the positive estimate for <emph>Everyday Mathematics</emph> (pre‐CCSS edition) is larger. Because many districts in California were switching out of the pre‐CCSS edition of <emph>Everyday Mathematics</emph>, this may simply reflect the fact that achievement fell in those districts in the first year of a new text. Given the fact that we do not see similar results in the other states for this same textbook, we hesitate to take these as evidence of the unusual efficacy of the pre‐CCSS edition of <emph>Everyday Mathematics</emph>. Rather, we take the findings from our school fixed effect model as consistent with what we see in the remainder of the table: that there are no substantial, consistent differences between the textbooks in our sample.</p> <p>We also report estimates from our textbook switcher models in the other five states (pooled). However, we are cautious in interpreting these estimates given the small number of schools in these five states that switched textbooks during the time frame of our study and the fact that these schools are not representative of broader state populations (see Table A4). Here, we observe statistically significant variation in student achievement gains across textbooks when we include school fixed effects (see model 1), though this estimate is driven entirely by the unique set of schools that switched textbooks. When we estimate the model excluding school fixed effects and limiting the sample to switchers (see model 2), we observe greater variation across textbooks than we observed in the full set of schools in these states (see model 2 in Table 7).</p> <hd id="AN0146104504-18">The Underlying Variation in Textbook Efficacy</hd> <p>It is easy to lose track of the underlying story amidst the large number of parameter estimates presented in Tables 7 and 8. Individual estimates may appear large in a given state or year even if they are being driven by estimation error. As a result, rather than report estimates of the efficacy of individual textbooks, in Table 9 we report estimates of the SD of the textbook random effect, which we take as an estimate of the underlying heterogeneity in textbook efficacy. The estimates in Table 9 are essentially the square root of the underlying variance in textbook efficacy, after adjusting for state, school, and district sampling errors. With no individual textbook as a consistent positive or negative outlier (see Tables 7 through 9), the model‐based estimate of the textbook‐level variance is, in fact, the exact estimand of interest.</p> <p>9 TableThe standard deviation in textbook efficacy (top 15)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">Pooled 5 States (Excluding California)</th><th align="center">California</th><th align="center">Louisiana</th><th align="center">Maryland</th><th align="center">New Mexico</th><th align="center">New Jersey</th><th align="center">Washington</th></tr><tr><th>Random Effects Parameters</th><th align="center">(1)</th><th align="center">(2)</th><th align="center">(3)</th><th align="center">(4)</th><th align="center">(5)</th><th align="center">(6)</th><th align="center">(7)</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.0270001</td><td>0.000</td><td>0.000</td><td>0.016</td><td>0.017</td><td>0.0280001</td></tr><tr><td /><td align="center">–</td><td>(0.008)</td><td align="center">–</td><td>(0.000)</td><td>(0.011)</td><td>(0.026)</td><td>(0.015)</td></tr><tr><td>State</td><td>0.013</td><td /><td /><td /><td /><td /><td /></tr><tr><td /><td>(0.010)</td><td /><td /><td /><td /><td /><td /></tr><tr><td>District</td><td>0.0540001</td><td>0.0830001</td><td>0.000</td><td>0.0950001</td><td>0.0490001</td><td>0.0620001</td><td>0.0520001</td></tr><tr><td /><td>(0.007)</td><td>(0.005)</td><td align="center">–</td><td>(0.031)</td><td>(0.015)</td><td>(0.021)</td><td>(0.022)</td></tr><tr><td>School</td><td>0.0710001</td><td>0.0600001</td><td>0.0730001</td><td>0.0910001</td><td>0.0520001</td><td>0.0810001</td><td>0.0650001</td></tr><tr><td /><td>(0.006)</td><td>(0.005)</td><td>(0.024)</td><td>(0.017)</td><td>(0.015)</td><td>(0.011)</td><td>(0.017)</td></tr><tr><td>Residual</td><td>0.1390001</td><td>0.1460001</td><td>0.1690001</td><td>0.1130001</td><td>0.1180001</td><td>0.1500001</td><td>0.1330001</td></tr><tr><td /><td>(0.003)</td><td>(0.002)</td><td>(0.011)</td><td>(0.009)</td><td>(0.004)</td><td>(0.005)</td><td>(0.005)</td></tr><tr><td>School*Year Observations</td><td align="center">2,676</td><td align="center">8,121</td><td align="center">292</td><td align="center">191</td><td align="center">840</td><td align="center">822</td><td align="center">531</td></tr></tbody></table> </ephtml> </p> <p>13 <emph>Notes</emph>: Estimates in each column come from separate models. Random effects are estimated from a multilevel mixed‐effects linear regression of school‐level value added on school‐by‐year demographic characteristics, 2010 to 2014 district census characteristics, and state‐by‐year fixed effects (restricted to year fixed effects only in specifications limited to a single state). The model also includes nested random effects for textbook, state, district, and school, nested in that order, with textbook as the top level of the nesting structure (state random effects are excluded from regressions that are limited to a single state). The sample is restricted to school‐by‐year observations with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year who are known to have used one of the top 15 textbooks by market share. Robust standard errors are in parentheses. "–" indicates that the relevant parameter could not be estimated. Empty cells indicate that the random effect parameter was not included in the model. <sups>∼</sups>z >1.64, *z >1.96, **z >2.58, ***z >3.29, where z equals the ratio of a given random effects parameter estimate to its standard error. These z‐scores do not correspond precisely to p‐values as in a traditional linear regression framework, as the confidence interval for a random effect estimate is not symmetric around the estimate (random effect estimates have a lower bound of zero). These traditional markers of significance are included as an aid to reader, but they should be interpreted with caution.</p> <p>We report random effect estimates for California separately from the remaining five states, as the data use agreement did not allow us to have access to the school‐level value‐added data for that state. Unlike for fixed effects specifications, where we could use variance‐covariance matrices to aggregate estimates across samples, we would have needed the unit‐level data to estimate a pooled random effect across all six states. For our textbook random effect models, we also focus on our primary specification that excludes school fixed effects given similarity in patterns of results for textbook fixed effect estimates that include versus exclude school fixed effects.</p> <p>In the pooled sample of five states (excluding California), we estimate that the variance (and SD) in efficacy across textbooks is zero SD. This is consistent with our failure to reject the joint hypothesis of no textbook differences in many of the models presented in Table 7. Although we found some individual differences with respect to the reference textbook, the overall pattern of differences between texts was not inconsistent with zero difference between texts. In California, while we found evidence of differences in textbook efficacy with our textbook fixed effects models (see Table 7), the underlying SD in textbook efficacy from our random effects specification is much smaller at 0.027 SD. In other words, California schools using a textbook at the 95th percentile—roughly 2 SD above the mean—would expect to perform 0.054 SD above schools using an average textbook. Although positive, this estimate still is smaller than the differences estimated in the two experiments that found textbook effects (i.e., Agodini et al., 2010; Jaciw et al., 2016). Our estimate of the SD in textbook efficacy for the state of Washington is similar to that in California (0.028 SD). The estimated SD in textbook efficacy in the four remaining states all are quite close to zero.</p> <p>Even though we see little variation in textbook efficacy overall, we could be overlooking small differences between subsets of texts due to lack of statistical power. To investigate, we conducted a series of simulations in which we assigned one text an efficacy estimate ranging from 0.02 to 0.15 student‐level SD above the reference textbook, and with a market share ranging from 1 to 25 percent (see Table A7). We assigned all other texts an efficacy of zero. We then used equation (<reflink idref="bib2" id="ref15">2</reflink>) to estimate the SD in underlying textbook efficacy. Above an estimated effect of 0.1 SD and 5 percent market share, or above 0.05 SD effect and 20 percent market share, detection rates reach 99 percent or higher. It was when the single textbook was equal to or below 0.03 SD more effective that we would have failed to reject zero variance in textbook efficacy more than 80 percent of the time (no matter the market share). Yet, effects of this magnitude are of less substantive significance relative to critical benchmarks including differences in teacher quality and the human resources investment that likely is needed to switch textbooks.</p> <hd id="AN0146104504-19">Heterogeneity in Textbook Efficacy</hd> <p>In addition to accounting appropriately for sampling error, our random effects specification allows us to easily disaggregate textbook effects by subgroups. It is possible, for example, that there may be larger differences in student achievement gains across textbooks for groups of students with specialized academic needs. In Table A8, we present estimates of the SD in underlying textbook efficacy by English Language Learner status, eligibility for special education services, eligibility for free or reduced‐price lunch, and median split of baseline math achievement. Here, we estimated equations (<reflink idref="bib1" id="ref16">1</reflink>) and (<reflink idref="bib2" id="ref17">2</reflink>) in a single step with student‐level data, limiting the sample to specific subgroups of students. Because we needed student‐level data to do so, we specified these models for the three states for which the primary research team had access to student‐level data: Maryland, New Jersey, and New Mexico; our partners in California did the same for that state on its own. The estimates of the SD in textbook efficacy were not statistically different by subgroup. In the pooled three‐state sample, point estimates and standard errors are zero to three decimal places; we generally observe non‐zero values at the fifth or sixth decimal place.</p> <p>All estimates presented thus far focus on the average efficacy of textbooks <emph>as implemented</emph> in classrooms. Accordingly, they reflect the difference in average achievement gains for schools adopting each text averaged across varying levels of fidelity of implementation. However, if teachers in a subset of schools were substituting other materials—as we observe in Tables 5 and 6—or if a subset of schools just adopted their text and were not yet familiar with it, or if teachers received little professional development in the use of the text, we may be understating the differences. As a result, in Table 10, we also present estimates of the underlying variation in textbook efficacy after dividing schools with regard to four dimensions of implementation: teacher‐reported usage, amount of textbook‐aligned professional development, pre‐ versus post‐CCSS textbook editions, and years of experience with a given text. Because of the endogeneity of implementation, we view these analyses as exploratory, highlighting possible mechanisms to explain our null findings presented thus far.</p> <p>10 TableHeterogeneity in textbook random effect estimates by measures of implementation</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Random Effects Parameters</th><th>Pooled 5 States (Excluding California)</th><th>California</th></tr><tr valign="bottom"><th>Panel A (Top 7 CCSS‐Edition Textbooks)</th><th>Above‐Median Teacher‐Reported Use</th><th>Below‐Median Teacher‐Reported Use</th><th>Above‐Median Teacher‐Reported Use</th><th>Below‐Median Teacher‐Reported Use</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.037</td></tr><tr><td /><td align="center">–</td><td align="center">–</td><td align="center">–</td><td>(0.046)</td></tr><tr><td>School*Year Observations</td><td>309</td><td>315</td><td>90</td><td>89</td></tr></tbody></table> </ephtml> </p> <p>10 TableHeterogeneity in textbook random effect estimates by measures of implementation</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel B (Top 7 CCSS‐Edition Textbooks)</th><th>Above‐Median Teacher‐Reported PD Aligned to Textbook</th><th>Below‐Median Teacher‐Reported PD Aligned to Textbook</th><th>Above‐Median Teacher‐Reported PD Aligned to Textbook</th><th>Below‐Median Teacher‐Reported PD Aligned to Textbook</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td /><td align="center">–</td><td align="center">–</td><td align="center">–</td><td>(0.000)</td></tr><tr><td>School*Year Observations</td><td>330</td><td>294</td><td>91</td><td>88</td></tr></tbody></table> </ephtml> </p> <p>10 TableHeterogeneity in textbook random effect estimates by measures of implementation</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel C (Top 15 Textbooks)</th><th>CCSS Edition</th><th>Not CCSS Edition</th><th>CCSS Edition</th><th>Not CCSS Edition</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.013</td><td>0.0220001</td><td>0.0420001</td></tr><tr><td /><td align="center">–</td><td>(0.043)</td><td>(0.008)</td><td>(0.023)</td></tr><tr><td>School*Year Observations</td><td>2,476</td><td>200</td><td>6,221</td><td>1,900</td></tr></tbody></table> </ephtml> </p> <p>10 TableContinued</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Random Effects Parameters</th><th>Pooled 5 States (Excluding California)</th><th>California</th></tr><tr valign="bottom"><th>Panel D (Top 15 Textbooks)</th><th>2+ Years Since Adoption</th><th>Year 1 of Adoption</th><th>2+ Years Since Adoption</th><th>Year 1 of Adoption</th></tr></thead><tbody><tr><td>Textbook</td><td>0.005</td><td>0.000</td><td>0.0350001</td><td>0.000</td></tr><tr><td /><td>(0.023)</td><td align="center">–</td><td>(0.011)</td><td align="center">–</td></tr><tr><td>School*Year Observations</td><td>1,590</td><td>81</td><td>2,482</td><td>979</td></tr></tbody></table> </ephtml> </p> <p>14 <emph>Notes</emph>: Estimates in each cell come from separate models. We estimate the standard deviation in textbook effects using a multilevel mixed‐effects linear regression of school‐level value added on school‐by‐year demographic characteristics, 2010 to 2014 district census characteristics, and state‐by‐year fixed effects (restricted to year fixed effects only in specifications limited to a single state). The model also includes nested random effects for textbook, state, district, and school, nested in that order, with textbook as the top level of the nesting structure (state random effects are excluded from regressions that are limited to a single state). The sample is restricted to school‐by‐year observations within the indicated subgroup with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year. In panels A and B, the sample includes schools that were sampled for the teacher survey and used one of the top seven CCSS‐edition textbooks by market share. In panels C and D, the sample includes schools known to have used one of the top 15 textbooks by market share. To identify above‐ and below‐median levels of textbook usage, we averaged teachers' response to five survey questions about frequency of textbook use (measured on a Likert scale from 0–4; see Table 5) across items and then across teachers within the school; then we split the sample of schools at the median value of this average (3.4, corresponding to 72.5 percent of lessons). Schools whose average response to the textbook usage questions were at or above the median are identified as "above‐median usage" schools (median = 3.7, corresponding to 80 percent of lessons), while schools whose average response were below the median are identified as "below‐median usage" schools (2.8, corresponding to 57.5 percent of lessons). To identify schools with above‐ or below‐median levels of professional development (PD) aligned to the textbook, we averaged teachers' response regarding the number of total days across their career they participated in textbook‐aligned PD, and we split the sample at the median value of this average (3 days). Schools whose average response to PD questions were at or above the median are identified as "above median" schools (median = 6 days), while schools whose average response were below the median are identified as "low PD" schools (median = 1.5 days). Textbooks identified as a "CCSS Edition" are those identified with "CC" in prior tables (i.e., published after 2011, and publisher indicated textbook was written for or adapted to the CCSS). Robust standard errors in parentheses; "–" indicates that standard errors could not be estimated. <sups>∼</sups>z > 1.64, *z > 1.96, **z > 2.58, ***z > 3.29, where z equals the ratio of a given random effects parameter estimate to its standard error. These z‐scores do not correspond precisely to p‐values as in a traditional linear regression framework, as the confidence interval for a random effect estimate is not symmetric around the estimate (random effect estimates have a lower bound of zero). These traditional markers of significance are included as an aid to reader, but they should be interpreted with caution.</p> <hd id="AN0146104504-20">Variation in Textbook Efficacy by Levels of Use</hd> <p>First, we created an index of textbook usage for different purposes, including choosing lesson objective, creating tasks and activities, and selecting examples (see Table 5 for survey item text), averaging across items within teachers and then across teachers within schools. Then, we split schools into two groups, depending on whether they were above or below the median in teacher‐reported textbook use. On average, teachers in above‐median usage schools used the textbook for one of a range of purposes in roughly 81 percent of lessons, while teachers in below‐median usage schools used the textbook in roughly 53 percent of lessons. Information on usage comes from our teacher survey, and thus was available for a subset of schools using one of the top seven CCSS‐edition textbooks by market share. To maximize the sample size, we categorized schools as above‐ or below‐median usage based on their reported usage in 2016/2017, but then used all available years of achievement gains.</p> <p>If teacher substitution of materials was blurring the differences between more and less effective texts, we might expect to see greater variation in textbook effects in the schools with above‐median levels of usage. However, we find little evidence of differences in textbook efficacy in either above‐ or below‐median usage schools. In panel A of Table 10, we estimated zero variance in underlying textbook efficacy for above‐median usage schools in our pooled sample of five states (excluding California), as well as in California on its own. We also estimated zero variation in textbook efficacy for below‐median usage schools in the five‐state sample. In California, the estimate for below‐median usage schools in California is 0.037 SD.</p> <hd id="AN0146104504-21">Variation in Textbook Efficacy by Days of Textbook‐Aligned Professional Development</hd> <p>To get the most out of high‐quality curriculum materials, districts or schools may have to provide more training and support for teachers (for a review of this large body of research, see O'Donnell, 2008). In panel B of Table 10, we disaggregated results for schools where teachers reported that they received above‐median levels of professional development aligned to their textbook versus schools where teachers reported below‐median levels of support. Across all schools in the teacher survey sample, the median amount of textbook‐aligned professional development reported by teachers was three days over the course of their careers. In the above‐median support schools, the median number of days of textbook‐aligned professional development was six; in the below‐median support schools, it was 1.5 days. Despite their differing levels of support, we found no difference in textbook efficacy within either of these two sets of schools, either in our pooled sample of five states (excluding California) or in California on its own.</p> <hd id="AN0146104504-22">Textbook Efficacy Among Pre‐ and Post‐CCSS Texts</hd> <p>With the advent of the CCSS, publishers had a single set of standards around which to write their texts. Before the CCSS, when states each had their own standards, publishers had an incentive to broaden their coverage to contain nearly every states' standards in a given grade. Every textbook was "a mile wide and an inch deep" to accommodate multiple state standards (to paraphrase Schmidt et al., 2001; Schmidt, McKnight, & Raizen, 1997), but on any given standard some textbooks may have gone deeper than others. One of the goals of the CCSS was to allow textbook publishers to focus on a shorter list of standards in more depth. Our failure to find a difference in textbook efficacy may simply mean that the policy goal was achieved: that textbooks differed less in their content and coverage—and, in turn, in their efficacy—under the CCSS than they had previously.</p> <p>Even after states adopted the CCSS, a nontrivial share of schools still were using pre‐CCSS textbook editions: 37 percent of schools across our six states were using pre‐CCSS editions in 2014/2015, and 16 percent still were using pre‐CCSS editions in the latest year of data available. In panel C of Table 10, we estimated the underlying variance in textbook efficacy separately for the samples of schools using pre‐ and post‐CCSS editions. Here, we could leverage our original school‐survey sample, and not be limited to the schools with teacher surveys. In California as well as in the remaining five states, the point estimate of the underlying variance in efficacy is larger among the pre‐CCSS texts than among the CCSS editions: 0.042 SD versus 0.022 SD in California, and 0.013 SD versus zero SD in the remaining five states. Although consistent with the hypothesis that the pre‐CCSS texts differed more from each other than the post‐CCSS texts, the differences are not large.</p> <hd id="AN0146104504-23">Variation in Textbook Efficacy by Years Since Adoption</hd> <p>Finally, it could be that the variation in textbook efficacy is muted during the first year following adoption, as teachers gain facility with the text and incorporate the new material into their lessons. Teachers and schools may have to use a given text for more than one year before the efficacy differences emerge. In panel D of Table 10, we split schools into two subgroups based on whether the school was in its first year of usage or not. Here, we leveraged our full school‐survey sample, but limited to the top 15 textbooks by market share. In the pooled five‐state sample (excluding California), there was no difference in the underlying variation in textbook efficacy, whether schools were in their first year of use or in their second year or higher. In California, however, we do see evidence that the SD in textbook efficacy is larger in the schools using a text for two or more years (0.035 SD, compared to zero SD for those schools using a textbook for one year). In other words, the non‐zero estimate of textbook efficacy in California from prior analyses (see Table 9) is coming from the schools that had been using the text for more than one year.</p> <p>In addition to returns for <emph>schools</emph> of using a text for more than one year, it also is possible that there is a benefit when <emph>students</emph> use the same textbook series across years and grades. The content would not be exactly the same, but the approach to the material would be similar. To examine this possibility, in Table A9 we disaggregate our efficacy estimates by grade (fourth versus fifth) and examine the sensitivity of these estimates to use of one‐ versus two‐year lagged test scores. If textbooks are a cross‐grade intervention, controlling for the prior‐year test score may attenuate the textbook effect. Because most states administer end‐of‐year tests starting in third grade, we could only compare results for one‐ versus two‐year test score lags for fifth grade. We present results for all school‐year observations, as well as those schools that kept the same text for at least two consecutive years; this approach ensures that 5th‐grade students were exposed to the same series in that grade and in the prior grade. We excluded Louisiana from these analyses, as the data use agreement did not allow for disaggregation and sharing of value‐added estimates by school, grade, and year. We also had to exclude some estimates for California due to data constraints (i.e., two rather than three years of textbook data, and a one‐year hiatus in test score reporting in the lead‐up to the new CCSS‐aligned assessment).</p> <p>Consistent with results presented in Table 10, in Table A9 we find some evidence of greater variation amongst 5th‐grade students, who had more exposure to a given text, relative to 4th‐grade students. In the pooled four‐state sample (excluding California and Louisiana) and in California on its own, point estimates for 4th‐grade students are zero. For 5th‐grade students in California, estimates range from 0.02 to 0.04 SD, with larger estimates when controlling for 3rd‐grade rather than 4th‐grade achievement and for 5th‐grade students in non‐switcher schools. In the other four states, we find small, non‐zero point estimates for 5th graders in some models.</p> <hd id="AN0146104504-24">Additional Robustness Checks to Test Identifying Assumptions</hd> <p>Interpreting textbook estimates as causal effects requires an assumption of conditional exogeneity. That is, controlling for observable school and district characteristics accounts sufficiently for nonrandom selection of textbooks by schools and districts. In addition to specifying models that replace observable characteristics with school fixed effects (see Table 8), we found that our primary random effect estimates were robust to different subsets of the school‐ and district‐level controls (see Table A10, panels A through C).</p> <p>Finally, we conducted a placebo test to examine whether math textbooks had an "impact" on ELA achievement, which they should not. To do this, we needed to condition our estimates on the ELA textbook used. Otherwise, we might simply find that the districts and schools that succeeded in choosing more effective math textbooks also were good at choosing ELA textbooks. It is only conditional on the ELA textbook that the math textbook should be irrelevant for gains in students' ELA test scores. In California, the only state where we had data on ELA textbooks, we found that math textbooks do appear to have a statistically significant relationship to ELA achievement (see Table A10, panel D). As expected, though, the relationship is small and roughly 50 percent of the magnitude of math textbooks on math achievement for this state (see Table 9).</p> <hd id="AN0146104504-25">RECONCILING WITH THE PREVIOUS LITERATURE</hd> <p>How do we reconcile our results with the previous literature, which suggested larger differences in student achievement gains between schools using different texts? There are several possibilities. First, the literature on the efficacy of alternative curricula is still in its infancy. While we did not find evidence of large differences in achievement gains for schools using different texts, it could be that the inclusion of more years and more states would point to larger differences than we have seen. Since we completed our data collection, new textbooks and curriculum materials have entered the market. Given the potential value of the curriculum lever, we hope future research will continue to try to resolve the differences between our findings and the earlier research.</p> <p>A second possibility is that our value‐added methodology is biased. As a result, we could be understating the differences in textbook efficacy. In contrast to the value‐added methodology, the randomized trials would have ensured that schools using different texts were similar. Our school fixed effects models aim to account for fixed, time‐invariant differences across schools, and we found that estimates from these models were similar to those from other models that excluded school fixed effects. However, school fixed effects cannot account for time‐varying characteristics such as turnover of school leadership that may be correlated with textbook selection and textbook efficacy. At the same time, if we were understating the efficacy of textbooks based on bias due to unmeasured school characteristics, it would be an unusual form of bias in which low‐growth schools were using better textbooks, and high‐growth schools were using less effective ones. The bias due to unmeasured school characteristics would have had to be of equal and opposite magnitude to the textbook effects. Typically, we would have expected to see unmeasured traits exaggerating the efficacy of interventions, with more advantaged (or better‐managed) schools compounding their advantage by purchasing more effective textbooks.</p> <p>A third possibility is that the findings from the randomized trials are not generalizable. Although randomized trials accurately reflect the causal effect of textbooks for the population studied, the population of schools that were willing to have their textbook randomly assigned may have been unusual. In the Agodini et al. (2010) experiment, the small percentage of districts (2.5 percent) that were willing to participate in such a study may have been particularly dissatisfied with the texts they were using and may have benefited more from the change than other districts would have. Similar to Agodini et al. (2010), Eddy et al. (2014) had to contact over 6,000 school districts and principals to recruit nine participating schools (less than 0.15 percent participation rate). Just as other non‐experimental studies have found, we see evidence of differences in achievement for some textbooks when focusing on one state. However, no single text proved to be more or less effective across multiple states. The fact that <emph>Saxon Math</emph> was among the most effective texts in the randomized trial by Agodini et al. (2010) but among the least effective texts in Bhatt and Koedel (2012) may reflect the same lack of generalizability. We recognize the usual trade‐off between internal and external validity. In our study, we placed a stronger priority on the latter, while still assessing identifying assumptions and threats to internal validity.</p> <p>A fourth, related possibility is that the answer is different in upper‐elementary grades (where our study focused because of our need for prior achievement controls) than in lower‐elementary grades (where much of the prior research was concentrated). In other words, the results of a given study may not generalize across grades. The randomized trial conducted by Agodini et al. (2010) focused on first and second graders, and the non‐experimental studies by Koedel and his co‐authors focused primarily on 3rd‐grade achievement (Bhatt & Koedel, 2012; Bhatt, Koedel, & Lehmann, 2013; Koedel et al., 2017). The types of math skills tested in early elementary grades (e.g., single digit addition and subtraction) may be more sensitive to interventions of all types than the math skills measured on the 4th‐ and 5th‐grade assessments (e.g., fractions, multi‐digit multiplication and division). In one of the randomized studies, Jaciw et al. (2016) did find modest textbook effects in third through fifth grade; but their analysis was limited to a single textbook, <emph>Math in Focus</emph>.</p> <p>A fifth possibility is that the answer has changed since the earlier studies were completed. In the pre‐CCSS era, when textbooks were covering a broader range of topics, textbooks may have differed more in their alignment with any specific test being used to measure efficacy. For instance, if a test included several items measuring students' ability to add fractions with unlike denominators, then the textbooks that emphasized that standard may appear more effective than another text. Yet, on a different test, with fewer items measuring that standard, the textbook rankings may change. This may explain inconsistent results in the Jaciw et al. (2016) randomized trial, where schools using the CCSS edition of <emph>Math in Focus</emph> outperformed on the Stanford Achievement Test but not on the state test. In our study, on the CCSS‐aligned tests, we also saw some evidence that this explanation may have been true. In California, where we had a large sample of schools using pre‐CCSS editions of textbooks, we did find more variation in student achievement gains for schools using these texts, compared to less variation for schools using a post‐CCSS edition textbook.</p> <hd id="AN0146104504-26">CONCLUSION</hd> <p>The adoption of the Common Core State Standards led schools in many states to switch curricula. Thus, the choice of textbook and curriculum has become much more salient, and often is perceived as a decision with high stakes (Pondiscio, 2017; Steiner, 2017). Contrary to prior research, though, we found little evidence of differences in average student achievement growth for elementary schools using different math textbooks in six states that adopted the CCSS standards and were using CCSS‐aligned assessments. We did find some evidence of greater variation in achievement gains among those using pre‐CCSS texts. Yet, the effect size was small.</p> <p>Some may interpret our findings as implying that curriculum choice does not matter. We believe that would be an overstatement. We remind readers that our estimates compare student achievement gains between schools using different textbooks, which say nothing about the difference between adopting a standard textbook versus adopting nothing. (Table 3 shows that very few schools adopted no textbook.) In exploratory analyses, we found some evidence of greater variation among schools and students who had spent more than one year using a specific text, suggesting that there may be returns to experience.</p> <p>It may also be that we just did not see sufficiently intensive usage or training in our sample to detect the differences in student achievement between texts. Although teachers in the vast majority of sampled schools (94 percent) reported using the official textbook for <emph>some</emph> purpose in a majority of their classes, few teachers hewed closely to the text. Teachers in just 25 percent of schools reported using the textbook in nearly all of their lessons and for multiple purposes: "to create mathematical tasks and activities," "to select examples to present in class," "as a source of practice problems that students work on independently during class time," <emph>and</emph> "as a source of problems for students to complete outside of class." Similarly, teachers reported modest amounts of training in the use of their texts. The average teacher received just one day of training in the current year, and fewer than four days over their entire careers. Even in the schools with above‐average levels of training, teachers reported receiving six days of training in their text over the course of their careers. Given districts' investments in curricula, these do not seem like large expenditures of time or funding.</p> <p>In light of the patterns of implementation, some readers may continue to advocate for the importance of curriculum choice, despite our overall null results related to efficacy. However, those who want to hold on to the importance of curriculum as a primary lever for reform need to be able to identify the level of support and training required for such curriculum changes to actually bear fruit in the classroom. It is possible that closer adherence to a high‐quality curriculum would produce benefits, but we still need to answer several questions: What levels of support are required to produce greater levels of adherence? Do the desired student achievement benefits appear afterwards? What are the costs associated with these supports, and are they justified given the observed effects?</p> <p>Citing the earlier research, Chingos and Whitehurst (2012) posed a choice between "challenging, expensive, and time consuming" efforts to improve teaching quality and the "relatively easy, inexpensive, and quick" choice of a higher‐quality curriculum. While our findings certainly cast doubt on the proposition that there are quick and easy payoffs to curriculum changes, the bigger error may be in thinking of curriculum choice and teaching reforms as alternatives. It could be that in order to gain the benefits of either, districts must do both.</p> <hd id="AN0146104504-27">ACKNOWLEDGMENTS</hd> <p>We gratefully acknowledge funding from the Bill & Melinda Gates Foundation, the Charles and Lynn Schusterman Foundation, the William and Flora Hewlett Foundation, and the Bloomberg Foundation. The research reported here also was supported by the Institute of Education Sciences, U.S. Department of Education, through grant R305B150010 to Harvard University and Grant R305E150006 to the Regents of the University of California (Michal Kurlaender, Principal Investigator, UC Davis School of Education) in partnership with the California Department of Education (Jonathan Isler, Co‐Principal Investigator). The opinions expressed are those of the authors and do not represent views of the Institute of Education Sciences, the philanthropic funders, the departments of education in any of the participating states, the research institutions, nor the advisory board members.</p> <p>Thomas Kelley‐Kemple, Jake Kramer, and Virginia Lovison at Harvard University; Lihan Liu at Tulane University; Matthew Naven and Derek Rury at the University of California–Davis provided excellent research assistance. Rachel Urso and Sophie Houstoun at the Center for Education Policy Research at Harvard University led the recruitment of schools and teachers for our surveys, and Nate Brown at the University of Washington led additional outreach in Washington state. Eric Hirsch, Lauren Weisskirk, and Mark LaVenia at EdReports provided invaluable support and feedback in understanding the breadth of curriculum choices and textbook alignment to the Common Core. Our project depended upon the collaboration and support of our state partners, including John White, Jessica Baghian, Kim Nesmith, Rebecca Kockler, and Alicja Witkowski at the Louisiana Department of Education; Carol Williamson and Debra Ward from the Maryland Department of Education; Peter Shulman, James Riddlesperger, LaShona Burke, and Jessica Merville at the New Jersey Department of Education; and Christopher Ruzkowski and Anthony Burns from the New Mexico Public Education Department. The study also benefited from the experience and feedback from an advisory board of experts on curriculum design and value‐added methodology, including: Matthew Chingos (Urban Institute), Erin Grogan and Dan Weisberg (TNTP), Cory Koedel (University of Missouri), Darleen Opfer and Julia Kaufman (RAND Corporation), Grover J. "Russ" Whitehurst (Brookings), and Jason Zimba (Student Achievement Partners). Finally, we thank the thousands of district leaders, school principals, administrators, and classroom teachers who generously provided input about their curriculum choices to ensure that our project was a success.</p> <p>APPENDIX</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/JPA/01sep20/pam22257-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="pam22257-fig-0001.jpg" title="A1 Clustered versus Non‐Clustered Standard Errors by Textbook. [Color figure can be viewed at wileyonlinelibrary.com]Notes: Textbook fixed effects are estimated from a mixed‐effects linear regression of school value‐added on a set of binary indicators for whether a school used a given textbook, school‐by‐year demographic characteristics, 2010 to 2014 district census characteristics, and state‐by‐year fixed effects. The omitted textbook category is enVision CC. The graph plots the ratio of standard errors clustered at the district level to the random effects standard errors for each textbook, against the number of districts (clusters) using each textbook. The sample is restricted to school‐by‐year observations in Louisiana, Maryland, New Jersey, New Mexico, and Washington with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year that are known to have used one of the top 15 textbooks by market share." /> </p> <p></p> <p>Table Characteristics of participating states</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>State Characteristics</th><th align="center">U.S.</th><th align="center">California</th><th align="center">Louisiana</th><th align="center">Maryland</th><th align="center">New Jersey</th><th align="center">New Mexico</th><th align="center">Washington</th></tr></thead><tbody><tr><td>Public School Enrollment, 4th and 5th Grade</td><td align="center">7,736,742</td><td align="center">958,634</td><td align="center">109,825</td><td align="center">136,269</td><td align="center">201,131</td><td align="center">52,050</td><td align="center">169,753</td></tr><tr><td>Free or Reduced‐Price Lunch (%)</td><td>52.3</td><td>58.1</td><td>63.0</td><td>46.7</td><td>37.9</td><td>71.4</td><td>43.6</td></tr><tr><td>Special Education (%)</td><td>13.7</td><td>12.2</td><td>11.8</td><td>12.1</td><td>16.9</td><td>15.8</td><td>12.9</td></tr><tr><td>English Language Learner (%)</td><td>9.6</td><td>20.2</td><td>3.2</td><td>7.8</td><td>5.0</td><td>13.4</td><td>11.1</td></tr><tr><td>Male (%)</td><td>51.3</td><td>51.3</td><td>51.2</td><td>51.0</td><td>51.3</td><td>51.1</td><td>51.5</td></tr><tr><td>African American (%)</td><td>15.2</td><td>5.5</td><td>43.6</td><td>33.7</td><td>15.3</td><td>1.9</td><td>4.4</td></tr><tr><td>Asian (%)</td><td>5.2</td><td>11.6</td><td>1.6</td><td>6.6</td><td>10.0</td><td>1.1</td><td>7.7</td></tr><tr><td>Hispanic (%)</td><td>26.7</td><td>54.3</td><td>6.9</td><td>17.4</td><td>28.7</td><td>61.7</td><td>23.2</td></tr><tr><td>Native American (%)</td><td>1.0</td><td>0.5</td><td>0.7</td><td>0.3</td><td>0.1</td><td>10.0</td><td>1.2</td></tr><tr><td>Mixed Race or Other (%)</td><td>3.9</td><td>4.5</td><td>2.5</td><td>4.6</td><td>2.0</td><td>1.9</td><td>8.0</td></tr><tr><td>NAEP, 4th Grade Math (Standardized)</td><td>0.000</td><td>−0.258</td><td>−0.355</td><td>0.032</td><td>0.258</td><td>−0.323</td><td>0.065</td></tr><tr><td>NAEP, 4th Grade Reading (Standardized)</td><td>0.000</td><td>−0.184</td><td>−0.263</td><td>0.079</td><td>0.289</td><td>−0.368</td><td>0.026</td></tr><tr><td>Per‐pupil Instructional Expenditure ($)</td><td align="center">7,445</td><td align="center">7,203</td><td align="center">6,354</td><td align="center">9,515</td><td align="center">11,762</td><td align="center">5,694</td><td align="center">6,940</td></tr><tr><td>Parents Married (%)</td><td>62.7</td><td>65.5</td><td>51.9</td><td>61.3</td><td>67.0</td><td>57.1</td><td>66.9</td></tr><tr><td>Speaks Language Other than English (%)</td><td>22.8</td><td>47.0</td><td>6.2</td><td>17.3</td><td>29.4</td><td>32.7</td><td>22.9</td></tr><tr><td>Median Household Income ($)</td><td align="center">55,600</td><td align="center">64,200</td><td align="center">46,200</td><td align="center">76,700</td><td align="center">74,500</td><td align="center">46,400</td><td align="center">63,600</td></tr><tr><td>Parent Attended College, no BA (%)</td><td>32.3</td><td>29.3</td><td>32.8</td><td>28.8</td><td>25.7</td><td>34.9</td><td>35.4</td></tr><tr><td>Parent Holds BA Degree + (%)</td><td>29.8</td><td>26.5</td><td>19.7</td><td>39.2</td><td>40.6</td><td>22.1</td><td>31.7</td></tr></tbody></table> </ephtml> </p> <p>15 <emph>Notes</emph>: Enrollment, student demographic, and expenditure data come from the 2016/2017 school year. Test score data come from the 2017 National Assessment of Educational Progress (NAEP), which is standardized based on the national mean and standard deviation. Additional data on parents and families come from the 2010 to 2014 American Community Survey.</p> <p>Table Observable characteristics of schools using different textbooks (top 15) by state</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Textbooks (Sorted by EdReports Ratings for Alignment to CCSS)</th><th align="center">California</th><th align="center">Louisiana</th><th align="center">Maryland</th><th align="center">New Jersey</th><th align="center">New Mexico</th><th align="center">Washington</th></tr></thead><tbody><tr><td>Panel A</td><td align="center">Prior Math Test Score (Standardized)</td></tr><tr><td>A. Minimum EdReports CCSS Alignment Rating: Meets Expectations</td></tr><tr><td><italic>Bridges in Mathematics</italic> CC</td><td>0.159</td><td /><td /><td /><td /><td>0.062</td></tr><tr><td><italic>Engage NY/Eureka</italic> CC</td><td>0.012</td><td>0.007</td><td>−0.624</td><td>−0.559</td><td /><td>−0.147</td></tr><tr><td><italic>My Math</italic> CC</td><td>−0.150</td><td>0.024</td><td>−0.017</td><td>−0.262</td><td>−0.063</td><td>−0.117</td></tr><tr><td><italic>Ready Common Core</italic> CC</td><td>0.278</td><td>−0.258</td><td>0.160</td><td /><td /><td /></tr><tr><td>B. Minimum EdReports CCSS Alignment Rating: Partially Meets Expectations</td></tr><tr><td><italic>Go Math</italic> CC</td><td>−0.027</td><td>−0.018</td><td>0.084</td><td>−0.065</td><td>−0.178</td><td>0.07</td></tr><tr><td><italic>Math Expressions</italic> CC</td><td>0.056</td><td>0.165</td><td /><td>0.040</td><td /><td>−0.061</td></tr><tr><td>C. Other CCSS Editions</td></tr><tr><td><italic>enVision</italic> CC</td><td>−0.014</td><td>−0.141</td><td>−0.092</td><td>0.037</td><td>0.05</td><td>0.183</td></tr><tr><td><italic>Everyday Mathematics</italic> CC</td><td>0.023</td><td /><td>−0.061</td><td>0.054</td><td>−0.135</td><td>0.366</td></tr><tr><td><italic>Math in Focus</italic> CC</td><td>0.161</td><td /><td /><td>−0.21</td><td>−0.181</td><td>0.011</td></tr><tr><td><italic>Stepping Stones</italic> CC</td><td>0.184</td><td /><td>−0.077</td><td /><td>−0.049</td><td>−0.101</td></tr><tr><td>D. Non‐CCSS Editions</td></tr><tr><td><italic>enVision</italic></td><td>−0.078</td><td>−0.518</td><td>0.116</td><td>0.328</td><td /><td>0.327</td></tr><tr><td><italic>Everyday Mathematics</italic></td><td>0.159</td><td /><td>0.55</td><td>−0.01</td><td>1.491</td><td>−0.067</td></tr><tr><td><italic>Houghton Mifflin Math</italic></td><td>0.173</td><td /><td /><td>0.326</td><td /><td /></tr><tr><td><italic>Math Connects</italic></td><td /><td /><td>0.068</td><td>0.006</td><td /><td>−0.059</td></tr><tr><td><italic>Math Expressions</italic></td><td>0.247</td><td /><td /><td>0.292</td><td /><td>−0.024</td></tr><tr><td>Panel B</td><td align="center">Free or Reduced‐Price Lunch (%)</td></tr><tr><td>A. Minimum EdReports CCSS Alignment Rating: Meets Expectations</td></tr><tr><td><italic>Bridges in Mathematics</italic> CC</td><td>43.4</td><td /><td /><td /><td /><td>43.4</td></tr><tr><td><italic>Engage NY/Eureka</italic> CC</td><td>59.9</td><td>69.1</td><td>74.6</td><td>61.1</td><td /><td>62.7</td></tr><tr><td><italic>My Math</italic> CC</td><td>72.4</td><td>72.4</td><td>45.5</td><td>51.6</td><td>81.9</td><td>53.5</td></tr><tr><td><italic>Ready Common Core</italic> CC</td><td>47.9</td><td>93.4</td><td>40.9</td><td /><td /><td /></tr><tr><td>B. Minimum EdReports CCSS Alignment Rating: Partially Meets Expectations</td></tr><tr><td><italic>Go Math</italic> CC</td><td>64.2</td><td>59.0</td><td>33.4</td><td>39.9</td><td>95.5</td><td>68.4</td></tr><tr><td><italic>Math Expressions</italic> CC</td><td>59.2</td><td>40.9</td><td /><td>25.7</td><td /><td>48.8</td></tr><tr><td>C. Other CCSS Editions</td></tr><tr><td><italic>enVision</italic> CC</td><td>65.0</td><td>84.0</td><td>50.9</td><td>38.0</td><td>89.5</td><td>38.2</td></tr><tr><td><italic>Everyday Mathematics</italic> CC</td><td>50.4</td><td /><td>65.2</td><td>35.5</td><td>83.1</td><td>18.7</td></tr><tr><td><italic>Math in Focus</italic> CC</td><td>50.3</td><td /><td /><td>49.0</td><td>67.5</td><td>47.9</td></tr><tr><td><italic>Stepping Stones</italic> CC</td><td>53.5</td><td /><td>76.4</td><td /><td>78.2</td><td>64.2</td></tr><tr><td>D. Non‐CCSS Editions</td></tr><tr><td><italic>enVision</italic></td><td>67.7</td><td>83.0</td><td>57.4</td><td>11.2</td><td /><td>30.2</td></tr><tr><td><italic>Everyday Mathematics</italic></td><td>54.7</td><td /><td>20.2</td><td>41.8</td><td>12.6</td><td>50.3</td></tr><tr><td><italic>Houghton Mifflin Math</italic></td><td>51.1</td><td /><td /><td>12.8</td><td /><td /></tr><tr><td><italic>Math Connects</italic></td><td /><td /><td>29.9</td><td>44.4</td><td /><td>56.0</td></tr><tr><td><italic>Math Expressions</italic></td><td>51.3</td><td /><td /><td>15.9</td><td /><td>48.1</td></tr><tr><td>School*Year Observations</td><td align="center">8,121</td><td align="center">292</td><td align="center">191</td><td align="center">840</td><td align="center">822</td><td align="center">531</td></tr></tbody></table> </ephtml> </p> <p>16 <emph>Notes</emph>: Estimates are weighted by the inverse of a school's sampling probability. The sample is restricted to school‐by‐year observations known to have adopted one of the top 15 textbooks by market share.</p> <p>Table Pairwise correlations between observable characteristics of schools choosing different textbooks (top 15) across states</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>States</th><th align="center">California</th><th align="center">Louisiana</th><th align="center">Maryland</th><th align="center">New Jersey</th><th align="center">New Mexico</th><th align="center">Washington</th></tr></thead><tbody><tr><td>Panel A</td><td align="center">Prior Math Test Score</td></tr><tr><td>California</td><td>1.000</td><td /><td /><td /><td /><td /></tr><tr><td>Louisiana</td><td>−0.082</td><td>1.000</td><td /><td /><td /><td /></tr><tr><td>Maryland</td><td>0.296</td><td>−0.496</td><td>1.000</td><td /><td /><td /></tr><tr><td>New Jersey</td><td>0.324</td><td>−0.606</td><td>0.642∼</td><td>1.000</td><td /><td /></tr><tr><td>New Mexico</td><td>0.365</td><td>−0.714</td><td>0.935**</td><td>0.279</td><td>1.000</td><td /></tr><tr><td>Washington</td><td>−0.281</td><td>−0.865*</td><td>0.142</td><td>0.566∼</td><td>−0.297</td><td>1.000</td></tr><tr><td>Panel B</td><td align="center">Free or Reduced‐Price Lunch (%)</td></tr><tr><td>California</td><td>1.000</td><td /><td /><td /><td /><td /></tr><tr><td>Louisiana</td><td>−0.162</td><td>1.000</td><td /><td /><td /><td /></tr><tr><td>Maryland</td><td>−0.086</td><td>0.031</td><td>1.000</td><td /><td /><td /></tr><tr><td>New Jersey</td><td>0.191</td><td>0.000</td><td>0.028</td><td>1.000</td><td /><td /></tr><tr><td>New Mexico</td><td>0.367</td><td>−0.477</td><td>0.544</td><td>−0.119</td><td>1.000</td><td /></tr><tr><td>Washington</td><td>0.152</td><td>−0.503</td><td>−0.147</td><td>0.493</td><td>−0.001</td><td>1.000</td></tr></tbody></table> </ephtml> </p> <ulist> <item>17 <emph>Notes</emph>: Correlations are estimated from a textbook‐level dataset where variables are the mean background characteristic (i.e., prior math test score, percent eligible for free or reduced‐price lunch) in each of six states, and each observation is one of the top 15 textbooks by market share.</item> <item>18 <sups>∼</sups>p <.10; *p <.05; **p <.01.</item> </ulist> <p>Table Comparing schools that switched textbooks to those that did not</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">Pooled 5 States (Excluding California)</th><th align="center">California</th></tr><tr valign="bottom"><th>School or District Characteristics</th><th align="center">Switchers</th><th align="center">Difference for Non‐Switchers</th><th align="center">Switchers</th><th align="center">Difference for Non‐Switchers</th></tr></thead><tbody><tr><td>Free or Reduced‐Price Lunch (%)</td><td>47.4</td><td>11.7</td><td>64.4</td><td>−2.3</td></tr><tr><td>Special Education (%)</td><td align="center">15</td><td>0.7</td><td>11.7</td><td>0.1</td></tr><tr><td>English Language Learner (%)</td><td>7.3</td><td>1.5</td><td>24.1</td><td>1.1</td></tr><tr><td>Male (%)</td><td>51.5</td><td>−0.3</td><td>50.9</td><td>0.1</td></tr><tr><td>African American (%)</td><td>15.3</td><td>−0.9</td><td>7.1</td><td>−1.7</td></tr><tr><td>Asian (%)</td><td>5.4</td><td>−0.5</td><td>11.8</td><td>−0.9</td></tr><tr><td>Hispanic (%)</td><td>17.9</td><td>13.1*</td><td>55.8</td><td>−2.6</td></tr><tr><td>Native American (%)</td><td>1.9</td><td>2</td><td>0.6</td><td>0.2</td></tr><tr><td>Mixed Race or Other (%)</td><td>3.2</td><td>−0.4</td><td>2.9</td><td>0.6</td></tr><tr><td>Prior Math Test Score (Standardized)</td><td>0.001</td><td>−0.009</td><td>−0.023</td><td>0.012</td></tr><tr><td>Prior Reading Test Score (Standardized)</td><td>0.016</td><td>−0.013</td><td>−0.037</td><td>0.029</td></tr><tr><td>Per‐pupil Instructional Expenditure ($)</td><td align="center">8,073</td><td align="center">−783</td><td align="center">5,818</td><td align="center">−162</td></tr><tr><td>Parents Married (%)</td><td>65.6</td><td>−4.1</td><td>63.9</td><td>1.7</td></tr><tr><td>Speaks Language Other than English (%)</td><td>18.7</td><td>5.4*</td><td>51.6</td><td>−7.2</td></tr><tr><td>Median Household Income ($)</td><td align="center">75,433</td><td align="center">−12,047</td><td align="center">60,658</td><td align="center">1,972</td></tr><tr><td>Parent Attended College, no BA (%)</td><td>30.6</td><td>0.2</td><td>26.6</td><td>3.0∼</td></tr><tr><td>Parent Holds BA Degree + (%)</td><td>31.7</td><td>−4.3</td><td>23.7</td><td>−0.2</td></tr><tr><td><italic>P</italic>‐Value from Test of Joint Significance</td><td /><td>0.053</td><td /><td>0.857</td></tr><tr><td>School Observations</td><td align="center">161</td><td align="center">987</td><td align="center">1,060</td><td align="center">4,047</td></tr></tbody></table> </ephtml> </p> <ulist> <item>19 <emph>Notes</emph>: Expenditure data and family characteristics comes from the 2010 to 2014 American Community Survey, captured at the district level. Other characteristics captured at the school level.</item> <item>20 <sups>∼</sups>p <.10; *p <.05.</item> </ulist> <p>Table Sensitivity of fixed effect estimates to textbook sample (pooled 6 states, all years)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">All Known Textbooks</th><th align="center">Top 15 Textbooks</th><th align="center">Top 10 Textbooks</th><th align="center">Top 5 Textbooks</th></tr><tr><th /><th align="center">(1)</th><th align="center">(2)</th><th align="center">(3)</th><th align="center">(4)</th></tr></thead><tbody><tr><td>A. Minimum EdReports CCSS Alignment Rating: Meets Expectations</td></tr><tr><td><italic>Bridges in Mathematics</italic> CC</td><td>0.011</td><td>0.010</td><td>0.01</td><td /></tr><tr><td /><td>(0.017)</td><td>(0.018)</td><td>(0.018)</td><td /></tr><tr><td><italic>Engage NY/Eureka</italic> CC</td><td>−0.004</td><td>−0.003</td><td>−0.001</td><td>−0.008</td></tr><tr><td /><td>(0.011)</td><td>(0.011)</td><td>(0.011)</td><td>(0.011)</td></tr><tr><td><italic>My Math</italic> CC</td><td>0.018∼</td><td>0.019∼</td><td>0.018*</td><td>0.013</td></tr><tr><td /><td>(0.010)</td><td>(0.010)</td><td>(0.01)</td><td>(0.01)</td></tr><tr><td><italic>Ready Common Core</italic> CC</td><td>−0.019</td><td>−0.020</td><td /><td /></tr><tr><td /><td>(0.039)</td><td>(0.039)</td><td /><td /></tr><tr><td>B. Minimum EdReports CCSS Alignment Rating: Partially Meets Expectations</td></tr><tr><td><italic>Go Math</italic> CC</td><td>0.003</td><td>0.001</td><td>0.006</td><td>0.007</td></tr><tr><td /><td>(0.010)</td><td>(0.010)</td><td>(0.01)</td><td>(0.01)</td></tr><tr><td><italic>Math Expressions</italic> CC</td><td>0.038**</td><td>0.039**</td><td>0.038***</td><td /></tr><tr><td /><td>(0.013)</td><td>(0.013)</td><td>(0.013)</td><td /></tr><tr><td>C. Other CCSS Editions</td></tr><tr><td><italic>Everyday Mathematics</italic> CC</td><td>−0.009</td><td>−0.009</td><td>−0.009</td><td /></tr><tr><td /><td>(0.012)</td><td>(0.012)</td><td>(0.012)</td><td /></tr><tr><td><italic>Math in Focus</italic> CC</td><td>0.003</td><td>0.002</td><td>0.004</td><td /></tr><tr><td /><td>(0.017)</td><td>(0.017)</td><td>(0.017)</td><td /></tr><tr><td><italic>Stepping Stones</italic> CC</td><td>0.009</td><td>0.010</td><td /><td /></tr><tr><td /><td>(0.030)</td><td>(0.031)</td><td /><td /></tr><tr><td>D. Non‐CCSS Editions</td></tr><tr><td><italic>enVision</italic></td><td>−0.028**</td><td>−0.029**</td><td>−0.028***</td><td>−0.013</td></tr><tr><td /><td>(0.010)</td><td>(0.010)</td><td>(0.011)</td><td>(0.011)</td></tr><tr><td><italic>Everyday Mathematics</italic></td><td>0.088***</td><td>0.086***</td><td>0.089***</td><td /></tr><tr><td /><td>(0.015)</td><td>(0.015)</td><td>(0.015)</td><td /></tr><tr><td><italic>Houghton Mifflin Math</italic></td><td>−0.018</td><td>−0.020</td><td /><td /></tr><tr><td /><td>(0.015)</td><td>(0.015)</td><td /><td /></tr><tr><td><italic>Math Connects</italic></td><td>0.012</td><td>0.015</td><td /><td /></tr><tr><td /><td>(0.027)</td><td>(0.028)</td><td /><td /></tr><tr><td><italic>Math Expressions</italic></td><td>−0.018</td><td>−0.016</td><td /><td /></tr><tr><td /><td>(0.033)</td><td>(0.034)</td><td /><td /></tr><tr><td><italic>P</italic>‐Value from Test of Joint Significance</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.026</td></tr><tr><td>SD of Textbook Fixed Effects</td><td>0.085</td><td>0.029</td><td>0.032</td><td>0.011</td></tr><tr><td>School*Year Observations</td><td align="center">11,516</td><td align="center">10,797</td><td align="center">9,911</td><td align="center">7,618</td></tr></tbody></table> </ephtml> </p> <ulist> <item>21 Notes: Estimates in each column come from separate linear regression models of school value‐added. We report coefficients for a set of binary indicators for each textbook. The omitted textbook category is <emph>enVision</emph> CC, which is in the "other CCSS editions" category of EdReports ratings. All models are multi‐level, mixed‐effects models that include random effects for schools nested within districts. We control for school‐by‐year demographic characteristics, 2010 to 2014 district census characteristics, and state‐by‐year fixed effects. The sample is restricted to school‐by‐year observations with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year. Model (<reflink idref="bib1" id="ref18">1</reflink>) includes school‐by‐year observations using any of the 38 known textbooks in our sample, though we only show estimates for the top 15 by market share given that most of these textbooks where used by very few schools; coefficients on these texts are not particularly informative. Empty cells indicate that a given textbook was not in the relevant sample for a given analysis (i.e., Top 10 or Top 5 by market share).</item> <item>22 <sups>∼</sups>p <.10; *p <.05; **p <.01; ***p <.001.</item> </ulist> <p>Table Correlations between textbook fixed effect estimates across states and years</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">Average Textbook Effects (Top 15 Textbooks)</th><th align="center">Marginal Effects for Textbook Switchers (All Known Textbooks)</th></tr><tr valign="bottom"><th /><th align="center">Pooled 6 States: All Years</th><th align="center">Pooled 5 States (Excluding California): All Years</th><th align="center">California: All Years</th><th align="center">Pooled 6 States: 2014‐15</th><th align="center">Pooled 6 States: 2015‐16</th><th align="center">Pooled 4 States (Excluding California and Louisiana): 2016‐17</th><th align="center">Pooled 5 States: With School Fixed Effects</th><th align="center">Pooled 5 States: Switchers Only, No School Fixed Effects</th><th align="center">California: With School Fixed Effects</th><th align="center">California: Switchers Only, No School Fixed Effects</th></tr></thead><tbody><tr><td>Average Textbook Effects (Top 15 Textbooks)</td></tr><tr><td>Pooled 6 States: All Years</td><td align="center">1</td><td /><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>Pooled 5 States (Excluding California): All Years</td><td>0.092</td><td align="center">1</td><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>California: All Years</td><td>0.872***</td><td>−0.159</td><td align="center">1</td><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>Pooled 6 States: 2014–15</td><td>0.419</td><td>−0.122</td><td>0.367</td><td align="center">1</td><td /><td /><td /><td /><td /><td /></tr><tr><td>Pooled 6 States: 2015–16</td><td>0.07</td><td>0.201</td><td>0.058</td><td>−0.023</td><td align="center">1</td><td /><td /><td /><td /><td /></tr><tr><td>Pooled 4 States (Excluding California and Louisiana): 2016–17</td><td>0.098</td><td>−0.062</td><td>−0.125</td><td>0.564*</td><td>0.058</td><td align="center">1</td><td /><td /><td /><td /></tr><tr><td>Marginal Effects for Textbook Switchers (All Known Textbooks)</td></tr><tr><td>Pooled 5 States: With School Fixed Effects</td><td>0.006</td><td>0.606*</td><td>−0.059</td><td>−0.167</td><td>0.322</td><td>−0.251</td><td align="center">1</td><td /><td /><td /></tr><tr><td>Pooled 5 States: Switchers Only, No School Fixed Effects</td><td>−0.182</td><td>0.503∼</td><td>−0.331</td><td>−0.568*</td><td>0.055</td><td>−0.581*</td><td>0.657*</td><td align="center">1</td><td /><td /></tr><tr><td>California: With School Fixed Effects</td><td>0.631*</td><td>−0.104</td><td>0.575∼</td><td>0.169</td><td>0.098</td><td>0.312</td><td>−0.414</td><td>−0.145</td><td align="center">1</td><td /></tr><tr><td>California: Switchers Only, No School Fixed Effects</td><td>0.771**</td><td>0.111</td><td>0.760**</td><td>0.346</td><td>0.193</td><td>0.326</td><td>−0.248</td><td>−0.568∼</td><td>0.571∼</td><td>1.000</td></tr></tbody></table> </ephtml> </p> <ulist> <item>23 <emph>Notes</emph>: Each cell calculates the unweighted correlation of textbook fixed effect estimates from Table 7 (Average Textbook Effects) or Table 8 (Marginal Effects for Textbook Switchers).</item> <item>24 <sups>∼</sups>p <.10; *p <.05; **p <.01; ***p <.001.</item> </ulist> <p>Table Power analyses</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">Market Share of Single "Effective" Textbook</th></tr><tr><th>Effect Size</th><th align="center">1%</th><th align="center">5%</th><th align="center">10%</th><th align="center">20%</th><th align="center">25%</th></tr></thead><tbody><tr><td /><td align="center">Mean Effect = 0.002</td><td>0.003</td><td>0.004</td><td>0.007</td><td>0.007</td></tr><tr><td>0.02 SD</td><td align="center">SD = (0.003)</td><td>(0.004)</td><td>(0.005)</td><td>(0.005)</td><td>(0.004)</td></tr><tr><td /><td align="center">1% p<0.05</td><td align="center">7%</td><td align="center">14%</td><td align="center">33%</td><td align="center">39%</td></tr><tr><td /><td>0.002</td><td>0.003</td><td>0.006</td><td>0.009</td><td>0.010</td></tr><tr><td>0.03 SD</td><td>(0.003)</td><td>(0.004)</td><td>(0.005)</td><td>(0.004)</td><td>(0.003)</td></tr><tr><td /><td align="center">2%</td><td align="center">8%</td><td align="center">34%</td><td align="center">61%</td><td align="center">73%</td></tr><tr><td /><td>0.002</td><td>0.008</td><td>0.013</td><td>0.016</td><td>0.017</td></tr><tr><td>0.05 SD</td><td>(0.003)</td><td>(0.006)</td><td>(0.004)</td><td>(0.002)</td><td>(0.002)</td></tr><tr><td /><td align="center">1%</td><td align="center">34%</td><td align="center">82%</td><td align="center">99%</td><td align="center">100%</td></tr><tr><td /><td>0.003</td><td>0.023</td><td>0.026</td><td>0.028</td><td>0.028</td></tr><tr><td>0.10 SD</td><td>(0.004)</td><td>(0.005)</td><td>(0.003)</td><td>(0.002)</td><td>(0.002)</td></tr><tr><td /><td align="center">3%</td><td align="center">99%</td><td align="center">100%</td><td align="center">100%</td><td align="center">100%</td></tr><tr><td /><td>0.006</td><td>0.036</td><td>0.038</td><td>0.040</td><td>0.040</td></tr><tr><td>0.15 SD</td><td>(0.010)</td><td>(0.004)</td><td>(0.003)</td><td>(0.002)</td><td>(0.002)</td></tr><tr><td /><td align="center">10%</td><td align="center">100%</td><td align="center">100%</td><td align="center">100%</td><td align="center">100%</td></tr></tbody></table> </ephtml> </p> <p>25 <emph>Notes</emph>: Each cell represents a summary of 100 simulations designed to test the sensitivity of our textbook random effect estimator in a simulated distribution of textbooks that vary in effectiveness. Rows vary the size of the simulated textbook effect, and columns vary the simulated market share of that text in the sample. The first value in each cell is the mean textbook random effect estimated in that set of 100 simulations (including zeros). The next value is the standard deviation of the simulated textbook random effects. The final value in each cell represents the proportion of simulated runs in that cell where the textbook random effect parameter was greater than 1.96 times larger than its standard error.</p> <p>Table Heterogeneity in textbook random effect estimates by student subgroups (top 15 textbooks)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Random Effects Parameters</th><th align="center">Pooled 3 States (Maryland, New Jersey, New Mexico)</th><th align="center">California</th></tr><tr valign="bottom"><th>Panel A</th><th align="center">English Language Learner</th><th align="center">Non‐English Language Learner</th><th align="center">English Language Learner</th><th align="center">Non‐English Language Learner</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.000</td><td>0.026**</td><td>0.030***</td></tr><tr><td /><td>(0.000)</td><td>(0.000)</td><td>(0.009)</td><td>(0.009)</td></tr><tr><td>Student*Year Observations</td><td align="center">22,727</td><td align="center">232,586</td><td align="center">307,952</td><td align="center">711,559</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel B</th><th align="center">Special Education</th><th align="center">Non‐Special Education</th><th align="center">Special Education</th><th align="center">Non‐Special Education</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.000</td><td>0.018*</td><td>0.029**</td></tr><tr><td /><td>(0.001)</td><td>(0.000)</td><td>(0.008)</td><td>(0.009)</td></tr><tr><td>Student*Year Observations</td><td align="center">40,416</td><td align="center">214,897</td><td align="center">108,476</td><td align="center">911,030</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel C</th><th align="center">Free or Reduced‐Price Lunch</th><th align="center">Non‐Free or Reduced‐Price Lunch</th><th align="center">Free or Reduced‐Price Lunch</th><th align="center">Non‐Free or Reduced‐Price Lunch</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.000</td><td>0.029***</td><td>0.028**</td></tr><tr><td /><td>(0.000)</td><td>(0.000)</td><td>(0.008)</td><td>(0.009)</td></tr><tr><td>Student*Year Observations</td><td align="center">136,646</td><td align="center">118,667</td><td align="center">650,167</td><td align="center">369,344</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel D</th><th align="center">High Prior Math Achievement</th><th align="center">Low Prior Math Achievement</th><th align="center">High Prior Math Achievement</th><th align="center">Low Prior Math Achievement</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.000</td><td>0.022**</td><td>0.014**</td></tr><tr><td /><td>(0.000)</td><td>(0.000)</td><td>(0.007)</td><td>(0.005)</td></tr><tr><td>Student*Year Observations</td><td align="center">129,667</td><td align="center">125,646</td><td align="center">505,981</td><td align="center">513,530</td></tr></tbody></table> </ephtml> </p> <p>26 <emph>Notes</emph>: Estimates in each cell come from separate models. We estimate the standard deviation in textbook effects with a multilevel mixed‐effects linear regression of student‐level standardized math test scores on student prior year math test scores, student demographic characteristics, school‐by‐year demographic characteristics, 2010 to 2014 district census characteristics, and state‐by‐year fixed effects (restricted to year fixed effects only in specifications limited to a single state). The model also includes nested random effects for textbook, state, district, and school, nested in that order, with textbook as the top level of the nesting structure (state random effects are excluded from regressions that are limited to a single state). Each subsample is restricted to student observations with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year who are known to have used one of the top 15 textbooks by market share. Robust standard errors are in parentheses. <sups>∼</sups>z > 1.64, *z > 1.96, **z > 2.58, ***z > 3.29, where z equals the ratio of a given random effects parameter estimate to its standard error. These z‐scores do not correspond precisely to p‐values as in a traditional linear regression framework, as the confidence interval for a random effect estimate is not symmetric around the estimate (random effect estimates have a lower bound of zero). These traditional markers of significance are included as an aid to reader, but they should be interpreted with caution.</p> <p>Table Heterogeneity in textbook random effect estimates by grade level (top 15 textbooks)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Random Effects Parameters</th><th align="center">Pooled 4 States (Excluding California and Louisiana)</th><th align="center">California</th></tr><tr valign="bottom"><th /><th align="center">5th Grade, Controlling for 4th Grade</th></tr><tr valign="bottom"><th>Panel A</th><th align="center">All Schools</th><th align="center">Non‐Switchers</th><th align="center">All Schools</th><th align="center">Non‐Switchers</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.024</td><td>0.017**</td><td>0.032**</td></tr><tr><td /><td>(0.000)</td><td>(0.015)</td><td>(0.006)</td><td>(0.012)</td></tr><tr><td>School*Year Observations</td><td align="center">2,322</td><td align="center">1,431</td><td align="center">3,507</td><td align="center">2,109</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">5th Grade, Controlling for 3rd Grade</th></tr><tr valign="bottom"><th>Panel B</th><th align="center">All Schools</th><th align="center">Non‐Switchers</th><th align="center">All Schools</th><th align="center">Non‐Switchers</th></tr></thead><tbody><tr><td>Textbook</td><td>0.020</td><td>0.000</td><td>0.037***</td><td align="center">NA</td></tr><tr><td /><td>(0.036)</td><td>(0.000)</td><td>(0.011)</td><td /></tr><tr><td>School*Year Observations</td><td align="center">1,985</td><td align="center">1,411</td><td align="center">3,415</td><td /></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th align="center">4th Grade, Controlling for 3rd Grade</th></tr><tr valign="bottom"><th>Panel C</th><th align="center">All Schools</th><th align="center">Non‐Switchers</th><th align="center">All Schools</th><th align="center">Non‐Switchers</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td align="center">NA</td><td>0.000</td><td align="center">NA</td></tr><tr><td /><td align="center">–</td><td /><td>(0.000)</td><td /></tr><tr><td>School*Year Observations</td><td align="center">2,310</td><td /><td align="center">3,505</td><td /></tr></tbody></table> </ephtml> </p> <p>27 <emph>Notes</emph>: Estimates in each cell come from separate models. We estimate the standard deviation in textbook effects using a multilevel mixed‐effects linear regression of school‐level value added on school‐by‐grade‐by‐year demographic characteristics, 2010 to 2014 district census characteristics, and state‐by‐year fixed effects (excluded from California estimates, which are limited to a single state and year). The model also includes nested random effects for textbook, state, district, and school, nested in that order, with textbook as the top level of the nesting structure (state random effects are excluded from regressions that are limited to a single state). The sample is restricted to school‐by‐year observations within the indicated subgroup with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year who are known to have used one of the top 15 textbooks by market share. "NA" indicates that estimates were not possible to estimate with available data. In all states, we did not collect textbook data in third grade and so cannot identify fourth grade non‐switchers. In California, the state did not record student‐level test scores in the spring of 2014, as schools prepared for the new CCSS‐aligned assessments to be administered in the spring of 2015; therefore, we did not have multiple years of test score and textbook data for fifth graders who also had third grade prior achievement, necessary for the non‐switcher model. Robust standard errors are in parentheses. <sups>∼</sups>z > 1.64, *z > 1.96, **z > 2.58, ***z > 3.29, where z equals the ratio of a given random effects parameter estimate to its standard error. These z‐scores do not correspond precisely to p‐values as in a traditional linear regression framework, as the confidence interval for a random effect estimate is not symmetric around the estimate (random effect estimates have a lower bound of zero). These traditional markers of significance are included as an aid to reader, but they should be interpreted with caution.</p> <p>Table Robustness of textbook random effect estimates to different sets of controls to limit selection bias (top 15 textbooks)</p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Random Effects Parameters</th><th align="center">Pooled 5 States (Excluding California)</th><th align="center">California</th></tr></thead><tbody><tr><td>Panel A</td><td align="center">School and District Covariates</td></tr><tr><td>Textbook</td><td>0.000</td><td>0.027**</td></tr><tr><td /><td align="center">–</td><td>(0.008)</td></tr><tr><td>School*Year Observations</td><td align="center">2,676</td><td align="center">8,121</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel B</th><th align="center">School Covariates</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.033***</td></tr><tr><td /><td align="center">–</td><td>(0.009)</td></tr><tr><td>School*Year Observations</td><td align="center">2,676</td><td align="center">8,121</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel C</th><th align="center">No Covariates</th></tr></thead><tbody><tr><td>Textbook</td><td>0.000</td><td>0.037***</td></tr><tr><td /><td align="center">–</td><td>(0.011)</td></tr><tr><td>School*Year Observations</td><td align="center">2,676</td><td align="center">8,121</td></tr></tbody></table> </ephtml> </p> <p></p> <p> <ephtml> <table><thead><tr valign="bottom"><th>Panel D</th><th align="center">ELA Achievement as Outcome, Controlling for ELA Textbooks</th></tr></thead><tbody><tr><td>Textbook</td><td>NA</td><td>0.017**</td></tr><tr><td /><td /><td>(0.006)</td></tr><tr><td>School*Year Observations</td><td /><td align="center">8,121</td></tr></tbody></table> </ephtml> </p> <p>28 <emph>Notes</emph>: Estimates in each cell come from separate models. Random effects are estimated from a multilevel mixed‐effects linear regression of school‐level value added on state‐by‐year fixed effects (restricted to year fixed effects only in specifications limited to a single state), and school‐by‐year demographic characteristics and/or 2010 to 2014 district census characteristics where indicated. The model also includes nested random effects for textbook, state, district, and school, nested in that order, with curriculum as the top level of the nesting structure (state random effects are excluded from regressions that are limited to a single state). The sample is restricted to school‐by‐year observations with value‐added data for the 2014/2015, 2015/2016, or 2016/2017 school year who are known to have used one of the top 15 textbooks by market share. Robust standard errors are in parentheses; "–" indicates that standard errors could not be estimated. <sups>∼</sups>z > 1.64, *z > 1.96, **z > 2.58, ***z > 3.29, where z equals the ratio of a given random effects parameter estimate to its standard error. These z‐scores do not correspond precisely to p‐values as in a traditional linear regression framework, as the confidence interval for a random effect estimate is not symmetric around the estimate (random effect estimates have a lower bound of zero). These traditional markers of significance are included as an aid to reader, but they should be interpreted with caution.</p> <ref id="AN0146104504-29"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Throughout the paper, we use the terms "textbook" and "curriculum" interchangeably. We recognize, though, that the physical textbook may be just one of multiple materials that make up a given curriculum. Curricula can include student and teacher editions of the textbook, formative assessment materials, manipulative sets, etc. In our survey to schools and teachers, we referred to the "primary textbook or curriculum materials" used by teachers, which could consist of "a printed textbook from a publisher, an online text, or a collection of materials assembled by the school, district, or individual teachers [but] does not include supplemental resources that individual teachers may use from time to time to supplement the curriculum materials."</bibtext> </blist> <blist> <bibl id="bib2" idref="ref3" type="bt">2</bibl> <bibtext> Since 2010, many of the states that initially adopted the CCSS have since revised their standards. Yet, several of the states that revised standards from the CCSS have landed on a close facsimile (Friedberg et al., [16]).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref8" type="bt">3</bibl> <bibtext> California schools are mandated under state law to report curriculum materials on school accountability report cards (Holden, [24]; Hutt & Polikoff, [25]). In Indiana and Florida, centralized adoption processes allow state agencies to capture information on districts' adoption of certain texts (Bhatt & Koedel, [6]; Bhatt, Koedel, & Lehmann, [7]). In the course of preparing for our study, we learned that New Mexico collects curriculum data based on purchasing records from a state‐organized curriculum warehouse, and these records can be attached to individual schools. Recently, Louisiana started to collect data on textbook adoptions through district and school surveys. Texas tracks adoption data based on requisitions and disbursements, and posts this information on a public website. In our study, we partnered with California and New Mexico to leverage their existing administrative data on textbook adoptions. We also partnered with Louisiana, though we administered our own survey to ensure representativeness. The other three states (Indiana, Florida, and Texas) did not administer student assessments from one of the two CCSS‐aligned testing consortium, which was a critical criterion for our work.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref9" type="bt">4</bibl> <bibtext> See the Appendix at the end of this article.</bibtext> </blist> <blist> <bibl id="bib5" type="bt">5</bibl> <bibtext> Restricting the sampling frame to schools with 10 or more students with test score data was relatively trivial, dropping fewer than 50 schools for whom we were able to capture textbook information (most from California).</bibtext> </blist> <blist> <bibl id="bib6" idref="ref10" type="bt">6</bibl> <bibtext> A related possibility is that the randomized trial by Agodini et al., while being unbiased, was conflating the efficacy of certain textbooks with schools' prior experience with the textbook. One‐quarter of schools had been using <emph>Saxon Math</emph> in the year before the experiment. <emph>Saxon Math</emph> also was the textbook with the highest measured "efficacy" in that experiment. Although the authors included a control for whether or not an <emph>individual teacher</emph> had used the text before, the final results did not include controls for whether or not <emph>the school</emph> had been using the randomly assigned curriculum previously. An individual teacher new to <emph>Saxon Math</emph> working in a school that had been using <emph>Saxon Math</emph> previously may not be as disadvantaged as those in schools where no one had been using this text. This raises the question of construct validity: Were the authors measuring the efficacy of <emph>Saxon Math</emph> or were they witnessing the advantage of not having to transition to a new curriculum?</bibtext> </blist> <blist> <bibl id="bib7" type="bt">7</bibl> <bibtext> Several additional studies that attempted to use randomized designs to evaluate specific curricula were rejected from the WWC for failing to meet inclusion standards, generally due to imbalance between groups at baseline. Two doctoral dissertations cited by WWC used experimental designs to evaluate textbook effectiveness but never were published and rely on extremely small samples (<emph>n</emph><100 students). WWC reviewed two additional studies with randomized designs that meet their evidence standards; however, these studies are not available online (Beck Evaluation & Testing Associates Inc., [5]; Gatti & Giordano, [18]). In a currently unpublished review, Pellegrini et al. ([37])—updated from an earlier published review of the same topic (Slavin & Lake, [52])—also cite a recent randomized evaluation of a post‐CCSS edition of <emph>enVision</emph> that is not available online (Strobel, Resendez, & DuBose, [55]). At the time of writing, we were unable to obtain access to review these studies. Additionally, Pellegrini et al. ([37]) cite results from randomized trials evaluating <emph>Everyday Mathematics</emph> and <emph>JUMP Math</emph> textbooks that were gleaned from conference presentations, where a full description of each study's experimental design and associated balance tests are not available online. There also are several randomized evaluations of math materials, which we see as different from the textbooks we examine in our studies. Math software products that sometimes are described as curriculum, including <emph>Cognitive Tutor Bridge to Algebra</emph>, <emph>Compass Learning's Odyseey Math</emph>, <emph>PLATO Achieve Now</emph>, and <emph>Larson Pre‐Algebra</emph>, have been subjected to randomized evaluations. In our study, we define these materials as supplemental and not the primary curriculum to teach mathematics. Jackson and Makarin ([27]) experimentally evaluated the effectiveness of "off‐the‐shelf" curriculum materials for middle school math teachers, which we distinguish from complete textbooks.</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> Prior to being updated to the CCSS, <emph>enVision</emph> was referred to as <emph>Scott Foresman‐Addison Wesley Elementary Math</emph>. This is the title used in several evaluations cited in our review (i.e., Agodini et al., [1]; Bhatt & Koedel, [6]; Bhatt et al., [7]).</bibtext> </blist> <blist> <bibl id="bib9" type="bt">9</bibl> <bibtext> WWC and Pellegrini et al. ([37]) cite several non‐experimental evaluations of single textbooks. We omit these evaluations from our literature review, preferencing the highest‐quality research designs (randomized trials) or multi‐textbook evaluations that are most similar to our own study.</bibtext> </blist> <blist> <bibtext> Another text, <emph>Investigations</emph>, showed up in both the multi‐textbook randomized trial (Agodini et al., [1]) and one of the non‐experimental studies (Bhatt et al., [7]). However, in the latter study, schools using <emph>Investigations</emph> were grouped together with schools using several other texts as part of a composite comparison group.</bibtext> </blist> <blist> <bibtext> In our usage index, we excluded two questions that refer to teachers' use of a textbook outside of class rather than for instructional activities with students: "choose objective" and "refresh content knowledge."</bibtext> </blist> <blist> <bibtext> Also to maximize statistical power, we included all possible schools no matter within‐school teacher response rate. Estimates for all subgroup analyses that split schools based on teacher survey responses were similar when we re‐ran estimates restricting to schools with two or more teacher responses, three or more teacher responses, 50 percent or higher teacher response rates, and 70 percent or higher teacher response rates.</bibtext> </blist> <blist> <bibtext> Boser, Chingos, and Straus ([9]) examined price data for 114 elementary math materials from 17 states that appeared on an approved list from more than one state. These products had an average per‐pupil cost of $34 (in 2014 dollars), excluding ancillary materials such as teachers' editions. Comparing costs for the same product across states, they found that the difference between the lowest and highest prices paid was less than 1 percent for 30 percent of products, and less than 10 percent for 85 percent of products.</bibtext> </blist> <blist> <bibtext> Purchasing data that link to individual schools are rare. In this study, we also examined the quality of purchasing data captured by the agency GovSpend, which uses Freedom of Information Act (FOIA) requests to capture purchases that are part of the public record. However, the data proved to be limited, as purchases generally came from district‐level offices, had no shipping information to connect to schools, and often did not provide sufficient information to identify the number of students covered by a given purchase. Agreement rates between these data and other adoption data at the school level generally were lower than 50 percent, and sometimes substantially lower than that.</bibtext> </blist> <blist> <bibtext> One exception was in Maryland, where initial conversations with state and district leaders indicated that very large districts in the state (several with over 100 schools that met our inclusion criteria) adopted a single textbook across the entire district. Therefore, we limited the random sample of schools to just 10, rather than 20 percent.</bibtext> </blist> <blist> <bibtext> In California and New Mexico, all schools had a weight of one as there was no sampling conducted in these two states.</bibtext> </blist> <blist> <bibtext> To distinguish between pre‐ and post‐CCSS editions of the same textbook series, we consulted with EdReports, did a high‐level content comparison of the content of individual contexts, and examined the distribution of textbook editions by year. Based on this information, we categorized new editions as "CC" if they were published during or after 2011. The start of the CCSS was in 2010.</bibtext> </blist> <blist> <bibtext> We include all district‐developed curricula in a single category, despite variation in materials within or across districts.</bibtext> </blist> <blist> <bibtext> We originally planned on administering the teacher survey in the spring of 2017, to occur during the same school year as the school survey. However, we decided to delay in order to first ensure sufficiently high response rates on the school survey, which provided data for our main analyses. This delay likely is one reason for lower response rates on the teacher survey relative to the school survey. For example, our population of interest was all 4th‐ and 5th‐grade math teachers working in randomly selected schools during the 2016/2017 school year; however, some of these teachers no longer were working in the same schools in the fall of 2017/2018 school year.</bibtext> </blist> <blist> <bibtext> For the teacher survey, we identified a target of 50 to 60 schools per textbook based on general benchmarks about sample sizes needed to approximate a population, as well as logistical constraints in recruiting, following up with, and compensating, teachers in these schools.</bibtext> </blist> <blist> <bibtext> Thirteen schools declined participation in the teacher survey as part of their response to our original school survey; 48 schools were from districts that declined participation; 20 school principals declined participation upon outreach; 21 schools did not provide sufficient information to identify teachers for recruitment; and 17 schools were no longer eligible to participate given that they switched textbooks from the prior year when we administered the school survey, the school closed from the prior year, or there no longer were eligible 4th‐ and 5th‐grade teachers who also worked in the school the prior year. School‐level participation rates in the teacher survey range from 44 to 80 percent across states (derived from numbers in Table 1). Participation rates by textbooks—which was the primary information we used to stratify the sample—are less variable, ranging from 59 to 86 percent (not shown in Table 1). As with school‐level participation rates, teacher response rates vary more across states (36 to 64 percent; derived from numbers in Table 1) than across textbooks (42 to 63 percent; not shown in Table 1).</bibtext> </blist> <blist> <bibtext> Louisiana developed and administrated a hybrid PARCC/state test.</bibtext> </blist> <blist> <bibtext> Per data use agreements with state agencies, our partners in Louisiana and Washington were allowed to share annual school‐level aggregates (derived from student‐level value‐added models described in the Empirical Methodology section). These data were pooled with school‐level aggregates from Maryland, New Jersey, and New Mexico to estimate models of textbook effects using all five states. In California, the agreement between partner researchers and the state agency did not allow for sharing of any student‐ or school‐level data with external teams. Therefore, the California team estimated textbook effects in those data alone and shared the estimates with the primary research team to pool with those from the other states.</bibtext> </blist> <blist> <bibtext> In California, the state did not record student‐level test scores in the spring of 2014, as schools prepared for the new CCSS‐aligned assessments to be administered in the spring of 2015. However, because California administers assessments to students in grades 2 through 8, we were able to use twice‐lagged scores for both fourth and fifth graders in 2014/2015. Other work using the California data shows that the school‐level value‐added scores are not particularly sensitive to use of once‐ or twice‐lagged achievement as a control (Carrell et al., [12]). For all states, if the prior score was missing in math (the primary outcome), we dropped that observation. If the prior score was missing in ELA, we created a flag for missing prior score, imputed the missing score to zero, and included the missing flag in the specification. We took the same approach for missingness of student demographic characteristics.</bibtext> </blist> <blist> <bibtext> An alternative approach would be to cluster standard errors at the district level. However, clustering can lead to overly optimistic standard errors when clusters have small numbers of observations (Cameron & Miller, [10]; Pustejovsky & Tipton, [43]). This is apparent in our data. In Figure A1, we plot the ratio of standard errors clustered at the district level to the random effects standard errors for each textbook, against the number of districts (clusters) using each textbook. For textbooks used in more than 15 districts, the standard errors were quite similar from the two methods. However, for the textbooks used in fewer than 15 districts, the standard error estimates differed dramatically depending upon the methods used, with standard errors for two of the textbooks falling by more than half when we clustered at the district level. We took this as evidence in favor of the random effects model.</bibtext> </blist> <blist> <bibtext> Because there are some districts using more than one text, we also explored models that crossed the state, district, and school random effects with the textbook random effects (e.g., non‐nested) with similar results.</bibtext> </blist> <blist> <bibtext> The full set of school‐by‐year characteristics included all available student‐level demographic characteristics averaged to the school‐year level. The full set of district characteristics from the census included instructional expenditure per‐pupil, median household income, percent of households that spoke a language other than English at home, percent of parents of school‐aged children who were married, percent of parents who attended some college or hold an associate's degree (but no bachelor's degree), and percent of parents who held at least a bachelor's degree.</bibtext> </blist> <blist> <bibtext> Use of a school‐level dataset in Tables 5 and 6 masks variation in teacher‐level response rates. However, patterns of textbook use are similar when drawing on the original teacher‐level dataset (and also applying sampling weights), suggesting that lower‐than‐desired teacher response rates in some schools is unlikely to alter our primary conclusions.</bibtext> </blist> <blist> <bibtext> As described in our Empirical Methodology section, fixed and random effects specifications make different assumptions—namely, whether the textbook effects are correlated with covariates included in the model. Several tests indicate that comparison of fixed and random effects specifications is reasonable in our data. First, we find correlations between textbook effects specified as a set of fixed versus random effects of 0.86 (for pooled five‐state sample excluding California) and 0.57 (for California). For this calculation, our "random effect" estimates are residuals averaged to the textbook level. We took this approach because hierarchical, random effects models employ a shrinkage factor, while fixed effect parameters do not. This shrinkage factor (a ratio less than one) would artificially attenuate the correlation between textbook fixed versus random effect estimates. In the teacher effectiveness literature, random effects and average residual models are described as comparable because they both calculate effects from the error rather than the fixed portion of the model (Guarino et al., [19]; Kane & Staiger, [28]). Second, we find a correlation of 0.99 (for both samples) on the coefficients for the set of school‐ and district‐level covariates included in both the random and fixed effect models. Finally, when conducting a Hausman test, we cannot reject the null hypothesis that one specification produces more or less efficient estimates than the other, at least in the pooled five‐state sample excluding California (<emph>p</emph> = 0.952). In California, despite superficially similar estimated textbook effects, we can reject this null hypothesis (<emph>p</emph> < 0.001), potentially driven by the very large number of school‐year observations in that state.</bibtext> </blist> <blist> <bibtext> Random effects models have known challenges when estimates are close to zero (Harville, [20]; Raudenbush & Bryk, [44]). For example, when the estimated variance approaches zero, the standard error is undefined (see Table 9, model 1, for example). To confirm that our estimates are true zeros, we estimated results to 10 decimal places, finding similar results. We also fit models using both full and restricted maximum likelihood. Because full maximum likelihood tends to produce estimates that are biased downward (Harville, [20]), we prefer (and present) estimates fit using restricted maximum likelihood. Indeed, in this exercise, estimates generated using restricted maximum likelihood are the same or slightly larger than estimates generated from full maximum likelihood.</bibtext> </blist> <blist> <bibtext> One explanation for differences in random effect estimates between California and Washington versus the other four states may be that the former two states used the SBAC assessment, while the remaining states used the PARCC test. However, California and Washington do not identify the same textbooks as most or least effective. The correlation of textbook fixed effect estimates between and fixed effect estimates from Washington is 0.3.</bibtext> </blist> <blist> <bibtext> To estimate statistical power, we generated 100 simulations for each combination of market share (1 percent, 5 percent, 10 percent, 20 percent and 25 percent) and single‐text effect size (0.02, 0.03, 0.05, 0.10, 0.15 SD). In each run, we stripped 2,676 school‐by‐year observations of their textbook data and randomly assigned one of 15 fake curricula within district‐by‐textbook clusters. That is, all schools within a given district that were observed using a given textbook were assigned a common fake textbook in each simulation for all years they appear in the data. To replicate real textbook adoption behavior, 6 percent of schools were chosen to switch to a new textbook in year two, and an additional 6 percent of schools were chosen to switch to a new textbook in year three. Schools that switched to a new text in year two kept that text in year three unless they were randomly chosen to switch in both years. Of the fake textbooks that schools were assigned, 14 of 15 were designed to have no effect on student outcomes in the simulation. As such, school‐years assigned one of these 14 textbooks kept their original value‐added. For school‐years randomly assigned the single textbook identified as effective in the simulation, their value‐added was increased by the amount in the effect size for that particular simulation. By design, this increase in value‐added is attributable to that textbook, so the simulation assesses whether our random effects model is able to detect and correctly attribute systematic variation in value‐added to a school's choice of textbook, for larger and smaller textbook effects distributed over larger and smaller shares of the sample of schools. The market‐share percentage indicates what percentage of schools were assigned the effective textbook for a set of simulations. The random effect estimate for each simulated run was stored, and this process was repeated 100 times for each combination of effect size and market‐share.</bibtext> </blist> </ref> <ref id="AN0146104504-30"> <title> REFERENCES </title> <blist> <bibtext> Agodini, R., Harris, B., Thomas, M., Murphy, R., & Gallagher, L. (2010). Achievement effects of four early elementary school math curricula: Findings for first and second graders. National Center for Education Evaluation and Regional Assistance. Retrieved June 24, 2020, from https://files.eric.ed.gov/fulltext/ED512551.pdf.</bibtext> </blist> <blist> <bibtext> Altonji, J. G., Blom, E., & Meghir, C. (2012). Heterogeneity in human capital investments: High school curriculum, college major, and careers. Annual Review of Economics, 4, 185 – 223.</bibtext> </blist> <blist> <bibtext> Angrist, J. D., Hull, P. D., Pathak, P. A., & Walters, C. R. (2017). Leveraging lotteries for school value‐added: Testing and estimation. The Quarterly Journal of Economics, 132, 871 – 919.</bibtext> </blist> <blist> <bibtext> Autor, D. H., Levy, F., & Murnane, R. J. (2003). The skill content of recent technological change: An empirical exploration. The Quarterly Journal of Economics, 118, 1279 – 1333.</bibtext> </blist> <blist> <bibtext> Beck Evaluation & Testing Associates, Inc. (2005). Progress in Mathematics 2006: Grade 1 pre‐post field test evaluation study. New York, NY : William H. Sadlier, Inc.</bibtext> </blist> <blist> <bibtext> Bhatt, R., & Koedel, C. (2012). Large‐scale evaluations of curricular effectiveness: The case of elementary mathematics in Indiana. Educational Evaluation and Policy Analysis, 34, 391 – 412.</bibtext> </blist> <blist> <bibtext> Bhatt, R., Koedel, C., & Lehmann, D. (2013). Is curriculum quality uniform? Evidence from Florida. Economics of Education Review, 34, 107 – 121.</bibtext> </blist> <blist> <bibtext> Bianchini, J. A., & Kelly, G. J. (2003). Challenges of standards‐based reform: The example of California's science content standards and textbook adoption process. Science Education, 87, 378 – 389.</bibtext> </blist> <blist> <bibtext> Boser, U., Chingos, M., & Straus, C. (2015). The hidden value of curriculum reform: Do states and districts receive the most bang for their curriculum buck? Washington, DC : Center for American Progress. Retrieved June 24, 2020, from https://cdn.americanprogress.org/wp-content/uploads/2015/10/06111518/CurriculumMatters-report.pdf.</bibtext> </blist> <blist> <bibtext> Cameron, A. C., & Miller, D. L. (2015). A practitioner's guide to cluster‐robust inference. Journal of Human Resources, 50, 317 – 372.</bibtext> </blist> <blist> <bibtext> Carmichael, S. B., Martino, G., Porter‐Magee, K., & Wilson, W. S., (2010). The state of state standards—and the Common Core—in 2010. Thomas B. Fordham Institute. Retrieved June 24, 2020, from https://files.eric.ed.gov/fulltext/ED516607.pdf.</bibtext> </blist> <blist> <bibtext> Carrell, S., Kurlaender, M., Martorell, P., & Naven, M. (2018). The impacts of high school quality on postsecondary outcomes: Evidence from California. Working Paper. Davis, CA : California Education Lab, University of California–Davis.</bibtext> </blist> <blist> <bibtext> Cavanaugh, S. (2015, June 9). N.Y. "open" education effort draws users nationwide. Education Week. Retrieved June 24, 2020, from https://<ulink href="http://www.edweek.org/ew/articles/2015/06/10/ny-open-education-effort-draws-users-nationwide.html">www.edweek.org/ew/articles/2015/06/10/ny-open-education-effort-draws-users-nationwide.html</ulink>.</bibtext> </blist> <blist> <bibtext> Chingos, M. M., & Whitehurst, G. J. (2012). Choosing blindly: Instructional materials, teacher effectiveness, and the Common Core. Washington, DC : Brookings. Retrieved June 24, 2020, from https://<ulink href="http://www.brookings.edu/wp-content/uploads/2016/06/0410%5fcurriculum%5fchingos%5fwhitehurst.pdf">www.brookings.edu/wp-content/uploads/2016/06/0410%5fcurriculum%5fchingos%5fwhitehurst.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Eddy, R. M., Hankel, N., Hunt, A., Goldman, A., & Murphy, K. (2014). Houghton Mifflin Harcourt GO Math! Efficacy study year one final report. La Verne, CA : Cobblestone Applied Research & Evaluation, Inc. Retrieved June 24, 2020, from https://<ulink href="http://www.hmhco.com/~/media/sites/home/educators/education-topics/hmh-efficacy/HMH%5fGo%5fMath%5fRCT%5fYr1%5f2014.pdf">www.hmhco.com/~/media/sites/home/educators/education-topics/hmh-efficacy/HMH%5fGo%5fMath%5fRCT%5fYr1%5f2014.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Friedberg, S., Barone, D., Belding, J., Chen, A., Dixon, L., Fennell, F., ... Shanahan, T. (2018). The state of state standards post‐Common Core. Thomas B. Fordham Institute. Retrieved June 24, 2020, from https://files.eric.ed.gov/fulltext/ED592393.pdf.</bibtext> </blist> <blist> <bibtext> Fryer Jr., R. G. (2017). The production of human capital in developed countries: Evidence from 196 randomized field experiments. In Handbook of Economic Field Experiments (Vol. 2, pp. 95 – 322). North‐Holland.</bibtext> </blist> <blist> <bibtext> Gatti, G., & Giordano, K. (2010). Pearson Investigations in Numbers, Data, & Space efficacy study: Final report. Pittsburgh, PA : Gatti Evaluation, Inc.</bibtext> </blist> <blist> <bibtext> Guarino, C. M., Maxfield, M., Reckase, M. D., Thompson, P. N., & Wooldridge, J. M. (2015). An evaluation of empirical Bayes's estimation of value‐added teacher performance measures. Journal of Educational and Behavioral Statistics, 40, 190 – 222.</bibtext> </blist> <blist> <bibtext> Harville, D. A. (1977). Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American statistical Association, 72, 320 – 338.</bibtext> </blist> <blist> <bibtext> Heck, D. J., Weiss, I. R., & Pasley, J. D. (2011). A priority research agenda for understanding the influence of the Common Core State Standards for Mathematics. Technical Report. Chapel Hill, NC : Horizon Research, Inc. Retrieved June 24, 2020, from https://files.eric.ed.gov/fulltext/ED544302.pdf.</bibtext> </blist> <blist> <bibtext> Herman, J., & Linn, R. (2014). New assessments, new rigor. Educational Leadership, 71, 34 – 37.</bibtext> </blist> <blist> <bibtext> Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2, 172 – 177.</bibtext> </blist> <blist> <bibtext> Holden, K. L. (2016). Buy the book? Evidence on the effect of textbook funding on school‐level achievement. American Economic Journal: Applied Economics, 8, 100 – 127.</bibtext> </blist> <blist> <bibtext> Hutt, E., & Polikoff, M. (Spring 2018). Reasonable expectations: A reply to Elmendorf and Shanske 2018. University of Illinois Law Review Online, 194 – 208. Retrieved June 24, 2020, from https://illinoislawreview.org/wp-content/uploads/2018/05/HuttPolikoff.pdf.</bibtext> </blist> <blist> <bibtext> Jaciw, A. P., Hegseth, W. M., Lin, L., Toby, M., Newman, D., Ma, B., & Zacamy, J. (2016). Assessing impacts of Math in Focus, a "Singapore Math" program. Journal of Research on Educational Effectiveness, 9, 473 – 502.</bibtext> </blist> <blist> <bibtext> Jackson, K., & Makarin, A. (2018). Can online off‐the‐shelf lessons improve student outcomes? Evidence from a field experiment. American Economic Journal: Economic Policy, 10, 226 – 254.</bibtext> </blist> <blist> <bibtext> Kane, T. J., & Staiger, D. O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation. NBER No. w14607. Cambridge, MA : National Bureau of Economic Research. Retrieved June 24, 2020, from https://<ulink href="http://www.nber.org/papers/w14607.pdf">www.nber.org/papers/w14607.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Kirst, M. W. (1982). How to improve schools without spending more money. The Phi Delta Kappan, 64, 6 – 8.</bibtext> </blist> <blist> <bibtext> Koedel, C., Li, D., Polikoff, M. S., Hardaway, T., & Wrabel, S. L. (2017). Mathematics curriculum effects on student achievement in California. AERA Open, 3. Retrieved June 24, 2020, from https://journals.sagepub.com/doi/full/10.1177/2332858417690511.</bibtext> </blist> <blist> <bibtext> McFarland, J., Hussar, B., de Brey, C., Snyder, T., Wang, X., Wilkinson‐Flicker, S., ... Hinz, S. (2017). The condition of education 2017. U.S. Department of Education. Washington, DC : National Center for Education Statistics. Retrieved June 24, 2020, from https://files.eric.ed.gov/fulltext/ED512551.pdf.</bibtext> </blist> <blist> <bibtext> National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA : Author.</bibtext> </blist> <blist> <bibtext> National Council of Teachers of Mathematics. (1991). Professional standards for teaching mathematics. Reston, VA : Author.</bibtext> </blist> <blist> <bibtext> National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA : Author.</bibtext> </blist> <blist> <bibtext> O'Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33 – 84.</bibtext> </blist> <blist> <bibtext> Opfer, V. D., Kaufman, J. H., & Thompson, L. E. (2016). Implementation of K–12 state standards for mathematics and English language arts and literacy. Santa Monica, CA : RAND. Retrieved June 24, 2020, from https://<ulink href="http://www.rand.org/content/dam/rand/pubs/research%5freports/RR1500/RR1529-1/RAND%5fRR1529-1.pdf">www.rand.org/content/dam/rand/pubs/research%5freports/RR1500/RR1529-1/RAND%5fRR1529-1.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Pellegrini, M., Lake, C., Inns, A., & Slavin, R. E. (2018). Effective programs in elementary mathematics: A best‐evidence synthesis. Baltimore, MD : Best Evidence Encyclopedia. Retrieved June 24, 2020, from <ulink href="http://www.bestevidence.org/word/elem%5fmath%5fOct%5f8%5f2018.pdf">http://www.bestevidence.org/word/elem%5fmath%5fOct%5f8%5f2018.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Polikoff, M. S. (2015). How well aligned are textbooks to the Common Core standards in mathematics? American Educational Research Journal, 52, 1185 – 1211.</bibtext> </blist> <blist> <bibtext> Polikoff, M. S. (2018). The challenges of curriculum materials as a reform lever. Washington, DC : Brookings. Retrieved June 24, 2020, from: https://<ulink href="http://www.brookings.edu/research/the-challenges-of-curriculum-materials-as-a-reform-lever/">www.brookings.edu/research/the-challenges-of-curriculum-materials-as-a-reform-lever/</ulink>.</bibtext> </blist> <blist> <bibtext> Polikoff, M. S., Campbell, S., Rabovsky, S., Koedel, C., Le, Q. T., Hardaway, T., & Gasparian, H. (2020). The formalized processes districts use to evaluate mathematics textbooks. Journal of Curriculum Studies, 52, 451 – 477.</bibtext> </blist> <blist> <bibtext> Pondiscio, R. (2017). Louisiana threads the needle on ed reform: Launching a coherent curriculum in a local‐control state. Education Next, 17, 8 – 15.</bibtext> </blist> <blist> <bibtext> Porter, A., McMaken, J., Hwang, J., & Yang, R. (2011). Common Core standards: The new U.S. intended curriculum. Educational Researcher, 40, 103 – 116.</bibtext> </blist> <blist> <bibtext> Pustejovsky, J. E., & Tipton, E. (2018). Small‐sample methods for cluster‐robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36, 672 – 683.</bibtext> </blist> <blist> <bibtext> Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis method. Thousand Oaks, CA : Sage Publications, Inc.</bibtext> </blist> <blist> <bibtext> Remillard, J. T., Harris, B., & Agodini, R. (2014). The influence of curriculum material design on opportunities for student learning. International Journal on Mathematics Education (ZDM), 46, 735 – 749.</bibtext> </blist> <blist> <bibtext> Reys, R. E. (2001). Curricular controversy in the math wars: A battle without winners. Phi Delta Kappan, 83, 255 – 258.</bibtext> </blist> <blist> <bibtext> Rockoff, J. E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94, 247 – 252.</bibtext> </blist> <blist> <bibtext> Schmidt, W. H., McKnight, C. C., Houang, R. T., Wang, H. C., Wiley, D. E., Cogan, L. S., & Wolfe, R. G. (2001). Why schools matter: A cross‐national comparison of curriculum and learning. San Francisco, CA : Jossey‐Bass.</bibtext> </blist> <blist> <bibtext> Schmidt, W. H., McKnight, C. C., & Raizen, S. (1997). A splintered vision: An investigation of U.S. science and mathematics education. Dordrecht, Netherlands : Kluwer.</bibtext> </blist> <blist> <bibtext> Schoenfeld, A. H. (2004). The math wars. Educational Policy, 18, 253 – 286.</bibtext> </blist> <blist> <bibtext> Seeley, C. L. (2003). Mathematics textbook adoption in the United States. In G. M. A. Stanic & J. Kilpatrick (Eds.), A history of school mathematics (Vol. 2, pp. 957 – 988). Reston, VA : National Council of Teachers of Mathematics.</bibtext> </blist> <blist> <bibtext> Slavin, R. E., & Lake, C. (2008). Effective programs in elementary mathematics: A best‐evidence synthesis. Review of Educational Research, 78, 427 – 515.</bibtext> </blist> <blist> <bibtext> Stein, M. K., Remillard, J., & Smith, M. S. (2007). How curriculum influences student learning. In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (Vol. 1, pp. 319 – 370). Reston, VA : National Council of Teachers of Mathematics.</bibtext> </blist> <blist> <bibtext> Steiner, D. (2017). Choosing a curriculum: A critical act. Education Next. Retrieved June 24, 2020, from https://<ulink href="http://www.educationnext.org/choosing-curriculum-critical-act/">www.educationnext.org/choosing-curriculum-critical-act/</ulink>.</bibtext> </blist> <blist> <bibtext> Strobel, A., Resendez, M., & DuBose, D. (2017). enVision Math 2.0 Year 2 RCT Study Final Report. Thayne, WY : Strobel Consulting, LLC.</bibtext> </blist> <blist> <bibtext> Tulley, M. A. (1985). A descriptive study of the intents of state‐level textbook adoption processes. Educational Evaluation and Policy Analysis, 7, 289 – 308.</bibtext> </blist> <blist> <bibtext> Whitehurst, G. J. (2009). Don't forget curriculum. Washington, DC : Brookings. Retrieved June 24, 2020, from https://<ulink href="http://www.brookings.edu/research/dont-forget-curriculum/">www.brookings.edu/research/dont-forget-curriculum/</ulink>.</bibtext> </blist> </ref> <aug> <p>By David Blazar; Blake Heller; Thomas J. Kane; Morgan Polikoff; Douglas O. Staiger; Scott Carrell; Dan Goldhaber; Douglas N. Harris; Rachel Hitch; Kristian L. Holden and Michal Kurlaender</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author; Author; Author; Author; Author</p> <p></p> <p>DAVID BLAZAR is an Assistant Professor in the College of Education at the University of Maryland, College Park, 2205 Benjamin Building, 3942 Campus Drive, College Park, MD 20742 (e‐mail:).</p> <p>BLAKE HELLER is a PH.D. Candidate in the Kennedy School of Government at Harvard University (e‐mail:).</p> <p>THOMAS J. KANE is a Professor in the Graduate School of Education at Harvard University, Center for Education Policy Research, 50 Church Street, 4th Floor, Cambridge, MA 02138 (e‐mail:).</p> <p>MORGAN POLIKOFF is an Associate Professor at the Rossier School of Education at the University of Southern California, Waite Phillips Hall #904A, 3470 Trousdale Parkway, Los Angeles, CA 90089 (e‐mail:).</p> <p>DOUGLAS O. STAIGER is a Professor in the Department of Economics at Dartmouth College, HB 61606, Rockefeller Hall, Dartmouth College, Hanover, NH 03755 (e‐mail:).</p> <p>SCOTT CARRELL is a Professor in the Department of Economics at the University of California at Davis, 1 Shields Avenue, Davis, CA 95616 (e‐mail:).</p> <p>DAN GOLDHABER is Director of the Center for Analysis of Longitudinal Data in Education Research at the American Institutes for Research and the Center for Education Data and Research at the University of Washington at Seattle, 3876 Bridge Way N., Suite 201, Seattle, WA 98103 (e‐mail:).</p> <p>DOUGLAS N. HARRIS is Professor and Chair of the Department of Economics at Tulane University, 302 Tilton Memorial Hall, 6823 St. Charles Avenue, New Orleans, LA 70118 (e‐mail:).</p> <p>RACHEL HITCH is Director of Program Management at Meteor Learning, 301 Edgewater Place, Suite 210, Wakefield, MA 01880.</p> <p>KRISTIAN L. HOLDEN is a Researcher at the Center for Analysis of Longitudinal Data in Education Research at the American Institutes for Research, 1050 Thomas Jefferson Avenue, Washington, DC 20007 (e‐mail:).</p> <p>MICHAL KURLAENDER is Professor and Department Chair in the School of Education at the University of California at Davis, 1 Shields Avenue, Davis, CA 95616 (e‐mail:).</p> </aug>
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED614241
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED614241
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Curriculum Reform in the Common Core Era: Evaluating Elementary Math Textbooks across Six U.S. States
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Blazar%2C+David%22">Blazar, David</searchLink><br /><searchLink fieldCode="AR" term="%22Heller%2C+Blake%22">Heller, Blake</searchLink><br /><searchLink fieldCode="AR" term="%22Kane%2C+Thomas+J%2E%22">Kane, Thomas J.</searchLink><br /><searchLink fieldCode="AR" term="%22Polikoff%2C+Morgon%22">Polikoff, Morgon</searchLink><br /><searchLink fieldCode="AR" term="%22Staiger%2C+Douglas+O%2E%22">Staiger, Douglas O.</searchLink><br /><searchLink fieldCode="AR" term="%22Carrell%2C+Scott%22">Carrell, Scott</searchLink><br /><searchLink fieldCode="AR" term="%22Goldhaber%2C+Dan%22">Goldhaber, Dan</searchLink><br /><searchLink fieldCode="AR" term="%22Harris%2C+Douglas+N%2E%22">Harris, Douglas N.</searchLink><br /><searchLink fieldCode="AR" term="%22Hitch%2C+Rachel%22">Hitch, Rachel</searchLink><br /><searchLink fieldCode="AR" term="%22Holden%2C+Kristian+L%2E%22">Holden, Kristian L.</searchLink><br /><searchLink fieldCode="AR" term="%22Kurlaender%2C+Michal%22">Kurlaender, Michal</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Grantee+Submission%22"><i>Grantee Submission</i></searchLink>. 2020 39(4).
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 54
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2020
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305B150010<br />R305E150006
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Curriculum+Development%22">Curriculum Development</searchLink><br /><searchLink fieldCode="DE" term="%22Common+Core+State+Standards%22">Common Core State Standards</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Materials%22">Mathematics Materials</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Gains%22">Achievement Gains</searchLink><br /><searchLink fieldCode="DE" term="%22Textbook+Selection%22">Textbook Selection</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Mathematics%22">Elementary School Mathematics</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/pam.22257
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Can a school or district improve student achievement simply by switching to a higher quality textbook or curriculum? We conducted the first multi-textbook, multi-state effort to estimate textbook efficacy following widespread adoption of the Common Core State Standards (CCSS) and associated changes in the textbook market. Pooling textbook adoption and student test score data across six geographically and demographically diverse U.S. states, we found little evidence of differences in average achievement gains for schools using different math textbooks. We found some evidence of greater variation in achievement gains among schools using pre-CCSS editions, which may have been more varied in their content than post-CCSS editions because they were written for a broader set of standards. We also found greater variation among schools that had more exposure to a given text. However, these differences were small. Despite considerable interest and attention to textbooks as a low-cost, "silver bullet" intervention for improving student outcomes, we conclude that the adoption of a new textbook or set of curriculum materials, on its own, is unlikely to achieve this goal. [This article was published in "Journal of Policy Analysis and Management" (EJ1268634).]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: What Works Clearinghouse (WWC) Reviewed
  Group: SrcInfo
  Data: Does Not Meet Evidence Standards
– Name: URL
  Label: WWC Study Page
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://ies.ed.gov/ncee/wwc/Study/90550" linkWindow="_blank">https://ies.ed.gov/ncee/wwc/Study/90550</link>
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED614241
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED614241
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/pam.22257
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 54
    Subjects:
      – SubjectFull: Curriculum Development
        Type: general
      – SubjectFull: Common Core State Standards
        Type: general
      – SubjectFull: Mathematics Materials
        Type: general
      – SubjectFull: Achievement Gains
        Type: general
      – SubjectFull: Textbook Selection
        Type: general
      – SubjectFull: Elementary School Mathematics
        Type: general
    Titles:
      – TitleFull: Curriculum Reform in the Common Core Era: Evaluating Elementary Math Textbooks across Six U.S. States
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Blazar, David
      – PersonEntity:
          Name:
            NameFull: Heller, Blake
      – PersonEntity:
          Name:
            NameFull: Kane, Thomas J.
      – PersonEntity:
          Name:
            NameFull: Polikoff, Morgon
      – PersonEntity:
          Name:
            NameFull: Staiger, Douglas O.
      – PersonEntity:
          Name:
            NameFull: Carrell, Scott
      – PersonEntity:
          Name:
            NameFull: Goldhaber, Dan
      – PersonEntity:
          Name:
            NameFull: Harris, Douglas N.
      – PersonEntity:
          Name:
            NameFull: Hitch, Rachel
      – PersonEntity:
          Name:
            NameFull: Holden, Kristian L.
      – PersonEntity:
          Name:
            NameFull: Kurlaender, Michal
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2020
          Numbering:
            – Type: volume
              Value: 39
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Grantee Submission
              Type: main
ResultId 1