View in EDS HTML Full Text PDF Full Text

Creating Actionable Human Capital Analytics for Studying School-Level Teacher Retention

Saved in:
Bibliographic Details
Title: Creating Actionable Human Capital Analytics for Studying School-Level Teacher Retention
Language: English
Authors: Robert Meyer, Anthony Milanowski, Ryan Veiga, Jessica Doherty
Source: Journal of Education Human Resources. 2025 43(2):390-421.
Availability: University of Toronto Press. 5201 Dufferin Street, Toronto, ON M3H 5T8, Canada. Tel: 416-667-7810; Fax: 800-221-9985; Fax: 416-667-7881; e-mail: journals@utpress.utoronco.ca; Web site: https://www.utpjournals.press/loi/jehr
Peer Reviewed: Y
Page Count: 32
Publication Date: 2025
Sponsoring Agency: Office of Elementary and Secondary Education (OESE) (ED)
Contract Number: 537A120095
Document Type: Journal Articles
Reports - Research
Descriptors: Teacher Persistence, Faculty Mobility, School Districts, Human Capital, Data Analysis, Educational Environment, Teaching Conditions, Administrator Attitudes, Evaluation Methods, Institutional Characteristics, Differences, Educational Policy, Educational Practices
Geographic Terms: Florida
DOI: 10.3138/jehr-2023-0063
ISSN: 2562-783X
Abstract: One potentially fruitful application for human capital analytics is to support policies and practices that might reduce undesirable teacher turnover. Teacher turnover can be harmful to student achievement and faculty cohesiveness and can exacerbate teacher shortages. This article describes an attempt to build a human capital analytics tool to help a school district better identify schools with problems retaining teachers as well as schools that are doing an especially good job of retaining them. It describes the theory of action for the school-level retention analytics tool, the models used to estimate persistent school effects and predict which schools would continue to have retention problems, and the features of a web-based tool developed to help district staff gain insights about the degree of variation among schools and likely contributing factors. We found that there were reliable differences in schools' persistent rates of teacher retention, the differences persisted after controlling for student demographics, teachers at different levels of effectiveness and experience have different predicted levels of retention, and there were substantial differences among schools in their ability to retain teachers at different levels of experience and effectiveness. Finally, we describe the initial reactions of the users for whom we designed the tool and what we learned about developing a retention analytic tool for use by school district administrators.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1467146
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHDU1-NnQe1SxT-FbcGcTmOAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI5g6vEZW8SncAVvxwIBEICBm0rsWETsjvRIHkn_L21pyttCzyoZjeB1917vqQ0r3YzLwmkBd87ToxQ_S-x_AnYPbUngiFYHS08UinXCXfq4Fad7s5cygm7PSqmZbbGTSMiZtcdyYRDf5VDwGZmaf_ruXq_wMi-193_NB_c9C4vhXgo2eE0iAODhOhBNKILCvoWRCq0ScjpSwlrheVjukGs8Y4KZtx0lt4774-05
Text:
  Availability: 1
  Value: <anid>AN0184324479;[mmhi]01apr.25;2025Apr10.04:42;v2.2.500</anid> <title id="AN0184324479-1">Creating Actionable Human Capital Analytics for Studying School-Level Teacher Retention </title> <p>One potentially fruitful application for human capital analytics is to support policies and practices that might reduce undesirable teacher turnover. Teacher turnover can be harmful to student achievement and faculty cohesiveness and can exacerbate teacher shortages. This article describes an attempt to build a human capital analytics tool to help a school district better identify schools with problems retaining teachers as well as schools that are doing an especially good job of retaining them. It describes the theory of action for the school-level retention analytics tool, the models used to estimate persistent school effects and predict which schools would continue to have retention problems, and the features of a web-based tool developed to help district staff gain insights about the degree of variation among schools and likely contributing factors. We found that there were reliable differences in schools' persistent rates of teacher retention, the differences persisted after controlling for student demographics, teachers at different levels of effectiveness and experience have different predicted levels of retention, and there were substantial differences among schools in their ability to retain teachers at different levels of experience and effectiveness. Finally, we describe the initial reactions of the users for whom we designed the tool and what we learned about developing a retention analytic tool for use by school district administrators.</p> <p>Keywords: assessment; compensation; educator evaluation; evaluation; evaluation design; human resource management; observation; performance; survey research; teaching</p> <p>One potentially fruitful application for human capital analytics is to support policies and practices that might reduce undesirable teacher turnover. Teacher turnover has been shown to be harmful to student achievement and faculty cohesiveness (e.g., [<reflink idref="bib10" id="ref1">10</reflink>]; [<reflink idref="bib21" id="ref2">21</reflink>]). Excessive turnover also exacerbates teacher shortages and diverts district resources to otherwise unnecessary recruitment and hiring of new teachers ([<reflink idref="bib12" id="ref3">12</reflink>]). [<reflink idref="bib22" id="ref4">22</reflink>] found that teacher turnover had negative consequences for teacher quality (measured by licensure, experience, out-of-field teaching, and certification test scores). This article describes an attempt to build a human capital analytics tool to help a school district better identify schools with problems retaining teachers as well as schools that are doing an especially good job of retaining them. The intent was that the tool would catalyze the sharing of promising practices and targeting of intensive support to schools with chronic retention challenges. In the sections that follow, we first describe the district the tool was developed to support and then our theory of action for the school-level retention analytics tool. We then present the models we used to estimate persistent school effects and predict which schools would continue to have retention problems, report on what applying these models told us about retention in the district, and describe the basic features of a web-based tool we developed for the district to allow staff supporting schools to interact with the results and gain insights about the degree of variation among schools and likely contributing factors. Finally, we describe the initial reactions of the users for whom we designed the tool and discuss what we learned about developing a retention analytic tool for use by school district administrators.</p> <hd id="AN0184324479-2">District Context and Theory of Action</hd> <p></p> <hd id="AN0184324479-3">District Context</hd> <p>Like many US school districts, Hillsborough County Public Schools (HCPS) has found it challenging to hire and retain enough qualified teachers to serve its students. HCPS serves Hillsborough County, Florida, which contains the city of Tampa and surrounding communities. It is the eighth-largest US school district, serving more than 210,000 students through 273 schools and programs. The district's population is geographically diverse, with farming areas, suburbs, and a dense urban core. Nearly two-thirds (61.34%) of students are classified as economically disadvantaged. The district employs over 15,500 certified teachers and makes approximately 1,000 new hires each year to fill teaching vacancies. However, the district has had increasing difficulties filling teaching positions. Improving retention was seen as a way to reduce the need for new hires as well as to improve instruction and culture at schools.</p> <p>The initial impetus for developing an analytics tool focused on teacher retention emerged from prior work developing estimates of school and teacher value-added that the organization employing the first three authors was doing for the district. The organization wanted to explore how teacher value-added estimates could be used in making human capital management decisions beyond contributing to teacher evaluation scores or allocating bonuses. The district was interested in identifying principals who had better experience with retaining effective teachers and whether these principals' schools might be using practices that promoted retention. The hope was that these practices could be replicated in schools that were weaker on retention. After some initial exploration showed that there was variation among schools, the idea of a tool for use by district staff overseeing schools to explore reasons for the variation emerged from discussions. The district agreed to use some funding from its Teacher Incentive Fund grant to support the development of the tool.</p> <p>The primary users envisioned for the tool were the six area superintendents, each of whom oversees a set of schools, and the six area specialists, district office staff assigned to help individual schools with human capital management issues such as hiring and retention. In order to help these area staff pinpoint those schools that needed help improving retention and identify those that could be models, the district and the tool development team decided to develop both analytical models that would reliably measure persistent school differences in retention and a visualization tool to allow exploration of differences in school-level retention and their potential causes.</p> <p>HCPS has several advantages that facilitated developing models for studying teacher retention. The district has been an innovator in human capital management, participating in the Gates Foundation–funded Empowering Effective Teachers initiative and three rounds of the US Department of Education's Teacher Incentive Fund grants. It has an extensive student data system with teacher–student links, a mature teacher evaluation system that uses both principals' ratings of teachers' instructional practice and teacher value-added (the latter covering almost all teachers), and multiple years of data. We were able to use the results of the teacher evaluations to explore whether schools were better or worse than expected in retaining highly effective, effective, and less than effective teachers. Because multiple years of student assessment data linked to teachers was available, we could also look at whether schools were better or worse than expected in retaining high-, average-, and low-value-added teachers.</p> <hd id="AN0184324479-4">Theory of Action</hd> <p>Our theory of action linking school-level retention analytics to improved teacher retention was that if districts could identify schools that appear to be doing better than expected at retaining teachers, given nonmalleable factors known to influence retention, such as proportions of students of color and economic disadvantaged students ([<reflink idref="bib3" id="ref5">3</reflink>]),[<reflink idref="bib1" id="ref6">1</reflink>] these schools could be contrasted with others doing worse than expected to see if the former have conditions or are using practices that encourage retention. These practices could then be shared with comparable lower-retention schools. In addition, identifying schools with persistently lower retention after controlling for such nonmalleable factors would allow the district to focus intensive support on these schools. It could also be useful to encourage leaders in low-retention schools to consider changes in practice by showing them that some peer schools (those with similar nonmalleable characteristics) are doing substantially better.</p> <p>We focused on school-level differences in retention instead of individual-level analysis and prediction of teacher-level turnover for several reasons. First, it is not easy to obtain information on malleable factors that influence turnover/retention at the individual level. Most districts do not collect information on teacher engagement, perceptions of leadership or climate, or perceptions of working conditions in ways that allow linking these factors to individuals. Second, many factors that have been shown to affect retention are school-level constructs such as school leadership and school climate and student demographics. Third, a great deal of human capital management takes place at the school level ([<reflink idref="bib16" id="ref7">16</reflink>]), and from a district perspective, actions to improve retention are likely to be directed at the school and not individual teachers.</p> <p>A key requirement for identifying exemplar schools and schools needing additional support is reliable identification of those schools that are doing better or worse than average at retaining teachers. Since school-level teacher retention can vary substantially from year to year ([<reflink idref="bib11" id="ref8">11</reflink>]), and may be especially variable for small schools, simply using last year's retention results to distinguish among schools could be "chasing noise" rather than identifying schools with chronic needs or proven strategies. Providing district leaders with information that is actionable requires estimating school effects that reliably show which schools have consistently better retention and which are most likely to continue to struggle. So, one of the priorities of the analytic tool described here was to use multiple years of data to estimate a reliable school effect on retention.</p> <p>We also wanted to distinguish between retaining more effective and less effective teachers. Some turnover is appropriate when it is ineffective teachers who leave. Research also suggests that retention can vary with teacher effectiveness. For example, [<reflink idref="bib7" id="ref9">7</reflink>] found that teachers in the top and bottom quartile of teacher value-added in Florida were more likely to leave their schools.[<reflink idref="bib2" id="ref10">2</reflink>] We also wanted to show users how well schools retain teachers at different experience levels, given that inexperienced teachers are often less effective ([<reflink idref="bib14" id="ref11">14</reflink>]; [<reflink idref="bib20" id="ref12">20</reflink>]). Differing retention rates by experience may also provide clues to potential causes of excessive turnover. Lower-than-average retention of new teachers but average retention of the more experienced might indicate problems with school-level induction programs, while lower retention of experienced teachers only could suggest veterans are using their experience to transfer out of less desirable schools. Lower-than-average retention across the experience spectrum could suggest problems with school leadership.</p> <p>To put this theory of action into action, we developed a tool for users to provide information on school-level retention. First, we had to develop reliable estimates of differences in school retention rates and predictions of future retention. We began by examining whether there were in fact reliable differences across schools in their ability to retain teachers. Focusing on school-level variation is a useful strategy only if there are reliable differences in retention that indicate the need for a school-specific intervention or the possibility that a school has characteristics or is using practices that plausibly lead to better-than-expected retention. We also needed to know whether these differences persist when controlling for schools' student population. One might expect schools with different student populations to have different levels of retention, and if we are to hold school leaders accountable for retention or identify schools that may be using practices or strategies that raise retention that might be shared with others, we need to be sure that differences in school demographics are not the predominant cause of differences.</p> <p>We also wanted target users to be able to see whether teachers of different levels of experience and effectiveness have different rates of retention and whether schools differ in persistent retention of teachers whose effectiveness or experience differs. Differential retention of teachers by level of experience may provide clues as to where to focus retention efforts. Showing retention by effectiveness and experience may help schools diagnose problems and target retention efforts. For example, schools that have trouble retaining both effective and less than effective teachers may need different support than those that are better at retaining ineffective compared to effective teachers. Then we had to develop a set of interactive visualizations that would allow the district staff responsible for overseeing and supporting the schools to explore the variation and recognize schools that were more and less successful in retaining teachers. Note that at this stage of development, we were primarily concerned with retention in the school, not retention in the district as a whole.</p> <hd id="AN0184324479-5">Estimating Persistent School Retention</hd> <p></p> <hd id="AN0184324479-6">The Analytic Foundation</hd> <p>The engine powering our retention analysis tool was a statistical model that estimated individual teachers' probability of retention by a school, using multiple years of data to differentiate between the persistent and transitory variation in teacher retention by school. The model included school- and teacher-level predictors of retention and school random effects that were allowed to be correlated across years. The covariance structure of these random effects was consistent with a variance components specification that allowed for a persistent school effect and transitory (year-by-year) effects. Typically, researchers would use a logit or probit specification since retention is a binary (yes or no) state. However, in this case, interpretability by users was a key consideration. It is far easier to interpret—and include in an interactive tool—a school retention rate derived from a linear probability model. Since our school-level retention rates were typically within the "rule of thumb" for acceptable use of a linear probability model (i.e., in the.20 to.80 range), we judged the gain in interpretability worth the loss from violation of linear model assumptions. A major advantage was the ability to interpret estimated school effects as deviations from a district-wide retention rate expressed as a proportion or percentage.</p> <p>Estimates of the persistent and transitory school effects were produced using multivariate shrinkage since the model allowed for different retention school effects for different levels of teacher experience. The use of shrinkage estimation produced estimates that were not dominated by statistical noise.</p> <p>The basic model used to estimate school persistent effects was:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>R</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mrow><mi>μ</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>S</mi></mrow><mrow><mi>k</mi></mrow></msub><mi mathvariant="normal">δ</mi><mo>+</mo><msub><mrow><mi>ν</mi></mrow><mrow><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi>τ</mi></mrow><mrow><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>r</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></math> </ephtml> </p> <p>where <emph>R</emph><subs><emph>jkt</emph></subs> is a binary indicator equal to 1 if teacher j at school k is retained; <emph>μ</emph><subs><emph>t</emph></subs> are year-specific means capturing district-wide average retention in a given year; <emph>S</emph><subs><emph>k</emph></subs> is composed of traits of the school thought to influence retention; <emph>ν</emph><subs><emph>k</emph></subs> is a time-invariant (i.e., "persistent") random school effect; <emph>τ</emph><subs><emph>kt</emph></subs> is a time-varying (i.e., "transitory") random school effect; and <emph>r</emph><subs><emph>jkt</emph></subs> is a model residual. In all our models, school characteristics, <emph>S</emph><subs><emph>k</emph></subs>, include the percentage of students at the school who are Hispanic, Black, English-language learners, or who qualify for free or reduced-price lunch. For purposes of comparison, we also report a version of this model in which these school characteristics are omitted.</p> <p>This model estimates the total rate of retention for a school but was refined to estimate school effects on teachers of varying quality or experience. Let <emph>γ</emph><subs><emph>jk</emph></subs> be a mean-zero measure of a persistent component of a teacher's value-added rating (the method by which this component is identified is discussed later in the section on persistent teacher effectiveness). We adjusted our model to reflect differential retention by quality as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>R</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mrow><mi>μ</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>S</mi></mrow><mrow><mi>k</mi><mi>t</mi></mrow></msub><mi>δ</mi><mo>+</mo><mi mathvariant="normal">β</mi><msub><mrow><mi>γ</mi></mrow><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi mathvariant="normal">β</mi></mrow><mrow><mi mathvariant="normal">k</mi></mrow></msub><msub><mrow><mi>γ</mi></mrow><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi mathvariant="normal">ν</mi></mrow><mrow><mi mathvariant="normal">k</mi></mrow></msub><mo>+</mo><msub><mrow><mi>τ</mi></mrow><mrow><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>r</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></math> </ephtml> In this model, we estimate the overall impact of quality on retention with the fixed-effect coefficient β and then further estimate a random school coefficient, <emph>β</emph><subs><emph>k</emph></subs>, to determine the degree to which schools vary in their ability to differentially retain the best teachers; the greater the value of <emph>β</emph><subs><emph>k</emph></subs>, the greater the school's ability to differentially retain its higher-quality teachers. An analogous model was also estimated where quality is measured by a persistent component estimated from the teacher's overall quality rating, a composite rating composed of value-added and observational metrics.</p> <p>To capture differential retention by experience, we estimated retention rates at 1, 5, 15, and 25 years of experience. Let <emph>p</emph> denote the years of experience from the list prior and <emph>x</emph><subs><emph>pkt</emph></subs> be equal to 1 if the teacher <emph>k</emph> has <emph>p</emph> years of experience. Teachers whose experience falls between two of these estimation points are weighted linearly to adjacent points; for example, if teacher <emph>k</emph> has 4 years of experience, then <emph>x</emph><subs>1<emph>kt</emph></subs> will equal.25 and <emph>x</emph><subs>2<emph>kt</emph></subs> will equal.75. In all other cases, <emph>x</emph><subs><emph>pkt</emph></subs> will equal 0. This is equivalent to a linear spline of experience from 1 to 25 years of experience with kink points at 5 and 15 years but allows retention rates for each level of experience to be more easily read from the model results.</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>R</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mrow><mi>μ</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>S</mi></mrow><mrow><mi>k</mi><mi>t</mi></mrow></msub><mi>δ</mi><mo>+</mo><mrow><munder><mo stretchy="false">∑</mo><mrow><mi>p</mi></mrow></munder><mrow><msub><mrow><mi>ϕ</mi></mrow><mrow><mi>p</mi></mrow></msub><msub><mrow><mi>x</mi></mrow><mrow><mi>p</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow></mrow><mo>+</mo><mrow><munder><mo stretchy="false">∑</mo><mrow><mi>p</mi></mrow></munder><mrow><msub><mrow><mi>ϕ</mi></mrow><mrow><mi>p</mi><mi>k</mi></mrow></msub><msub><mrow><mi>x</mi></mrow><mrow><mi>p</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow></mrow><mo>+</mo><msub><mrow><mi>τ</mi></mrow><mrow><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>r</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></math> </ephtml> </p> <p>where <emph>ϕ</emph><subs><emph>p</emph></subs> is a fixed effect coefficient corresponding to experience level <emph>p</emph> ∈ {1, 5, 15, 25} and <emph>ϕ</emph><subs><emph>pk</emph></subs> is a random school coefficient.</p> <hd id="AN0184324479-7">Measures</hd> <p> <emph>Teacher retention:</emph> Teacher retention was defined as retention with the same school. For the purpose at hand, we were interested in school retention differences irrespective of whether those who left were movers (changed schools) or leavers (left the district). We used teacher rosters provided by the district to track teacher movement out of schools or out of the district.</p> <p> <emph>Teacher experience:</emph> We relied on district administrative data to provide a value for each teacher's total number of years of experience in teaching. As described prior, instead of grouping teachers into experience categories and using dummies to indicate category membership, we used linear splines to estimate a continuous effect of experience on retention and chose to report retention effects at three experience points: 1 year (to represent novice teachers), 5 years (to represent early-career teachers, and 15 years (to represent midcareer teachers). We dropped teachers with more than 25 years of experience from the analysis sample in order to simplify reporting and interpretation, since this group of teachers is expected to have higher rates of turnover due to retirement, which was not considered a preventable or undesirable type of turnover.</p> <p> <emph>Teacher effectiveness:</emph> We used two different measures of teacher effectiveness in our analyses. First, we used teachers' ratings on the district's multiple method teacher evaluation system. This system combined ratings of classroom practice by school leaders using a practice rubric based on the Danielson Framework for Teaching ([<reflink idref="bib5" id="ref13">5</reflink>]). Based on these ratings, teachers received a score ranging from 0 to 60. The second component was a teacher-level value-added score, rescaled to run from 0 to 40. The two scores were added together to create an overall rating on a 0-to-100-point scale. The point values were developed so that the practice rating carried a 60% weight and the value-added score a 40% weight. In our dataset, the mean of the overall scores across all teachers and years was 65.0 and the standard deviation was 7.9. The district defined less than effective teachers as earning less than 46 points, effective teachers as earning to 46 to less than 63 points, and highly effective teachers as earning 63 or more points. For the following analyses, we grouped teachers into these three categories based on these cutoffs and used dummy variables to indicate the category into which each teacher's performance fell. Note that in this district, almost all classroom teachers have value-added scores due to district efforts to develop standardized tests for all courses. These tests are given in addition to the standard state tests given in math and English language arts (ELA).</p> <p>The second effectiveness measure was teachers' value-added in either mathematics or English language arts (or both) based on state standards-based assessments. We were able to estimate value-added for teachers of Grades 3 to 8 in mathematics and Grades 3 to 10 in ELA. We used this second measure to provide another perspective on teacher effectiveness, one that was less skewed toward higher scores than the performance rating. The downside is that fewer teachers were covered and thus included in the analysis. In particular, few high school teachers could be included.</p> <p> <emph>Estimating persistent teacher effectiveness:</emph> One major complication we needed to address was that just as retention rates fluctuate over time, so do teachers' levels of effectiveness, both as measured by instructional practice ratings (e.g., [<reflink idref="bib19" id="ref14">19</reflink>]) and teacher value-added (e.g., Goldhaber & Hanson, 2013; [<reflink idref="bib15" id="ref15">15</reflink>]). Since we wanted to provide actionable results that reflected the stable component of teacher performance, we also developed models to estimate the stable components of the district's overall effectiveness ratings and value-added. We did this by using multiple years of data and multilevel modeling to decompose teacher value-added into time-varying and time-invariant components.</p> <p>The true value-added of teacher <emph>j</emph> in year <emph>t</emph>, denoted <emph>α</emph><subs><emph>jt</emph></subs>, is conceptualized as the sum of two random variables: a time-invariant component, <emph>μ</emph><subs><emph>j</emph></subs>, and a time-varying component, <emph>η</emph><subs><emph>jt</emph></subs>:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>α</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mrow><mi>γ</mi></mrow><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi>η</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></math> </ephtml> Available data for this project included previously calculated value-added estimates, as estimated from the following model:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>y</mi></mrow><mrow><mi>i</mi><mi>j</mi><mi>k</mi><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><msub><mrow><mi>y</mi></mrow><mrow><mi>i</mi><mi>j</mi><mi>k</mi><mo>,</mo><mn>0</mn></mrow></msub><mi>λ</mi><mo>+</mo><msub><mrow><mi>X</mi></mrow><mrow><mi>i</mi><mi>j</mi><mi>k</mi></mrow></msub><mi>β</mi><mo>+</mo><msub><mrow><mi>α</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>e</mi></mrow><mrow><mtext mathvariant="italic">ijkt</mtext></mrow></msub></math> </ephtml> where <emph>y</emph><subs><emph>ij</emph>,0</subs> and <emph>y</emph><subs><emph>ij</emph>,1</subs> are exam scores for student <emph>i</emph> in the prior and current year and <emph>X</emph><subs><emph>ij</emph></subs> are student-level variables controlling for student characteristics thought to influence student achievement growth. The fixed effects estimator</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mover accent="true"><mrow><mi>α</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>j</mi><mi>t</mi></mrow></msub></math> </ephtml> for <emph>α</emph><subs><emph>jt</emph></subs> includes an error term due to attributing the mean of the student residuals to the teacher effect:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mover accent="true"><mrow><mi>α</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mrow><mi>α</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mover accent="true"><mrow><mi>e</mi></mrow><mo>¯</mo></mover></mrow><mrow><mo>.</mo><mi>j</mi><mi>k</mi></mrow></msub></math> </ephtml> Given our assumptions about the components of the value-added, this implies that:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mover accent="true"><mrow><mi>α</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>=</mo><msub><mrow><mi>γ</mi></mrow><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi>η</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mover accent="true"><mrow><mi>e</mi></mrow><mo>¯</mo></mover></mrow><mrow><mo>.</mo><mi>j</mi><mi>k</mi></mrow></msub></math> </ephtml> Estimation of the random, time-invariant teacher effect, <emph>γ</emph><subs><emph>jk</emph></subs>, first requires accurate estimation of all variance components. Because the variance of the mean residual depends on the number of students assigned to the particular teacher, it is therefore necessary to employ a strategy to account for heteroskedasticity. Our approach is to construct an alternative variable that does not include heteroskedastic components but still correctly identifies the variance of the time-invariant teacher effect. Let ϵ<subs>ijt</subs> be the conditional student growth, defined as:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>ϵ</mi></mrow><mrow><mtext mathvariant="italic">ijkt</mtext></mrow></msub><mo>=</mo><msub><mrow><mi>α</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>e</mi></mrow><mrow><mtext mathvariant="italic">ijkt</mtext></mrow></msub><mo>=</mo><msub><mrow><mi>γ</mi></mrow><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi>η</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mrow><mi>e</mi></mrow><mrow><mtext mathvariant="italic">ijkt</mtext></mrow></msub></math> </ephtml> With the student-level data used to estimate the original value-added data, <emph>ϵ</emph><subs><emph>ijkt</emph></subs> is identified as <emph>y</emph><subs><emph>ijk</emph>,1</subs> – <emph>y</emph><subs><emph>ijk</emph>,0</subs><emph>λ</emph> + <emph>X</emph><subs><emph>ijk</emph></subs><emph>β</emph>. Unfortunately, available data do not include student-level data in all years. To circumvent this problem, we create an analogous variable using available teacher value-added estimates and match the variance components by introducing a normally distributed, mean-zero simulated residual, <emph>ρ</emph><subs><emph>ijkt</emph></subs>, which is generated to have the same known variance as <emph>e</emph><subs><emph>ijkt</emph></subs>, or</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>e</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> , and then demeaned by teacher:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mtable columnalign="left"><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mover><mi>ϵ</mi><mo>˜</mo></mover></mrow><mrow><mi>i</mi><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow></mtd><mtd columnalign="left"><mrow><mo>=</mo><msub><mover accent="true"><mi>α</mi><mo>^</mo></mover><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mi>ρ</mi><mrow><mi>i</mi><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>−</mo><msub><mover accent="true"><mi>ρ</mi><mo>¯</mo></mover><mrow><mo>.</mo><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow></mtd></mtr><mtr columnalign="left"><mtd /><mtd columnalign="left"><mrow><mo>=</mo><msub><mi>α</mi><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mover accent="true"><mi>e</mi><mo>¯</mo></mover><mrow><mo>.</mo><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>+</mo><msub><mi>ρ</mi><mrow><mi>i</mi><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>−</mo><msub><mover accent="true"><mi>ρ</mi><mo>¯</mo></mover><mrow><mo>.</mo><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow></mtd></mtr></mtable></mrow></math> </ephtml> It can readily be shown, then, that the variance of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>ρ</mi></mrow><mrow><mtext mathvariant="italic">ijkt</mtext></mrow></msub><mo>-</mo><msub><mrow><mover accent="true"><mrow><mi>ρ</mi></mrow><mo>¯</mo></mover></mrow><mrow><mo>.</mo><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></math> </ephtml> is equal to</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mfrac><mrow><msub><mrow><mi>n</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub><mo>-</mo><mn>1</mn></mrow><mrow><msub><mrow><mi>n</mi></mrow><mrow><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow></mfrac><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>e</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> , where <emph>n</emph><subs><emph>jkt</emph></subs> is the number of students assigned to teacher <emph>j</emph> in time <emph>t</emph>. Thus, the variance of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mover accent="true"><mrow><mi>ϵ</mi></mrow><mo>˜</mo></mover></mrow><mrow><mtext mathvariant="italic">ijkt</mtext></mrow></msub></math> </ephtml> is equal to:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mtable columnalign="left"><mtr columnalign="left"><mtd columnalign="left"><mrow><mi>v</mi><mi>a</mi><mi>r</mi><mrow><mo>(</mo><mrow><msub><mover accent="true"><mi>ϵ</mi><mo>˜</mo></mover><mrow><mi>i</mi><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow><mo>)</mo></mrow></mrow></mtd><mtd columnalign="left"><mrow><mo>=</mo><msubsup><mi>σ</mi><mi>α</mi><mn>2</mn></msubsup><mo>+</mo><msubsup><mi>σ</mi><mi>e</mi><mn>2</mn></msubsup></mrow></mtd></mtr><mtr columnalign="left"><mtd /><mtd columnalign="left"><mrow><mo>=</mo><msubsup><mi>σ</mi><mi>γ</mi><mn>2</mn></msubsup><mo>+</mo><msubsup><mi>σ</mi><mi>η</mi><mn>2</mn></msubsup><mo>+</mo><msubsup><mi>σ</mi><mi>e</mi><mn>2</mn></msubsup></mrow></mtd></mtr><mtr columnalign="left"><mtd /><mtd columnalign="left"><mrow><mo>=</mo><mi>v</mi><mi>a</mi><mi>r</mi><mrow><mo>(</mo><mrow><msub><mi>ϵ</mi><mrow><mi>i</mi><mi>j</mi><mi>k</mi><mi>t</mi></mrow></msub></mrow><mo>)</mo></mrow></mrow></mtd></mtr></mtable></mrow></math> </ephtml> This demonstrates that</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mover accent="true"><mrow><mi>ϵ</mi></mrow><mo>˜</mo></mover></mrow><mrow><mtext mathvariant="italic">ijkt</mtext></mrow></msub></math> </ephtml> has the appropriate variance components. The variance of the persistent teacher quality,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>γ</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> , and the associated best linear unbiased predictors (BLUPs) of school persistent effects, may then be calculated using a standard mixed modeling estimation package, in this case the lme4 package in R.</p> <p>Persistent components for the overall rating were determined analogously to the procedure described for value-added previously. However, to reflect the district's internal teacher classification system, models based on the overall rating translated the estimated persistent rating (a score on a 0 to 100 scale) into three performance categories, as described in the next section.</p> <hd id="AN0184324479-8">Data</hd> <p>Data was supplied by the district as part of another cooperative project that involves estimating individual teacher value-added. For analyses, the effects data from the 2013–14, 2014–15, 2015–16, 2016–17, and 2017–18 school years was used, along with data from the fall of 2018, which showed which teachers from 2017–18 were retained in the fall of 2018. This dataset included 13,900 unique teachers and 45,352 teacher-year records. The median number of years each teacher was observed was 3. For retention analyses using value-added in ELA and math, we were able to include an earlier year, 2012–13, in the analysis. A total number of 4,399 unique teachers and 12,987 teacher-year records were included in the math teacher value-added dataset, while for ELA there were 5,092 unique teachers and 15,744 observations. Table 1 presents a descriptive summary of the data used in the models. Teacher-level data was used to estimate the models, with school-level data (e.g., percentage of students who were English learners) added to some of the models and used for visualizations that showed how retention was related to school characteristics.</p> <p> <bold>Table 1:</bold> Selected descriptive statistics for teacher retention models</p> <p> <ephtml> <table rules="groups"><colgroup span="1"><col align="left" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /></colgroup><thead><tr><th align="left" valign="top" rowspan="1" colspan="1" /><th align="center" valign="top" rowspan="1" colspan="1">Mean</th><th align="center" valign="top" rowspan="1" colspan="1">Standard deviation</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">Teachers<italic>n</italic> = 13,900</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Number per year</td><td align="center" valign="top" rowspan="1" colspan="1">12,204</td><td align="center" valign="top" rowspan="1" colspan="1">344.1</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Years of experience</td><td align="center" valign="top" rowspan="1" colspan="1">12.1</td><td align="center" valign="top" rowspan="1" colspan="1">8.9<xref ref-type="table-fn" rid="tfn2">a</xref></td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Yearly retention rate (based on individual data)</td><td align="center" valign="top" rowspan="1" colspan="1">82.3</td><td align="center" valign="top" rowspan="1" colspan="1">2.3<xref ref-type="table-fn" rid="tfn2">a</xref></td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Schools<italic>n</italic> = 224</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School-level retention rate</td><td align="center" valign="top" rowspan="1" colspan="1">83.1</td><td align="center" valign="top" rowspan="1" colspan="1">8.5</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Number of students per school</td><td align="center" valign="top" rowspan="1" colspan="1">916.9</td><td align="center" valign="top" rowspan="1" colspan="1">586.1</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Percentage of students who were Black</td><td align="center" valign="top" rowspan="1" colspan="1">24.2</td><td align="center" valign="top" rowspan="1" colspan="1">20.7</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Percentage of students who were Latino</td><td align="center" valign="top" rowspan="1" colspan="1">36.1</td><td align="center" valign="top" rowspan="1" colspan="1">18.4</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Percentage of students who were economically disadvantaged</td><td align="center" valign="top" rowspan="1" colspan="1">31.9</td><td align="center" valign="top" rowspan="1" colspan="1">24.3</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Percentage of students who were English learners</td><td align="center" valign="top" rowspan="1" colspan="1">74.7</td><td align="center" valign="top" rowspan="1" colspan="1">15.3</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note:</emph></p> <p>2 a Standard deviations of years of experience and yearly retention rate reflect cross-year variation.</p> <hd id="AN0184324479-9">Targeting Accuracy</hd> <p>In many cases, it may be of interest for district administration to identify schools that may lag behind their peers in retention so that they may be targeted for intervention. An aim of the current research is to study whether school-level retention estimates may be used to identify a subset of schools according to low (or high) retention rates and to investigate the accuracy with which such targeted selection may be accomplished. To show how this may be done, we use as an example the model that identifies school effects on total retention rates (i.e., not conditional on quality or experience). We begin with <emph>η</emph><subs><emph>k</emph></subs>, the true school effect, whose shrunk estimate (described in the following) is</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mover accent="true"><mrow><mi>η</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>k</mi></mrow></msub></math> </ephtml> , and allow <emph>C(p)</emph> to be the value of the school effect at the <emph>p</emph>'th quantile, which may be calculated from the variance estimate for the school effects,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>η</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> , and the assumption of normal distribution. We need to estimate the probability of each school falling below this threshold. Denoting this probability for school <emph>k</emph> as <emph>P</emph><subs><emph>kp</emph></subs>:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>P</mi></mrow><mrow><mi>k</mi><mi>p</mi></mrow></msub><mo>=</mo><mrow><mrow><mtext mathvariant="italic">Pr</mtext></mrow><mo>⁡</mo><mrow><mfenced separators="|"><mrow><msub><mrow><mi>η</mi></mrow><mrow><mi>k</mi></mrow></msub><mo><</mo><mi>C</mi><mo>(</mo><mi>p</mi><mo>)</mo><mo>|</mo><msub><mrow><mover accent="true"><mrow><mi>η</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>k</mi></mrow></msub></mrow></mfenced></mrow></mrow></math> </ephtml> In shrinkage estimation, the shrunk school effect estimate,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mover accent="true"><mrow><mi>η</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>k</mi></mrow></msub></math> </ephtml> , is calculated to be the expected value of the true school effect given the overall variance of school effects. This allows us to write the true school effect as a sum of the shrunk estimate and a mean-zero estimation error:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>η</mi></mrow><mrow><mi>k</mi></mrow></msub><mo>=</mo><msub><mrow><mover accent="true"><mrow><mi>η</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi>ε</mi></mrow><mrow><mi>k</mi></mrow></msub></math> </ephtml> where <emph>ε</emph><subs><emph>k</emph></subs> represents the estimation error for school <emph>k</emph> with variance</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>k</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> . Substitution gives:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>P</mi></mrow><mrow><mi>k</mi><mi>p</mi></mrow></msub><mo>=</mo><mrow><mrow><mtext mathvariant="italic">Pr</mtext></mrow><mo>⁡</mo><mrow><mfenced separators="|"><mrow><msub><mrow><mover accent="true"><mrow><mi>η</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>k</mi></mrow></msub><mo>+</mo><msub><mrow><mi>ε</mi></mrow><mrow><mi>k</mi></mrow></msub><mo><</mo><mi>C</mi><mfenced separators="|"><mrow><mi>p</mi></mrow></mfenced></mrow></mfenced></mrow></mrow></math> </ephtml> This in turn implies that:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>P</mi></mrow><mrow><mi>k</mi><mi>p</mi></mrow></msub><mo>=</mo><mrow><mrow><mtext mathvariant="italic">Pr</mtext></mrow><mo>⁡</mo><mrow><mfenced separators="|"><mrow><mfrac><mrow><msub><mrow><mi>ε</mi></mrow><mrow><mi>k</mi></mrow></msub></mrow><mrow><msub><mrow><mi>σ</mi></mrow><mrow><mi>k</mi></mrow></msub></mrow></mfrac><mo><</mo><mfrac><mrow><mi>C</mi><mfenced separators="|"><mrow><mi>p</mi></mrow></mfenced><mo>-</mo><msub><mrow><mover accent="true"><mrow><mi>η</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>k</mi></mrow></msub></mrow><mrow><msub><mrow><mi>σ</mi></mrow><mrow><mi>k</mi></mrow></msub></mrow></mfrac></mrow></mfenced></mrow></mrow></math> </ephtml> If all components are assumed to be normal, then the targeting accuracy for school k is given by</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>P</mi></mrow><mrow><mi>k</mi><mi>p</mi></mrow></msub><mo>=</mo><mi mathvariant="normal">Φ</mi><mfenced separators="|"><mrow><mfrac><mrow><mi>C</mi><mfenced separators="|"><mrow><mi>p</mi></mrow></mfenced><mo>-</mo><msub><mrow><mover accent="true"><mrow><mi>η</mi></mrow><mo>^</mo></mover></mrow><mrow><mi>k</mi></mrow></msub></mrow><mrow><msub><mrow><mi>σ</mi></mrow><mrow><mi>k</mi></mrow></msub></mrow></mfrac></mrow></mfenced></math> </ephtml> This gives us an estimate of the probability that any individual school falls in the lowest <emph>p</emph>'th quantile. If we assume that schools indexed by <emph>k</emph> are ordered from lowest to highest performing, the expected number of schools in a selection of <emph>M</emph> schools that are correctly identified in the lowest <emph>p</emph>'th quantile is therefore:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi>E</mi><mfenced open="[" close="]" separators="|"><mrow><mtext mathvariant="italic">Number of correctly identified schools</mtext></mrow></mfenced><mo>=</mo><mrow><munderover><mo stretchy="false">∑</mo><mrow><mi>k</mi></mrow><mrow><mi>M</mi></mrow></munderover><mrow><msub><mrow><mi>P</mi></mrow><mrow><mi>k</mi><mi>p</mi></mrow></msub></mrow></mrow></math> </ephtml> </p> <p>We believe that this calculation may be extremely useful to school districts in planning interventions aimed at increasing retention rates at schools that struggle most by providing a way for the district to estimate the marginal benefit of expanding the program to include additional schools. Conversely, this development may also be followed for identifying schools with the <emph>highest</emph> retention rates so that districts may choose subsets of schools that provide the best environments for studying what makes schools successful at retaining their teachers.</p> <hd id="AN0184324479-10">Evaluation of the Analytical Approach</hd> <p>Before building our estimates into a tool for the intended users, we examined whether the model did indeed provide reliable estimates with meaningful cross-school variation and whether the model could estimate credible differential effects by experience and teacher effectiveness. Before presenting the results from the retention models, we first show that in this district, schools' yearly retention rates did vary substantially over time, supporting the need for the relatively complex modeling we used to estimate persistent school effects. Table 2 shows the correlation of school retention rates between succeeding years in our dataset.</p> <p> <bold>Table 2:</bold> Correlations of school retention rates from year to year</p> <p> <ephtml> <table rules="groups"><colgroup span="1"><col align="left" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /></colgroup><thead><tr><th align="left" valign="top" rowspan="1" colspan="1" /><th colspan="2" align="center" valign="bottom" rowspan="1">All schools</th><th colspan="2" align="center" valign="top" rowspan="1">Schools with fewer than 25 teachers</th></tr><tr><th align="left" valign="top" rowspan="1" colspan="1">Year pair</th><th align="center" valign="top" rowspan="1" colspan="1">Correlation</th><th align="center" valign="top" rowspan="1" colspan="1">Number</th><th align="center" valign="top" rowspan="1" colspan="1">Correlation</th><th align="center" valign="top" rowspan="1" colspan="1">Number</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">2014 and 2015</td><td align="center" valign="top" rowspan="1" colspan="1">.45</td><td align="center" valign="top" rowspan="1" colspan="1">236</td><td align="center" valign="top" rowspan="1" colspan="1">.41</td><td align="center" valign="top" rowspan="1" colspan="1">34</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">2015 and 2016</td><td align="center" valign="top" rowspan="1" colspan="1">.52</td><td align="center" valign="top" rowspan="1" colspan="1">235</td><td align="center" valign="top" rowspan="1" colspan="1">.57</td><td align="center" valign="top" rowspan="1" colspan="1">30</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">2016 and 2017</td><td align="center" valign="top" rowspan="1" colspan="1">.44</td><td align="center" valign="top" rowspan="1" colspan="1">235</td><td align="center" valign="top" rowspan="1" colspan="1">.30</td><td align="center" valign="top" rowspan="1" colspan="1">35</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">2017 and 2018</td><td align="center" valign="top" rowspan="1" colspan="1">.46</td><td align="center" valign="top" rowspan="1" colspan="1">238</td><td align="center" valign="top" rowspan="1" colspan="1">.23</td><td align="center" valign="top" rowspan="1" colspan="1">38</td></tr></tbody></table> </ephtml> </p> <p>Column 2 shows the correlations for all schools in the dataset. While there is substantial correlation of retention rates across consecutive years, one year by itself does not predict the next too well. Column 4 shows that the year-to-year correlations are typically smaller for schools with fewer than 25 teachers. There was also a small correlation between the number of teachers in a school and retention in each year, ranging from.17 to.33. This might be expected because loss of one teacher from a 15-teacher school produces a lower retention rate than a loss of one teacher from a 100-teacher school. This suggests that focusing on one year's retention rate, especially for small schools, will make it difficult to correctly identify those schools that are consistently doing better at retaining and those that need more support.</p> <p>Next, we estimated a model that decomposed school retention into stable and transitory components, without controlling for student demographics. The variance components for school and for the school X year combinations are shown in Column 2 of Table 3 following. We then added three major school-level demographic controls to the model: percentage free and reduced-price lunch, percentage English learners, and percentage African American/Black and Hispanic/Latino. As expected, characteristics of schools' student populations influenced teacher retention and changed the variance decomposition. Column 3 of Table 3 shows the coefficients for school characteristics and the school-level variance in retention from a model including these characteristics.</p> <p> <bold>Table 3:</bold> Results of retention models with and without school characteristics, teacher effectiveness ratings, and teacher experience</p> <p> <ephtml> <table rules="groups"><colgroup span="1"><col align="left" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /></colgroup><thead><tr><th align="left" valign="bottom" rowspan="1" colspan="1">Parameter</th><th align="center" valign="top" rowspan="1" colspan="1">Model without school characteristics</th><th align="center" valign="top" rowspan="1" colspan="1">Model with school characteristics</th><th align="center" valign="top" rowspan="1" colspan="1">Model with teacher experience effects</th><th align="center" valign="bottom" rowspan="1" colspan="1">Model with teacher effectiveness ratings</th></tr></thead><tbody><tr><td colspan="5" align="left" valign="top" rowspan="1">Standard deviations of random effects</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School/year</td><td align="center" valign="top" rowspan="1" colspan="1">0.048</td><td align="center" valign="top" rowspan="1" colspan="1">0.048</td><td align="center" valign="top" rowspan="1" colspan="1">0.049</td><td align="center" valign="top" rowspan="1" colspan="1">0.048</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect</td><td align="center" valign="top" rowspan="1" colspan="1">0.071</td><td align="center" valign="top" rowspan="1" colspan="1">0.037</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 1st-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.103</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 5th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.044</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 15th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.022</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for less than effective teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.081</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for effective teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.047</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for highly effective teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.037</td></tr><tr><td colspan="5" align="left" valign="top" rowspan="1">Fixed effect coefficients</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent free and reduced-price lunch</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.169(0.029)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.131(0.027)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.154(0.028)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent Black/African American</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.129(0.033)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.119(0.031)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.092(0.033)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent Hispanic/Latino</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.021(0.054)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.034(0.049)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.006(0.053)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent English learner</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.108(0.051)</td><td align="center" valign="top" rowspan="1" colspan="1">0.104(0.045)</td><td align="center" valign="top" rowspan="1" colspan="1">0.107(0.107)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Teacher experience</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 1st year</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.769(0.014)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 5th year</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.920(0.011)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 15th year</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.947(0.010)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Teacher effectiveness category</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Less than effective</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.536(0.030)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Effective</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.865(0.012)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Highly effective</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.942(0.011)</td></tr></tbody></table> </ephtml> </p> <p>While the variance of school-level persistent effects decreases when controls for student demographics are added, there is still a substantial level of variance in these effects. This shows that while school ethnic composition, percentage of English learners, and student economic disadvantage influenced retention in schools, there was still substantial stable variation among schools at similar levels of these characteristics.</p> <p>Column 4 of Table 3 shows the average effects of experience. As might be expected, retention increases substantially for 5th-year teachers over 1st-year teachers but then less between 5th- and 15th-year teachers. Column 5 shows the average effects of teachers' rated effectiveness on retention: less than effective teachers have, on average, only a little more than a 50% retention rate, while rates for effective and highly effective teachers are considerably higher. Reassuringly, the rate for teachers rated as highly effective is higher than for those rated effective.</p> <p>Tables 4 and 5 report on results of analyses using the more restricted sample of Grades 3–10 teachers with ELA value-added (Table 4) and Grades 3–8 teachers with math (Table 5). Standard deviations of school effects for overall retention are largely similar. So is the pattern of fixed effects for teacher experience, with district average retention rates greater at each level of experience.</p> <p> <bold>Table 4:</bold> Results of retention models with and without school characteristics and teacher value-added: ELA teacher subset</p> <p> <ephtml> <table rules="groups"><colgroup span="1"><col align="left" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /></colgroup><thead><tr><th align="left" valign="bottom" rowspan="1" colspan="1">Parameter</th><th align="center" valign="top" rowspan="1" colspan="1">Model without school characteristics</th><th align="center" valign="bottom" rowspan="1" colspan="1">Model with school characteristics</th><th align="center" valign="top" rowspan="1" colspan="1">Model with teacher experience</th><th align="center" valign="top" rowspan="1" colspan="1">Model with ELA teacher value-added</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">Standard deviations of random effects</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School/year</td><td align="center" valign="top" rowspan="1" colspan="1">0.043</td><td align="center" valign="top" rowspan="1" colspan="1">0.043</td><td align="center" valign="top" rowspan="1" colspan="1">0.046</td><td align="center" valign="top" rowspan="1" colspan="1">0.044</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect</td><td align="center" valign="top" rowspan="1" colspan="1">0.069</td><td align="center" valign="top" rowspan="1" colspan="1">0.038</td><td align="center" valign="top" rowspan="1" colspan="1">-</td><td align="center" valign="top" rowspan="1" colspan="1">0.037</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 1st-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.074</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 5th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.046</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 15th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.035</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School coefficient on persistent teacher value-added component</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.038</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">School-level fixed effect coefficients</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent free and reduced-price lunch</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.093(0.039)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.075(0.038)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.083(0.039)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent Black/African American</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.211(0.045)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.202(0.044)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.212(0.045)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent Hispanic/Latino</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.091(0.072)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.090(0.069)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.080(0.071)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent English learner</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.109(0.066)</td><td align="center" valign="top" rowspan="1" colspan="1">0.096(0.063)</td><td align="center" valign="top" rowspan="1" colspan="1">0.087(0.065)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 1st-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.820(0.018)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 5th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.868(0.016)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 15th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.918(0.015)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Persistent teacher value-added</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">-</td><td align="center" valign="top" rowspan="1" colspan="1">0.023<xref ref-type="table-fn" rid="tfn4">a</xref>(0.008)</td></tr></tbody></table> </ephtml> </p> <ulist> <item>3 <emph>Note:</emph></item> <item>4 a This slope coefficient indicates the percent change in retention per standard deviation in teacher quality.</item> </ulist> <p> <bold>Table 5:</bold> Results of retention models with and without school characteristics and teacher value-added: Math teacher subset</p> <p> <ephtml> <table rules="groups"><colgroup span="1"><col align="left" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /></colgroup><thead><tr><th align="left" valign="bottom" rowspan="1" colspan="1">Parameter</th><th align="center" valign="top" rowspan="1" colspan="1">Model without school characteristics</th><th align="center" valign="bottom" rowspan="1" colspan="1">Model with school characteristics</th><th align="center" valign="top" rowspan="1" colspan="1">Model with teacher experience</th><th align="center" valign="top" rowspan="1" colspan="1">Model with math teacher value-added</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">Standard deviations of random effects</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School/year</td><td align="center" valign="top" rowspan="1" colspan="1">0.061</td><td align="center" valign="top" rowspan="1" colspan="1">0.060</td><td align="center" valign="top" rowspan="1" colspan="1">0.061</td><td align="center" valign="top" rowspan="1" colspan="1">0.060</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect</td><td align="center" valign="top" rowspan="1" colspan="1">0.077</td><td align="center" valign="top" rowspan="1" colspan="1">0.048</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.047</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 1st-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.097</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 5th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.053</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School stable effect for 15th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.037</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School coefficient on persistent teacher value-added component</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.029</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">School-level fixed effect coefficients</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1" /></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent free and reduced-price lunch</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.117(0.047)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.091(0.044)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.105(0.046)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent Black/African American</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.172(0.053)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.178(0.050)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.168(0.053)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent Hispanic/Latino</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">−0.071(0.083)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.094(0.078)</td><td align="center" valign="top" rowspan="1" colspan="1">−0.073(0.082)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> School percent English learner</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.172(0.075)</td><td align="center" valign="top" rowspan="1" colspan="1">0.142(0.069)</td><td align="center" valign="top" rowspan="1" colspan="1">0.126(0.074)</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 1st-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.828(0.020)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 5th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.886(0.018)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> 15th-year teachers</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">0.913(0.017)</td><td align="center" valign="top" rowspan="1" colspan="1">–</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1"> Persistent teacher value-added</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1">–</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1">0.030<xref ref-type="table-fn" rid="tfn6">a</xref>(0.007)</td></tr></tbody></table> </ephtml> </p> <ulist> <item>5 <emph>Note:</emph></item> <item>6 a This slope indicates the percent change in retention per standard deviation in teacher quality.</item> </ulist> <p>In these models, the standard deviation of the school coefficient on teacher value-added (0.038 for ELA and 0.029 for math) shows how far different schools typically are from the district mean in how teachers' persistent value-added effects retention. The fixed effect coefficients are positive and show that retention is more likely as value-added increases. One standard deviation of teacher value added is associated with an increase in retention of 2.9 (math) or 2.3 (ELA) percentage points. The standard deviations, which are about the same size as the coefficients for persistent teacher value-added, suggest that there is substantial school-level variation in the relationship between value-added and retention across schools. For example, a school with a retention slope that is one standard deviation more than the district-wide estimate for the relationship of math value-added to retention shows a stronger effect of value-added on retention, with a one standard deviation increase in value-added associated with a 6.7 percentage point increase in retention rate. A school with a school effect one standard deviation less would have a slight negative relationship between teacher value-added and retention in that a one standard deviation increase in value-added would be associated with a −0.9 percentage point decrease in retention rate. It seems that on this measure, as well as the rated effectiveness measures, there is substantively important variation across schools.</p> <p>Whether schools differ in their retention of teachers at different experience or effectiveness levels may also be useful in suggesting reasons for retention problems. Table 6 shows the correlations of the noise-corrected (persistent) school retention effects for different levels of teacher experience, while Table 7 shows the same for the three different levels of rated teacher effectiveness.</p> <p> <bold>Table 6:</bold> Correlation of noise-corrected school retention rates for teachers at different experience levels</p> <p> <ephtml> <table rules="groups"><colgroup span="1"><col align="left" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /></colgroup><thead><tr><th align="left" valign="top" rowspan="1" colspan="1">Experience</th><th align="center" valign="top" rowspan="1" colspan="1">5th year</th><th align="center" valign="top" rowspan="1" colspan="1">15th year</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">1st year</td><td align="center" valign="top" rowspan="1" colspan="1">0.30</td><td align="center" valign="top" rowspan="1" colspan="1">0.25</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">5th year</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1">0.84</td></tr></tbody></table> </ephtml> </p> <p> <bold>Table 7:</bold> Correlation of noise-corrected school retention rates for teachers at different effectiveness levels</p> <p> <ephtml> <table rules="groups"><colgroup span="1"><col align="left" valign="top" span="1" /><col align="center" valign="top" span="1" /><col align="center" valign="top" span="1" /></colgroup><thead><tr><th align="left" valign="top" rowspan="1" colspan="1">Effectiveness rating</th><th align="center" valign="top" rowspan="1" colspan="1">Effective</th><th align="center" valign="top" rowspan="1" colspan="1">Highly effective</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">Less than effective</td><td align="center" valign="top" rowspan="1" colspan="1">−0.20</td><td align="center" valign="top" rowspan="1" colspan="1">0.34</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Effective</td><td align="center" valign="top" rowspan="1" colspan="1" /><td align="center" valign="top" rowspan="1" colspan="1">0.48</td></tr></tbody></table> </ephtml> </p> <p>Table 6 suggests that there is not as much consistency as might be expected between retention probabilities at different experience levels. Schools that are relatively better at retaining 1st-year teachers are not, on average, also better at retaining 5th- and 15th-year teachers, though schools that are relatively better at retaining 5th-year teachers do tend to be better at retaining 15th-year teachers. Since we know that new teacher attrition from teaching is relatively high, often due to individuals' discovering they are not cut out for teaching, new teacher attrition may be less related to individual school characteristics, especially after controlling for characteristics of the student population. In addition, this may be partly due to the tendency of new teachers to change schools while looking for a good fit, while 5th-year teachers likely have found an acceptable fit and then continue at the school.</p> <p>As mentioned prior, schools likely want to retain effective and highly effective teachers and not less effective ones. Table 7 suggests that schools that are relatively better at retaining effective teachers tend to be less likely to retain those rated less than effective. But unexpectedly, schools that are relatively better at retaining highly effective teachers are also relatively better at retaining less than effective teachers. This finding deserves more examination, since it may be due to schools that retain highly effective teachers trying harder (and hopefully succeeding) in remediating, rather than removing, ineffective teachers, instead of or as well as such school leaders being more reluctant or unable to remove them. More reassuringly, schools that are relatively better at retaining effective teachers are also better at retaining highly effective teachers. But again, the correlations suggest that school effects on retention are not uniform across levels of teachers' rated effectiveness.</p> <p>These results led us to conclude that we had a potentially viable modeling approach upon which to build the user-facing analytic tool.</p> <p></p> <ulist> <item> There were reliable differences in schools' persistent rates of teacher retention.</item> </ulist> <p>Despite considerable year-to-year variation in school retention rates, the standard deviation of the school stable effects is larger than those of the yearly (transitory effects. The standard deviations for the school effects show that the typical school retention rate deviation from the district average is between 7.7 and 6.9 percentage points (depending on the model and sample). The average reliability of the school effects (as shown in the second column of Tables 2, 3, and 4) was.65 for overall retention and.61 for math and ELA teacher retention.</p> <p></p> <ulist> <item> These differences persisted after controlling for student demographics.</item> </ulist> <p>While the standard deviations of the school effects decline after adding student population characteristics to the model, there is still substantial variation across schools. The standard deviations for the school effects indicate that the typical school retention rate deviation from the district average is between 3.7 and 4.8 percentage points after controlling for student characteristics. Average reliability of the school effects was.45 for overall retention,.42 for Grades 3–8 math teachers, and.39 for Grades 3–10 ELA teachers.</p> <p></p> <ulist> <item> District-wide, teachers at different levels of effectiveness and experience have different predicted levels of retention.</item> </ulist> <p>As expected, 1st-year teachers have the lowest district-wide retention and 15th-year teachers the highest, with 5th-year teachers being retained about as well as 15th-year teachers. The results of the models shown in Tables 2 and 3 suggest that the district's primary overall retention issue is with 1st-year teachers. This may be due in part to the district's pay schedule being somewhat behind those of its neighbors in the first few years of teacher experience.</p> <p></p> <ulist> <item> There are substantial differences among schools in their ability to retain teachers at different levels of experience and effectiveness.</item> </ulist> <p>With respect to teacher experience, the school effect standard deviations in Table 3 show that the typical school varies from the district average for 1st-year teachers by 10.3 percentage points, for 5th-year teachers by 4.4 percentage points, and for 15th-year teachers by 2.2 percentage points. The school effect standard deviations for the smaller ELA and math teacher samples in Tables 4 and 5 are similar in size. In all the models, the variation goes down as experience increases, suggesting that schools differ most in retaining 1st-year teachers. This implies that there may be more benefit in examining how schools retain 1st-year teachers and working to bring those with much lower rates up to at least the average, especially since this is the least likely to be retained group district-wide.</p> <p>Similarly, variability was found for rated effectiveness. The typical school varies from the district average retention rate for less than effective teachers by 8.1 percentage points, for effective teachers by 4.7 percentage points, and for highly effective teachers by 3.7 percentage points. (See the rightmost column in Table 3.) Since the most variability is for less than effective teachers, it may be appropriate to look at how schools make decisions to retain these teachers and to explore where these teachers go when they do leave from the schools that are least likely to retain them. If the schools that do not retain less effective teachers send them to other schools, and these teachers do not improve, then leaders in the original school may need support in removing less effective teachers from the district rather than passing them on to other schools. If the teachers leave the district, it could be useful to find out how some schools are encouraging these teachers to leave and whether others have special circumstances that encourage them to retain those who are less than effective.</p> <hd id="AN0184324479-11">Visualizations and Reporting</hd> <p>Having accomplished our first goal of developing reliable estimates of differences in school retention rates, we also wanted to ensure the results were useful to district staff responsible for overseeing and supporting the schools. Our next step was to develop an interactive web-based visualization tool to present the information. This section shows some of the visualizations that appeared in the tool. They were deigned to help users understand the nature of the variation in school retention and to encourage hypotheses about potential causes of high and low retention rates.</p> <p>An important influence on whether and how potential users of analytic results become actual users is the clarity of presentation of the information and the ease of navigation through the presentation tool. In the course of development, we mocked up reports and visualizations, sharing them with potential users for their feedback. This resulted in adding several features to the tool, including more extensive context information about the school, such as whether the principal changed during the period, the actual number of teachers who left in each year, and distributions of teacher experience.</p> <p>The backbone of the tool is a series of scatterplots that show the relationship among the predicted stable retention rates and other factors at the school and teacher levels.</p> <p>Figure 1 shows schools' predicted stable retention rates plotted against the schools' percentage of students eligible for free/reduced-price lunch.</p> <p>Graph: Figure 1: Source: Copyright 2017 by Education Analytics Inc.</p> <p>The tool also shows schools' overall retention plotted with three other school population characteristics, the percentage of students who are English learners, the percentage who are white, and the percentage receiving special education services. The intent of these views is to show the overall relationship between retention and these characteristics, as well as to show how much retention varies, even where schools have very similar student populations. Each plot shows that there is considerable variation in retention among schools, typically at least plus and minus 10 percentage points around the typical value for schools with more disadvantaged students. Thus, these visualizations make the point that school demographics are not destiny with respect to retention and that at most levels, there are schools that might be models and others that might need extra support. In the tool, users can highlight specific schools, specific areas (recall that the district is divided into six of these), and also which schools are most likely to have higher or lower retention rates.</p> <p>Because districts and schools will want to explore retention by experience and effectiveness, the tool shows school predictions plotted against school characteristics by three sets of teacher characteristics: total teaching experience, rated teacher effectiveness, and, for teachers in the covered grades and subjects, value-added in Grades 3–8 math and 3–10 ELA. Figures 2 through 5 show some of the characteristics plotted against schools' percentage of students eligible for free or reduced-price lunch.</p> <p>Graph: Figure 2: Source: Copyright 2017 by Education Analytics Inc.</p> <p>Graph: Figure 3: Source: Copyright 2017 by Education Analytics Inc.</p> <p>Graph: Figure 4: Source: Copyright 2017 by Education Analytics Inc.</p> <p>Graph: Figure 5: Source: Copyright 2017 by Education Analytics Inc.</p> <p>The tool also provides a school view, which includes context information about the school and a specific prediction from the analysis model on the number of vacancies expected for the next school year. Context information includes the actual number of teachers who left the school in the prior school year, their experience, and whether the principal was new to the school in the last year. Figures 4 and 5 are screenshots from the school view.</p> <p>Figure 4 shows histograms representing the distribution of experience of the school's teachers from the prior school year, both for all teachers and for those with value-added on state tests in math or ELA. This part of the school view also shows whether the principal changed during the years used to make the retention rate estimate. This information is intended to provide some context to help users interpret the school effects shown on the scatterplots and the specific predictions shown lower down in the school view.</p> <p>Figure 5 shows the estimated number of vacancies for the next school year. This prediction is made by multiplying the school's persistent predicted retention rate by the number of teachers in the school, then subtracting this from the number of teachers. It also shows the retention rate predictions for that school (based on the stable school effect estimated by the analytic models) for teachers at different levels of experience. The view can be changed (not shown) to show predictions for teachers of different levels of rated effectiveness and value-added.</p> <p>The various views offered in the tool provide potential users with a large amount of information about school retention, including specific estimates of stable school effects on retention, which can also be used to predict future retention, as well as the influence of student demographics and aspects of school context (like principal change) that can help interpret the school effects and predictions.</p> <hd id="AN0184324479-12">User Reactions to the Retention Analytic Tool</hd> <p></p> <hd id="AN0184324479-13">Intended Applications</hd> <p>During the development process, a primary application we envisioned was for area leaders (area superintendents) and central office support staff (area leadership specialists and central office HR staff) to identify those schools likely to need extra support due to historically low retention and a high number of expected vacancies. Given the effort this district has to go through to fill its positions, these predictions were expected to help prioritize recruiting efforts for schools that are likely to have many vacancies. We also envisioned that the tool could be used to identify schools that are "beating the odds" given their student populations. Area superintendents and leadership specialist could use their knowledge of these schools to begin to determine whether there were any special characteristics (e.g., a long-serving and popular principal) or policies (e.g., particularly intensive new teacher induction) that might explain the higher retention level, compared to schools with similar student populations. We developed a school visit protocol to help these staff examine school policies and practices and find any that might be shared with similar schools that have much lower-than-expected retention rates. After determining whether high-retention schools are using practices that seem related to higher retention, the results could be used to pair high with low-retention schools to help the latter adapt the practices used by the former.</p> <p>We also hypothesized that users would find it useful to examine whether retention differs by experience or performance levels. As suggested in the introduction, low retention of new teachers but average or better retention of the more experienced might indicate problems with school-level support for new teachers or hiring of new teachers who are not committed to or ill adapted to the school's mission or culture. Low retention of experienced teachers only could suggest veterans are using their experience to transfer out of less desirable schools. Low retention across the experience spectrum, compared to schools with similar student demographics, could be due to problems with the school climate or school leadership. Similarly, differences in retention by teacher performance are likely to be of interest. Low retention of high performers compared to similar schools coupled with high retention of low performers compared to similar schools could suggest that the school has a hard time keeping good teachers and may not want to encourage lower performers to leave because of the difficulty of replacing them. This pattern may be more likely for struggling schools. In contrast, schools that retain average and high performers at greater-than-expected rates and lose lower performers more often than expected may be better at "counseling out" the latter. If the highest performers are the only group retained at lower-than-expected rates, this could be due to lack of recognition for their efforts or their being poached by other schools who can offer more desirable working conditions. There are many possible stories that could explain observed patterns, and part of the value we expected for the tool was to encourage administrators to develop and test these sorts of hypotheses.</p> <p>Another purpose we envisioned was to provide district administrators with more objective information about schools to compliment the clinical judgments or gut feelings they develop about the schools they oversee or support. "Actuarial" predictions of which schools are likely to have lower retention might be a good way to confirm clinical judgements of which schools need help. The predictions in the tool may also be more accurate, especially since yearly retention rates are fairly noisy. The predictions in the tool can provide another perspective, and, when combined with administrators' knowledge about individual schools, would provide more efficient prioritization of support.</p> <hd id="AN0184324479-14">Initial Use of the Tool by Intended Users</hd> <p>We held an orientation session with each of two sets of potential district users: area assistant superintendents (responsible to the superintendent for overseeing a group of the district's schools) and leadership specialists (central office staff assigned to support each area in improving school leadership and human capital management). While we were unable to systematically collect information about user reactions to the tool, review of our notes from the two orientation sessions did provide some indication of whether the tool was likely to be of use and how it might be improved. After an initial familiarization, most found the tool easy to use and quickly became engaged with the tool, exploring the predicted retention of the schools for which each was responsible. Many indicated that the tool quantified what they felt they already knew about the school (though there were some surprises). Users appreciated this confirmation but also mentioned that the metrics would provide a good basis for conversations with principals about how their schools could improve. The ability to show principals that schools with similar student demographics could have higher retention rates was seen as a good conversation starter for leaders of schools with lower-than-expected retention rates. Users also appreciated the tool's inclusion of school context information such as whether the principal had recently changed and how many teachers left in each of the years used to estimate the metrics. There was also substantial interest in improvements, including adding information about school climate, student attendance, and discipline, and providing breakouts by school grade span (i.e., high, middle, or elementary school).</p> <p>Despite this promising start, this tool did not accomplish the purposes intended. A combination of district leadership change, COVID-19 disruptions, and a potential mismatch between administrators' interests and what the tool provided led to the abandonment of the tool after the 2019–20 school year. In March of 2020, the district leadership change brought a new superintendent, a new leadership cabinet, and nearly complete turnover of one of the key user groups we had identified, regional administrators. The new leaders had different priorities and did not have the time or initial interest to get familiar with the tool. COVID-19 also changed priorities, with learning about persistent retention overshadowed by developing health and safety policies, implementing distance learning, and complying with state reopening policy. Few if any of the new regional leaders used the tool or asked for information about how to use it. District sponsors were therefore reluctant to spend time and resources on training or updating the tool.</p> <p>The mismatch between what the tool provided and how district administrators thought about retention was also a likely factor in the lack of use. We noted in the initial orientation that some users, area superintendents especially, may not have fully appreciated the distinction between the stable/persistent school effect on which the retention metric was based and yearly observed retention rates. Our initial concern was that identification of schools as doing better than average or needing more help should be reliable, so we estimated models that used multiple years of data, as well as shrinkage, to try to separate signal from noise. However, when administrators' priority was filling immediate vacancies, they may have been more interested in the influence of shorter-term factors. While the tool predicted the number of likely vacancies based on several years of data, users were concerned about the effects of more recent conditions such as school climate and principal turnover. The district experiences substantial movement of principals among schools, and anecdotal evidence suggested that teachers move with principals to new schools. Some administrators were also more interested in seeing trends in retention rather than static indicators of stable retention. While we planned to include principal turnover in the predictive model underlying the tool, as well as additional metrics, we were not able to do so before district leadership changes and COVID disruptions reduced interest in it. For the 2021–22 school year, the district did not have the funding to support further development or updating the underlying statistical models with an additional year of data. This was effectively the end of the tool's implementation, though we continued to develop the tool by exploring adding school climate and principal turnover to the models.</p> <hd id="AN0184324479-15">Potential Tool Improvements</hd> <p>As mentioned prior, users suggested improvements, including breakouts by school grade span and adding information about school climate, student attendance, and discipline. Reviewers of this article also suggested adding school size and principal turnover as controls. The [<reflink idref="bib17" id="ref16">17</reflink>] meta-analysis found that grade span was important, as were school resources including work environment, administrative support, professional development, induction/mentoring, and provision of teaching materials. That these school resources appear more influential than school demographics is not inconsistent with the large variation in persistent retention rates at similar levels of school demographics shown in Figures 1–3 and in the other scatterplot views the tool provided.</p> <p>Our agenda for the second version included addressing grade span, teacher perceptions of climate, and principal turnover. We did conduct exploratory analyses on the effects of climate and principal turnover and found that, as might be expected, principal turnover was negatively related to teacher retention ([<reflink idref="bib24" id="ref17">24</reflink>]) and positive teacher perceptions of school climate were positively related. School size was a lower priority because, as explained in our targeting accuracy section prior, the instability in retention due in part to small size was taken into account when the probability of a school being near the top or bottom of the ordering of school-level retention was calculated.</p> <p>One issue we would have needed to resolve was how to portray any additional factors that could be related to retention. School level is relatively simple: the tool could be modified so the user could choose which level to show in the existing scatterplot views (i.e., for retention by percentage of students eligible for free/reduced-price lunch, English learners, and ethnicity). This approach could work well for similar categorical factors but less well for a factor such as school culture. These continuous factors could be shown in additional bivariate scatter plots (e.g., retention by some index of school culture), which would allow users to explore more relationships at the cost of making the tool more complex. An alternative would be to allow users to specify a multivariate retention model with a set of "control" variables of their choice, then show a bivariate plot of the regression residuals versus an additional factor of their choice. This would allow the user to explore the effect of a factor of interest control for the effects of multiple other potential factors. While this may seem like an obvious improvement to researchers, district administrators unfamiliar with this methodology might have difficulty interpreting the results. Pursuing this approach would require more training of users as well as substantial additional programing. It would also require some agreement about which additional factors should be available and whether data on additional factors such as those identified in the [<reflink idref="bib17" id="ref18">17</reflink>] review (e.g., school variation in professional development availability or induction/mentoring) could be collected.</p> <p>We also note that the data we had available for the initial tool build did not allow distinguishing between voluntary and involuntary turnover. For version two, we aimed at assessing how differentiating between them would have changed the results. This was not a priority for version one because the district's teaching force was relatively young and firings relatively rare. Since we did represent teacher effectiveness in the tool, it does to some extent recognize that some nonretention may be desirable and allowed users to focus on retention of more effective teachers.</p> <hd id="AN0184324479-16">Lessons from Our Experience</hd> <p>Our experience with this tool provides several potential lessons for developers of human capital analytics for use by US local education agencies. First, once developers and practitioners begin to interact around a tool like the one described here, many ideas for additional analyses and modifications are likely to emerge. As mentioned previously, practitioners suggested adding more school context information. The designers recognized that information about additional school characteristics such as principal retention, school leadership quality, and school climate, which research has shown to influence school retention (e.g., [<reflink idref="bib13" id="ref19">13</reflink>]; [<reflink idref="bib8" id="ref20">8</reflink>]; [<reflink idref="bib2" id="ref21">2</reflink>]), could improve retention rate predictions. Second, shifts in management priorities and changes in the external environment can threaten implementation. The relevance of analytic systems is likely to depend on the problems users face at a given time. Since these problems change with external events (such as COVID) and district leadership, it could be necessary to design systems that are adaptable to easily address multiple human capital issues rather than building single-issue applications. For example, school districts or other local education authorities could build comprehensive data warehouses or data marts and then build relatively simple and low-cost "apps" to answer specific user questions. But this would require a substantial up-front investment in data warehouse/mart development. Third, it is important to recognize the potential mismatch between the designers' and users' perspectives early in the development process. The developers were strongly concerned with the reliability of identifying schools with retention strengths and weaknesses and expected that the tool would prompt the target users (district-level middle managers) to examine schools with higher- and lower-than-predicted retention to find positive models and identify schools needing support. This intent was based on the assumption that these users would have the time and interest in a longer-term process of using data to understand root causes of retention or nonretention. In contrast, the target users seem to have been more concerned with shorter-term issues like filling this year's vacancies. Since the initiation of the project was not based on an in-depth needs assessment or extensive interaction with the intended users, this difference only appeared near the end of the development process and limited our ability to build user support for the tool. Had users been part of the initiation process, their preferences might have received more emphasis and they might have been a stronger voice for continuing the project.</p> <p>While involving users earlier in the development process might have surfaced the difference in perspective earlier, there may be a trade-off between developing tools that address immediate user priorities and tools that afford new perspectives and opportunities. While meeting potential users where they are is likely to generate more buy-in, focusing on supporting existing HR practices could close the door to developing tools that help users dig deeper into systemic weaknesses and underlying causes. Introducing human capital analytics that attempt to tease out causal relationships may require figuring out how to design analytic tools that begin with simpler layers that help users with immediate problems while providing "deeper" layers providing the capacity for and guidance in using more sophisticated analytics.</p> <hd id="AN0184324479-17">Potential Lessons from Design-Based Research</hd> <p>A suggestion made by reviewers of this article was to think about this work as an example of design-based research. While the tool described was not intentionally developed using this paradigm, it could be useful to compare its development to what might have occurred had design-based ideas guided the process.</p> <p>While there appears to be no single canonical exposition of what design-based research should look like, the literature we consulted ([<reflink idref="bib23" id="ref22">23</reflink>]; [<reflink idref="bib18" id="ref23">18</reflink>]; [<reflink idref="bib1" id="ref24">1</reflink>]; [<reflink idref="bib4" id="ref25">4</reflink>]; [<reflink idref="bib6" id="ref26">6</reflink>]) suggests that key characteristics relevant to this work include a situation in a real educational context; a theory of action guiding initial development; multiple iterations of prototyping, testing, and redesign; systematic data collection using multiple methods; a collaborative partnership between researchers and practitioners; and the development of theories or principles that can be applied by practitioners as well as general theory.</p> <p>In the case discussed in this article, some of the elements of design-based research were present: the work was embedded in a real educational context, the district's need to keep schools staffed with qualified teachers. There was also an initial theory of action that underlay the tool's development: that district staff would use the tool to identify schools with better-than-expected retention for emulation and worse-than-expected retention for additional support. The intention was to engage in a process of prototyping, testing, and redesign. The tool as described here was considered version 1.0. Missing or underemphasized were the collaborative partnership with users, systematic data collection, and the development of theories or principles, though the intent was to use the tool to help users develop and test local theories about school characteristics that would promote retention using the school visit protocol we developed. Had we thought of this as design research, we would have had to work earlier and more directly with users to define the problem, possibly leading to more user buy-in. We might have started from a broader question such as how data analytics can be used to help improve retention. This in turn might have surfaced user preferences earlier in the development process.</p> <p>Potential advantages to thinking about human capital analytics as design-based research include ensuring stakeholder participation to surface users' needs and constraints, recognition of the need for iteration, encouraging planned and systematic data collection around both how the users use analytics and whether use facilitates action toward improving retention. The latter would lead to further tool development as a better understanding of what influences retention emerges.</p> <p>However, developing human capital analytics as design-based research presents challenges. First, the user partners must see the analytics as tools for research and be willing to engage in an ongoing research process, rather than concentrating on using the tools to do their day-to-day jobs. This will require the commitment of likely scarce user time and attention. Second, grafting a research process onto the existing practices and expectations that guide information system development will require setting some new expectations for information technology staff. The more typical system build involves developing relatively static, backward-facing products to agreed-on specifications within a short time frame, rather than continual iteration as new insights emerge. Third, it may be difficult to embed systematic data collection about how users use the tool and what they learn by doing so into the process. It is difficult to get user time even for conventional system development activities such as interface reviews.</p> <ref id="AN0184324479-18"> <title> Notes </title> <blist> <bibl id="bib1" idref="ref6" type="bt">1</bibl> <bibtext> A more recent review by [17] concluded that race/ethnic proportions and economic disadvantage were not as influential as found by the earlier meta-analysis. This later review also found evidence for the influence of several other school-level characteristics, most of which (besides grade level) were school resources or organizational characteristics (malleable factors) that could be targets of school or district policy. The implications of these results for the development of retention analytics are discussed in the "Potential Tool Improvements" section near the end of this article.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref10" type="bt">2</bibl> <bibtext> Driven by the tendency of these teachers to leave teaching altogether rather than changing schools.</bibtext> </blist> </ref> <ref id="AN0184324479-19"> <title> References </title> <blist> <bibtext> Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research?.. Educational Researcher, 41(1), 16–25. https://doi.org/10.3102/0013189x11428813</bibtext> </blist> <blist> <bibtext> Beteille, T., Kalogrides, D., & Loeb, S. (2011).. Stepping stones: Principal career paths and school outcomes (NBER Working Paper No. 17243). National Bureau of Economic Research.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref5" type="bt">3</bibl> <bibtext> Borman, G. D., & Dowling, N. M. (2008). Teacher attrition and retention: A meta-analytic and narrative review of the research.. Review of Educational Research, 78(3), 367–409. https://doi.org/10.3102/0034654308321455</bibtext> </blist> <blist> <bibl id="bib4" idref="ref25" type="bt">4</bibl> <bibtext> Collins, A., Joseph, D., & Bielaczyc, K. (2004) Design research: Theoretical and methodological issues.. Journal of the Learning Sciences, 13(1), 15–42. https://doi.org/10.1207/s15327809jls1301_2</bibtext> </blist> <blist> <bibl id="bib5" idref="ref13" type="bt">5</bibl> <bibtext> Danielson, C. (2007).. Enhancing professional practice: A framework for teaching (2nd edition). Association for Supervision and Curriculum Development.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref26" type="bt">6</bibl> <bibtext> Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry.. Educational Researcher, 32(1), 5–8. https://doi.org/10.3102/0013189x032001005</bibtext> </blist> <blist> <bibl id="bib7" idref="ref9" type="bt">7</bibl> <bibtext> Feng, L., & Sass, T. R. (2017). Teacher quality and teacher mobility.. Education Finance and Policy, 12(3), 396–418. https://doi.org/10.1162/edfp_a_00214</bibtext> </blist> <blist> <bibl id="bib8" idref="ref20" type="bt">8</bibl> <bibtext> Fuller, B., Waite, A., & Irribarra, D. T. (2016). Explaining teacher turnover: School cohesion and intrinsic motivation in Los Angeles.. American Journal of Education, 122(4), 537–567. https://doi.org/10.1086/687272</bibtext> </blist> <blist> <bibl id="bib9" type="bt">9</bibl> <bibtext> Goldhaber, D., & Hansen, M. (2012).. Is it just a bad class? Assessing the long-term stability of estimated teacher performance (Working Paper 73). National Center for Analysis of Longitudinal Data in Education Research.</bibtext> </blist> <blist> <bibtext> Hanushek, E. A., Rivkin, S. G., & Schiman, J. C. (2016).. Dynamic effects of teacher turnover on the quality of instruction (Working Paper 170). National Center for Analysis of Longitudinal Data in Education Research.</bibtext> </blist> <blist> <bibtext> Holme, J. J., Jabbar, H., Germain, E., & Dinning, J. (2018). Rethinking teacher turnover: Longitudinal measures of instability in schools.. Educational Researcher, 47(1), 62–75. https://doi.org/10.3102/0013189x17735813</bibtext> </blist> <blist> <bibtext> Ingersoll, R. M. (1997). Teacher turnover and teacher quality: The recurring myth of teacher shortages.. Teachers College Record, 99(1), 41–44. https://doi.org/10.1177/016146819709900113</bibtext> </blist> <blist> <bibtext> Kraft, M. A., Marinell, W. H., & Shen-Wei Yee, D. (2016). School organizational contexts, teacher turnover, and student achievement: Evidence from panel data.. American Educational Research Journal, 53(5), 1411–1449. https://doi.org/10.3102/0002831216667478</bibtext> </blist> <blist> <bibtext> Ladd, H. F., & Sorensen, L. C. (2017). Returns to teacher experience: Student achievement and motivation in middle school.. Education Finance and Policy, 12(2), 241–279. https://doi.org/10.1162/edfp_a_00194</bibtext> </blist> <blist> <bibtext> McCaffrey, D. F., Sass, T. R., Lockwood, J. R., & Mihaly, K. (2009).. The inter-temporal variability of teacher effect estimates (Working Paper 2009–03). National Center on Performance Incentives.</bibtext> </blist> <blist> <bibtext> Milanowski, A.T., & Kimball, S.M. (2010). The principal as human capital manager; Lessons from the private sector. In R., Curtis and J., Wurtzel (Eds.), Teaching talent: A visionary framework for human capital in Education. Cambridge, MA: Harvard Education Press.</bibtext> </blist> <blist> <bibtext> Ngeyen, T. D., Pham, L., Springer, M., & Crouch, M. (2019).. The factors of teacher attrition and retention: An updated and expanded meta-analysis of the literature (EdWorkingPaper 19–149). Annenberg Institute at Brown University. https://edworkingpapers.com/ai19-149</bibtext> </blist> <blist> <bibtext> Plomp, T. (2013). Design research: An introduction. In T., Plomp & N., Nieveen (Eds.). Educational design research. Part A: An introduction (pp. 10–51). SLO. <ulink href="http://downloads.slo.nl/Documenten/educational-design-research-part-a.pdf">http://downloads.slo.nl/Documenten/educational-design-research-part-a.pdf</ulink></bibtext> </blist> <blist> <bibtext> Polikoff, M. S. (2015). The stability of observational and student survey measures of teaching effectiveness.. American Journal of Education, 121, 183–212. https://doi.org/10.1086/679390</bibtext> </blist> <blist> <bibtext> Rice, J. K. (2013). Learning from experience? Evidence on the impact and distribution of teacher experience and the implications for teacher policy.. Education Finance and Policy, 8(3), 332–348. https://doi.org/10.1162/edfp_a_00099</bibtext> </blist> <blist> <bibtext> Ronfeldt, M., Loeb, S., & Wyckoff, J. (2013). How teacher turnover harms student achievement.. American Educational Research Journal, 50(1), 4–36. https://doi.org/10.3102/0002831212463813</bibtext> </blist> <blist> <bibtext> Sorensen, L. C., & Ladd, H. F. (2020). The hidden costs of teacher turnover.. AERA Open, 6(1). https://doi.org/10.1177/2332858420905812</bibtext> </blist> <blist> <bibtext> Stahl, N. A., King, J. R., & Lampi, J. P. (2019). Expanding approaches for research: Design research.. Journal of Developmental Education, 42(3), 29–30.</bibtext> </blist> <blist> <bibtext> Veiga, R, Meyer, R. H., & Milanowski, A. (2020, March19–21). Principal turnover and the retention of quality teachers [Conference presentation]. 2020 Annual Conference of the Association for Education Finance and Policy, Fort Worth, TX, United States.</bibtext> </blist> </ref> <aug> <p>By Robert Meyer; Anthony Milanowski; Ryan Veiga and Jessica Doherty</p> <p>Reported by Author; Author; Author; Author</p> <p></p> <p>Robert Meyer is a Senior Fellow in the Education and Child Development Department at NORC at the University of Chicago. Prior to joining NORC, Meyer was a Research Professor (now Emeritus) and Director of the Value-Added Research Center at the University of Wisconsin-Madison and Founder and former President and CEO of Education Analytics.</p> <p>Anthony Milanowski is a researcher with Education Analytics. Inc. a non-profit research and analytics organization headquartered in Madison, WI. He is also affiliated with the Wisconsin Center for Education Research at the University of Wisconsin, Madison. He has been conducting research on human resource management topics in education since 1997 and has provided technical assistance on teacher compensation, evaluation, and HR program alignment.</p> <p>Ryan Veiga earned his PhD in Economics at the University of Wisconsin-Madison, where he worked closely with Professor Robert H. Meyer on charter school effectiveness, teacher quality, and human capital transitions in public schools. Upon receiving his degree, he continued his collaboration with Meyer as a postdoctoral researcher at Education Analytics. He is currently employed as a Data Science Specialist at Tufts University.</p> <p>Jessica Doherty currently serves as the Director of Performance Evaluation for Hillsborough County Public Schools in Tampa, Florida. During this research, she was the director of the district's Federal Teacher Incentive Fund grant project. Some of the results herein were presented at the Association for Education Finance and Policy Annual Meeting, March 21–23, 2019. Thise research was supported in part by the US Department of Education, Office of Elementary and Secondary Education to the Hillsborough Public Schools (Grant 537A120095). Opinions expressed are those of the authors and do not necessarily reflect the view of the Office of Elementary and Secondary Education, the US Department of Education, or the Hillsborough Public Schools.</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib21" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib12" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib22" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib11" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib14" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib19" firstref="ref14"></nolink> <nolink nlid="nl10" bibid="bib15" firstref="ref15"></nolink> <nolink nlid="nl11" bibid="bib17" firstref="ref16"></nolink> <nolink nlid="nl12" bibid="bib24" firstref="ref17"></nolink> <nolink nlid="nl13" bibid="bib13" firstref="ref19"></nolink> <nolink nlid="nl14" bibid="bib23" firstref="ref22"></nolink> <nolink nlid="nl15" bibid="bib18" firstref="ref23"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1467146
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Creating Actionable Human Capital Analytics for Studying School-Level Teacher Retention
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Robert+Meyer%22">Robert Meyer</searchLink><br /><searchLink fieldCode="AR" term="%22Anthony+Milanowski%22">Anthony Milanowski</searchLink><br /><searchLink fieldCode="AR" term="%22Ryan+Veiga%22">Ryan Veiga</searchLink><br /><searchLink fieldCode="AR" term="%22Jessica+Doherty%22">Jessica Doherty</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Education+Human+Resources%22"><i>Journal of Education Human Resources</i></searchLink>. 2025 43(2):390-421.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: University of Toronto Press. 5201 Dufferin Street, Toronto, ON M3H 5T8, Canada. Tel: 416-667-7810; Fax: 800-221-9985; Fax: 416-667-7881; e-mail: journals@utpress.utoronco.ca; Web site: https://www.utpjournals.press/loi/jehr
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 32
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Office of Elementary and Secondary Education (OESE) (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 537A120095
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Teacher+Persistence%22">Teacher Persistence</searchLink><br /><searchLink fieldCode="DE" term="%22Faculty+Mobility%22">Faculty Mobility</searchLink><br /><searchLink fieldCode="DE" term="%22School+Districts%22">School Districts</searchLink><br /><searchLink fieldCode="DE" term="%22Human+Capital%22">Human Capital</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Analysis%22">Data Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Environment%22">Educational Environment</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Conditions%22">Teaching Conditions</searchLink><br /><searchLink fieldCode="DE" term="%22Administrator+Attitudes%22">Administrator Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Characteristics%22">Institutional Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Differences%22">Differences</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Policy%22">Educational Policy</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Practices%22">Educational Practices</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Florida%22">Florida</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.3138/jehr-2023-0063
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2562-783X
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: One potentially fruitful application for human capital analytics is to support policies and practices that might reduce undesirable teacher turnover. Teacher turnover can be harmful to student achievement and faculty cohesiveness and can exacerbate teacher shortages. This article describes an attempt to build a human capital analytics tool to help a school district better identify schools with problems retaining teachers as well as schools that are doing an especially good job of retaining them. It describes the theory of action for the school-level retention analytics tool, the models used to estimate persistent school effects and predict which schools would continue to have retention problems, and the features of a web-based tool developed to help district staff gain insights about the degree of variation among schools and likely contributing factors. We found that there were reliable differences in schools' persistent rates of teacher retention, the differences persisted after controlling for student demographics, teachers at different levels of effectiveness and experience have different predicted levels of retention, and there were substantial differences among schools in their ability to retain teachers at different levels of experience and effectiveness. Finally, we describe the initial reactions of the users for whom we designed the tool and what we learned about developing a retention analytic tool for use by school district administrators.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1467146
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1467146
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.3138/jehr-2023-0063
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 32
        StartPage: 390
    Subjects:
      – SubjectFull: Teacher Persistence
        Type: general
      – SubjectFull: Faculty Mobility
        Type: general
      – SubjectFull: School Districts
        Type: general
      – SubjectFull: Human Capital
        Type: general
      – SubjectFull: Data Analysis
        Type: general
      – SubjectFull: Educational Environment
        Type: general
      – SubjectFull: Teaching Conditions
        Type: general
      – SubjectFull: Administrator Attitudes
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Institutional Characteristics
        Type: general
      – SubjectFull: Differences
        Type: general
      – SubjectFull: Educational Policy
        Type: general
      – SubjectFull: Educational Practices
        Type: general
      – SubjectFull: Florida
        Type: general
    Titles:
      – TitleFull: Creating Actionable Human Capital Analytics for Studying School-Level Teacher Retention
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Robert Meyer
      – PersonEntity:
          Name:
            NameFull: Anthony Milanowski
      – PersonEntity:
          Name:
            NameFull: Ryan Veiga
      – PersonEntity:
          Name:
            NameFull: Jessica Doherty
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 2562-783X
          Numbering:
            – Type: volume
              Value: 43
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Journal of Education Human Resources
              Type: main
ResultId 1