View in EDS HTML Full Text PDF Full Text

Testing for Sequential Bias in School Inspections

Saved in:
Bibliographic Details
Title: Testing for Sequential Bias in School Inspections
Language: English
Authors: Christian Bokhove (ORCID 0000-0002-4860-8723), John Jerrim (ORCID 0000-0001-5705-7954), Maria Palma Carvajal (ORCID 0000-0001-8306-7515), Sam Sims (ORCID 0000-0002-5585-8202)
Source: Oxford Review of Education. 2025 51(5):785-809.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 25
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Education
Descriptors: Foreign Countries, Institutional Evaluation, Elementary Schools, Bias, Sequential Approach, Behavior Patterns
Geographic Terms: United Kingdom (England)
DOI: 10.1080/03054985.2024.2410270
ISSN: 0305-4985
1465-3915
Abstract: Inspectors are tasked with judging the quality of provision based on visits to schools. They conduct these inspections sequentially, completing one before moving on to the next. However, empirical research in a range of settings outside education suggests that prior judgements in a sequence can influence subsequent judgements, despite being logically irrelevant. We investigate whether school inspectors in England display such sequential bias by testing whether they judge similar schools differently, depending on the judgements they reached in prior inspections. We find only limited evidence of sequential bias in primary school inspections. In particular, an inspector reaching an 'Inadequate' judgement in their previous inspection is associated with a 42 per cent reduction in the odds of reaching another 'Inadequate' judgement in their next inspection. Only around 5 per cent of inspection judgements result in an 'Inadequate' and we do not find consistent evidence of sequential bias at other grades, meaning this bias only affects a small minority of judgements. We also do not find the same results for secondary schools, albeit in a much smaller sample.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1500700
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwExjge0GOv1nJnycVIXrGONAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDBtNRhtjofEkVNjfQIBEICBm3Rt7qhewUhN5lYgTPdDxRspT8np6rlBFQpkd-rB31qpjfcqLMGUIfTkDhr1GVB-FSE78d40bPFGDBadx7ZLp5Lm6857J2Q6BTXUXs_fEWAf0MPKgsjHDni1d7F4OnkvT4KyQ4oe2R9SMvLuvxIhjzTqtUyO1rvzrvF-fZrA0rOzr__Ex1WpyEvxETdQU6EBoUOAQ8pkz3haPOWw
Text:
  Availability: 1
  Value: <anid>AN0188054557;oxr01oct.25;2025Sep22.06:14;v2.2.500</anid> <title id="AN0188054557-1">Testing for sequential bias in school inspections </title> <p>Inspectors are tasked with judging the quality of provision based on visits to schools. They conduct these inspections sequentially, completing one before moving on to the next. However, empirical research in a range of settings outside education suggests that prior judgements in a sequence can influence subsequent judgements, despite being logically irrelevant. We investigate whether school inspectors in England display such sequential bias by testing whether they judge similar schools differently, depending on the judgements they reached in prior inspections. We find only limited evidence of sequential bias in primary school inspections. In particular, an inspector reaching an 'Inadequate' judgement in their previous inspection is associated with a 42 per cent reduction in the odds of reaching another 'Inadequate' judgement in their next inspection. Only around 5 per cent of inspection judgements result in an 'Inadequate' and we do not find consistent evidence of sequential bias at other grades, meaning this bias only affects a small minority of judgements. We also do not find the same results for secondary schools, albeit in a much smaller sample.</p> <p>Keywords: School inspections; validity; sequential bias; ofsted</p> <hd id="AN0188054557-2">Introduction</hd> <p>Many countries send inspectors to visit schools to collect evidence, form judgements about the quality of their provision, and then report on their findings (OECD, [<reflink idref="bib21" id="ref1">21</reflink>]). The judgements formed by inspectors often have important consequences for schools (Altrichter & Kemethofer, [<reflink idref="bib1" id="ref2">1</reflink>]; Ehren et al., [<reflink idref="bib10" id="ref3">10</reflink>]; Jones et al., [<reflink idref="bib18" id="ref4">18</reflink>]). Indeed, in some jurisdictions, failing a school inspection can lead to the replacement of the headteacher (Eyles & Machin, [<reflink idref="bib12" id="ref5">12</reflink>]). Given the high-stakes nature of this process, it is crucial that school inspections provide a valid, consistent and reliable measure of school quality. By valid, we mean that inspection outcomes should covary with genuine differences in school quality but should not be influenced by irrelevant factors such as the specific inspector(s) that happen to be conducting the inspection (Borsboom et al., [<reflink idref="bib7" id="ref6">7</reflink>]; Haladyna & Downing, [<reflink idref="bib14" id="ref7">14</reflink>]).</p> <p>There are longstanding concerns about the validity of inspection judgements (Maw, [<reflink idref="bib20" id="ref8">20</reflink>]). However, hard evidence on this issue has been slow to accumulate. In England, the setting for this research, the national inspectorate, Ofsted [the Office for Standards in Education, Children's Services and Skills], decided to conduct research into the reliability of its own inspections (Ofsted, [<reflink idref="bib22" id="ref9">22</reflink>]). This led to the publication of research in which two teams of inspectors were sent to inspect the same school. In 22 of the 24 schools that took part, the two sets of inspectors reached the same conclusion (Ofsted, [<reflink idref="bib22" id="ref10">22</reflink>]). However, this study only focused upon a subset of schools previously judged to be 'Good', focused upon Ofsted's 'Short' (rather than full) inspections, and involved only a select subset of the inspection workforce. More recently, researchers have documented small correlations between inspection outcomes and logically irrelevant aspects of the inspection, such as the gender of the lead inspector (Bokhove et al., [<reflink idref="bib6" id="ref11">6</reflink>]). However, there is still a dearth of evidence in this area, particularly given the central role of inspection in school accountability (Richmond, [<reflink idref="bib27" id="ref12">27</reflink>]).</p> <p>We seek to fill part of this gap in the existing literature by testing whether inspection judgements are affected by one variable they clearly should not be affected by: the sequence in which inspectors visit schools. For example, if a lead inspector has judged the previous school positively ('Good' or 'Outstanding') does this make the present school look less good by comparison, making a negative ('Requires Improvement' or 'Inadequate') more likely? In short, perhaps a very positive inspection is a 'hard act to follow'. Conversely, if a lead inspector has judged the previous school very negatively, does this affect the way in which they judge the present school? If a prior negative inspection triggered the headteacher to lose their job, perhaps the inspector would be reluctant to give another negative judgement soon afterwards? Of course, the judgement given to the last school should not affect the judgement that an inspector gives to the next school. However, empirical research in a range of settings, including baseball, speed-dating, courtrooms, and corporate settings, has found such sequential bias to exist (Bindler & Hjalmarsson, [<reflink idref="bib5" id="ref13">5</reflink>]; Chen et al., [<reflink idref="bib8" id="ref14">8</reflink>]).</p> <p>This paper makes two contributions to the literature. First, we investigate whether school inspectors' judgements in prior school inspections are correlated with their judgement in current school inspections, conditional on characteristics of the current school. This is, to our knowledge, the first such paper to address this question in relation to school inspections. Our data allow us to control for many observable indicators of school quality, as well as inspector fixed effects. This helps us to differentiate between sequential bias and underlying correlation between the quality of schools that happen to be inspected one after the other. Our second contribution is to explore the pattern of findings to investigate the potential mechanisms underpinning any sequential bias.</p> <hd id="AN0188054557-3">Sequential bias</hd> <p></p> <hd id="AN0188054557-4">Theory and mechanisms</hd> <p>Sequential bias occurs when past judgements influence present judgements, despite the judgements being logically independent of one other (Goldbach et al., [<reflink idref="bib13" id="ref15">13</reflink>]). In the context of school inspections in England, sequential bias would occur if the judgements reached by a lead inspector in prior inspections influenced the judgement that the same lead inspector reached in a subsequent inspection. This sequential bias can go in two directions. Past inspection judgements can be positively autocorrelated with present inspection judgements. That is, present judgements become more like past judgements. For example, it may be that a lead inspector reaching negative judgements in prior inspections makes it more likely they will reach a negative judgement in the present inspection. Conversely, past inspection judgements can be negatively autocorrelated with present inspection judgements. That is, present judgements become more different to past judgements. For example, a lead inspector reaching negative judgements in prior inspections makes it more likely they will reach a positive judgement in the inspection that follows.</p> <p>Prior judgements can affect present judgements for cognitive (e.g. Stewart et al., [<reflink idref="bib29" id="ref16">29</reflink>]) or affective (e.g. Bindler & Hjalmarsson, [<reflink idref="bib5" id="ref17">5</reflink>]) reasons. Indeed, our review of the existing theoretical literature suggests six specific mechanisms by which past judgements might influence independent future judgements. Two of these mechanisms – the <emph>inference effect</emph> and the <emph>consistency effect</emph> – lead to positive autocorrelation. An <emph>inference effect</emph> occurs when prior judgements are assumed to be informative about what is likely to be the case in the current situation (e.g. Ibanez & Toffel, [<reflink idref="bib17" id="ref18">17</reflink>]). For example, a string of negative (e.g. 'Inadequate') school inspection judgements may lead an inspector to expect that future schools are unlikely to be much better, and vice versa. A <emph>consistency effect</emph> occurs when two similar cases are judged one after the other and, for fear of being seen as inconsistent, they are judged more similarly (Bindler & Hjalmarsson, [<reflink idref="bib5" id="ref19">5</reflink>]). For example, if an inspector visits two single-form primary schools in the same affluent suburb with similar levels of achievement in examinations, they may be more likely to give the second school the same inspection judgement as the first school <emph>relative to a situation where the two schools had not been inspected one after the other</emph>. In sum, both an <emph>inference effect</emph> and a <emph>consistency effect</emph> would lead present inspection judgements to become more similar to prior inspection judgements.</p> <p>The remaining four sequential bias mechanisms would lead to negative autocorrelation. First, the <emph>gambler's fallacy</emph> occurs when the person making a judgement concludes that a run of 'bad luck' must turn soon (Tversky & Kahneman, [<reflink idref="bib30" id="ref20">30</reflink>], [<reflink idref="bib31" id="ref21">31</reflink>]). For example, a lead inspector who has made a series of negative ('Inadequate' or 'Requires Improvement') judgements may believe they are 'overdue' a positive inspection, and vice versa. Second, <emph>norm referencing</emph> occurs when the person making a judgement has some implicit sense of what proportion of cases should be awarded each judgement, and over time they calibrate their judgements to achieve the desired distribution of judgements (see Chen et al., [<reflink idref="bib8" id="ref22">8</reflink>]). For example, a series of negative ('Inadequate' or 'Requires Improvement') judgements would prompt the lead inspector to lower their implicit threshold for cases that should be awarded a positive judgement, to maintain their implicit target for the proportion of positive judgements over time. Third, a <emph>contrast effect</emph> occurs when the present case is evaluated relative to a previous case (Bhargava & Fisman, [<reflink idref="bib4" id="ref23">4</reflink>]; Hartzmark & Shue, [<reflink idref="bib15" id="ref24">15</reflink>]; Pepitone & DiNubile, [<reflink idref="bib24" id="ref25">24</reflink>]). For example, a lead inspector may be so impressed with the previous school (e.g. awarding an 'Outstanding') the current school looks more negative by comparison. Finally, an <emph>emotional cost effect</emph> occurs when the person reaching a negative judgement cannot stomach reaching another negative judgement because of the psychological consequences of doing so (see Bindler & Hjalmarsson, [<reflink idref="bib5" id="ref26">5</reflink>]). For example, a lead inspector who has just given an 'Inadequate' judgement, leading to anger and resentment, might have less resolve to reach a negative judgement in subsequent inspections. In sum, the <emph>gambler's fallacy, norm referencing, contrast effects</emph> and <emph>emotional cost effects</emph> would lead to present inspection judgements becoming less similar to prior inspection judgements.</p> <hd id="AN0188054557-5">Empirical evidence from different field settings</hd> <p>Ibanez and Toffel ([<reflink idref="bib17" id="ref27">17</reflink>]) study sequential bias in restaurant food safety inspections. This setting is similar to ours in that inspectors move between different establishments making high-stakes judgements. They find a positive autocorrelation, such that an additional violation in the prior establishment increases by 1.7 per cent the number of violations found in the subsequent establishment. However, there are very few observable indicators of restaurant food safety standards available, making it hard for them to rule out autocorrelation due to the underlying order in which inspectors are assigned to cases. Lahtinen and Lushaku ([<reflink idref="bib19" id="ref28">19</reflink>]) study driving test examiners and find a similar positive autocorrelation. They also find that a) the immediate prior test result has the strongest relationship with the current test result, and b) a string of positive prior test results has a stronger positive correlation with the current test result than a single positive prior test result. However, their dataset has similar limitations to Ibanez and Toffel ([<reflink idref="bib17" id="ref29">17</reflink>]) in that they cannot control for the assignment of examiners to test takers. A third paper documenting positive autocorrelation is Bindler and Hjalmarsson ([<reflink idref="bib5" id="ref30">5</reflink>]), which looks at 18<sups>th</sups> century English jury decisions. They go a step further than other papers in using permutation tests to establish that the sequence in which cases are heard by the court appears to be randomly ordered. Interestingly, they also find greater positive autocorrelation when two similar cases are presented one after another (in line with a <emph>consistency effect</emph>).</p> <p>In contrast to the above, empirical studies have also documented negative autocorrelation between judgements. Bhargava and Fisman ([<reflink idref="bib4" id="ref31">4</reflink>]) study speed-dating and find that people are more likely to reject a candidate date if they rated their prior candidate date as attractive. Consistent with a <emph>contrast effect</emph>, the more attractive the prior candidate date is, the more likely they are to reject the current candidate date. Criscuolo et al. ([<reflink idref="bib9" id="ref32">9</reflink>]) study R&D project evaluations at a professional services firm, in which staff pitch for money to research problems faced by the organisation. Crucially, the authors randomise the order in which the pitches are considered by the panel, which leaves sequential bias as the most likely explanation for the autocorrelation. They find a pitch considered immediately after a successful pitch to be 23 per cent less likely to be funded. Chen et al. ([<reflink idref="bib8" id="ref33">8</reflink>]) study judges' decisions on refugee asylum cases, which are also randomly assigned to judges within the relevant court. They find judges are 3.3 per cent less likely to approve a claim if they approved the prior claim. In a separate study, Chen et al. ([<reflink idref="bib8" id="ref34">8</reflink>]) also investigate whether loan officers approve loan applications, where the quality of loan applications is observable and can therefore be controlled for in the analysis. Again, they find a negative autocorrelation, with 'streaks' of decisions in a certain direction showing a stronger correlation with the subsequent decision (see also Stewart et al., [<reflink idref="bib29" id="ref35">29</reflink>]).</p> <p>In summary, existing empirical research outside of education reveals a varied pattern of results. More rigorous studies, including random allocation of cases to judges, or extensive controls for underlying case quality, tend to find negative autocorrelation (Chen et al., [<reflink idref="bib8" id="ref36">8</reflink>]; Criscuolo et al., [<reflink idref="bib9" id="ref37">9</reflink>]). However, there are exceptions to this (e.g. Bindler & Hjalmarsson, [<reflink idref="bib5" id="ref38">5</reflink>]). Based on our discussion of the mechanisms above, we suspect that the net empirical effect depends on which of our six potential mechanisms are most important in the specific empirical settings under study. We are not aware of any empirical studies of sequence effects for school inspections in the existing literature, which suggests that new empirical research is required to understand the existence, direction, and mechanisms behind any sequential bias in this particular setting.</p> <p>Our setting (school inspection) provides some important advantages. We can control for observable measures (test scores) of the quality of cases (schools) being judged, as in the loan applications example in Chen et al. ([<reflink idref="bib8" id="ref39">8</reflink>]). We can also use this information to test for underlying autocorrelation in the assignment of cases to judges, as in Bindler and Hjalmarsson ([<reflink idref="bib5" id="ref40">5</reflink>]). However, our setting does not incorporate random assignment of judges to cases, as in the courtroom study in Chen et al. ([<reflink idref="bib8" id="ref41">8</reflink>]). We therefore cannot entirely rule out that the set of schools assigned to a lead inspector with a positive previous inspection would have different unobservable characteristics to the set of schools assigned to a lead inspector with a negative previous inspection. Such selection of unobservables would, of course, bias our results.</p> <hd id="AN0188054557-6">Inspection in England</hd> <p>All state schools in England are inspected by Ofsted. The inspectorate sends teams of inspectors into schools and awards a grade ('Outstanding', 'Good', 'Requires Improvement' or 'Inadequate') based on their observations. Inspection teams are always led by a named lead inspector and are comprised of HMIs (inspectors employed directly by Ofsted) and OIs (freelance inspectors contracted to run inspections as and when needed, many of whom also work in schools). To reduce travel time, inspectors usually work within a given Ofsted region. The number of inspectors in an Ofsted inspection team and the number of days for which an inspection lasts varies across types of inspections, and across academic years. After an inspection, the inspection grade is made publicly available on government websites (Bokhove et al., [<reflink idref="bib6" id="ref42">6</reflink>]). A poor inspection grade can also result in the leadership of the school being replaced. Inspections are therefore high stakes for schools.</p> <p>Ofsted is a non-ministerial department of the national government and, legally speaking, conducts two broad types of inspections. The first type is a 'Section 5' (or 'graded') inspection, which results in the school being awarded one of the four inspection grades: 'Outstanding', 'Good', 'Requires Improvement', 'Inadequate'. The second type of inspection is a 'Section 8' (or 'ungraded'), which does not result in a school being awarded a grade. Instead, the inspectors seek to establish whether there has been any change since the last inspection. After a Section 8 inspection, the school either continues with their previous grade, or the inspection is 'converted' to a Section 5 inspection. In the latter case, the school can then be awarded a new inspection grade. In 2015/16, Ofsted introduced a new type of Section 8 inspection known as a 'Short' inspection. These are one-day inspections for schools that had previously been judged to be 'Good' and can either result in a school retaining its 'Good' grade, or conversion to a Section 5 (graded) inspection.</p> <p>In the 2017/18 academic year, Ofsted revised the inspection framework so that a Section 8 inspection could only be converted to a Section 5 inspection immediately in exceptional circumstances. Instead, inspectors could recommend that a Section 5 inspection occur within the next year. To ensure that Short inspections still resulted in some sort of judgement, new judgements were introduced. Specifically, Short inspections would now result in three outcomes: 'Remains good', 'Remains good with concerns' or 'Remains good with potential improvement'. These new 'Short' inspection judgements are not directly comparable with 'Short' inspection judgements prior to January 2018 (where a conversion would lead to a 'Outstanding', 'Good', 'Requires Improvement' or 'Inadequate' grade). For this reason, we run our main analysis using data prior to January 2018 and then report supplementary analysis using the full sample.</p> <hd id="AN0188054557-7">Empirical approach</hd> <p></p> <hd id="AN0188054557-8">Data</hd> <p>We draw on three sources of data for this project. The first is data extracted from the 'Watchsted' website, from which we take details about individual Ofsted inspectors. In particular, we extracted all inspections done by inspectors who had conducted at least five inspections between the 2011/12 (which is as far back as the Watchsted database goes) and 2018/19 academic years. The second source of data is publicly available information from Ofsted, from which we take information on judgements reached following inspections.</p> <p>We linked these first two datasets using the date on which each inspection started and fuzzy matching on school names. We then checked that the information on inspection outcomes – including sub-judgements – was consistent across the two sources. After linking, we were able to observe the lead inspector for 99.2% of the primary inspections and 98.1% of the secondary inspections resulting in an overall effectiveness judgement. To further check the quality of our data, we randomly sampled 300 inspections, accessed the relevant inspection reports from the Ofsted website, and manually checked whether they corresponded with the information in our data, finding 97% agreement.</p> <p>The third and final dataset used in this project is the Department for Education's (DfE) School Performance Tables, from which we take information about school type (e.g. admissions policy), the composition of the student body (e.g. percentage of pupils eligible for Free School Meals) and the schools' average performance in national examinations. We exclude infant schools from the analysis on the basis that we lack good endline pupil achievement measures for such schools.</p> <hd id="AN0188054557-9">Analysis</hd> <p>Our analysis tests for sequential bias based on the sequence of schools judged by a given lead inspector. To test for sequential bias, we could have used ordered logistic regression. However, the proportional odds assumptions required for such a model were violated (<emph>p</emph> < 0.001). Instead, we estimate multinomial logistic regression models of the following form:</p> <p>Model 1:</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mo>ln</mo><mfenced open="(" close=")"><mrow><mrow><mfrac><mrow><mi mathvariant="italic">P</mi><mfenced open="(" close=")"><mrow><mrow><msub><mi mathvariant="italic">Y</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow><mo>=</mo><mi mathvariant="italic">j</mi></mrow></mfenced></mrow><mrow><mi mathvariant="italic">P</mi><mfenced open="(" close=")"><mrow><mrow><msub><mi>Y</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow><mo>=</mo><mi mathvariant="italic">J</mi></mrow></mfenced></mrow></mfrac></mrow></mrow></mfenced><mo>=</mo><mrow><msub><mi>β</mi><mrow><mn>0</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mrow><mn>1</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mi mathvariant="italic">Outstan</mi><mrow><msub><mi>d</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mrow><mn>2</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mi mathvariant="italic">ReqIm</mi><mrow><msub><mi>p</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mrow><mn>3</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mi mathvariant="italic">Ina</mi><mrow><msub><mi>d</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mrow><mn>4</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mrow><msub><mi>X</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow></math> </ephtml> </p> <p>Where:</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>Y</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow></math> </ephtml> represents the Ofsted judgement given by lead inspector <emph>i</emph> in period <emph>t</emph></p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">j</mi></math> </ephtml> represents the four possible Ofsted judgement categories: 'Inadequate'; 'Requires Improvement'; 'Good'; 'Outstanding'</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">J</mi></math> </ephtml> is the reference category ('Good', or 3)</p> <p></p> <ulist> <item> The left-hand side of the equation represents the log odds of being in a given Ofsted category relative to the reference category</item> <p></p> </ulist> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mrow><mn>0</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow></math> </ephtml> -</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mrow><mn>4</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow></math> </ephtml> represent the intercepts and coefficients for each of the</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">j</mi></math> </ephtml> categories of the outcome variable</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">Outstan</mi><mrow><msub><mi>d</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> is a dummy which takes the value 1 if the lead inspector <emph>i</emph> reached an 'Outstanding' judgement in their prior school inspection (<emph>t-1</emph>), and 0 otherwise</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">ReqIm</mi><mrow><msub><mi>p</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">Ina</mi><mrow><msub><mi>d</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> are the equivalent of</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">Outstan</mi><mrow><msub><mi>d</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> for the other prior inspection judgements</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">X</mi></math> </ephtml> is set of covariates, none of which could have been affected by the inspection result: lead inspector status (HMI or OI), lead inspector gender, percentage of pupils at inspected school with English as an additional language (EAL), percentage of pupils on free school meals (FSM), percentage of pupils with special educational needs (SEN), the Ofsted region in which the inspection takes place, the inspected school's previous overall Ofsted judgement, and whether the school is all-boys, all-girls or mixed.</p> <p></p> <ulist> <item> Achievement at age 7 and 11 (if a primary school) or age 16 (if a secondary school), in the year prior to the inspection occurring</item> </ulist> <p>All standard errors are clustered at the lead inspector level.</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mrow><mn>1</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow></math> </ephtml> -</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mrow><mn>3</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow></math> </ephtml> from Model 1 provide our estimates of the association between the different prior inspection judgements and the present inspection judgement, relative to a 'Good' judgement (the reference category). By including the different judgements separately, we can detect non-linear effects. Such non-linearities are plausible because mechanisms such as emotional costs are likely to apply much more strongly to the extreme inspection judgements. If</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>1</mn></msub></mrow></math> </ephtml> to</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>3</mn></msub></mrow><mo>=</mo><mn>0</mn></math> </ephtml> then we find no autocorrelation and therefore no evidence of sequential bias. If</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>1</mn></msub></mrow></math> </ephtml> to</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>3</mn></msub></mrow><mo>></mo><mn>0</mn></math> </ephtml> then we observe positive autocorrelation, and if</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>1</mn></msub></mrow></math> </ephtml> to</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>3</mn></msub></mrow><mo><</mo><mn>0</mn></math> </ephtml> then we observe negative autocorrelation.</p> <p>For</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>1</mn></msub></mrow></math> </ephtml> to</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>3</mn></msub></mrow><mo>≠</mo><mn>0</mn></math> </ephtml> to be indicative of sequential bias, it must be the case that the assignment of inspectors to inspections is essentially random, conditional on the covariates. One strength of our data relative to the existing literature is that we observe several characteristics (</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>X</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow></math> </ephtml> ) of the inspector, the school intake, and school quality as captured by exam results. However, there may still be unobservable characteristics that affect both the assignment of inspectors to inspections and the judgements reached in those inspections. For example, if some inspectors are naturally harsher in their judgements, then this would create positive autocorrelation even without sequential bias, since harsh inspectors would be more likely to give negative judgement in prior inspections and in present inspections. In our preferred specification (Model 2), we address this threat using inspector fixed effects. Unfortunately, our multinomial logistic regression models do not converge when inspector fixed effects are added. We therefore estimate a series of logistic regression models (one for each level of <emph>j</emph>) incorporating inspector fixed effects</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>I</mi><mi>i</mi></msub></mrow></math> </ephtml> . All other variables are defined as in Model 1.</p> <p>Model 2:</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mo>ln</mo><mfenced open="(" close=")"><mrow><mrow><mfrac><mrow><mi mathvariant="italic">P</mi><mfenced open="(" close=")"><mrow><mrow><msub><mi mathvariant="italic">Y</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow><mo>=</mo><mi mathvariant="italic">j</mi></mrow></mfenced></mrow><mrow><mn>1</mn><mo>−</mo><mi mathvariant="italic">P</mi><mfenced open="(" close=")"><mrow><mrow><msub><mi>Y</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow><mo>=</mo><mi mathvariant="italic">j</mi></mrow></mfenced></mrow></mfrac></mrow></mrow></mfenced><mo>=</mo><mrow><msub><mi>β</mi><mn>0</mn></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mrow><mn>1</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mi mathvariant="italic">Outstan</mi><mrow><msub><mi>d</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mn>2</mn></msub></mrow><mi mathvariant="italic">ReqIm</mi><mrow><msub><mi>p</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mn>3</mn></msub></mrow><mi mathvariant="italic">Ina</mi><mrow><msub><mi>d</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mrow><mn>4</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mrow><msub><mi>X</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mrow><mn>5</mn><mi mathvariant="italic">j</mi></mrow></msub></mrow><mrow><msub><mi>I</mi><mi>i</mi></msub></mrow></math> </ephtml> </p> <p>Even if we can account for differences in harshness/leniency across inspectors, we might still observe autocorrelation in the absence of sequential bias if the underlying sequence of schools that need inspecting is autocorrelated. For example, a string of poorly performing schools might be due for inspection in the area in which an inspector operates. We assess this threat by directly testing for autocorrelation in the underlying case quality using variations of the following ordinary least squares regression model:</p> <p>Model 3:</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>Y</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow><mo>=</mo><mrow><msub><mi>β</mi><mn>0</mn></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mn>1</mn></msub></mrow><mrow><msub><mi>Y</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mn>2</mn></msub></mrow><mrow><msub><mi>X</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mi>β</mi><mn>3</mn></msub></mrow><mrow><msub><mi>I</mi><mi>i</mi></msub></mrow><mo>+</mo><mrow><msub><mi>ε</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow></math> </ephtml> </p> <p>Where:</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>Y</mi><mrow><mi mathvariant="italic">it</mi></mrow></msub></mrow></math> </ephtml> represents a measure of average pupil achievement for the school, determined in the year before the inspection occurred. In primary schools, this is either the national quintile of Key Stage 2 English results or the national quintile of Key Stage 2 mathematics results, captured in the first such tests that occurred after the inspection. In secondary schools, this is either the national quintile Key Stage 4 (GCSE) Progress 8 score, or the national quintile Key Stage 4 (GCSE) total points score, captured in the first such tests that occurred after the inspection.</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>Y</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> represents the equivalent outcome measures at the school previously inspected by the inspector.</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>X</mi><mrow><mi mathvariant="italic">it</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> is a vector of school average pupil characteristics: national quintile of proportion of pupils eligible for free school meals (FSM), the proportion of pupils with special educational needs (SEN), the proportion of pupils with English as an additional language (EAL). These are measured in the year prior to the inspection occurring.</p> <p></p> <p>• </p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>I</mi><mi>i</mi></msub></mrow></math> </ephtml> is again a vector of inspector fixed effects.</p> <hd id="AN0188054557-10">Results</hd> <p></p> <hd id="AN0188054557-11">Descriptives</hd> <p>We begin by presenting some simple descriptive statistics, showing the relationship between the judgement awarded by an inspector in their 'previous' inspection and that in their 'current' inspection. We include every pair of sequential inspections conducted by a given inspector in the tables. That is to say, many inspections feature twice, once as a 'previous' inspection and then again as a 'current' inspection. Table 1 shows all cases where the 'current' inspection is for a primary school, and Table 2 shows all cases where the 'current' inspection is for a secondary school. The tables show that 'Good' and 'Requires Improvement' are the most frequently occurring judgements. Although we observe all possible pairs of judgements, moving between the less frequently occurring judgements ('Outstanding' and 'Inadequate') is relatively rare, resulting in small cell sizes. Generally, the number of observations is also much smaller in secondary schools, which reflects there being far fewer secondary schools than primary schools in England.</p> <p>Table 1. Inspectors' previous and current inspection judgements (primary school inspections).</p> <p> <ephtml> <table><thead><tr><td /><td>Current Inspection</td></tr><tr><td>Outstanding</td><td>Good</td><td>Requires Imp.</td><td>Inadequate</td><td>Total</td></tr></thead><tbody><tr><td><bold>Previous Inspection</bold></td><td /><td /><td /><td /><td /></tr><tr><td>Outstanding</td><td>188 (0.7%)</td><td>1,156 (4.5%)</td><td>393 (1.5%)</td><td>85 (0.3%)</td><td><bold>1,822</bold></td></tr><tr><td>Good</td><td>1,078 (4.2%)</td><td>10,150 (39.5%)</td><td>3,685 (14.3%)</td><td>733 (12.9%)</td><td><bold>15,646</bold></td></tr><tr><td>Requires Imp.</td><td>438 (1.7%)</td><td>3,820 (14.9%)</td><td>2,092 (8.1%)</td><td>415 (1.6%)</td><td><bold>6,765</bold></td></tr><tr><td>Inadequate</td><td>72 (0.3%)</td><td>813 (3.2%)</td><td>452 (1.8%)</td><td>110 (0.4%)</td><td><bold>1,447</bold></td></tr><tr><td><bold>Total</bold></td><td><bold>1,776</bold><bold>(6.9%)</bold></td><td><bold>15,939</bold><bold>(62.1%)</bold></td><td><bold>6,622</bold><bold>(25.8%)</bold></td><td><bold>1,343</bold><bold>(5.2%)</bold></td><td><bold>25,680</bold></td></tr></tbody></table> </ephtml> </p> <p>1 Notes: Unbolded percentages refer to the proportion of transitions out of 25,680 total transitions. Bolded percentages in the bottom row refer to the proportion of current inspections receiving each judgement.</p> <p>Table 2. Inspectors' previous and current inspection judgements (secondary school inspections).</p> <p> <ephtml> <table><thead><tr><td /><td>Current Inspection</td></tr><tr><td>Outstanding</td><td>Good</td><td>Requires Imp.</td><td>Inadequate</td><td>Total</td></tr></thead><tbody><tr><td><bold>Previous Inspection</bold></td><td /><td /><td /><td /><td /></tr><tr><td>Outstanding</td><td>68 (1.3%)</td><td>219 (4.0%)</td><td>143 (2.6%)</td><td>55 (1.0%)</td><td><bold>485</bold></td></tr><tr><td>Good</td><td>240 (4.4%)</td><td>1,350 (24.9%)</td><td>844 (15.6%)</td><td>271 (5.0%)</td><td><bold>2,705</bold></td></tr><tr><td>Requires Imp.</td><td>152 (2.8%)</td><td>760 (%)</td><td>589 (10.9%)</td><td>185 (3.4%)</td><td><bold>1,686</bold></td></tr><tr><td>Inadequate</td><td>42 (0.8%)</td><td>215 (%)</td><td>179 (3.3%)</td><td>100 (1.8%)</td><td><bold>536</bold></td></tr><tr><td><bold>Total</bold></td><td><bold>502</bold><bold>(9.3%)</bold></td><td><bold>2,544</bold><bold>(47.0%)</bold></td><td><bold>1,755</bold><bold>(32.4%)</bold></td><td><bold>611</bold><bold>(11.3%)</bold></td><td><bold>5,412</bold></td></tr></tbody></table> </ephtml> </p> <p>2 Notes: Unbolded percentages refer to the proportion of transitions out of 5,412 total transitions. Bolded percentages in the bottom row refer to the proportion of current inspections receiving each judgement.</p> <hd id="AN0188054557-12">Modelling</hd> <p>Figure 1 plots coefficients</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>1</mn></msub></mrow></math> </ephtml> to</p> <p>Graph</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>β</mi><mn>3</mn></msub></mrow></math> </ephtml> (Model 1) for all primary school inspections in our dataset prior to January 2018. The coefficients are risk ratios and have been estimated using multinomial regression. The reference category for the outcome (current inspection judgement) is 'Good' and the reference category for the predictors (prior inspection judgement) is also 'Good'. Hence, the top coefficient should be interpreted as 'the risk of the inspected school being awarded an "Outstanding" grade, relative to a "Good" grade, is 26% higher if the inspector awarded an "Outstanding" grade in their previous inspection, relative to if the inspector awarded a "Good" grade in their previous inspection'. Looking across Figure 1, the results show positive autocorrelation. That is, inspectors are more likely to award high inspection judgements if they awarded high inspection judgements in their prior inspection, and vice versa. Figure 2 shows the equivalent for secondary school inspections and displays a similar pattern. However, the estimates are less precisely estimated, due to our smaller sample of secondary school inspections.</p> <p>Graph: Figure 1. Autocorrelation in inspectors' inspection judgements (primary schools).</p> <p>Graph: Figure 2. Autocorrelation in inspectors' inspection judgements (secondary schools).</p> <p>Recall that positive autocorrelation in Model 1 could be the result of inspectors differing in their average levels of harshness/leniency, even in the absence of sequence effects. The next stage of our analysis attempts to address this by estimating a set of binary logistic regression models (Model 2) incorporating inspector fixed effects. The coefficients are now odds ratios and are estimated using within-inspector, between-inspection variation.</p> <p>The top panel of Figure 3 focuses on an 'Outstanding' judgement in the current inspection. The results are somewhat different to the analysis without inspector fixed effects. The only prior inspection judgement that shows a statistically significant relationship with the probability of this outcome is awarding an 'Outstanding' judgement in the previous inspection. More precisely, we find that when an inspector awarded an 'Outstanding' judgement in the previous inspection, the odds of them doing so in the next inspection are 32 per cent lower. The second panel down in Figure 3 focuses on a 'Good' judgement in the present inspection. Awarding any grade other than 'Good' in the previous inspection is associated with an increase in the odds of this outcome. The odds are 16 per cent higher when the previous inspection reached an 'Outstanding' judgement, 10 per cent higher when the previous inspection reached a 'Requires Improvement' and 17 per cent higher when the previous inspection reached an 'Inadequate' Judgement. The third panel down in Figure 3 focuses on a 'Requires Improvement' judgement in the current inspection. The only prior inspection judgement that shows a statistically significant relationship with the probability of this outcome is awarding a 'Requires Improvement' judgement in the previous inspection. We find that when an inspector awarded a 'Requires Improvement' judgement in the previous inspection, the odds of them doing so again in the next inspection are 12 per cent lower. Finally, the bottom panel in Figure 3 focuses on an 'Inadequate' judgement. The only prior inspection judgement that shows a statistically significant relationship with the probability of this outcome is awarding an 'Inadequate' judgement in the previous inspection. More precisely, we find that when an inspector awarded an 'Inadequate' judgement in the previous inspection, the odds of them doing so again in the next inspection fall by 42 per cent. Broadly speaking, the picture in Figure 3 (negative autocorrelation) is the reverse of that in Figure 1 (positive autocorrelation). This is consistent with the fixed effects accounting for differences in leniency/harshness across inspectors.</p> <p>Graph: Figure 3. Autocorrelation in inspectors' inspection judgements (primary schools).</p> <p>Figure 4 shows the analogous results for secondary schools. The results are again somewhat different to the analysis without inspector fixed effects (Figure 2). None of the prior inspection grades shows a statistically significant association with the award of either an 'Outstanding' or a 'Good' judgement in the current inspection. The results for 'Requires Improvement' (third panel down in Figure 4) are similar to those for primary schools in Figure 3, with a 'Requires Improvement' in the prior inspection associated with a 19 per cent reduction of the odds of the same grade being awarded in the current inspection. The bottom panel of Figure 4 focuses on 'Inadequate' judgements. A prior 'Outstanding' judgement is associated with an 86 per cent increase in the odds of an 'Inadequate' judgement in the current inspection. When looked at in the round, the results in Figure 4 show no clear overall pattern of either positive or negative autocorrelation.</p> <p>Graph: Figure 4. Autocorrelations in inspectors' inspection judgements (secondary schools).</p> <hd id="AN0188054557-13">Robustness and sensitivity checks</hd> <p>The results in Figures 3 and 4 might still not reflect sequence effects if there is autocorrelation in the sequence of schools assigned to inspectors. For example, our finding that in primary schools, inspectors are less likely to award an 'Inadequate' grade if they awarded an 'Inadequate' grade in their prior inspection, could potentially reflect a decision by Ofsted staff not to assign inspectors to two struggling schools in a row. We test this in Table 3 using variations on Model 3 (which includes inspector fixed effects) to check whether objective measures of school quality are autocorrelated between 'current' and 'previous' schools inspected by a given inspector. Columns 1 and 2 show results for primary schools. There is no clear relationship between the Key Stage (KS) 2 English results (Column 1) or the KS2 mathematics results (Column 2) in the current inspection and the characteristics of the school in the previous inspection. None of the coefficients is statistically significant, despite being precisely estimated. Column 3 shows results for secondary schools. There is no clear relationship between pupil progress in the current inspection and pupil progress in the prior inspection. Two of the pupil characteristics variables show some correlation. However, these are substantively very small. This suggests that the results in Figures 3 and 4 cannot easily be explained by autocorrelation in the underlying sequence of schools assigned to inspectors.</p> <p>Table 3. Testing for autocorrelation in the assignment of schools to inspectors.</p> <p> <ephtml> <table><thead><tr><td /><td>Primary</td><td>Secondary</td></tr><tr><td>(1) Age 11 English</td><td>(2) Age 11 mathematics</td><td>(3) Age 11–16 Progress</td></tr></thead><tbody><tr><td>Prior inspection KS2 English</td><td>0.002</td><td>−0.0003</td><td /></tr><tr><td>(0.01)</td><td>(0.01)</td><td /></tr><tr><td>Prior inspection KS2 mathematics</td><td>0.003</td><td>−0.004</td><td /></tr><tr><td>(0.01)</td><td>(0.01)</td><td /></tr><tr><td>Prior inspection KS4 Progress 8</td><td /><td /><td>−0.04</td></tr><tr><td /><td /><td>(0.02)</td></tr><tr><td>Prior inspection FSM</td><td>−0.016</td><td>−0.01</td><td>−0.05**</td></tr><tr><td>(0.008)</td><td>(0.008)</td><td>(0.02)</td></tr><tr><td>Prior inspection SEN</td><td>0.005</td><td>0.004</td><td>0.01**</td></tr><tr><td>(0.003)</td><td>(0.002)</td><td>(0.006)</td></tr><tr><td>Prior inspection EAL</td><td>−0.00004</td><td>−0.0003</td><td>−0.0001</td></tr><tr><td>(0.0004)</td><td>(0.001)</td><td>(0.001)</td></tr><tr><td>N (inspections)</td><td>23,190</td><td>23,190</td><td>4,098</td></tr><tr><td>N (inspectors)</td><td>1,232</td><td>1,232</td><td>524</td></tr></tbody></table> </ephtml> </p> <p>3 Notes: Each column is a separate OLS regression. Age 11–16 progress is based on average pupil progress across eight subjects. FSM = free school meals. SEN = special educational needs. EAL = English as an additional language. Standard errors – clustered at inspector level – shown in parentheses. ** = <emph>p</emph> < 0.01. * = <emph>p</emph> < 0.05. The sequence of inspections is defined based on the order in which a given lead inspector judges schools.</p> <p>As previously discussed, after January 2018 Section 8 inspections (of previously 'Good' schools) resulted in either a 'Remains good', 'Remains good with concerns' or 'Remains good with potential improvement'. Since these outcomes are not directly comparable to the four judgements awarded in other inspections, we excluded all inspections after January 2018 from our analysis above. To check the sensitivity of our results in this decision, we recoded all inspection judgements in our dataset to express them in terms of whether the inspector 'Upgraded' the school, 'Downgraded' the school, or if the school 'Stayed'. This allows us to express all inspection judgements before and after January 2018 in a common metric. For example, a Section 8 'Remains good with potential improvement' is coded as an 'Upgrade', along with any Section 5 inspections in which a school went from, for example, 'Good' to 'Outstanding'.</p> <p>We now report the results from an analysis based on all the inspections in our sample, using Model 2 (which includes inspector fixed effects) and this new comparable outcome variable. The reference category for the prior inspection judgement is 'Stay'. Figure 5 shows the results for primary schools. The top panel shows that the probability of an inspector awarding an 'Upgrade' does not show a statistically significant relationship with the judgement reached in their prior inspection. The middle panel shows that the probability of an inspector awarding a 'Stay' is positively associated with whether they awarded a 'Down' in the previous inspection. More precisely, a 'Down' in the prior inspection is associated with a 10 per cent increase in the odds of a 'Stay' in the current inspection. Finally, the bottom panel of Figure 5 shows that the probability of an inspector awarding a 'Downgrade' in the current inspection is negatively associated (18% lower odds) if the inspector awarded a 'Downgrade' in the previous inspection. This result aligns with the results observed for prior 'Inadequate' judgements in primary schools in Figure 3. The size of the association is smaller in Figure 5 (odds ratio of 0.82) than in Figure 3 (odds ratio of 0.58). However, this is to be expected since the 'Downgrade' in Figure 5 includes all downward moves, not just those to 'Inadequate'. Taken in the round, the findings in Figure 5 display negative autocorrelation, at least among the 'Downgrade' and 'Stay' judgements.</p> <p>Graph: Figure 5. Autocorrelation including inspections after January 2018 (primary schools).</p> <p>Figure 6 shows the analogous results for secondary schools. The top panel shows that the probability of an inspector awarding an 'Upgrade' is negatively associated with reaching a judgement other than 'Stay' in the prior inspections. More precisely, an 'Upgrade' in the prior inspection is associated with a 29 per cent reduction in the odds of an 'Upgrade' in the next inspection. Likewise, a 'Downgrade' in the prior inspection is associated with a 24 per cent reduction in the odds of an 'Upgrade' in the current inspection. The middle panel shows that the probability of an inspector awarding a 'Stay' is positively associated (26% increase in odds) with the inspector reaching a 'Stay' judgement in the prior inspection. Finally, the bottom panel shows that there is no statistically significant relationship between the judgement in the prior inspection and the judgement in the current inspection. Again, the secondary findings do not display a clear pattern of either negative or positive autocorrelation.</p> <p>Graph: Figure 6. Autocorrelation including inspections after January 2018 (secondary schools).</p> <p>We observe somewhat different findings for primary (Figure 5) and secondary (Figure 6) schools. However, the secondary school sample is smaller than the primary school sample, which limits the precision of the estimates. To investigate this further, and for completeness, Appendix Figure A1 shows the results pooled across primary and secondary schools. There are two clear findings in the pooled sample. First, consistent with Figures 3 and 4, a 'Requires Improvement' judgement in the prior inspection was associated with a 9 per cent reduction in the odds of a 'Requires Improvement' judgement in the current inspection. Second, consistent with Figures 3 and 5, an 'Inadequate' judgement in the prior inspection was associated with a 30 per cent reduction in the odds of an 'Inadequate' judgement in the current inspection.</p> <p>We also checked whether our main findings for primary schools were sensitive to excluding all inspections occurring before the introduction of short inspections in 2015/16. Appendix Figure A2 shows the results for primary schools with this sample restriction imposed. The overall pattern of findings remains qualitatively very similar.</p> <hd id="AN0188054557-14">Further exploration of mechanisms</hd> <p>Our results for secondary schools display no consistent pattern. However, looking across our inspector fixed effect analyses for primary schools, we observe a fairly consistent negative autocorrelation, but only for the high-stakes 'Inadequate' inspection outcome. This is consistent with an <emph>emotional cost</emph> mechanism, in which inspectors are reluctant to deliver two of the high stakes 'Inadequate' judgements in back-to-back inspections but are happy to deliver any of the three lower-stakes judgements in back-to-back inspections. By contrast, this pattern of results is not consistent with any of the other three negative autocorrelation mechanisms outlined above. This is because all three of the other mechanisms would predict that awarding an 'Outstanding' judgement in the previous inspection would make an 'Outstanding' judgement in the current inspection less likely. With the <emph>gambler's fallacy</emph>, this would be due to 'luck running out'; with <emph>norm referencing</emph> this would be to avoid awarding 'too many' 'Outstanding' judgements; and with <emph>contrast effects</emph> this would be because an 'Outstanding' previous judgement would be a 'hard act to follow'.</p> <p>If <emph>emotional costs</emph> are the correct explanation for our findings, it seems likely that inspectors with longer periods between their current and previous (lead) inspection would be less prone to sequential bias. Our reasoning here is that an inspector who has had, for example, two months since awarding an 'Inadequate' judgement would have had more time to (psychologically) recover compared to an inspector who had, for example, two days since awarding an 'Inadequate' judgement. Figure 7 reports an analysis designed to test this hypothesis by running the regressions across different subsamples of inspections, which vary based on the number of days since the inspector's last inspection occurred. The results of interest can be found at the bottom of the right panel. If the blue dots (inspections less than 10 days since the previous inspection) are further from the vertical red line than the red dots (less than 30 days) and the green dots (less than 60 days), then this would be consistent with the emotional costs hypothesis. In practice, we observe little difference, with similar point estimates and overlapping confidence intervals across the three.</p> <p>Graph: Figure 7. Variation in autocorrelation by length of time since last inspection (primary schools).</p> <hd id="AN0188054557-15">Discussion</hd> <p>School inspections are an important part of the school accountability system in many countries (OECD, [<reflink idref="bib21" id="ref43">21</reflink>]). Yet, despite the often high-stakes nature of inspections, little is known about the reliability and validity of the judgements reached by inspectors. We set out to address this gap in the evidence by testing for the presence of a bias that has been shown to operate in many other settings involving repeated judgement calls – sequential bias. The judgements reached by inspectors in prior inspections are logically irrelevant to the judgements that they should reach in the next school they visit. The presence of sequential bias in school inspection judgements would therefore be evidence of (some degree of) invalidity in the school inspection process.</p> <p>We found some limited evidence of sequential bias in primary school inspections but not in secondary schools. More precisely, among primary schools we found that inspectors who reached an 'Inadequate' judgement in their previous inspection are less likely to reach another 'Inadequate' judgement in their next inspection. This result holds when accounting for objective indicators of school quality and differences in harshness/leniency across different inspectors. The result also does not seem to be driven by the way in which schools are assigned to inspectors. The odds of inspectors reaching an inadequate judgement are about 40 per cent lower if they also reached an inadequate judgement in the previous inspection. However, it should also be noted that only around 5 per cent of inspection judgements result in an 'Inadequate', meaning this bias only affects a small minority of judgements given. We do not detect a similar relationship in secondary schools, albeit with a much smaller sample.</p> <p>We only detect sequential bias among 'Inadequate' inspection judgements. These often result in headteachers losing their job, which suggests that the sequential bias might reflect an 'emotional costs' mechanism. This is consistent with a large qualitative literature which has documented the considerable emotional strain experienced by teachers (Hopkins et al., [<reflink idref="bib16" id="ref44">16</reflink>]; Perryman, [<reflink idref="bib25" id="ref45">25</reflink>]; Quintelier et al., [<reflink idref="bib26" id="ref46">26</reflink>]), school leaders (Penninckx & Vanhoof, [<reflink idref="bib23" id="ref47">23</reflink>]; Segerholm & Hult, [<reflink idref="bib28" id="ref48">28</reflink>]), and inspectors themselves (Elonga Mboyo, [<reflink idref="bib11" id="ref49">11</reflink>]; Segerholm & Hult, [<reflink idref="bib28" id="ref50">28</reflink>]), particularly where an adverse judgement has been reached. This suggests that it may simply be too psychologically costly for some inspectors to reach an 'Inadequate' judgement twice in a row, even when they would otherwise have judged the second school 'Inadequate'. However, contrary to our expectations around the emotional costs mechanism, we did not find that a longer gap after a previous 'Inadequate' inspection moderated the sequential bias. This interpretation of our findings should therefore be considered tentative and is in need of further testing.</p> <hd id="AN0188054557-16">Limitations</hd> <p>This research should, of course, be interpreted in light of the limitations of this study. Foremost among these is the lack of random assignment of inspectors to school inspections. This means we cannot entirely rule out unobserved variables driving the relationship that we observe between prior and current inspection grades awarded by an inspector. Having said that, we consider this to be unlikely for two reasons. First, we provide empirical evidence that there are no clear correlations between the objectively assessed characteristics of the 'prior' and 'current' schools visited by a given inspector. Second, our dataset allows us to control for a rich set of inspector and school characteristics. Any remaining unobserved confounders would have to be strongly correlated with the characteristics of the 'prior' and 'current' school to explain the 42 per cent reduction in the odds of reaching a second 'Inadequate' judgement.</p> <p>A second important caveat relates to the lack of an equivalent finding for the secondary schools in our sample. One potential explanation for this is that our secondary inspection dataset is just much smaller than our primary inspection dataset. Having said that, our point estimate for the relationship between a prior 'Inadequate' and subsequent 'Inadequate' in secondaries does not even have the same sign as the point estimate for primaries. An alternative explanation is that secondary inspections are just different to primary school inspections in important ways. For example, primary school inspections often only involve one or two inspectors, whereas secondary school inspections often involve larger teams of inspectors (Bokhove et al., [<reflink idref="bib6" id="ref51">6</reflink>]). While we can only speculate as to how this would interact with sequential bias, it may be that lone inspectors are more susceptible to bias because their judgements are less subject to moderation by colleagues. We attempted to test this in our data, but the sample size was not large enough to support the analysis with the inclusion of inspector fixed effects.</p> <p>A third limitation relates to our focus on lead inspectors. For each inspection, we observe only the name of the lead inspector. If inspectors are also influenced by prior inspections in which they are not the lead inspector, this could affect our results. We can only check this empirically if Ofsted releases more complete and granular data about all the inspections in which all inspectors take part. A fourth and final limitation is that our data pertain only to England and our empirical findings therefore cannot be straightforwardly generalised to other countries where inspection systems differ in several ways, in particular around the stakes attached to receiving a negative judgement.</p> <hd id="AN0188054557-17">Conclusion</hd> <p>In terms of inspection policy, our findings tentatively suggest that there may be a trade-off between the stakes attached to an inspection process and how valid it is. If inspectors feel that the consequences of reaching a negative judgement are too emotionally costly, then they may sometimes shy away from reaching a negative judgement. If this is true, then policymakers face a trade-off between a more accurate inspection system that is less consequential, or a more consequential system that is less accurate. This trade-off runs counter to the conventional view that the more consequential a decision, the greater we would consider the need for it to be accurate (AERA, APA, & NCME, [<reflink idref="bib2" id="ref52">2</reflink>], [<reflink idref="bib3" id="ref53">3</reflink>]).</p> <hd id="AN0188054557-18">Acknowledgement</hd> <p>The Nuffield Foundation is an independent charitable trust with a mission to advance socialwellbeing. It funds research that informs social policy, primarily in Education, Welfare, andJustice. It also funds student programmes that provide opportunities for young people todevelop skills in quantitative and scientific methods. The Nuffield Foundation is the founderand co-funder of the Nuffield Council on Bioethics and the Ada Lovelace Institute. The Foun-dation has funded this project, but the views expressed are those of the authors and notnecessarily the Foundation. Visit https://<ulink href="http://www.nuffieldfoundation.org/">www.nuffieldfoundation.org/</ulink>. We are grateful for theirsupport. Helpful comments have been received on the draft from our project advisorygroup, whom we would like to thank for their input and support.</p> <hd id="AN0188054557-19">Disclosure statement</hd> <p>No potential conflict of interest was reported by the author(s). John Jerrim worked on secondment at Ofsted during the period in which this work was conducted.</p> <hd id="AN0188054557-20">Appendix A</hd> <p>Graph: Figure A1. Autocorrelation in inspectors' judgements (pooled primary and secondary schools).</p> <p>Graph: Figure A2. Autocorrelation in inspectors' judgements (primary schools) – before short inspections introduced in 2015/16.</p> <ref id="AN0188054557-21"> <title> References </title> <blist> <bibl id="bib1" idref="ref2" type="bt">1</bibl> <bibtext> Altrichter, H., & Kemethofer, D. (2015). Does accountability pressure through school inspections promote school improvement? School Effectiveness and School Improvement, 26 (1), 32 – 56. https://doi.org/10.1080/09243453.2014.927369</bibtext> </blist> <blist> <bibl id="bib2" idref="ref52" type="bt">2</bibl> <bibtext> American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. American Educational Research Association.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref53" type="bt">3</bibl> <bibtext> American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref23" type="bt">4</bibl> <bibtext> Bhargava, S., & Fisman, R. (2014). Contrast effects in sequential decisions: Evidence from speed dating. The Review of Economics and Statistics, 96 (3), 444 – 457. https://doi.org/10.1162/REST_a_00416</bibtext> </blist> <blist> <bibl id="bib5" idref="ref13" type="bt">5</bibl> <bibtext> Bindler, A., & Hjalmarsson, R. (2019). Path dependency in jury decision making. Journal of the European Economic Association, 17 (6), 1971 – 2017. https://doi.org/10.1093/jeea/jvy046</bibtext> </blist> <blist> <bibl id="bib6" idref="ref11" type="bt">6</bibl> <bibtext> Bokhove, C., Jerrim, J., & Sims, S. (2023). Are some school inspectors more lenient than others? School Effectiveness and School Improvement, 34 (4), 419 – 441. https://doi.org/10.1080/09243453.2023.2240318</bibtext> </blist> <blist> <bibl id="bib7" idref="ref6" type="bt">7</bibl> <bibtext> Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2004). The concept of validity. Psychological Review, 111 (4), 1061. https://doi.org/10.1037/0033-295X.111.4.1061</bibtext> </blist> <blist> <bibl id="bib8" idref="ref14" type="bt">8</bibl> <bibtext> Chen, D. L., Moskowitz, T. J., & Shue, K. (2016). Decision making under the gambler's fallacy: Evidence from asylum judges, loan officers, and baseball umpires. Quarterly Journal of Economics, 131 (3), 1181 – 1242. https://doi.org/10.1093/qje/qjw017</bibtext> </blist> <blist> <bibl id="bib9" idref="ref32" type="bt">9</bibl> <bibtext> Criscuolo, P., Dahlander, L., Grohsjean, T., & Salter, A. (2021). The sequence effect in panel decisions: Evidence from the evaluation of research and development projects. Organization Science, 32 (4), 987 – 1008. https://doi.org/10.1287/orsc.2020.1413</bibtext> </blist> <blist> <bibtext> Ehren, M. C., Gustafsson, J. E., Altrichter, H., Skedsmo, G., Kemethofer, D., & Huber, S. G. (2015). Comparing effects and side effects of different school inspection systems across Europe. Comparative Education, 51 (3), 375 – 400. https://doi.org/10.1080/03050068.2015.1045769</bibtext> </blist> <blist> <bibtext> Elonga Mboyo, J. P. (2017). The place of emotions while inspecting schools: Reflections of two ofsted inspectors. School Leadership and Management, 37 (3), 267 – 287. https://doi.org/10.1080/13632434.2017.1324841</bibtext> </blist> <blist> <bibtext> Eyles, A., & Machin, S. (2019). The introduction of academy schools to England's education. Journal of the European Economic Association, 17 (4), 1107 – 1146. https://doi.org/10.1093/jeea/jvy021</bibtext> </blist> <blist> <bibtext> Goldbach, C., Sickmann, J., & Pitz, T. (2022). Sequential decision bias–evidence from grading exams. Applied Economics, 54 (32), 3727 – 3739. https://doi.org/10.1080/00036846.2021.1976390</bibtext> </blist> <blist> <bibtext> Haladyna, T. M., & Downing, S. M. (2004). Construct‐irrelevant variance in high‐stakes testing. Educational Measurement Issues & Practice, 23 (1), 17 – 27. https://doi.org/10.1111/j.1745-3992.2004.tb00149.x</bibtext> </blist> <blist> <bibtext> Hartzmark, S. M., & Shue, K. (2018). A tough act to follow: Contrast effects in financial markets. The Journal of Finance, 73 (4), 1567 – 1613. https://doi.org/10.1111/jofi.12685</bibtext> </blist> <blist> <bibtext> Hopkins, E., Hendry, H., Garrod, F., McClare, S., Pettit, D., Smith, L., Burrell, H., & Temple, J. (2016). Teachers' views of the impact of school evaluation and external inspection processes. Improving Schools, 19 (1), 52 – 61. https://doi.org/10.1177/1365480215627894</bibtext> </blist> <blist> <bibtext> Ibanez, M. R., & Toffel, M. W. (2020). How scheduling can bias quality assessment: Evidence from food-safety inspections. Management Science, 66 (6), 2396 – 2416. https://doi.org/10.1287/mnsc.2019.3318</bibtext> </blist> <blist> <bibtext> Jones, K. L., Tymms, P., Kemethofer, D., O'Hara, J., McNamara, G., Huber, S., Myrberg, E., Skedsmo, G., & Greger, D. (2017). The unintended consequences of school inspection: The prevalence of inspection side-effects in Austria, the Czech Republic, England, Ireland, the Netherlands, Sweden, and Switzerland. Oxford Review of Education, 43 (6), 805 – 822.</bibtext> </blist> <blist> <bibtext> Lahtinen, T., & Lushaku, F. (2019). The gambler's fallacy and decision fatigue in decision-making: A study on driving license examiners. https://<ulink href="http://www.diva-portal.org/smash/get/diva2:1348232/FULLTEXT01.pdf">www.diva-portal.org/smash/get/diva2:1348232/FULLTEXT01.pdf</ulink></bibtext> </blist> <blist> <bibtext> Maw, J. (1995). The handbook for the inspection of schools: A critique. Cambridge Journal of Education, 25 (1), 75 – 87. https://doi.org/10.1080/0305764950250108</bibtext> </blist> <blist> <bibtext> OECD. (2015). Education at a glance 2015.</bibtext> </blist> <blist> <bibtext> Ofsted. (2017). Do two inspectors inspecting the same school make consistent decisions?</bibtext> </blist> <blist> <bibtext> Penninckx, M., & Vanhoof, J. (2015). Insights gained by schools and emotional consequences of school inspections. A review of evidence. School Leadership and Management, 35 (5), 477 – 501. https://doi.org/10.1080/13632434.2015.1107036</bibtext> </blist> <blist> <bibtext> Pepitone, A., & DiNubile, M. (1976). Contrast effects in judgments of crime severity and the punishment of criminal violators. Journal of Personality & Social Psychology, 33 (4), 448. https://doi.org/10.1037/0022-3514.33.4.448</bibtext> </blist> <blist> <bibtext> Perryman, J. (2007). Inspection and emotion. Cambridge Journal of Education, 37 (2), 173 – 190. https://doi.org/10.1080/03057640701372418</bibtext> </blist> <blist> <bibtext> Quintelier, A., Vanhoof, J., & De Maeyer, S. (2019). A full array of emotions: An exploratory mixed methods study of teachers' emotions during a school inspection visit. Studies in Educational Evaluation, 63, 83 – 93. https://doi.org/10.1016/j.stueduc.2019.07.006</bibtext> </blist> <blist> <bibtext> Richmond, T. (2019). Requires improvement: A new role for ofsted and school inspections. EDSK.</bibtext> </blist> <blist> <bibtext> Segerholm, C., & Hult, A. (2018). Learning from and reacting to school inspection–two Swedish case narratives. Scandinavian Journal of Educational Research, 62 (1), 125 – 139. https://doi.org/10.1080/00313831.2016.1212257</bibtext> </blist> <blist> <bibtext> Stewart, N., Chater, N., & Brown, G. D. (2006). Decision by sampling. Cognitive Psychology, 53 (1), 1 – 26. https://doi.org/10.1016/j.cogpsych.2005.10.003</bibtext> </blist> <blist> <bibtext> Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76 (2), 105. https://doi.org/10.1037/h0031322</bibtext> </blist> <blist> <bibtext> Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185 (4157), 1124 – 1131. https://doi.org/10.1126/science.185.4157.1124</bibtext> </blist> </ref> <aug> <p>By Christian Bokhove; John Jerrim; Maria Palma Carvajal and Sam Sims</p> <p>Reported by Author; Author; Author; Author</p> <p></p> <p>Christian Bokhove is Professor in Mathematics Education at Southampton Education School within the University of Southampton. He is a specialist on international comparisons in mathematics education, the use of technology, and innovative methodologies.</p> <p>John Jerrim is a Professor of Education and Social Statistics at the University College London Institute of Education (IOE). John's research interests include the economics of education, access to higher education, intergenerational mobility, cross-national comparisons and educational inequalities.</p> <p>Maria Palma is a Research Fellow at the University College London Institute of Education (IOE). Maria's research interests include the economics of education and educational inequalities.</p> <p>Sam Sims is an Associate Professor of Education at the University College London Institute of Education (IOE). Sam's research interests include education policy, teachers, teaching and teacher education.</p> </aug> <nolink nlid="nl1" bibid="bib21" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib10" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib18" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib12" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib14" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib20" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib22" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib27" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib13" firstref="ref15"></nolink> <nolink nlid="nl10" bibid="bib29" firstref="ref16"></nolink> <nolink nlid="nl11" bibid="bib17" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib30" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib31" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib15" firstref="ref24"></nolink> <nolink nlid="nl15" bibid="bib24" firstref="ref25"></nolink> <nolink nlid="nl16" bibid="bib19" firstref="ref28"></nolink> <nolink nlid="nl17" bibid="bib16" firstref="ref44"></nolink> <nolink nlid="nl18" bibid="bib25" firstref="ref45"></nolink> <nolink nlid="nl19" bibid="bib26" firstref="ref46"></nolink> <nolink nlid="nl20" bibid="bib23" firstref="ref47"></nolink> <nolink nlid="nl21" bibid="bib28" firstref="ref48"></nolink> <nolink nlid="nl22" bibid="bib11" firstref="ref49"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1500700
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Testing for Sequential Bias in School Inspections
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Christian+Bokhove%22">Christian Bokhove</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4860-8723">0000-0002-4860-8723</externalLink>)<br /><searchLink fieldCode="AR" term="%22John+Jerrim%22">John Jerrim</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5705-7954">0000-0001-5705-7954</externalLink>)<br /><searchLink fieldCode="AR" term="%22Maria+Palma+Carvajal%22">Maria Palma Carvajal</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8306-7515">0000-0001-8306-7515</externalLink>)<br /><searchLink fieldCode="AR" term="%22Sam+Sims%22">Sam Sims</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-5585-8202">0000-0002-5585-8202</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Oxford+Review+of+Education%22"><i>Oxford Review of Education</i></searchLink>. 2025 51(5):785-809.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 25
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Evaluation%22">Institutional Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Schools%22">Elementary Schools</searchLink><br /><searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Sequential+Approach%22">Sequential Approach</searchLink><br /><searchLink fieldCode="DE" term="%22Behavior+Patterns%22">Behavior Patterns</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22United+Kingdom+%28England%29%22">United Kingdom (England)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/03054985.2024.2410270
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0305-4985<br />1465-3915
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Inspectors are tasked with judging the quality of provision based on visits to schools. They conduct these inspections sequentially, completing one before moving on to the next. However, empirical research in a range of settings outside education suggests that prior judgements in a sequence can influence subsequent judgements, despite being logically irrelevant. We investigate whether school inspectors in England display such sequential bias by testing whether they judge similar schools differently, depending on the judgements they reached in prior inspections. We find only limited evidence of sequential bias in primary school inspections. In particular, an inspector reaching an 'Inadequate' judgement in their previous inspection is associated with a 42 per cent reduction in the odds of reaching another 'Inadequate' judgement in their next inspection. Only around 5 per cent of inspection judgements result in an 'Inadequate' and we do not find consistent evidence of sequential bias at other grades, meaning this bias only affects a small minority of judgements. We also do not find the same results for secondary schools, albeit in a much smaller sample.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1500700
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1500700
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/03054985.2024.2410270
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 25
        StartPage: 785
    Subjects:
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Institutional Evaluation
        Type: general
      – SubjectFull: Elementary Schools
        Type: general
      – SubjectFull: Bias
        Type: general
      – SubjectFull: Sequential Approach
        Type: general
      – SubjectFull: Behavior Patterns
        Type: general
      – SubjectFull: United Kingdom (England)
        Type: general
    Titles:
      – TitleFull: Testing for Sequential Bias in School Inspections
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Christian Bokhove
      – PersonEntity:
          Name:
            NameFull: John Jerrim
      – PersonEntity:
          Name:
            NameFull: Maria Palma Carvajal
      – PersonEntity:
          Name:
            NameFull: Sam Sims
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0305-4985
            – Type: issn-electronic
              Value: 1465-3915
          Numbering:
            – Type: volume
              Value: 51
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Oxford Review of Education
              Type: main
ResultId 1