View in EDS HTML Full Text PDF Full Text

Is England's Office for Students Likely to Falsely Identify Courses as below Threshold on the B3 Progression Metric?

Saved in:
Bibliographic Details
Title: Is England's Office for Students Likely to Falsely Identify Courses as below Threshold on the B3 Progression Metric?
Language: English
Authors: Vladislav Areshka (ORCID 0009-0006-6369-9324), Alex Bradley (ORCID 0000-0003-4304-7653)
Source: Studies in Higher Education. 2025 50(7):1471-1487.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 17
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Foreign Countries, Outcomes of Education, College Graduates, Measurement, Classification, Confidence Testing, Employment Potential, Education Work Relationship, Educational Indicators, Institutional Evaluation, Public Agencies
Geographic Terms: United Kingdom, United Kingdom (England)
DOI: 10.1080/03075079.2024.2382258
ISSN: 0307-5079
1470-174X
Abstract: Rising inflation and slow growth globally have led to governments focussing on value for money, and in the sphere of higher education policy that is often translated into graduate outcomes. In England, the Office for Students (OfS) has created the progression metric which for full-time, first degree and UK domiciled students requires courses to have 60% of graduates with positive outcomes 15 months after graduation. This paper utilises simulations to evaluate the likelihood that the OfS's approach will fail to identify courses below the threshold (false positives) and incorrectly identify courses at or above the threshold (false negatives). The simulation adapts the levels of positive outcomes within the population (20-90%), sample size (40-1000), percentage sampled (30-90%) and crucially confidence intervals applied (90%, 95% and 99%) to identify courses below threshold. The three main findings are: the choice of 90% CI minimises the likelihood of false positives compared to 95% or 99% CIs; second, larger samples are essential to reduce likelihood of false positives; and third, preferring the 95% confidence interval instead of 99% for taking regulatory action will result in more courses incorrectly identified as below threshold. Governments that choose a statistical approach to regulate educational standards, like the application of confidence intervals to identify provision below quality standards, should utilise simulations to check the likelihood of false positives and false negatives since failure to do so could lead to both: (a) courses below standard not being identified and (b) regulatory action taken incorrectly against provision at or above standards.
Abstractor: As Provided
Notes: https://osf.io/pd43c/?view_only=bb46a701d69846d99c0518616737df4f
Entry Date: 2026
Accession Number: EJ1502810
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFS43lPT3qw9c0uRjE4ZbgeAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI24yrT6EdUerShY9wIBEICBm0tMOC9dR18sCC2_mBhIUgjviHb5TjXWhzLBK0TcrHa4TYyMRz6d14oVtKq5HKwQ6wLyfdF0OQaGZ1fIJeBFIbu_5oexcCZFSjzoLubWfXhzNEF18TPRRV3OylZ1IS20MnWAoRhiwkW7qgjpQc048HMVS_QbzEae6dotnQ9Gi3fWWPZglMk_uPyiEXhBWXZSDw84Zuv5ViaKAEy9
Text:
  Availability: 1
  Value: <anid>AN0186503156;she01jul.25;2025Jul11.03:29;v2.2.500</anid> <title id="AN0186503156-1">Is England's Office for Students likely to falsely identify courses as below threshold on the B3 progression metric? </title> <p>Rising inflation and slow growth globally have led to governments focussing on value for money, and in the sphere of higher education policy that is often translated into graduate outcomes. In England, the Office for Students (OfS) has created the progression metric which for full-time, first degree and UK domiciled students requires courses to have 60% of graduates with positive outcomes 15 months after graduation. This paper utilises simulations to evaluate the likelihood that the OfS's approach will fail to identify courses below the threshold (false positives) and incorrectly identify courses at or above the threshold (false negatives). The simulation adapts the levels of positive outcomes within the population (20-90%), sample size (40–1000), percentage sampled (30-90%) and crucially confidence intervals applied (90%, 95% and 99%) to identify courses below threshold. The three main findings are: the choice of 90% CI minimises the likelihood of false positives compared to 95% or 99% CIs; second, larger samples are essential to reduce likelihood of false positives; and third, preferring the 95% confidence interval instead of 99% for taking regulatory action will result in more courses incorrectly identified as below threshold. Governments that choose a statistical approach to regulate educational standards, like the application of confidence intervals to identify provision below quality standards, should utilise simulations to check the likelihood of false positives and false negatives since failure to do so could lead to both: (a) courses below standard not being identified and (b) regulatory action taken incorrectly against provision at or above standards.</p> <p>Keywords: Employability; OfS; B3 regulations; progression metric; graduate outcomes</p> <hd id="AN0186503156-2">Introduction</hd> <p>Over the last couple of years, countries around the world have experienced challenging economic conditions often with high inflation and slowing growth rates, which has led to an increased focus from the government and the public on the value for money (IMF [<reflink idref="bib17" id="ref1">17</reflink>]; ONS [<reflink idref="bib27" id="ref2">27</reflink>]). In higher education policy this has led to a drive to measure and evaluate the value for money of higher education. Whilst higher education has many benefits (e.g. often creating citizens who are more likely to volunteer, vote, engage in healthier lifestyles), the debate is often reduced to economic benefits of higher education, like the likelihood of graduates gaining a professional occupation (Brand [<reflink idref="bib4" id="ref3">4</reflink>]; De Walque [<reflink idref="bib10" id="ref4">10</reflink>]; Droomers et al. [<reflink idref="bib11" id="ref5">11</reflink>]; Feinstein and Duckworth [<reflink idref="bib13" id="ref6">13</reflink>]; Ma, Pender, and Welch [<reflink idref="bib18" id="ref7">18</reflink>]). One example of this approach is England's Office for Students (OfS) B3 progression metric which requires that all UK domiciled full time and first-degree graduate courses[<reflink idref="bib1" id="ref8">1</reflink>] have 60% of their students gain a positive outcome (OfS [<reflink idref="bib24" id="ref9">24</reflink>]) which is defined as professional occupation or further study[<reflink idref="bib2" id="ref10">2</reflink>] within 15 months of their degree (OfS [<reflink idref="bib23" id="ref11">23</reflink>]). Failure to attain the threshold could lead to financial penalties and, in the worst-case scenario, the removal of degree-awarding powers. We know from previous work by Bradley and Quigley ([<reflink idref="bib3" id="ref12">3</reflink>]) that the progression metric is at a population level accurate; however, the precision of its confidence intervals drops with smaller sample sizes. This imprecision makes classification and detection of underperforming courses increasingly unreliable the closer one gets to a threshold like 60% (leading to false positives). This analysis does not address the important question regarding how such imprecision impacts false negative misclassifications, or – to put more simply – whether a university provision will be investigated when it is actually above the threshold; nor Bradley and Quigley ([<reflink idref="bib3" id="ref13">3</reflink>]) tackle the issue of how regulatory preference of one confidence level to another may affect false positive and false negative rates.</p> <p>The present research aims to explore how the OfS's choices of confidence intervals for investigation and regulatory action impact the correct classification of courses as above or below the threshold. We know from a consultation with the sector (containing responses of 244 participants) that the OfS's selection of levels at which statistical confidence is set, handling of statistical significance and sample size were all important concerns raised within the consultation (Alma Economics [<reflink idref="bib1" id="ref14">1</reflink>]). These choices are of fundamental importance to providers who may receive financial and reputational damage should incorrect classification occur (i.e. identifying a provider below baseline despite their true population value being at or above a threshold). We shall now review international approaches to measuring and evaluating employability before looking in detail at England's approach to regulating employability, and outlining the current study.</p> <hd id="AN0186503156-3">Measuring employability internationally</hd> <p>There are a number of different metrics that can be used to assess employability from whether a graduate is employed or unemployed, to how much a graduate gets paid, if the work is professional (as for teachers, doctors, solicitors) or nonprofessional (i.e. customer service advisor, bar staff), and whether there are horizontal (e.g. graduates employed in an area unrelated to their studies) or vertical mismatches (such as a job which does not match the level of education attained) (McGuinness, Pouliakas, and Redmond [<reflink idref="bib19" id="ref15">19</reflink>]; Somers et al. [<reflink idref="bib29" id="ref16">29</reflink>]). These metrics are either tracked through administrative data collected by government bodies or through large-scale surveys of graduates. As an example of administrative data in the UK, the Longitudinal Educational Outcomes (LEO) dataset collects earnings one, three and five years after graduation (Department for Education [<reflink idref="bib7" id="ref17">7</reflink>], [<reflink idref="bib9" id="ref18">9</reflink>]). However, administrative data is unlikely to be able to tell us how meaningful graduates find their careers, the extent to which they utilise skills from their degree, or the indicators of the quality of the jobs they do – their level of autonomy, or amount of responsibilities, etc.</p> <p>To address these types of questions and many others, a large-scale survey can be conducted by governments, sector bodies or implemented by higher education institutions. For example, the EU has launched Eurograduate Pilot survey which sampled 21,000 bachelor and master's graduates from eight EU countries (Austria, Czechia, Croatia, Germany, Greece, Lithuania, Malta and Norway) with plans to have a comprehensive graduate tracking system – survey and administrative – across the European Education Area by 2025 (European Commission [<reflink idref="bib12" id="ref19">12</reflink>]; European Commission et al. [<reflink idref="bib20" id="ref20">20</reflink>], [<reflink idref="bib2" id="ref21">2</reflink>]). Data when gathered from administrative records or a survey is rarely inert; instead, it is often used to evaluate current educational practices and encourage change through different policy levers.</p> <hd id="AN0186503156-4">How employability is evaluated across different countries</hd> <p>Governments have a range of different mechanisms at their disposal to ensure that higher education institutions focus on the outcomes of their graduates: from soft touch, information-based approaches (i.e. providing employment rates or salaries of graduates from universities/courses to prospective students) to harder, more financial based approaches; we review three harder approaches.</p> <p>The first approach uses employability outcomes to directly impact the funding that higher education institutions receive. For example, in Denmark, funding of higher education programmes is dependent on graduate employment rates at 12–23 months after graduation, or in Finland tracking of graduate outcomes will determine 2% of the performance-based funding for universities (European Commission et al. [<reflink idref="bib2" id="ref22">2</reflink>]).</p> <p>The second approach uses labour market information to identify skills needs within the economy, which is then used to adjust the number of places on courses available or which impacts the costs of courses depending upon how sought for those graduates are in the labour market. For example, a consultation in 2022 by the English government suggested reducing the number of places available on courses which are known to have poor labour market outcomes (Department for Education [<reflink idref="bib8" id="ref23">8</reflink>]).</p> <p>The third approach is to design specific targets that universities and their courses have to meet or surpass in order to avoid financial or regulatory consequences, like fines or the removal of degree-awarding powers. The problem with setting targets or having performance-related funding when graduate outcomes are measured using self-report surveys (often with small samples of the population) is that their ability to estimate the true employability values within the population can be inaccurate, thus leading to incorrect imposition of fines or erroneous reductions to performance-based financial funding.</p> <hd id="AN0186503156-5">England's approach to measuring and regulating employability</hd> <p>This work focuses on the Office for Students (OfS) who regulate English HE providers. The OfS have in the past three years brought in a series of reforms, such as the B3 condition of registration, modified the Teaching Excellence Framework (TEF) and introduced Access and Participation Plans (APP) to address widening participation. These three reforms contain the progression metric; however, it is only the B3 that currently has scope to incur serious financial penalties and more existential threats (like the revoking of degree-awarding powers), hence this has become the focus of the current research.</p> <p>The progression metric is the percentage of graduates who are either in professional work, further study, or are caring, retired or travelling after 15 months. The thresholds applied to the progression metric vary by mode of study (full time, part time or apprenticeship) as well as by level of study (first degree, masters, other undergraduate, etc.) (OfS [<reflink idref="bib24" id="ref24">24</reflink>]). So, for example, a full-time first degree has a threshold of 60% in a positive outcome, which is also the centre of interest for the present study since it represents the largest population of students (in particular, 58% of the total student population in 2021/2022 academic year (HESA [<reflink idref="bib15" id="ref25">15</reflink>])) – as compared to a full-time Post Graduate Certificate in Education which has an 85% threshold (OfS [<reflink idref="bib24" id="ref26">24</reflink>]). To further complicate the picture, in addition to different thresholds by mode and level of study, there are also split indicators which include different subject areas of study, gender, ethnicity, disability and relative deprivation measures (OfS [<reflink idref="bib22" id="ref27">22</reflink>]).</p> <p>In all instances where a threshold exists, the OfS will begin investigation when the upper edge of the 90% confidence interval falls below a threshold and can take regulatory action when the upper 95% confidence interval is below the threshold (OfS [<reflink idref="bib22" id="ref28">22</reflink>]). The implications of such actions for English providers are serious, with potentially fines up to £500,000, restricting students access to finances, removing degree-awarding powers (OfS [<reflink idref="bib22" id="ref29">22</reflink>]). With more countries now collecting graduate outcomes data (European Commission et al. [<reflink idref="bib20" id="ref30">20</reflink>]), there will be a temptation for policymakers to possibly follow a path similar to the Office for Students in England, therefore it is essential that the impact of sampling error and sample size on the application of confidence intervals to adjudicate provision above or below thresholds are more clearly understood.</p> <hd id="AN0186503156-6">Current study</hd> <p>Previously, Bradley and Quigley ([<reflink idref="bib3" id="ref31">3</reflink>]) assessed accuracy of the composite Proceed metric (in the present research we refer directly to its revised 'Progression' constituent) through the following measures: bias (difference between sample and population estimate), coverage probability (proportion of true population estimates within confidence intervals) and precision (average range of confidence intervals). Three main findings were highlighted: (a) the Progression metric is accurate in terms of projected bias and coverage probability, (b) the estimates lack precision, especially with small sample sizes and (c) the imprecision will impact the Progression metric ability to correctly classify courses as below or above a set numeric threshold. However, this study did not identify the likelihood of false negatives which is crucial to understanding whether university provision will be falsely investigated or have regulator action taken against them. Second, this study does not ask the question of whether a 99% confidence interval would be more appropriate than the current levels of 90% and 95%; therefore, it was not in a position to provide alternatives to the OfS current paradigm.</p> <p>In light of Bradley and Quigley's analysis of how this inherent inexactness of statistical measurements and confidence intervals bears non-negligible challenges for the OfS to evaluate providers' performance effectively, the present study aims to explore the consequences of regulatory choices in setting thresholds for investigating and taking enforcement measures against universities for poor graduate outcomes. Thus, it investigates if and what changes can be deployed to mitigate related problems. Currently, the OfS uses a 90% confidence interval (CI) below a 60% threshold[<reflink idref="bib3" id="ref32">3</reflink>] to trigger an investigation and a 95% CI to provide evidence for taking regulatory action which can include financial penalties and even, at the extreme, revoking degree-awarding powers. The decision of the choice of a confidence interval is therefore an important consideration for the OfS and providers. Simulation of varying levels of confidence intervals are used to show how choice of 90%, 95% or 99% confidence levels influences risks of misclassifying courses:</p> <p></p> <ulist> <item> which fall below a 60% threshold for positive outcomes and</item> <p></p> <item> incorrectly investigating and acting against courses at or above the threshold.</item> </ulist> <p>First, we expect 99% CI will have a greater likelihood of misclassifying those courses under the 60% threshold (<emph>false positive</emph>) as being potentially above the threshold whilst 90% CI will have the lowest number of misclassifications.</p> <p>Second, we predict 90% CI will have the greatest likelihood of misclassifying courses above the threshold whilst 99% (<emph>false negative</emph>) will have the lowest likelihood.</p> <hd id="AN0186503156-7">Method</hd> <p></p> <hd id="AN0186503156-8">Simulation description</hd> <p>This study is a simulation study designed to replicate the OfS B3 progression metric. For the present simulation, we needed to reproduce three key variables used by Bradley and Quigley ([<reflink idref="bib3" id="ref33">3</reflink>]): the percentage with a positive outcome, population size and percentage sample. Each is outlined in more detail below.</p> <p>First, the percentage of graduates in the population with a positive outcome varied from 20% up to 95% increasing by 5% increments (20%, 25%, ... , 95%). These levels were chosen as limits because anything less than 20% was not likely to be identified as above the 60% target and anything greater than 95% would be unlikely to be identified as below the threshold.</p> <p>Second, we vary the population size from 40 students up to 1000 students going up in increments of 10. These limits were chosen as it was thought to be unlikely that courses below 40 students would be included as the OfS does not use samples less than 23 students (OfS [<reflink idref="bib23" id="ref34">23</reflink>]).</p> <p>Third, we varied the percentage sampled from the population from a minimum of 30%, as set by the OfS, to 90% going up in increments of 5% (30%, 35%, ... , 90%) (OfS [<reflink idref="bib23" id="ref35">23</reflink>]).</p> <hd id="AN0186503156-9">Simulation procedure</hd> <p>To create the simulation, we took as a model the simulation procedure and code produced in Bradley and Quigley ([<reflink idref="bib3" id="ref36">3</reflink>]). Similarly, we implemented the following three steps: data generation, sampling of the data and calculating statistics from each of the samples.</p> <p>Data generation involved the creation of 97 data frames each with 16 variables (columns). The 97 datasets, representing student outcomes, went from 40 rows in length up to 1000 rows long in ten increments per dataset (e.g. the second dataset had 50 rows). Each row symbolises a graduate with a positive outcome (<reflink idref="bib1" id="ref37">1</reflink>) or negative outcome (0). Each variable within a data frame represented a population with a certain percentage of positive outcomes. The first variable in a dataset had 20% of the rows with a 1 symbolising a population with 20% positive outcomes, whilst the second variable had 25% with a positive outcome, and this percentage increased by 5% in each variable till the 16th variable which had 95% with a positive outcome.</p> <p>The second step was to take a randomly selected sample, without replacement, from each of the 16 variables in each of the 97 datasets. This sample varied from 30% of the population to 90% of the population. This process was then repeated 100 times for each of the 16 variables in all 97 datasets.</p> <p>Finally, from each of the samples, four statistics were calculated. The first one is the percentage with a positive outcome/graduate job. Then, three levels of confidence intervals – 90%, 95% and 99%. The Jeffreys method has been used to calculate confidence intervals, since Jeffreys confidence intervals were chosen by the OfS as they are known to have favourable properties when estimating intervals on binomial proportions and when used on small samples (Brown, Cai, and DasGupta [<reflink idref="bib5" id="ref38">5</reflink>]; OfS [<reflink idref="bib23" id="ref39">23</reflink>]). Due to the OfS not using a student population of less than 23, any samples with less than this number were removed before analysis (20,800 cases were removed). The final dataset had 1,996,800 observations. Simulations were performed with R Studio (version 2023.06.2 Build 561) running R version 4.3.0 and using four packages ('tidyverse', 'DescTools', 'kableExtra', 'modelsummary'). All the data and code for simulation are available from the Open Science Framework (OSF): https://osf.io/pd43c/?view%5fonly = bb46a701d69846d99c0518616737df4f.</p> <hd id="AN0186503156-10">Confidence interval measure</hd> <p>Unlike Bradley and Quigley ([<reflink idref="bib3" id="ref40">3</reflink>]), in addition to the 90% and 95% confidence intervals we also included a 99% confidence interval and, crucially, calculated measures of false positives and false negatives which are described in more detail below.</p> <hd id="AN0186503156-11">False positives</hd> <p>In this study, false positive misclassifications describe the cases when an upper limit of a confidence interval, that is calculated for an estimate from a population with less than 60% with a positive outcome, crosses the 60% threshold, and so there could be not enough statistical certainty to believe that a provider's underlying performance can indeed be below the threshold. For instance, Figure 1 shows different estimates for a population with 50% in positive outcomes and their respective confidence intervals set at 90% level of statistical confidence. It can be observed that in 72 out of 100 cases, the upper limit of the confidence interval crosses the 60% threshold. Therefore, University 'X' could have an underperforming course with 50% of positive outcomes, yet – due to sampling and the quality of statistical evaluations around uncertainty levels – confidence intervals may fail to identify it correctly (or with strong enough statistical evidence) as below.</p> <p>Graph: Figure 1. This graph depicts 100 estimates with 90% CI from a population with 50% of positive outcomes and a sample size of 80 students from a course of 160 students. The figure shows how many samples would not be properly identified as below the 60% threshold.</p> <hd id="AN0186503156-12">False negatives</hd> <p>False negative misclassifications denote in this study the cases when a population level of positive outcomes is at or above the 60% threshold, but the sampled estimate and the upper limit of its respective confidence interval suggest the course is below the threshold. For example, Figure 2 has a population of 65% with positive outcomes and shows that 5 times out of 100 the samples with the upper limits of their 90% CIs falsely suggested that the underlying performance of the courses were below the threshold. Thus, University 'Y' could be wrongly identified as underperforming due to sampling when, in fact, the unknown level of positive outcomes was 65% – above the threshold.</p> <p>Graph: Figure 2. This graph depicts 100 estimates with 90% CI from a population with 65% of positive outcomes and a sample size of 108 students from a course of 360 students. Five samples would be identified as being below the 60% limit with an upper confidence interval below the 60% threshold.</p> <hd id="AN0186503156-13">Odds ratio</hd> <p>In the end, binomial logistic regression modelling was used to determine odds of false positives and false negatives at 90%, 95% and 99% levels of confidence intervals. In addition, for each type of misclassifications odds ratios across the three confidence intervals were calculated. This was done in order to quantify and provide a clearer explanation of the change in risk rates associated with the variables under study.</p> <hd id="AN0186503156-14">Results</hd> <p></p> <hd id="AN0186503156-15">False positive misclassification</hd> <p></p> <hd id="AN0186503156-16">Mean risk levels</hd> <p>Risks of false positive misclassifications present themselves at different degrees for different levels of confidence intervals: the lowest risks are recorded for 90% CI, while the highest – for 99% CI. In fact, Table 1 demonstrates that on average 8.22% of courses will have an upper confidence interval above the 60% threshold if 90% CI is applied to an estimation. When 95% CI is used to measure providers' performance, the mean value for such false positives will rise by 28.8% to 10.59, and if 99% CI is used for evaluation, then the percentage of false positives almost doubles and reaches 15.71%.</p> <p>Table 1. Demonstrates the number and percentage of false positives by confidence intervals.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td>Confidence Interval</td><td>N° total estimations</td><td>Mean CI range</td><td>N° false positives</td><td>% false positives</td></tr></thead><tbody><tr><td>90</td><td>998,400</td><td char=".">9.76</td><td>82,115</td><td char=".">8.22</td></tr><tr><td>95</td><td>998,400</td><td char=".">11.61</td><td>105,777</td><td char=".">10.59</td></tr><tr><td>99</td><td>998,400</td><td char=".">15.18</td><td>156,804</td><td char=".">15.71</td></tr></tbody></table> </ephtml> </p> <p>A logistic regression model was performed using 90, 95 and 99% CIs to predict the likelihood of misclassifying a course as being above the 60% threshold when it is not. Table 2 shows that choosing 95% CI over 90% CI as the measure of performance, the odds of misclassifying courses as being potentially above the threshold will grow on average by 32%. At the same time, odds of false positives are more than 2 times higher if 99% CI is set compared to the 90% CI.</p> <p>Table 2. Results of a logistic regression model predicting the likelihood of misclassification by 95% and 99% CIs compared to 90%.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td /><td>Odds ratio to 90% CI</td></tr></thead><tbody><tr><td>95% CI</td><td char=".">1.32</td></tr><tr><td /><td>SE = 0.006 (<italic>p</italic> = <.001)</td></tr><tr><td>99% CI</td><td char=".">2.08</td></tr><tr><td /><td>SE = 0.009 (<italic>p</italic> = <.001)</td></tr><tr><td>Num. Obs.</td><td>2,995,200</td></tr></tbody></table> </ephtml> </p> <hd id="AN0186503156-17">Risks of false positives using 90%, 95%, 99% confidence intervals with varying percentages in...</hd> <p>As the percentage in positive outcomes approaches the 60% threshold so do the risks of false positives. Table 3 shows that, when 95% CI is deployed for samples from the population level of 55% with a positive outcome, more than a half of courses (56.27%) will not be classified as being sufficiently below the 60% threshold. Using a lower 90% CI, this number will decrease to 45.13%. Setting a higher 99% level, instead, means that more than three quarters (76.41%) of courses will be classified as not having very strong statistical evidence of low performance. The issue of appropriate choice of confidence interval level is more apparently acute since, even for a population with as low as 45% of courses with a positive outcome (15% lower than the threshold), degrees of false positive classifications are identified to be non-trivial at 12.65% for the 99% CI, 7.78% for the 95% and 5.67% for the 90% CI.</p> <p>Table 3. Illustrates how the percentages in graduate jobs impact the number of false positives for 90%, 95% and 99% CIs.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td>% Population in graduate job</td><td>N° total estimations</td><td>N° false positives 99CI</td><td>% false positives 99CI</td><td>N° false positives 95CI</td><td>% false positives 95CI</td><td>N° false positives 90CI</td><td>% false positives 90CI</td></tr></thead><tbody><tr><td>20</td><td>124,800</td><td char=".">30</td><td char=".">0.02</td><td char=".">4</td><td char=".">0.00</td><td char=".">1</td><td char=".">0.00</td></tr><tr><td>25</td><td>124,800</td><td char=".">69</td><td char=".">0.06</td><td char=".">15</td><td char=".">0.01</td><td char=".">4</td><td char=".">0.00</td></tr><tr><td>30</td><td>124,800</td><td char=".">765</td><td char=".">0.61</td><td char=".">224</td><td char=".">0.18</td><td char=".">109</td><td char=".">0.09</td></tr><tr><td>35</td><td>124,800</td><td char=".">1840</td><td char=".">1.47</td><td char=".">564</td><td char=".">0.45</td><td char=".">282</td><td char=".">0.23</td></tr><tr><td>40</td><td>124,800</td><td char=".">8028</td><td char=".">6.43</td><td char=".">3972</td><td char=".">3.18</td><td char=".">2456</td><td char=".">1.97</td></tr><tr><td>45</td><td>124,800</td><td>15,788</td><td char=".">12.65</td><td char=".">9712</td><td char=".">7.78</td><td char=".">7073</td><td char=".">5.67</td></tr><tr><td>50</td><td>124,800</td><td>34,920</td><td char=".">27.98</td><td char=".">21060</td><td char=".">16.88</td><td>15,864</td><td char=".">12.71</td></tr><tr><td>55</td><td>124,800</td><td>95,364</td><td char=".">76.41</td><td char=".">70226</td><td char=".">56.27</td><td>56,326</td><td char=".">45.13</td></tr></tbody></table> </ephtml> </p> <hd id="AN0186503156-18">Risks of false positives using small sample sizes</hd> <p>Figure 3 demonstrates a significant inability for small sample sizes (especially, of 100 or less) to accurately identify courses below the 60% threshold, even when the graduate outcomes are as low as 40 or 45% with positive outcomes. For example, when 45% have a positive outcome within the population with a sample of 50 or less, there is an 80% chance that a course will not be correctly identified as being below the threshold if 99% CI is chosen. Setting lower confidence levels however, such as 95% and 90%, a lesser degree of considerably underperforming courses will remain undetected – respectively, 61% and 50%. Yet, the numbers remain very high when the percentage with positive outcomes gets closer to the 60% threshold. Even using 90% CI for estimation of populations with 55% of students in positive outcomes, 87% of courses with sample sizes of less than 50 students and three quarters (74%) of those comprising 51–100 will not provide enough statistical certainty to be classified as below the threshold. Under these circumstances, deployment of 99% confidence level would lead to almost all of the 'low-value' courses (98.73% for 24–50 sample size and 94.4% for 51–100) being unidentified.</p> <p>Graph: Figure 3. These graphs depict the percentage of false positive cases by confidence interval and sample size that is sampled from the population. Each graph represents the percentage with a positive outcome in the population starting from 40% top left to 55% bottom right. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI.</p> <p>Figure 3 and Figure 4 clearly show how in many situations distinctively higher rates of false positive classifications are for 99% CI, while 90% and 95% CIs are at much closer levels. Nevertheless, in certain occasions increasingly higher sample sizes grant similarly good results in false-positive performance regardless of a confidence interval level applied. For example, when there are 50% of positive outcomes in a population, sample sizes over 550 help identify markedly as below the threshold more than 99% of courses even if 99% CI is set. At 45% of graduates with positive outcomes the same result is achieved already with sample sizes over 350. Yet, any similar extent of considerable reduction and assimilation in CI efficiency is hardly detectable in case of various response rates (as represented by percentages of populations sampled in Figure 4).</p> <p>Graph: Figure 4. These graphs depict the percentage of false positive cases by confidence interval and percentage of the population that is sampled. Each graph represents the percentage with a positive outcome/graduate job in the population starting from 40% top left to 55% bottom right. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI.</p> <p>Overall, across populations with different levels of attaining a positive outcome, there are increasing rates of false positives from 90% to 95% to 99% CIs. For instance, Table 4 highlights that on average, when a population with 55% of positive outcomes is sampled, using 99% CI rather than 90% will increase the odds of false positives by almost 4 times, while applying 95% CI – only by a factor of 1.56. Even for courses lying farther from the 60% threshold, such as with 50% of positive outcomes in a population, a similar picture is observed: the respective values of growth in odds for 95% and 99% CIs as compared to 90% are 1.39 and 2.67.</p> <p>Table 4. Results of two logistic regression models on positive outcomes at 50% and 55% in the population with 90%, 95% and 99% CIs.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td /><td>Odds ratio to 90% CI: 50% of a positive outcome</td><td>Odds ratio to 90% CI: 55% of a positive outcome</td></tr></thead><tbody><tr><td>95% CI</td><td>1.39</td><td char=".">1.56</td></tr><tr><td /><td>SE = 0.016 (<italic>p</italic> = <.001)</td><td>SE = 0.013 (<italic>p</italic> = <.001)</td></tr><tr><td>99% CI</td><td>2.67</td><td char=".">3.94</td></tr><tr><td /><td>SE = 0.028 (<italic>p</italic> = <.001)</td><td>SE = 0.035 (<italic>p</italic> = <.001)</td></tr><tr><td>Num. Obs.</td><td>374,400</td><td>374,400</td></tr></tbody></table> </ephtml> </p> <hd id="AN0186503156-19">False negatives</hd> <p></p> <hd id="AN0186503156-20">Mean risk levels</hd> <p>Table 5 shows that risks of false negative misclassifications represent much lower levels in comparison to false positive classifications. When 90% CI is selected, only 0.57% of courses will be treated incorrectly as those falling below the 60% threshold; while measuring their performance via 95% and 99% CIs, the amount of such misclassifications will be merely 0.24% and 0.03% respectively. Notably, this false negative phenomenon presents an inverted dependency on confidence intervals used upon estimations as opposed to the cases of false positives: the higher the level of a confidence interval is, the lower is the rate of misclassifications.</p> <p>Table 5. Demonstrates the number and percentage of false negatives by confidence intervals on samples with 60% or above in positive outcomes.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td>Confidence Interval</td><td>N° total estimations</td><td>Mean CI range</td><td>N° false negatives</td><td>% false negatives</td></tr></thead><tbody><tr><td>90</td><td>998,400</td><td char=".">9.76</td><td char=".">5656</td><td char=".">0.57</td></tr><tr><td>95</td><td>998,400</td><td char=".">11.61</td><td char=".">2372</td><td char=".">0.24</td></tr><tr><td>99</td><td>998,400</td><td char=".">15.18</td><td char=".">258</td><td char=".">0.03</td></tr></tbody></table> </ephtml> </p> <hd id="AN0186503156-21">Risks of false negatives using 90%, 95%, 99% confidence intervals for percentages with positi...</hd> <p>The risk of misclassifying courses as underperforming reaches quite tangible numbers when percentages of a population with a positive outcome are very close to the threshold. For instance, Table 6 demonstrates that rates of false negatives will be zero or close to zero for all three levels of confidence intervals when the populations of 70% or even 65% of graduates with positive outcomes are sampled. However, for the population of 60% with a positive outcome the usage of 90% CI will markedly increase the risk to 4.41%. 95% CI observes 1.86% of misclassifications and 99% CI – only 0.2%.</p> <p>Table 6. Illustrates how the number of false negatives fluctuates for 90%, 95% and 99% CIs depending on the percentages in graduate jobs.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td>% Population in graduate job</td><td>N° total estimations</td><td>N° false negatives 99CI</td><td>% false negatives 99CI</td><td>N° false negatives 95CI</td><td>% false negatives 95CI</td><td>N° false negatives 90CI</td><td>% false negatives 90CI</td></tr></thead><tbody><tr><td>60</td><td>124,800</td><td char=".">255</td><td char=".">0.2</td><td char=".">2327</td><td char=".">1.86</td><td char=".">5502</td><td char=".">4.41</td></tr><tr><td>65</td><td>124,800</td><td char=".">3</td><td char=".">0.0</td><td char=".">43</td><td char=".">0.03</td><td char=".">148</td><td char=".">0.12</td></tr><tr><td>70</td><td>124,800</td><td char=".">0</td><td char=".">0.0</td><td char=".">2</td><td char=".">0.00</td><td char=".">6</td><td char=".">0.00</td></tr><tr><td>75</td><td>124,800</td><td char=".">0</td><td char=".">0.0</td><td char=".">0</td><td char=".">0.00</td><td char=".">0</td><td char=".">0.00</td></tr><tr><td>80</td><td>124,800</td><td char=".">0</td><td char=".">0.0</td><td char=".">0</td><td char=".">0.00</td><td char=".">0</td><td char=".">0.00</td></tr><tr><td>85</td><td>124,800</td><td char=".">0</td><td char=".">0.0</td><td char=".">0</td><td char=".">0.00</td><td char=".">0</td><td char=".">0.00</td></tr><tr><td>90</td><td>124,800</td><td char=".">0</td><td char=".">0.0</td><td char=".">0</td><td char=".">0.00</td><td char=".">0</td><td char=".">0.00</td></tr><tr><td>95</td><td>124,800</td><td char=".">0</td><td char=".">0.0</td><td char=".">0</td><td char=".">0.00</td><td char=".">0</td><td char=".">0.00</td></tr></tbody></table> </ephtml> </p> <p>Figure 5 demonstrates that selecting 90% CI for different sample sizes can even lead to the evaluation of more than 5% of courses as underperforming and, in case of our simulation, this choice resulted in up to 7.71% of misclassifications (for samples between 551 and 600). Unlike in the case of false positive classifications, various levels of sample sizes or percentages of a population sampled (see Figure 6) presented less apparent impact on false negative misclassifications, decreasing at higher sample sizes and response rates. Yet, importantly, Figures 5 and 6 illustrate how the false negative rates are greatly reduced with the deployment of 95% CI and almost completely eliminated when 99% CI is applied.</p> <p>Graph: Figure 5. These graphs illustrate the percentage of false negative cases by confidence interval and sample size that is sampled from the population. The two graphs represent the percentage with a positive outcome/graduate job in the population: 60% top and 65% bottom. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI; the straight line (━) – 1% error for 99% CI.</p> <p>Graph: Figure 6. These graphs illustrate the percentage of false negative cases by confidence interval and the percentage of the population that is sampled. The two graphs represent the percentage with a positive outcome/graduate job in the population: 60% top and 65% bottom. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI; the straight line (━) – 1% error for 99% CI.</p> <p>Table 7 summaries that, in general, deploying 95% CI we experienced a decrease of 58.2% in the odds of false negative misclassification, while a reduction of the odds of 95.5% is observed when setting 99% CI.</p> <p>Table 7. Results of a logistic regression model predicting the likelihood of false negatives by 95% and 99% CIs as compared to 90%.</p> <p> <ephtml> <table><thead valign="bottom"><tr><td /><td>Odds ratio to 90% CI</td></tr></thead><tbody><tr><td>95% CI</td><td char=".">0.418</td></tr><tr><td /><td>SE = 0.010 (<italic>p</italic> = <.001)</td></tr><tr><td>99% CI</td><td char=".">0.045</td></tr><tr><td /><td>SE = 0.003 (<italic>p</italic> = <.001)</td></tr><tr><td>Num. Obs.</td><td>2,995,200</td></tr></tbody></table> </ephtml> </p> <hd id="AN0186503156-22">Discussion</hd> <p>The OfS's approach to using statistical measures when considering a provider's performance has not faced unanimous support: a considerable share of respondents to their consultation on the metrics in question either disagreed with the approach (17%), or abstained from declaring their position (13%) – in total, almost a third of all respondents (Alma Economics [<reflink idref="bib1" id="ref41">1</reflink>]). Therefore, our research tries to address prevalent doubts with regard to the OfS's numeric thresholds and indicators, bearing in mind that the choice by the OfS of parameters determining the progression metric is of substantial importance both for higher education providers and for the regulatory body itself. For the former, the confidence interval applied could make the difference between financial penalties and even revocation of degree-awarding powers though universities' effective results, without sample error, may meet the established targets. For the OfS, inaccurate calculations could mean an inability to identify low-value courses resulting for the government in poor returns on investment in funding students on these courses. This simulation provides three main findings: (<reflink idref="bib1" id="ref42">1</reflink>) the 90% CI is better at reducing false positives; (<reflink idref="bib2" id="ref43">2</reflink>) the size of the sample matters since larger samples see reductions in false positives; (<reflink idref="bib3" id="ref44">3</reflink>) 99% CI can substantially reduce false negatives. These three findings will now be discussed in more detail.</p> <p>One prominent theme within the OfS's sector consultation centred around the handling of statistical uncertainty with respondents frequently questioning the choice of 90% and 95% as being too low to be optimal (Alma Economics [<reflink idref="bib1" id="ref45">1</reflink>]). Our simulation directly addresses this commonly raised concern showing that the 90% CI delivers better results in terms of prevention of false positive cases. 95% and 99% CIs due to wider ranges of values will usually have a greater likelihood of misclassifying courses as potentially above an established threshold while their underlying performance is below. This illustrates that the regulator by choosing the 90% level as the criterion for launching investigation of underperforming providers should be able to more accurately discern courses as those below the threshold. On average, using 90% CI instead of 99% CI will result in an almost two-fold decrease in unidentified courses, and they will reduce by a third when the 90% level is preferred to 95%. This guarantees a greater degree of regulatory efficiency at the first stage of intervention (i.e. time and money saved on investigating courses).</p> <p>The consultation with the sector also found a number of respondents voicing concerns around small samples (especially from smaller providers or those 'providing highly specialist/technical courses where small cohorts are unavoidable') leading to unfair judgements of outcomes (Alma Economics [<reflink idref="bib1" id="ref46">1</reflink>], 44). The results from the simulation validate these concerns and highlight the importance of larger samples to reduce the risk of false positives. For example, for sample sizes between 24 and 50 students of the population with 50% in positive outcomes, the rate of false positives ranged widely from 71.42% (90% CI) to 90.41% (99% CI), and the equivalent range was from 49.13% (90% CI) to 79.21% (99% CI) with samples having 51–100 students. But for sample sizes comprising from 551 to 600 respondents, the number of false positive misclassifications was below 1% even with 99% CI. Small sample sizes – in particular, those involving less than 100 respondents – incur significant challenges in identifying courses that have fallen below the threshold. This issue is particularly pressing considering that the latest OfS data for the progression metric based on 2017–2020 surveys of courses at all levels and modes of study (excluding those with no outcome indicator values) revealed 47% of samples being under 100 students. This finding has two implications: first, every effort possible should be made to enhance the response rates to the graduate outcomes surveys which have over recent years seen declining numbers of respondents (HESA [<reflink idref="bib16" id="ref47">16</reflink>]). One way to do this is through the use of financial incentives to increase response rates (Church [<reflink idref="bib6" id="ref48">6</reflink>]; Singer et al. [<reflink idref="bib28" id="ref49">28</reflink>]). Second, regulators ought to be wary of trying to regulate at micro-levels, such as courses, since their ability to identify courses' true population value accurately will be impaired (Bradley and Quigley [<reflink idref="bib3" id="ref50">3</reflink>]). Inherently, smaller universities and courses are disadvantaged since, in view of the statistical nature of the progression metric, their survey results are more prone to be erroneous.</p> <p>Respondents to the consultation were right to highlight that lower confidence intervals can at times have adverse consequences (Alma Economics [<reflink idref="bib1" id="ref51">1</reflink>]). For example, the simulations show that choosing 99% CI instead of 95% to take concrete regulatory actions against courses would help to reduce from 2% to 0.2% the risk of false negatives. As it currently stands, if we estimate using the last four years of data across all modes and levels of study, split indicators for each provider across all the years of the survey from the OfS data dashboard ([<reflink idref="bib25" id="ref52">25</reflink>]) and select progression indicator values between 0-1% above threshold, we estimate 534 cases. Using the 95% CI, 10 would be incorrectly identified as below the level – compared to 1 if 99% CI were used. That means nine additional providers (or multiple areas within a provider) could have part of their provision incorrectly identified as below the threshold who could be liable to investigation and, if these did not come back positively, financial and regulatory penalties.</p> <p>It should be noted that in between identification and financial/regulatory penalties the OfS does conduct an investigation with inspectors on the ground (for example, see OfS [<reflink idref="bib26" id="ref53">26</reflink>]), nevertheless, the procedures this follows are yet to be fully spelt out. It would seem prudent for the OfS to adopt 99% CI instead of 95% CI to provide greater confidence and trust within its regulatory framework since it can't be sure which area of provision that is below the threshold with 95% CI is below due to sample error, as opposed to their real population value falling short of the threshold.</p> <hd id="AN0186503156-23">Limitations</hd> <p>Whilst this research accounts for sampling error, it should be recognised that measurement error is also important for accurate assessment (Mokkink et al. [<reflink idref="bib21" id="ref54">21</reflink>]), and we know from an evaluation conducted by HESA and the Office for National Statistics that there was around a 10% error in positive outcomes (HESA [<reflink idref="bib14" id="ref55">14</reflink>]) – in this case, professional occupations – being categorised as a negative outcome (i.e. non-professional job). Measurement error would add further noise to false positives and false negatives, therefore this work could well represent a lower bound to these phenomena.</p> <p>Finally, a broad point is that the OfS's conceptualisation of a positive outcome primarily in terms of professional occupation or further study is rather narrow since the benefits of HE extend well beyond attaining professional-level jobs: contributing to citizens being more socially and politically conscious, leading healthier lifestyles, engaging in more volunteering and providing more educational activities for their children (Ma, Pender, and Welch [<reflink idref="bib18" id="ref56">18</reflink>]). It would be useful to create a more comprehensive measure of the value of HE that would be holistically designed to capture a broad range of outcomes beyond professional-level employment.</p> <hd id="AN0186503156-24">Conclusions and recommendations</hd> <p>The findings of this article address important concerns raised by the sector on the management of statistical uncertainty, the challenge of smaller samples and the accuracy with which smaller provider results and smaller areas of provision can be understood. The results showed the 90% confidence interval was a good choice as an investigation threshold since it represents a reasonable level of certainty that the expected value lies within its range and minimises false positives. But the simulated effects of sample sizes and the latest graduate outcomes survey results suggest that the metric is to be dealt with care. The simulation clearly highlights that the best way to minimise false negatives is to use the 99% confidence interval which helps to fairly adjudicate where provision is at the threshold. If the OfS continues to use the 95% confidence level, around 2% of courses at the threshold – as the simulation study indicates – would be incorrectly identified as below which, given financial and reputational consequences of investigation, ought not to be taken lightly. Therefore, we recommend that the OfS continues to use the 90% threshold for investigation, but adopts the 99% threshold for regulatory action. We also recommend that the OfS works closely with Jisc/HESA who run the graduate outcomes survey to ensure actions are taken to boost response rates which have been consistently falling since the 2019/2020 release.</p> <p>Another major contribution this article aspires to provide is demonstration of how the simulation technique can be a valuable and useful tool to inspect the implications of higher education policy decisions in real time and cost-effective manner. In the field of higher education policy, resorting to simulations could help to clearly showcase the relative trade-offs of different policy decisions and provide an empirical evidence base for those policy decisions.</p> <hd id="AN0186503156-25">Author contributions</hd> <p>VA is the guarantor. AB and VA created the simulator data. VA analysed the data. VA and AB drafted the manuscript. All authors read, provided feedback and approved the final version of the manuscript.</p> <hd id="AN0186503156-26">Consent to publish</hd> <p>All authors approved the final manuscript before it was sent off for review.</p> <hd id="AN0186503156-27">Data availability statement</hd> <p>The dataset and code to reproduce the results are available from the Open Science Framework at: https://osf.io/pd43c/?view%5fonly = bb46a701d69846d99c0518616737df4f.</p> <hd id="AN0186503156-28">Ethics approval</hd> <p>No ethics approval was required since no data was collected.</p> <hd id="AN0186503156-29">Disclosure statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <ref id="AN0186503156-30"> <title> Notes </title> <blist> <bibl id="bib1" idref="ref8" type="bt">1</bibl> <bibtext> Courses are assessed at the Common Aggregation Hierarchy 3 level. For more details see: https://<ulink href="http://www.hesa.ac.uk/support/documentation/hecos/cah">www.hesa.ac.uk/support/documentation/hecos/cah</ulink></bibtext> </blist> <blist> <bibl id="bib2" idref="ref10" type="bt">2</bibl> <bibtext> Those graduates that are caring, retired or travelling are also counted as a positive outcome (OfS [23]).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref12" type="bt">3</bibl> <bibtext> As said above, threshold levels vary by mode and level of study. However, for readability the 60% threshold is chosen as this applies to first-degree, full-time students who make up the majority of the student population.</bibtext> </blist> </ref> <ref id="AN0186503156-31"> <title> References </title> <blist> <bibtext> Alma Economics. 2022. "A New Approach to Regulating Student Outcomes: Analysis of Consultation Responses." https://<ulink href="http://www.officeforstudents.org.uk/media/7301/alma%5fstudent%5foutcomes.pdf">www.officeforstudents.org.uk/media/7301/alma%5fstudent%5foutcomes.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> European Commission, Shane Beadle, Patricia Vale, Sigrid Mannsberger-Nindl, Arthur Hannah, Ali Zaidi, Claudia Abdallah and Andrea Kottmann. 2020b. "Mapping the State of Graduate Tracking Policies and Practices in the EU Member States and EEA Countries." https://doi.org/10.2766/09070.</bibtext> </blist> <blist> <bibtext> Bradley, Alex, and Martyn Quigley. 2023. " Governments Harnessing the Power of Data to Get 'Value for Money': A Simulation Study of England's Office for Students B3 Proceed Metric." Studies in Higher Education 48 (8): 1289 – 302. https://doi.org/10.1080/03075079.2023.2196292.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref3" type="bt">4</bibl> <bibtext> Brand, Jennie E. 2010. " Civic Returns to Higher Education: A Note on Heterogeneous Effects." Social Forces 89 (2): 417 – 33. https://doi.org/10.1353/sof.2010.0095.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref38" type="bt">5</bibl> <bibtext> Brown, Lawrence D., T. Tony Cai, and Anirban DasGupta. 2001. " Interval Estimation for a Binomial Proportion." Statistical Science 16 (2): 101 – 33. https://doi.org/10.1214/ss/1009213285.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref48" type="bt">6</bibl> <bibtext> Church, Allan H. 1993. " Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis." Public Opinion Quarterly 57 (1): 62 – 79. https://doi.org/10.1086/269355.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref17" type="bt">7</bibl> <bibtext> Department for Education. 2017. "Employment and Earnings Outcomes of Higher Education Graduates by Subject and Institution: Experimental Statistics Using the Longitudinal Education Outcomes (LEO) Data." https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/718225/SFR_18_2017_LEO_mainText.pdf.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref23" type="bt">8</bibl> <bibtext> Department for Education. 2023. "Higher Education Policy Statement and Reform: Government Consultation Response." https://assets.publishing.service.gov.uk/media/6516a49b6a423b0014f4c5fe/CP876_Higher_Education_policy_statement_and_reform_government_consultation_response_web_.pdf.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref18" type="bt">9</bibl> <bibtext> Department for Education. 2024. "LEO Graduate Outcomes Provider Level Data." Department for Education: Explore Education Statistics. https://explore-education-statistics.service.gov.uk/find-statistics/graduate-outcomes-leo-provider-level-data/2021-22.</bibtext> </blist> <blist> <bibtext> De Walque, Damien. 2004. Education, Information, and Smoking Decisions Evidence from Smoking Histories, 1940–2000. Washington, DC : World Bank Group. <ulink href="http://documents.worldbank.org/curated/en/408411468760767791/Education-information-and-smoking-decisions-evidence-from-smoking-histories-1940-2000">http://documents.worldbank.org/curated/en/408411468760767791/Education-information-and-smoking-decisions-evidence-from-smoking-histories-1940-2000</ulink>.</bibtext> </blist> <blist> <bibtext> Droomers, Mariël, Carola T. M. Schrijvers, Karien Stronks, Dike van de Mheen, and Johan P. Mackenbach. 1999. " Educational Differences in Excessive Alcohol Consumption: The Role of Psychosocial and Material Stressors." Preventive Medicine 29 (1): 1 – 10. https://doi.org/10.1006/pmed.1999.0496.</bibtext> </blist> <blist> <bibtext> European Commission. 2021. "European Graduate Tracking Initiative Demystified." https://data.europa.eu/doi/10.27665990.</bibtext> </blist> <blist> <bibtext> Feinstein, Leon, and Kathryn Duckworth. 2006. "Are There Effects of Mothers' Post-16 Education on the Next Generation?: Effects on Children's Development and Mothers' Parenting." https://discovery.ucl.ac.uk/id/eprint/10005969/1/Feinstein2006Are.pdf.</bibtext> </blist> <blist> <bibtext> HESA (Higher Education Statistics Authority). 2021. Graduate Outcomes Soc Coding - Independent Verification Analysis Report. https://<ulink href="http://www.hesa.ac.uk/files/Graduate-Outcomes-SOC-coding-Independent-verification-analysis-report-20210429.pdf">www.hesa.ac.uk/files/Graduate-Outcomes-SOC-coding-Independent-verification-analysis-report-20210429.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> HESA (Higher Education Statistics Authority). 2023a. "Who's Studying in HE?" HESA. https://<ulink href="http://www.hesa.ac.uk/data-and-analysis/students/whos-in-he">www.hesa.ac.uk/data-and-analysis/students/whos-in-he</ulink>.</bibtext> </blist> <blist> <bibtext> HESA (Higher Education Statistics Authority). 2023b. "Graduate Outcomes 2020/21: Summary Statistics – Summary." HESA. https://<ulink href="http://www.hesa.ac.uk/news/31-05-2023/sb266-higher-education-graduate-outcomes-statistics">www.hesa.ac.uk/news/31-05-2023/sb266-higher-education-graduate-outcomes-statistics</ulink>.</bibtext> </blist> <blist> <bibtext> IMF (International Monetary Fund). 2023. "World Economic Outlook: Navigating Global Divergences." https://<ulink href="http://www.imf.org/-/media/Files/Publications/WEO/2023/October/English/text.ashx">www.imf.org/-/media/Files/Publications/WEO/2023/October/English/text.ashx</ulink>.</bibtext> </blist> <blist> <bibtext> Ma, Jennifer, Matea Pender, and Meredith Welch. 2016. "Education Pays 2016: The Benefits of Higher Education for Individuals and Society." https://files.eric.ed.gov/fulltext/ED572548.pdf.</bibtext> </blist> <blist> <bibtext> McGuinness, Seamus, Konstantinos Pouliakas, and Paul Redmond. 2018. " Skills Mismatch: Concepts, Measurement and Policy Approaches." Journal of Economic Surveys 32 (4): 985 – 1015. https://doi.org/10.1111/joes.12254.</bibtext> </blist> <blist> <bibtext> European Commission, Christoph Meng, Katarina Wessling, Kai Mühleck, and Martin Unger. 2020a. "Eurograduate Pilot Survey: Design and Implementation of a European Graduate Survey." https://doi.org/10.2766/149071.</bibtext> </blist> <blist> <bibtext> Mokkink, Lidwine B., Iris Eekhout, Maarten Boers, Cees P. M. van der Vleuten, and Henrica C. W. de Vet. 2023. " Studies on Reliability and Measurement Error of Measurements in Medicine – From Design to Statistics Explained for Medical Researchers." Patient Related Outcome Measures 14 : 193 – 212. https://doi.org/10.2147/prom.s398886.</bibtext> </blist> <blist> <bibtext> OfS (Office for Students). 2022a. "Consultation on a New Approach to Regulating Student Outcomes: Analysis of Responses to Consultation and Decisions." https://<ulink href="http://www.officeforstudents.org.uk/media/7362/regulating-student-outcomes-analysis-of-responses-reformatted.pdf">www.officeforstudents.org.uk/media/7362/regulating-student-outcomes-analysis-of-responses-reformatted.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> OfS (Office for Students). 2022b. "Description of Student Outcome and Experience Measures Used in OfS Regulation: Definition of Measures and Methods Used to Construct and Present Them." https://<ulink href="http://www.officeforstudents.org.uk/media/8060/description-of-student-outcome-and-experience-measures.pdf">www.officeforstudents.org.uk/media/8060/description-of-student-outcome-and-experience-measures.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> OfS (Office for Students). 2022c. "Setting Numerical Thresholds for Condition B3." https://<ulink href="http://www.officeforstudents.org.uk/media/7522/setting-minimum-numerical-thresholds-for-condition-b3.pdf">www.officeforstudents.org.uk/media/7522/setting-minimum-numerical-thresholds-for-condition-b3.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> OfS (Office for Students). 2023a. "Student Outcomes Data Dashboard." Accessed November 30, 2023. https://<ulink href="http://www.officeforstudents.org.uk/data-and-analysis/student-outcomes-data-dashboard/get-the-data/">www.officeforstudents.org.uk/data-and-analysis/student-outcomes-data-dashboard/get-the-data/</ulink>.</bibtext> </blist> <blist> <bibtext> OfS (Office for Students). 2023b. "Quality Assessments." Office for Students. https://<ulink href="http://www.officeforstudents.org.uk/for-providers/quality-and-standards/quality-assessments">www.officeforstudents.org.uk/for-providers/quality-and-standards/quality-assessments</ulink>.</bibtext> </blist> <blist> <bibtext> ONS (Office for National Statistics). 2022. "Global Inflation: 1970 to 2022." Office for National Statistics. https://<ulink href="http://www.ons.gov.uk/economy/inflationandpriceindices/articles/globalinflation/1970to2022">www.ons.gov.uk/economy/inflationandpriceindices/articles/globalinflation/1970to2022</ulink>.</bibtext> </blist> <blist> <bibtext> Singer, Eleanor, John van Hoewyk, Nancy Gebler, Trivellore Raghunathan, and Katherine McGonagle. 1999. " The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys." Journal of Official Statistics 15 (2): 217 – 30.</bibtext> </blist> <blist> <bibtext> Somers, Melline A., Sofie J. Cabus, Wim Groot, and Henriëtte Maassen van den Brink. 2019. " Horizontal Mismatch Between Employment and Field of Education: Evidence From a Systematic Review." Journal of Economic Surveys 33 (2): 567 – 603. https://doi.org/10.1111/joes.12271.</bibtext> </blist> </ref> <aug> <p>By Vladislav Areshka and Alex Bradley</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib17" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib27" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib10" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib11" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib13" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib18" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib24" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib23" firstref="ref11"></nolink> <nolink nlid="nl9" bibid="bib19" firstref="ref15"></nolink> <nolink nlid="nl10" bibid="bib29" firstref="ref16"></nolink> <nolink nlid="nl11" bibid="bib12" firstref="ref19"></nolink> <nolink nlid="nl12" bibid="bib20" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib15" firstref="ref25"></nolink> <nolink nlid="nl14" bibid="bib22" firstref="ref27"></nolink> <nolink nlid="nl15" bibid="bib16" firstref="ref47"></nolink> <nolink nlid="nl16" bibid="bib28" firstref="ref49"></nolink> <nolink nlid="nl17" bibid="bib25" firstref="ref52"></nolink> <nolink nlid="nl18" bibid="bib26" firstref="ref53"></nolink> <nolink nlid="nl19" bibid="bib21" firstref="ref54"></nolink> <nolink nlid="nl20" bibid="bib14" firstref="ref55"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1502810
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Is England's Office for Students Likely to Falsely Identify Courses as below Threshold on the B3 Progression Metric?
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Vladislav+Areshka%22">Vladislav Areshka</searchLink> (ORCID <externalLink term="https://orcid.org/0009-0006-6369-9324">0009-0006-6369-9324</externalLink>)<br /><searchLink fieldCode="AR" term="%22Alex+Bradley%22">Alex Bradley</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4304-7653">0000-0003-4304-7653</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Studies+in+Higher+Education%22"><i>Studies in Higher Education</i></searchLink>. 2025 50(7):1471-1487.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 17
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Outcomes+of+Education%22">Outcomes of Education</searchLink><br /><searchLink fieldCode="DE" term="%22College+Graduates%22">College Graduates</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement%22">Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink><br /><searchLink fieldCode="DE" term="%22Confidence+Testing%22">Confidence Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Employment+Potential%22">Employment Potential</searchLink><br /><searchLink fieldCode="DE" term="%22Education+Work+Relationship%22">Education Work Relationship</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Indicators%22">Educational Indicators</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Evaluation%22">Institutional Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Public+Agencies%22">Public Agencies</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22United+Kingdom%22">United Kingdom</searchLink><br /><searchLink fieldCode="DE" term="%22United+Kingdom+%28England%29%22">United Kingdom (England)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/03075079.2024.2382258
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0307-5079<br />1470-174X
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Rising inflation and slow growth globally have led to governments focussing on value for money, and in the sphere of higher education policy that is often translated into graduate outcomes. In England, the Office for Students (OfS) has created the progression metric which for full-time, first degree and UK domiciled students requires courses to have 60% of graduates with positive outcomes 15 months after graduation. This paper utilises simulations to evaluate the likelihood that the OfS's approach will fail to identify courses below the threshold (false positives) and incorrectly identify courses at or above the threshold (false negatives). The simulation adapts the levels of positive outcomes within the population (20-90%), sample size (40-1000), percentage sampled (30-90%) and crucially confidence intervals applied (90%, 95% and 99%) to identify courses below threshold. The three main findings are: the choice of 90% CI minimises the likelihood of false positives compared to 95% or 99% CIs; second, larger samples are essential to reduce likelihood of false positives; and third, preferring the 95% confidence interval instead of 99% for taking regulatory action will result in more courses incorrectly identified as below threshold. Governments that choose a statistical approach to regulate educational standards, like the application of confidence intervals to identify provision below quality standards, should utilise simulations to check the likelihood of false positives and false negatives since failure to do so could lead to both: (a) courses below standard not being identified and (b) regulatory action taken incorrectly against provision at or above standards.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/pd43c/?view_only=bb46a701d69846d99c0518616737df4f
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1502810
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1502810
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/03075079.2024.2382258
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 17
        StartPage: 1471
    Subjects:
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Outcomes of Education
        Type: general
      – SubjectFull: College Graduates
        Type: general
      – SubjectFull: Measurement
        Type: general
      – SubjectFull: Classification
        Type: general
      – SubjectFull: Confidence Testing
        Type: general
      – SubjectFull: Employment Potential
        Type: general
      – SubjectFull: Education Work Relationship
        Type: general
      – SubjectFull: Educational Indicators
        Type: general
      – SubjectFull: Institutional Evaluation
        Type: general
      – SubjectFull: Public Agencies
        Type: general
      – SubjectFull: United Kingdom
        Type: general
      – SubjectFull: United Kingdom (England)
        Type: general
    Titles:
      – TitleFull: Is England's Office for Students Likely to Falsely Identify Courses as below Threshold on the B3 Progression Metric?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Vladislav Areshka
      – PersonEntity:
          Name:
            NameFull: Alex Bradley
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0307-5079
            – Type: issn-electronic
              Value: 1470-174X
          Numbering:
            – Type: volume
              Value: 50
            – Type: issue
              Value: 7
          Titles:
            – TitleFull: Studies in Higher Education
              Type: main
ResultId 1