'Phantom' Compositional Effects in English School Value-Added Measures: The Consequences of Random Baseline Measurement Error
Saved in:
| Title: | 'Phantom' Compositional Effects in English School Value-Added Measures: The Consequences of Random Baseline Measurement Error |
|---|---|
| Language: | English |
| Authors: | Perry, Thomas (ORCID |
| Source: | Research Papers in Education. 2019 34(2):239-262. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 24 |
| Publication Date: | 2019 |
| Document Type: | Journal Articles Reports - Descriptive |
| Education Level: | Secondary Education |
| Descriptors: | Value Added Models, Correlation, Individual Characteristics, Peer Influence, Error of Measurement, Simulation, Databases, Scores, Academic Ability, High Achievement, Selective Admission, Institutional Characteristics, Foreign Countries, Secondary School Students |
| Geographic Terms: | United Kingdom (England) |
| DOI: | 10.1080/02671522.2018.1424926 |
| ISSN: | 0267-1522 |
| Abstract: | A compositional effect is when pupil attainment is associated with the characteristics of their peers, over and above their own individual characteristics. Pupils at academically selective schools, for example, tend to out-perform similar-ability pupils who are educated with mixed-ability peers. Previous methodological studies however have shown that 'phantom' compositional effects can arise purely from measurement error. Through simulations using English National Pupil Database data, this paper demonstrates that moderate rates of measurement error in baseline scores can produce a systemic bias in value-added ('Progress') scores that favours higher ability pupils. This bias is large enough to explain the ostensibly superior performance of selective 'grammar' schools that is seen in the English data and create an appreciable bias across the school performance distribution. This paper describes how such biases can arise from 'regression attenuation' and discusses the more general methodological implications of the results. The paper concludes that the consequences of baseline measurement error on school value-added scores is greatly mitigated by controls for school-level prior attainment, but that the English 'Progress' value-added measures, as they do not control for prior attainment at school-level, are seriously biased and that the observed 'grammar school effect' is likely to be largely or wholly spurious. |
| Abstractor: | As Provided |
| Number of References: | 49 |
| Entry Date: | 2019 |
| Accession Number: | EJ1205325 |
| Database: | ERIC |
| Abstract: | A compositional effect is when pupil attainment is associated with the characteristics of their peers, over and above their own individual characteristics. Pupils at academically selective schools, for example, tend to out-perform similar-ability pupils who are educated with mixed-ability peers. Previous methodological studies however have shown that 'phantom' compositional effects can arise purely from measurement error. Through simulations using English National Pupil Database data, this paper demonstrates that moderate rates of measurement error in baseline scores can produce a systemic bias in value-added ('Progress') scores that favours higher ability pupils. This bias is large enough to explain the ostensibly superior performance of selective 'grammar' schools that is seen in the English data and create an appreciable bias across the school performance distribution. This paper describes how such biases can arise from 'regression attenuation' and discusses the more general methodological implications of the results. The paper concludes that the consequences of baseline measurement error on school value-added scores is greatly mitigated by controls for school-level prior attainment, but that the English 'Progress' value-added measures, as they do not control for prior attainment at school-level, are seriously biased and that the observed 'grammar school effect' is likely to be largely or wholly spurious. |
|---|---|
| ISSN: | 0267-1522 |
| DOI: | 10.1080/02671522.2018.1424926 |
