View in EDS HTML Full Text PDF Full Text

Treason and Revenge: The Emergence and Continuation of ILSA Contracting

Saved in:
Bibliographic Details
Title: Treason and Revenge: The Emergence and Continuation of ILSA Contracting
Language: English
Authors: Camilla Addey (ORCID 0000-0001-8431-5568)
Source: Critical Studies in Education. 2025 66(3):411-428.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 18
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Secondary Education
Elementary Secondary Education
Descriptors: International Assessment, Contracts, Networks, Educational Assessment, Educational Testing, Social Capital, Cultural Capital, Ethnography, Trust (Psychology), National Competency Tests, Communities of Practice, Educational History, Achievement Tests, Foreign Countries, Secondary School Students, Elementary Secondary Education, Mathematics Tests, Mathematics Achievement, Science Tests, Science Achievement
Assessment and Survey Identifiers: National Assessment of Educational Progress, Program for International Student Assessment, Trends in International Mathematics and Science Study
DOI: 10.1080/17508487.2024.2357120
ISSN: 1750-8487
1750-8495
Abstract: OECD and IEA International Large-Scale Assessments (ILSAs) contractors have remained largely invisible despite playing a key role in the making of education assessment data. With the help of Bourdieu's thinking, this paper analyses data generated through document analysis of ILSA Reports and in-depth interviews with staff working on ILSAs at the IEA, the OECD and ILSA contractors. The paper traces education assessment developments in the USA and at the IEA and the OECD since the 1980s, and points to the critical role of Education Testing Services (ETS). Although the selection of ILSA contractors ostensibly occurs through a globally competitive tendering process, this paper reveals the pressures and struggles that emerged when economic capital became available at the IEA and the OECD. The struggles reveal PISA was created as a competing project to TIMSS. The paper demonstrates how the social and cultural capitals that individuals acquired, and in particular trust, shaped who can and who cannot produce ILSA data today. Overall, the paper makes visible the ILSA data chefs and their recipes: ILSAs are not raw, they are cooked under pressure with personal and organizational tensions, struggles, conflicts, and emotional bonds.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1493726
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHuSE_u66EYLX4Lt6MQF29dAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDAg7YaNZA2Ri1zTDVAIBEICBm--1DIErxWAEPckOIHfjpZJjbyx6iWHu97gulUTSHk45CoEael-flbNFb4F3OhZgmBMshSMxz12ZoSzbzQZ7WZ0daiiL7MuEddH6W_1qig19kFV3i6hBao-Ap2UleRepp9yAM-P1Qz201Inb949RZ7LFHLmNhXfCTejztL-gVds-amhgBPf2bpBUpeNVHmoLX6Nhsbxxm5DkZDDg
Text:
  Availability: 1
  Value: <anid>AN0186746490;[30nj]01aug.25;2025Jul22.02:41;v2.2.500</anid> <title id="AN0186746490-1">Treason and revenge: the emergence and continuation of ILSA contracting </title> <p>OECD and IEA International Large-Scale Assessments (ILSAs) contractors have remained largely invisible despite playing a key role in the making of education assessment data. With the help of Bourdieu's thinking, this paper analyses data generated through document analysis of ILSA Reports and in-depth interviews with staff working on ILSAs at the IEA, the OECD and ILSA contractors. The paper traces education assessment developments in the USA and at the IEA and the OECD since the 1980s, and points to the critical role of Education Testing Services (ETS). Although the selection of ILSA contractors ostensibly occurs through a globally competitive tendering process, this paper reveals the pressures and struggles that emerged when economic capital became available at the IEA and the OECD. The struggles reveal PISA was created as a competing project to TIMSS. The paper demonstrates how the social and cultural capitals that individuals acquired, and in particular trust, shaped who can and who cannot produce ILSA data today. Overall, the paper makes visible the ILSA data chefs and their recipes: ILSAs are not raw, they are cooked under pressure with personal and organizational tensions, struggles, conflicts, and emotional bonds.</p> <p>Keywords: ILSAs; IEA; OECD; Bourdieu; contracting; network ethnography; trust</p> <hd id="AN0186746490-2">Introduction</hd> <p>Inspired by Critical Data Studies, this paper takes a critical approach to the making of education assessment data and infrastructures (the instruments and processes through which data are made). Critical Data Studies challenges the view that data are neutral and objective on the basis that this ignores the role of larger power structures which become embedded in data. Inspired by Gitelman's ([<reflink idref="bib9" id="ref1">9</reflink>]) rejection of 'raw' data, a key tenet of Critical Data Studies is that data are 'never raw, but always cooked to some recipe by chefs embedded within institutions that have certain aspirations and goals and operate within wider frameworks' (Kitchin & Lauriault, [<reflink idref="bib15" id="ref2">15</reflink>]). In particular, Critical Data Studies highlights the wide array of interests vis-à-vis data collection and processing, and invite scholars to investigate how different actors, their interests, and their struggles become embedded in these very data. Kitchin and Lauriault ([<reflink idref="bib15" id="ref3">15</reflink>]) and Williamson ([<reflink idref="bib28" id="ref4">28</reflink>]) suggest focusing on the privileged positions of business interests and asking questions about who can and who cannot produce data (and, by implication, knowledge). It is striking that the contractors that develop ILSA infrastructures and data went unstudied until recently (Addey, [<reflink idref="bib2" id="ref5">2</reflink>], [<reflink idref="bib3" id="ref6">3</reflink>]).</p> <p>Addey ([<reflink idref="bib2" id="ref7">2</reflink>]) has shown ILSA contractors carry out cut-even or loss ILSA contracts. Their aims are to develop improved or new products and processes, gain access to networks of potential buyers, manage human capital, create direct business opportunities, and fulfil individual rationales such as professional opportunities, lifestyles, and emotional bonds. In other words, ILSA contracts are vested in goals and aspirations. The importance of this paper lies in its critical contribution to understanding how contractors' goals and aspirations and consequent struggles become embedded in ILSA infrastructures and data. This paper reveals the 'data chefs' hidden away in their privileged kitchens where they assemble their 'ILSA recipes'. It responds to questions about who can make ILSA data and infrastructure and what struggles are involved. The importance of this work is twofold. Firstly, by shedding light on actors who create ILSA infrastructure and data and showing the processes of their involvement, the paper holds up a mirror to ILSA contractors and contracting. Secondly, this paper builds on the assumption that education policies and practices are transferred in the making of education data and infrastructures (in other words, before the data is even analysed) Morgan and Shahjahan ([<reflink idref="bib21" id="ref8">21</reflink>]), hence the importance of a critical analysis of the different actors and their involvement in these processes.</p> <p>Bottani ([<reflink idref="bib6" id="ref9">6</reflink>], [<reflink idref="bib7" id="ref10">7</reflink>]) was one of the first scholars to critically address the making of OECD education data. He showed the extent to which the development of assessment frameworks is political, controversial, full of tensions and power struggles, methodologically limited, creative, and subjective. Focusing specifically on the making of PISA, Gorur has shown how 'constraints were addressed, interests translated, categories defined, classifications negotiated, frameworks agreed upon, choices made, methodologies established and protocols developed' ([<reflink idref="bib10" id="ref11">10</reflink>], p. 578). Also analysing the making of PISA, Morgan ([<reflink idref="bib18" id="ref12">18</reflink>], [<reflink idref="bib19" id="ref13">19</reflink>]) states that the education assessment community struggled 'to render their science factual' (p. 52). In particular, she describes the struggles of early ILSA experts as they dealt with technical difficulties and controversies that were temporarily stabilized: how they came under extensive political pressures and how political compromises were reached. Morgan states that the science of international educational measurement is a 'societal construction' (p. 245), that ILSAs are socially constructed objects 'created through a negotiated process by an elite group of experts and policymakers' (p. 245). However, no scholarship has addressed an important group of those involved in the making of ILSA infrastructure and data, ILSA contractors. This paper focuses on this elite group of actors. Although the main ILSAs are administered by the International Association for the Evaluation of Educational Achievement (IEA) and the Organization for Economic Cooperation and Development (OECD), ILSAs are mostly developed, implemented, and analysed by ILSA contractors – a well-hidden group of actors (Addey, [<reflink idref="bib2" id="ref14">2</reflink>]). Their invisibility to date is surprising since ILSA contractors are <emph>the</emph> actors taking daily mundane decisions about how to construct ILSA infrastructures and data, and they are potentially driven by business interests as opposed to education as a public good.</p> <p>In the next section, the paper explores how the thinking of Bourdieu helps make sense of the way ILSA contracting emerged and how ILSA contractor struggles stabilised, followed by a methodological account of the data generating and analysis processes. The paper then examines: 1) the development of new assessment approaches in the USA in the 1980s and the first large-scale assessments at IEA; 2) how ILSAs at IEA came under pressure to keep up with assessment developments in the USA, which were then picked up in adult literacy ILSAs at the OECD; and 3) how individuals and organizations that had previously collaborated on academic projects won the first ILSA contracts by virtue of accumulated capitals. The paper analyses how interpersonal struggles shaped the emergence of PISA and its contractors. Finally, an analysis of how PISA was pressured into diversifying ILSA contractors shows how the legacy of the early days continues to shape current ILSAs.</p> <hd id="AN0186746490-3">Theoretical and methodological approach</hd> <p>This paper brings together Critical Data Studies and a Bourdieuian conceptual frame. CDS motivated the critical approach and shaped the focus of this paper (i.e. if data are not neutral, then who made them?; what were their recipes?; what were their interests?). Bourdieu's work resonates with the CDS approach, whilst providing theoretical tools to analyse the way data is made (if data are not neutral, how did some actors become privileged in the process? And, how can we unpack the process through which some actors make data and dominate this field?). Together CDS and Bourdieu allow the paper to critically engage with the making of data, highlighting struggles that have less to do with what is worth measuring in education and more to do with larger power structures and competing interests.</p> <p>I draw on Bourdieu's ([<reflink idref="bib8" id="ref15">8</reflink>]) concept of <emph>field</emph>— described as dynamic social space of struggles with its own laws of functioning and unequally distributed power (there are relationships of inequality between dominated and dominating actors) – to understand how contactors first came together and relate. In the Bourdieuian conceptual frame, the unifying principle is the struggle to transform or preserve the field; the task of the researcher is to describe it. Struggles result in position-taking, which is progressively instituted and contains a whole past and potential history, and the actors' resulting strategies. Actors compete for control over their interests (Lingard et al., [<reflink idref="bib16" id="ref16">16</reflink>]). What is at stake in the struggle to win ILSA contracts is the power to define assessment methodologies and 'delimit the population of those entitled to take part in the struggle' (Bourdieu, [<reflink idref="bib8" id="ref17">8</reflink>], 43). The field is protected by tacit conditions and terms of entry (i.e. a certain cultural capital) which are practically required (knowledge of what actually matters in bids) or explicitly codified and legally guaranteed (i.e. through a bidding process).</p> <p>How the struggle evolves amongst actors does not depend exclusively on the authority inherent in each actor's recognition, consecration, and prestige. Their authority depends on their habitus: a set of predispositions that generate a set of practices and perceptions. These can be understood as an internal law, a 'feel for the game' that guides actors' actions and reactions. Each actor's habitus depends on different kinds and levels of social, economic, and cultural capital: respectively, actors' networks; monetary resources; and forms of cultural knowledge, competences, and dispositions (in other words, knowing the right cultural codes, what works, and how to behave). These require a long process of acquisition. Scholars have described trust as one dimension of social capital (Bakker et al., [<reflink idref="bib4" id="ref18">4</reflink>]; Gubbins & MacCurtain, [<reflink idref="bib11" id="ref19">11</reflink>]; Putnam, [<reflink idref="bib27" id="ref20">27</reflink>]). Huang and Wilkinson ([<reflink idref="bib13" id="ref21">13</reflink>]) describe cognitive trust as the 'evaluation of the competence, responsibility, and dependability' (p. 456) of actors, and affective trust as deriving from emotional bonds and 'the belief that an exchange partner cares about your welfare, will act positively towards it and take care to avoid harming it' (p. 456). The acquisition of capital and the role of trust are central to the emergence of ILSA contracting.</p> <p>This paper used Ball's ([<reflink idref="bib5" id="ref22">5</reflink>]) network ethnography, an approach which requires mapping, following, questioning, and visiting people and nodal actors, their lives, stories, conflicts, money, and things. Junemann et al. describe it as focusing on 'the content, nature, and meaning of the exchanges and transactions between network participants, the roles, actions, motivations, discourses, and resources of the different actors involved ([<reflink idref="bib5" id="ref23">5</reflink>], p. 539). Rather than simply identifying which actors were involved, understanding the struggles at play in ILSA contracting required digging into the relationships, how they formed, how they evolved, and how they continued to shape each other ('O'Connor & Addey, [<reflink idref="bib22" id="ref24">22</reflink>]).</p> <p>In practice, we[<reflink idref="bib1" id="ref25">1</reflink>] set out to identify all contractors, consultants, and experts for OECD and IEA ILSAs from 1990 to 2020,[<reflink idref="bib2" id="ref26">2</reflink>] with particular attention to the three ILSAs with the most implementations: PISA (seven completed cycles), TIMSS (seven cycles), and PIRLS (four cycles). Information was drawn from <emph>Technical Reports</emph> published for each ILSA implementation. In a spreadsheet with sheets for each ILSA, we marked individuals credited in the technical report, their institutional affiliation, and their official roles. This allowed us to see at a glance, on an organisational and individual level, which contractors were involved and how their participation remained stable or changed over time. We sought to identify struggles through points of rupture or discontinuity, but also repeated collaborations.</p> <p>To supplement the mapping, we made a visual representation of the ILSA network (Figures 1 and 2). Visual representations are a combination of social network analysis <emph>techniques</emph> with network ethnography (as also done in Ball, [<reflink idref="bib5" id="ref27">5</reflink>]). The visualisation process was intended to display the actors involved, represent patterns of continuity, interruptions, or irregularities in relationships, and compare the network across assessment implementations and over time (ex., OECD vs. IEA assessments). We used NodeXL software to create network diagrams. This 'works' by entering information about actors (nodes) and relationships (connections) – collected from technical reports, interviews, and internet sources – into a spreadsheet. The NodeXL software determined what data could and must be entered. We used a directional graph with arrows flowing in the direction of payment (from the contracting organisation towards the contracted one) to represent the existence of a contractual relationship. Arrow width represents the importance or significance of a relationship, calculated by the number of distinct contracts between two actors.</p> <p>Graph: Figure 1. All contractors over all OECD ILSA implementations.</p> <p>Graph: Figure 2. All contractors over all IEA ILSA implementations.</p> <p>While we mapped and visualised all actors mentioned in the Technical Reports, we chose to interview actors who had been involved in the network the longest. Addey carried out approximately 35 interviews with high-level staff at IEA, OECD, and ILSA contractors. Open-ended interview questions focused on interviewees' perceptions and experience of the network, relationships, and struggles amongst actors. Interviewees are identified by randomly assigned letters or institutional affiliation to ensure anonymity is not compromised.</p> <p>The main theoretical concepts used in the project design – like struggles, habitus, and capitals – were used to code the data with NVivo, a software for data analysis. Further to these concepts, there were a number of repeated ideas that emerged. For example, interviewees described unethical behaviours like stealing ideas, plagiarism, and corruption in tendering processes. These themes were also used in the data coding process. Reading extracts by theoretical code and theme helped identify a story-one of inter-personal and inter-organizational struggles and emotional bonds, as presented in the next sections.</p> <hd id="AN0186746490-4">The development of assessment methodologies in the USA in the 1980s</hd> <p>In 1969, the National Assessment of Educational Progress (NAEP) was carried out in the United States by the Education Commission for the States (ECS). It was one of the first learning assessments in the world. In the late 1970s, the U.S. Department of Education called for a new design after a review of NAEP. Educational Testing Services (known by its abbreviation, ETS, is the world's largest private non-profit educational testing and assessment organization) replaced the former NAEP contractor in the 1980s and re-developed NAEP using new methodologies. ETS chose to use Item Response Theory (IRT) scaling of learning outcomes and connected outcomes to variables in background questionnaires.</p> <p>So the first big change is the introduction of the methodologies that were used in NAEP in the 1980s, because they [ETS] introduced IRT scaling; they introduced what has come to be known as population modelling and plausible values; because of the scaling that was done it was possible to connect or to look at the relationships between cognitive performance and background questionnaires. So those methodologies resulted in what I would describe as a key inflection point. [...] The real shift was toward this information that became much more relevant, much more useful to policy-makers. [...] It's not the methodology that drove the change, it was the fact that policy-makers were interested in an emerging set of questions Interviewee R</p> <p>This extract describes the methodological changes in NAEP as an inflection point in the history of education assessment. The interviewee, who was involved in this historical shift, states that it was not the methodology that drove this change but policy-makers' growing interest in questions about learning outcomes and their relationship to students' background. This coincided with the growing interest in policies that are now described as 'evidence-based'. What underlies this key inflection moment, is a struggle in which ETS was able to define <emph>the</emph> education assessment methodology. From NAEP, as we will see, IRT scaling spread to other NLSAs and then ILSAs, allowing ETS to acquire a dominant position in the field.</p> <p>These developments in learning assessments in the USA coincided with the publication of <emph>A Nation at Risk</emph> in 1983. The report resulted in a crisis about learning outcomes in the USA and a growing interest in the achievements of students completing college. The then-Secretary of Education, Terrence Bell, provided ETS with funding to develop and implement the Young Adult Literacy Survey (YALS). YALS drew on the experience of NAEP and adopted its methodologies. Still in the USA, in 1992 the YALS was followed by the development of the National Adult Literacy Survey (NALS) which tested 16 to 65 year-olds. With the redevelopment of NAEP at ETS and the spread of NAEP's methodologies to YALS and NALS, psychometric scaling of achievement and background data became recognized as the most rigorous and policy relevant approach in education assessment. By privileging and spreading this approach, ETS now played an influential role in delimiting who could be part in the field of international education assessment. Tacit terms of entry into this field were being set; terms that would come into play when funding became available at IEA in the early 1990s. The methodologies used in NAEP, YALS and NALS became an exclusive area of expertise, a capital that few organizations around the world could deploy. At the time, the only big organizations with psychometric scaling expertise were ETS, ACER, and possibly IEA. Ultimately, ETS was establishing who would make education assessment infrastructure and data over the next thirty years.</p> <hd id="AN0186746490-5">ILSAs at IEA: how a community of practice came together and split</hd> <p>According to a number of interviewees involved in the IEA's early ILSAs, the Six-Subject Study and the Reading Literacy Study (1990/1991) were the result of a network of passionate academics with limited funding but big, comparative, research ideas in education (see also Morgan, [<reflink idref="bib19" id="ref28">19</reflink>]; Pizmony-Levy, [<reflink idref="bib25" id="ref29">25</reflink>]; Pizmony-Levy et al., [<reflink idref="bib26" id="ref30">26</reflink>]). The development, implementation, and analysis of IEA ILSAs depended mostly on personal networks and researchers' good will. Costs were absorbed by university departments and national research centres where staff donated time. Interviewees state that it was unsustainable for IEA to continue with little funding because it meant not integrating the methodological and technological developments in education assessment taking place in the U.S. and, more specifically, at ETS.</p> <p>'There were lots of technological developments that were happening, both in terms of computing and in terms of scaling methodologies [...], which in many ways happened at ETS. [...] We had to find ways of paying for those'. Interviewee G</p> <p>The network of passionate scholars at IEA was under pressure to obtain financial capital and integrate assessment developments ongoing at ETS. IEA's ILSAs were starting to lack the recognition that ETS methodologies had now achieved. The position of ETS in education assessment was starting to define the position of all actors in the international education assessment field, including those who had not adopted ETS methodologies.</p> <p>Until August 1993, the main IEA TIMSS study centre was based at the International Coordinating Center at the University of British Columbia. However, it emerged that the little funds that were available at IEA were going into '<emph>flying around the world drinking good Burgundy'</emph> (<emph>Interviewee I</emph>) and a '<emph>very scathing evaluation' (Interviewee I)</emph> was published.[<reflink idref="bib3" id="ref31">3</reflink>] This report fed into struggles in the international education assessment field, allowing actors to be displaced whilst others took up new positions. It was at this stage that the U.S. Department of Education, through the National Center for Education Statistics (NCES), funded TIMSS entirely. The funding amounted to 10 million USD, '<emph>which 25 years ago was a lot of money' (Interviewee P)</emph>, and allowed a transition from a network of academics working together '<emph>to do what they think is good work'</emph> to more <emph>'robustly-funded activities that actually permitted the hiring of companies to do the work' (Interviewee T)</emph>. Interestingly, the financial capital the IEA was under pressure to obtain came from the USA, where ETS is based, and it was attached to three conditions:</p> <p></p> <ulist> <item> TIMSS had to be carried out to high standards (i.e. adopt the latest assessment developments);</item> <p></p> <item> Actors had to have the know-how (i.e. the U.S. Department of Education would oversee the choice of contractors); and</item> <p></p> <item> One of the study centres had to be in the USA (thus maintaining the position of U.S. contractors).</item> </ulist> <p>The conditions guaranteed some actors a dominant position in the international education assessment field. It is reasonable to state that through these conditions, the U.S. government consolidated the position of ETS and its methodological choices.</p> <p>During the time it took for TIMSS to be moved, Andreas Schleicher and Ray Adams '<emph>kept the project running' (Interviewee T)</emph> from the IEA. Andreas Schleicher, was a key figure at IEA, where he started the IEA Data Processing Center in Hamburg and replaced the international coordinator, Neville Postlethwaite. A future competitor of Andreas Schleicher, Al Beaton, was on the Steering Committee of the IEA Reading Literacy Study. When TIMSS was moved, Schleicher left IEA to set up PISA at the OECD, marking the beginning of a competitive relationship between the IEA and the OECD. Ray Adams, like others, moved between both communities as he carried out a TIMSS contract through ACER, whilst assisting Andreas Schleicher in developing PISA and becoming a PISA contractor. At Boston College, TIMSS was under the leadership of Al Beaton (formerly based at ETS), Ina Mullis (formerly based at ETS, with extensive NAEP experience), and Michael Martin (Ireland's National Research Coordinator at the start of TIMSS). IEA was now part of the ETS international education assessment field, its habitus redefined. Alternative education assessment infrastructures and data were no longer desirable at IEA, the ETS-privileged assessment approach had been adopted.</p> <p>The community of practice that grew around IEA's first ILSAs had now split (Morgan, [<reflink idref="bib19" id="ref32">19</reflink>]). Within the network that had assembled around the first IEA ILSAs, there had been substantive theoretical and methodological debate on testing theories, indicated by the struggle between IEA's first contractors, the conditions attached to the 10 million USD contract, and the competing PISA project. As funding became available to the IEA community of practice and contracts were signed, passionate debates were settled and methodological choices defined who was in and who was out of the international education assessment field. The former loosely structured academic projects started to resemble more formalized political and business ventures. Some academics and national experts involved in IEA studies were employed by IEA. Many interviewees describe a natural transition from pro-bono basis work or consultancies to bigger contracts. For example, one interviewee describes contractors as the '<emph>people who were there, that we were working with, and we knew were trustworthy, and took care' (Interviewee P)</emph>, showing the value of social and cultural capitals in relationships. Trust appears to have played a key role in the international education assessment field. Many interviewees state that contracts were not based on an evaluation of competence, responsibility, and dependability; these qualities were assumed based on emotional bonds. They add that specific individuals drove the choice of contractor organizations, highlighting the role of interpersonal, affective trust built during the first IEA ILSAs and then extended to individuals' affiliations. For example, Ray Adams, who had been involved in IEA ILSAs as the National Research Coordinator for Australia, described how he was contracted to do scaling of achievement data because he had the required capitals. As Ray Adams was based at the Australian Council for Educational Research (ACER), the contract was established with ACER, and the organization eventually took on the technical lead.</p> <p>The first TIMSS was carried out by Boston College, the IEA Secretariat and the IEA Data Processing and Research Centre, ETS, Westat,[<reflink idref="bib4" id="ref33">4</reflink>] Statistics Canada, and ACER. It was with U.S. funding and the first consortium of contractors that the IEA adopted the technological developments that had come to dominate education assessment in the USA. The US economic capital was used to establish which cultural and social capitals counted in TIMSS; as the approach to educational assessment was now being set beyond the US borders. Trust stands out as a key component of social capital when TIMSS contracts were first sealed: it was not only who had the exclusive expertise, but who was trustworthy. As the international education assessment field started consolidating with the first ILSA contracts, it becomes evident that philosophical and methodological questions were not determining the positions of actors: there were other aspirations and goals underpinning how education came to be measured and acted upon. One of these ambitions is clearly the power to define who can produce data about education worldwide and how that is done.</p> <hd id="AN0186746490-6">How developments in education assessment continued to travel</hd> <p>Preoccupation with poor science results in the USA emerged with the IEA's Six-Subject Study, and led the US government to pressure the OECD into developing comparative education indicators (what became INES, the OECD's Indicators of Education Statistics in 1988) (Martens [<reflink idref="bib17" id="ref34">17</reflink>]; Ydesen & Grek [<reflink idref="bib29" id="ref35">29</reflink>]). Interviewees also describe Albert Tuijnman, one of the founders of INES, as captivated by the idea of the OECD playing a role in developing an ILSA. In 1990, the Canadian Survey of Literacy Skills Used in Daily Activities (LSUDA) had been developed with ETS methodologies in French and English, demonstrating valid results across languages. Following LSUDA's shocking results, Statistics Canada wanted to carry out a similar assessment in multiple languages. Together, the YALS, NALS, and LSUDA led to the OECD's first ILSA: the Adult Literacy and Lifeskills Survey (IALS, then renamed ALL). It was decided that the ILSA would be developed as an OECD ILSA due to its ability to bring governments together, showing the need to pool different capitals. ETS, on the other hand, had the ability to obtain economic capital from the US government. An interviewee described how an ETS staff member's connections in the US government were used to obtain funding from the National Center for Education Statistics and the Department of Labor, whilst Statistics Canada obtained funding from the Canadian government. Merging their different capitals, Statistics Canada, ETS and the OECD obtained the funding and global reach they were seeking. At this early stage in ILSA contracting, ETS appeared to already have an established position in this field as it generated and distributed economic capital, chose methodological approaches, and defined terms of entry. The approach to educational assessment privileged at ETS in the 1980s in the redevelopment of NAEP had travelled to national assessments in the US and in Canada through the YALS, NALS, and LSUDA and was now consolidating internationally through TIMSS and IALS. In a period of approximately fifteen years, education assessment had become one in which some actors, networks, monetary resources, knowledge, competences, and dispositions had acquired value, whilst devaluing all others. Although ETS played an influential role in delimiting the actors in this field and preserving the existing state of struggles, the OECD was set to transform them.</p> <hd id="AN0186746490-7">Treason and the birth of PISA</hd> <p>Morgan states that the PISA community of practice emerged from 'the members who were originally involved in the IEA' (Morgan, [<reflink idref="bib20" id="ref36">20</reflink>], p. 53), where there were two communities of practice with competing ideas. One got funded and was moved to Boston College; the other, led by Andreas Schleicher, started PISA with former IEA colleagues. A number of interviewees describe the birth of PISA as an interpersonal struggle; with one interviewee describing the general perception of PISA's birth at IEA as treason.</p> <p>'Schleicher was not satisfied with the way of working of the IEA, so he started a competing project, and that was PISA, and he asked Aletta Grisay and Ray Adams to go with him. So within IEA they considered it as treason. So there is always, still nowadays, there is still a kind of bad atmosphere between IEA and OECD'. Interviewee AA</p> <p>PISA's birth was perceived by the members of one community of practice, the one populated by IEA and ETS, as a form of treason, which continues to fuel a difficult relationship between the IEA and OECD. If PISA was a form of treason for the IEA community, it is likely that PISA was a form of revenge for the other community. Although Andreas Schleicher set out to develop a competing project, he involved key actors from IEA. Like himself, Ray Adams and Aletta Grisay knew the methodological debates, and TIMSS from the inside. Importantly, they had the habitus required to struggle and transform the international educational assessment field.</p> <p>Andreas Schleicher's arrival at the OECD coincided with Network A[<reflink idref="bib5" id="ref37">5</reflink>] writing a data strategy which focused on the regular production of data on student achievement (OECD, [<reflink idref="bib23" id="ref38">23</reflink>]). The data strategy laid the foundations for PISA's original approach: measuring how well students could use the skills and knowledge acquired in the first fifteen years of life. The strategy completion coincided with an inter-organizational struggle between IEA and Network A. The OECD had requested full access to IEA's database to carry out analysis before data were released, but IEA refused (IEA, [<reflink idref="bib14" id="ref39">14</reflink>]). As a consequence, Tom Alexander, the OECD Director of the Directorate for Education, Employment, Labour and Social Affairs (DEELSA) chose to develop its own ILSA (IEA, [<reflink idref="bib14" id="ref40">14</reflink>]).</p> <p>One interviewee states that the USA (it can be assumed it was the NCES, which was funding TIMSS) expected IEA to develop PISA through Boston College upon completion of the OECD data strategy. The OECD responded that internal OECD regulations required a bidding process. The PISA Terms of Reference, informed by the OECD data strategy, were written up. As opposed to the tacit terms of entry in IEA ILSAs, PISA introduced explicitly codified and legally guaranteed terms of entry. The choice of PISA's contractors was presented as rational, scientific, and market-driven.</p> <p>'[In] '96, [...] the United States said 'okay, we have the data strategy and now we ask the IEA to implement it' because they were the organization with the expertise, and actually at that point my organization said 'we can't do that, we have to go out for a call for tender". OECD Interviewee</p> <p>Another reading of this extract, substantiated by a number of interviewees, is that these explicit terms of entry into the international education assessment field covered up tacit terms of (non) entry: the OECD regulations were used to oppose the US government pressures, IEA and Boston College. One interviewee stated that the legacy of the relationship between Andreas Schleicher and the IEA meant that <emph>'it wasn't going to work with IEA being the source' (Interviewee T)</emph>, whilst another interviewee stated that there was '<emph>no way of having an arrangement between the two organizations' (Interviewee Z)</emph>. The formal bidding process thus concealed interpersonal and inter-organizational struggles, and shaped the faux formality of formal networks.</p> <p>The first PISA Terms of Reference stated that the main international contractor had 'ultimate decision-making authority and responsibility for operationalising the overall project design' (OECD, [<reflink idref="bib24" id="ref41">24</reflink>]). Contractors were granted autonomy and flexibility. It is reasonable to ask if this autonomy was intended to take away decisional power from experts and national representatives in the PISA Board of Participating Countries (now the PISA Governing Board); whose members were simultaneously involved at the IEA.</p> <p>When the PISA Terms of Reference were published, contractors sought to establish consortia, groups of organizations working together to obtain and implement ILSA contracts. The process saw three bidding consortia: one led by the University of Bourgogne, one led by ACER, and one led by IEA. Interviewees describe the process as a legacy of the TIMSS consortium, a struggle between two strong personalities at the OECD and IEA, and related to the nationalities represented in the consortia. ACER decided to put together an offer after individuals at DEELSA informed ACER that they 'found the American presence too dominant' (Morgan, [<reflink idref="bib18" id="ref42">18</reflink>], p. 152) and were keen on a greater European presence. Interview data shows that Boston College and IEA had assumed they would develop PISA as they had a dominant position in the field of international education assessment. Boston College and IEA soon discovered that their former partners used the global tendering process to bid against them. This led all contractors to regroup as they struggled for new positions in the changing international education assessment field. With PISA, struggles were growing, the positions of actors were shifting, and the value of actors' capitals had changed.</p> <p>ACER contacted Aletta Grisay (University of Liège, Belgium) who had been involved in IEA ILSAs and was closely involved in the design of the PISA call for tender, to manage all the translation work. Aletta Grisay involved Steve Dept, a freelance translator who was working with several universities to develop guidelines to evaluate translations. This led to the establishment and involvement of cApStAn. ACER contacted Cito, which agreed to join the ACER-led consortium; this new relationship was described as based on affective trust. ACER also contacted Statistics Canada, despite it being on the Boston College-led bid. Statistics Canada declined this invitation on the basis of its relationship with Boston College, paying tribute to interpersonal emotional bonds. ACER was also seeking experts to put together a bid that would win.</p> <p>'We decided that it would be better to have an American partner, because the choice was going to be made [...] by the Board of Participating Countries [...], it was chaired by Eugene Owen, from the National Center for Education Statistics, so I thought, "Well, we're going to have an American in the chair at the meeting, we'd better have an American partner, so we got Westat"'. ACER Interviewee</p> <p>The extract shows ACER had the required social capital (access to the 'right' organizations) and cultural capital (knowing what counted in the evaluation process). In explaining how their organizations became part of consortia, most interviewees state that they (or their organizations) were already working closely together in other large-scale assessments or with the OECD. This shows the importance of affective trust and also that contractors had built a shared vision of research, methods, and objectives.</p> <p>Eventually, the ACER-led bid represented Australia, the Netherlands, Belgium, and the USA, whereas the Boston College-led bid represented the U.S., England, and Canada (Boston College, ETS, Statistics Canada, and the National Foundation for Education Research, NFER). One interviewee involved in the evaluation of bids stated that after heated debates, the ACER-led offer was chosen because their approach was novel. It is also possible to read this as ACER offering a different approach, knowing that the OECD intended to move away from IEA and Boston College and TIMSS. According to other interviewees, the status of the project director (Ray Adams) won ACER the PISA contract. In this case, it can be read as affective trust and 'a feel for the game' determining the outcome of the struggle: Andreas Schleicher and Ray Adams had worked together for many years in IEA and had designed PISA together. Despite the ACER-led bid winning the first PISA contract, IALS (developed by ETS and Statistics Canada) was used as a template for PISA (Morgan, [<reflink idref="bib18" id="ref43">18</reflink>], p. 180). The ETS approach, first set out in NAEP, had now travelled even further into the international education assessment field.</p> <p>PISA contracts further consolidated assessment debates and interpersonal tensions, with actors moving into new positions. As with IEA ILSA contracting, PISA contracts were sealed with those who were trusted and had a 'feel for the game'. The formal global tendering process had allowed the OECD to choose ILSA contractors from their network – those who had acquired the cultural and social capitals from the inside – without submitting to the US government pressures. What stands out with the first PISA contracts are the role of interpersonal struggles, the faux formality of the formal contractor selection processes, and possibly, the use of contractor autonomy to shift away from IEA's education assessment approach. Through PISA, the OECD had now introduced new rules of the game whilst at the same time not changing what capitals were valued in this field. The number of organizations which had accumulated these capitals in educational assessment remained small, and the ETS approach was now consolidated. The dominant position ETS acquired in the field allowed the organization to monopolize the valued capitals, determine the field's laws of functioning, and delimit who produces assessment data. Indeed, revenge at PISA did not lead to struggles with ETS but against IEA and Boston College.</p> <hd id="AN0186746490-8">Pressure to diversify and the tall walls of ILSA contracting</hd> <p>In this section, the paper analyses how ILSA contracting has evolved, and does this by focusing only on the case of <emph>PISA</emph> contracting. The visualizations of IEA and OECD contracting between 1990 and 2020 (Figures 1 and 2 above) demonstrate what interviewees describe: the main ILSA contractors are always the same ones. They are ACER and ETS, Boston College, IEA-DPC, Statistics Canada, Westat, cApStAn, and NFER. It is a small group of experts and organizations which have the sophisticated methodological expertise required for ILSAs. A number of OECD interviewees refer to them as 'the usual suspects', a name that insinuates dissatisfaction. The repetitiveness of ILSAs, carried out every few years for almost three decades, means that individuals and organizations come together regularly. At the same time, the size, complexity and influence of ILSAs put extensive pressure on these actors who develop strong bonds but also lead to conflicts.</p> <p>Growing discontent among OECD staff and members of the PISA Governing Board about lack of competition among PISA contractors led the OECD to carry out an internal audit before contracts were sealed for PISA 2015. The audit concluded that the PISA procurement process was lumpy because only big organizations could compete for PISA contracts, thus limiting competition. The audit recommended PISA contracts be made more accessible by breaking the PISA 2015 Call for Tender into smaller contracts to allow more and smaller contractors to bid. To make this possible in practice, one OECD interviewee describes providing potential contractors with as much insider knowledge as they could and 'a feel for the game' by inviting potential contractors to attend PISA Governing Board meetings and contractor workshops.</p> <p>However, OECD interviewees stated that this was not enough. The habitus of the usual suspects was hard to acquire and compete with. Multiple interviewees state that in order to successfully obtain a contract, organizations had to employ members of ILSA expert groups or staff who had already worked for ILSA contractors. Interviewees insisted on the importance of 'having an insider' and 'someone known to the program. This underlines the importance of affective trust and shows how social and cultural capitals are accumulated through a long process of acquisition, not by attending an insider's workshop or a board meeting. Indeed, another contractor points out the importance of 'being known by PISA' since the beginning of ILSAs.</p> <p>He was known by PISA, and he'd been in that circle pretty much ever since. [...] Him being known, and having the expertise to do it, well, they just fell into place. We did go through the formal application of responding to the call for tender and all that stuff, but it would have been a very, very big surprise had we not gotten that contract. Interviewee H</p> <p>This interview also demonstrates the faux formality of the bidding process. Another interviewee described being hired by a company that '<emph>asked me to get them into PISA' (Interviewee S)</emph> because she had substantial experience working on PISA advisory groups and with contractors. One interviewee states that a contract was won because of the person who had been hired, despite the company being viewed with suspicion. This suggests that a lack of inter-organizational trust was compensated with interpersonal, affective trust.</p> <p>'I didn't trust the organization but I trusted him. So as long he was there, we thought, "Okay, everything is fine". And everything was fine'. Interviewee N</p> <p>These extracts show how, even when the OECD seeks to diversify ILSA contractors, efforts are not always sufficient in the face of ILSA contractors who have acquired capitals in the international education assessment field and used them to institute a dominant position. Interestingly though, one big change in this field did take place: the main contractor, ACER was replaced by ETS, thus further indicating that struggles do occur, albeit only amongst insiders.</p> <p>An OECD interviewee stated that the OECD was <emph>'getting nervous'</emph> about its level of dependency on ACER (main contractor from PISA 2000 to 2012) and <emph>'felt that they couldn't have enough influence and that they were becoming, kind of, bound hand and feet, on ACER'</emph> (Interviewee S). The OECD gave the main PISA contract to ETS for PISA 2015 to PISA 2022. The above-mentioned interviewee stated that now <emph>'ETS has really input a lot of their specialized expertise, and so OECD is now becoming more bound to ETS than they ever were to ACER'</emph> (Interviewee S). One OECD interviewee stated that any contractors seeking to compete with ETS are unlikely to succeed because of ETS's management role in PISA but also because it is difficult to compete against the expertise of the ETS, which the organization has made dominant across national and international large-scale assessments.</p> <p>Despite efforts to diversify the usual suspects, few contractors enter this international education assessment field: the walls are high for contractors who have not acquired capitals over many ILSA implementations – so high that organizations have had to employ insiders to enter this field, whilst outsiders do not manage. Between 2018 and 2020, the OECD developed the Socio-Emotional Skills Assessment (SSES). It is an example of how a lack of cultural and social capitals makes it difficult for new contractors to carry out an ILSA contract. A consortium of new actors (only cApStAn was an insider) obtained the SSES contracts only because the competing bid was withdrawn according to an OECD interviewee. A year into the SSES contract, the consortium's contract was terminated by the OECD and given to the usual suspects (ACER as the main contractor). Despite ILSAs extending into new areas of expertise, new contractors who appear to have the required expertise cannot acquire the feel for the game. However, for those who have acquired a set of predispositions from the inside over a long period of ILSA contracting, the struggles continue even in the face of the dominant position of ETS. In 2025, the main PISA contract was given to ACER.</p> <hd id="AN0186746490-9">Conclusions – the legacy of the early ILSA contracting days</hd> <p>The larger power structures, in particular, the crisis about educational outcomes in the USA, coupled with the methodological shift in measuring learning outcomes and increased policy interest, meant the educational data world was set for big changes in the 1980s. This history has been studied: scholars highlighted the role of the US government in pressuring both the OECD and IEA towards measuring learning outcomes and a shift from philosophical doubt to statistical certainty when comparatively measuring learning outcomes at the IEA and the OECD (Martens [<reflink idref="bib17" id="ref44">17</reflink>]; Addey, [<reflink idref="bib1" id="ref45">1</reflink>]; Pizmony-Levy et al., [<reflink idref="bib26" id="ref46">26</reflink>]; Heyneman [<reflink idref="bib12" id="ref47">12</reflink>]). Pizmony-Levy et al. ([<reflink idref="bib26" id="ref48">26</reflink>]) also demonstrated how IEA went from a question-driven approach led by scholars to a data-driven approach led by policy actors. These studies focus on IOs and government actors involved in ILSAs. But as Critical Data Studies suggest, we must also look at the privileged positions of business interests. It is surprising no scholarship had highlighted what a decisive role business interests played in the history of assessment of educational outcomes. This paper shows how ETS played – with the support of the US government – a central role in this history. It is through ETS that NAEP methodologies travelled to YALS and then NALS in the USA, and then to LSUDA in Canada. In the meantime, IEA was developing ILSAs without adopting the latest ETS developments in educational assessment. After developing the Six-Subject Study and the Reading Literacy Study, the passionate network of scholars assembled around TIMSS. IEA was under pressure to adopt the approach at ETS but could not afford to contract ETS. It was at this stage that the US government offered IEA a way out: it would provide massive economic capital for IEA to carry out TIMSS but there were conditions. These conditions allowed the USA to consolidate the cultural and social capital ETS had acquired. The community of practice at the IEA had ongoing substantive theoretical and methodological debates about how to assess learning outcomes. The economic capital that became available for TIMSS settled these debates: the conditions attached to the funding indirectly led the approach at ETS to be adopted. The IEA community of practice split: one community assembled around IEA and Boston College (managing TIMSS), the other at the OECD. Trust stands out as a key component in the sealing of IEA contracts. The TIMSS consortium allowed actors to put their capitals to fruition and move into new positions of power in the international education assessment field. The US economic capital had played an important role herein.</p> <p>As actors moved into new positions in the education assessment contracting field, others found support at the OECD where the experts of Network A (part of INES) were putting together a data strategy. PISA, an alternative to TIMSS was developed by former IEA community members: it was a response to a personal struggle between Andreas Schleicher and staff at IEA. Under further pressure from the USA to develop PISA with the TIMSS consortium of contractors, OECD regulations were used to resist this pressure. Formal regulations were used to cover up interpersonal struggles. However, the OECD had already adopted the ETS methodological approach in IALS, its first ILSA, in collaboration with Statistics Canada and ETS; and PISA used IALS as a template. ETS had made its approach dominant at the international level, even though PISA was contracted to ACER as the main contractor. With PISA, contractors regrouped. The choice of PISA contractors consolidated choices in assessment debates but also emotional bonds and tensions. PISA contracting instituted new positions in the international education assessment field. However, both IEA and OECD contractors shared cultural and social capitals that had been acquired over a long period of time starting out at IEA before the split. Indeed, it emerges in this paper that the international education assessment field is one of struggles among insiders only. Capitals cannot be transferred through efforts to share 'a feel for the game' with potential ILSA contractors; the international education assessment field requires people acquire capitals through extensive participation in this field. In the international education assessment field, it appears that social and cultural capitals, especially trust, only travel embodied in people. This is another contribution this paper makes: it highlights the importance of trust in social capital; the lengthy process of acquisition of capitals and how these cannot be transferred quickly and through knowledge-transfer activities: they are embodied in individuals. Finally, the paper also shows how struggles in the field are bound up in interpersonal tensions and emotional bonds.</p> <p>Ultimately, this paper has held up a mirror to the data chefs that had so far remained invisible in their kitchens: it has shown the ILSA chefs and their recipes. ILSAs are not raw: data chefs cook emotional bonds, treason, and revenge in their data. This paper has shown the chefs who had gone so far unseen, whilst at the same time highlighting the interlinked goals and aspirations of business interests, governments and international organizations. This paper does not draw on data that can show how ETS gained extensive support from the US government to spread and consolidate its assessment approach both nationally and internationally (empirical research is needed), but it does demonstrate how both government and business interests and aspirations are embedded and interlinked in the dominant approach to educational assessment, ILSA infrastructures and data. Knowledge is made by those who can acquire capitals and struggle into positions of power; all others are kept out of the struggle, their knowledge cannot be made. Some chefs have ingredients, recipes and kitchens to cook their data; others cannot get into the kitchens.</p> <p>Scholarship by Bottani, Morgan and Gorur (see Introduction) showing the political nature of data making at the OECD demonstrates the limits of educational data. This paper further demonstrates the limits of ILSA data – both at IEA and OECD – whilst also demonstrating how interests and aspirations embedded in data are what allow some data to emerge and dominate all other data. It is these politics embedded in data that allow dominant data to displace all other data. At the same time, the making of this dominant data is moving policies and practices and it does so whilst also transferring the larger power structures and struggles which dominant data encodes.</p> <hd id="AN0186746490-10">Disclosure statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <ref id="AN0186746490-11"> <title> Notes </title> <blist> <bibl id="bib1" idref="ref25" type="bt">1</bibl> <bibtext> This methodological section is written in the first person plural, as Addey carried out the data generating process with O'Connor (see our methodological paper, O'Connor and Addey, [3]).</bibtext> </blist> <blist> <bibl id="bib2" idref="ref5" type="bt">2</bibl> <bibtext> For the OECD, this encompassed IALS in 1994 and 1998, ALL in ~ 2005, PIAAC between 2008-~2019, PISA 2000, 2003, 2006, 2009, 2012, 2015, 2018, 2021, and 2024, PISA-D in 2018, TALIS in 2008 and 2018, IELS in 2020*, and SSES in 2020. For IEA, this encompassed TIMSS 1995, 1999, 2003, 2007, 2011, 2015, and 2019, PIRLS 2001, 2006, 2011, and 2016, CIVED ~ 1998, ICCS 2009 and 2016, and ICILS 2013 and 2018.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref6" type="bt">3</bibl> <bibtext> There is a parallel between the 'very scathing' report that was used to discredit IEA's management and the Puryear and Guthrie/Hansen 1995 reports that discredited UNESCO's Institute for Statistics (UIS) (Addey, [1] and Heyneman [12]). In both cases, U.S. government bodies steered reports that led to the discrediting organizations producing international education data, followed shortly after by substantial U.S. funding to develop new education data.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref18" type="bt">4</bibl> <bibtext> ETS and Westat already a had long-standing partnership, including their NAEP work.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref22" type="bt">5</bibl> <bibtext> Network A was the INES group that focused on Student Achievement Outcomes and was established in 1989. It was led by U.S. Department of Education/NCES. In 1995, the year Eugene Owen from NAEP took over the leadership of Network A, there was a pivotal Third General Assembly and the Network established that the OECD would develop a 'new data strategy for collecting indicators on student outcomes' (OECD/INES [23]: 129).</bibtext> </blist> </ref> <ref id="AN0186746490-12"> <title> References </title> <blist> <bibtext> Addey, C. (2018). The assessment culture of international organizations: From philosophical doubt to statistical certainty through the appearance and growth of international education assessments. In C. Alarcón & M. Lawn Frankfurt am Main (Eds.), Assessment cultures – historical perspectives (pp. 379 – 408). Peter Lang.</bibtext> </blist> <blist> <bibtext> Addey, C. (2022). Halos, love boats, and the PISA family: Understanding contractors' rationales for carrying out international large-scale assessment contracts, compare. Compare: A Journal of Comparative and International Education, 1 – 18. https://doi.org/10.1080/03057925.2022.2152278</bibtext> </blist> <blist> <bibtext> Addey, C. (2024). Lost in translation: PISA experts, brokers, and marionettes. Journal of Education Policy, 1 – 20. https://doi.org/10.1080/02680939.2024.2321335</bibtext> </blist> <blist> <bibtext> Bakker, M., Leenders, R. T. A. J., Gabbay, S. M., Kratzer, J., & Van Engelen, J. M. L. (2006). Is trust really social capital? Knowledge sharing in product development projects. The Learning Organization, 13 (6), 594 – 605. https://doi.org/10.1108/09696470610705479</bibtext> </blist> <blist> <bibtext> Ball, S. (2016). Following policy: Networks, network ethnography and education policy mobilities. Journal of Education Policy, 31 (5), 549 – 566. https://doi.org/10.1080/02680939.2015.1122232</bibtext> </blist> <blist> <bibl id="bib6" idref="ref9" type="bt">6</bibl> <bibtext> Bottani, N. (1994). The OECD international education indicators, assessment in education: Principles. Policy & Practice, 1 (3), 333 – 350.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref10" type="bt">7</bibl> <bibtext> Bottani, N. (1996). OECD international education indicators. International Journal of Educational Research, 25 (3), 279 – 288. https://doi.org/10.1016/0883-0355(96)82857-6</bibtext> </blist> <blist> <bibl id="bib8" idref="ref15" type="bt">8</bibl> <bibtext> Bourdieu, P. (1993). The field of cultural production. Columbia University Press.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref1" type="bt">9</bibl> <bibtext> Gitelman, L. (2013). Raw data is an oxymoron (pp. 15 – 40). The MIT Press.</bibtext> </blist> <blist> <bibtext> Gorur, R. (2015). Producing calculable worlds: Education at a glance. Discourse: Studies in the Cultural Politics of Education, 36 (4), 578 – 595. https://doi.org/10.1080/01596306.2015.974942</bibtext> </blist> <blist> <bibtext> Gubbins, C., & MacCurtain, S. (2008). Understanding the dynamics of collective learning: The role of trust and social capital. Advances in Developing Human Resources, 10 (4), 578 – 599. https://doi.org/10.1177/1523422308320372</bibtext> </blist> <blist> <bibtext> Heyneman, S. (1999). The sad story of UNESCO's education statistics. Internatinal Journal of Educational Development, 19, 65 – 74.</bibtext> </blist> <blist> <bibtext> Huang, Y., & Wilkinson, I. F. (2013). The dynamics and evolution of trust in business relationships. Industrial Marketing Management, 42 (3), 455 – 465. https://doi.org/10.1016/j.indmarman.2013.02.016</bibtext> </blist> <blist> <bibtext> IEA. (1995). The IEA reading literacy study: technical report. The Hague.</bibtext> </blist> <blist> <bibtext> Kitchin, R., & Lauriault, T. P. (2018). Towards critical data studies: Charting and unpacking data assemblages and their work. In J. Thatcher, J. Eckert, & A. Shears (Eds.), Thinking Big Data in Geography (pp. 3 – 20). University of Nebraska Press.</bibtext> </blist> <blist> <bibtext> Lingard, B., Rawolle, S., & Taylor, S. (2005). Globalizing policy sociology in education: Working with Bourdieu. Journal of Education Policy, 20 (6), 6, 759 – 777. https://doi.org/10.1080/02680930500238945</bibtext> </blist> <blist> <bibtext> Martens, K. (2007). How to become an influential actor - the 'comparative turn' in OECD education policy. In K. Martens, A. Rusconi, & K. Leuze (Eds.), New Arenas in Education Governance. Palgrave Macmillan.</bibtext> </blist> <blist> <bibtext> Morgan, C. (2007). OECD programme for international student assessment: unraveling a knowledge network [ PhD thesis ]. School of Public Policy and Administration, Carleton University.</bibtext> </blist> <blist> <bibtext> Morgan, C. (2011). Constructing the OECD PISA. In M. A. Pereyra, H. G. Kotthoff, & R. Cowen (Eds.), PISA under examinarion: changing knowledge, changing tests, and changing schools (pp. 47 – 59). Sense Publishers.</bibtext> </blist> <blist> <bibtext> Morgan, C. (2011). Constructing the OECD Programme for International Student Assessment. In M. A. Pereyra (Ed.), PISA Under Examinarion: Changing Knowledge, Changing Tests, and Changing Schools (pp. 47 – 59).</bibtext> </blist> <blist> <bibtext> Morgan, C., & Shahjahan, R. A. (2014). The legitimation of OECD's globalc educational governance: Examining PISA and AHELO test production. Comparative Education, 50 (2), 192 – 205. https://doi.org/10.1080/03050068.2013.834559</bibtext> </blist> <blist> <bibtext> O'Connor, C., & Addey, C. (2024). A network ethnography of international large-scale assessment contracting: Scientific knowledge as messy, provisional, complex, and subjective. International Journal of Research & Method in Education, 1 – 14. https://doi.org/10.1080/1743727X.2023.2294778</bibtext> </blist> <blist> <bibtext> OECD. (1995, June 12–15). Data strategy for network A. In Presented at the general assembly of the INES project (pp. GA-95 – 8). OECD Publishing.</bibtext> </blist> <blist> <bibtext> OECD. (1997). Data strategy for the development of student achievement indicators on a regular basis: terms of reference for the tendering procedure [DEELSA/ED/CERI/CD(97)6]. OECD Publishing.</bibtext> </blist> <blist> <bibtext> Pizmony-Levy, O. (2013). Testing for all: The emergence and development of international assessments of student achievement, 1958-2012 [ Ph.D thesis ]. IN, Indiana University.</bibtext> </blist> <blist> <bibtext> Pizmony-Levy, O., & Harvey, J. (2014). On the merits of, and myths about, international assessments. Quality Assurance in Education, 22 (4), 319 – 338.</bibtext> </blist> <blist> <bibtext> Putnam, R. D. (2000). Bowling alone: The collapse and revival of American community. Simon & Schuster.</bibtext> </blist> <blist> <bibtext> Williamson, B. (2017). Big data in education: The digital future of learning, policy and practice. SAGE.</bibtext> </blist> <blist> <bibtext> Ydesen, C., & Grek, S. (2019). Securing Organisational Survival – a historical inquiry into the configurations and positions of the OECD's work in education during the 1960s. Paedagogica Historica. https://doi.org/10.1080/00309230.2019.1604774</bibtext> </blist> </ref> <aug> <p>By Camilla Addey</p> <p>Reported by Author</p> <p></p> <p>Camilla Addey is a Marie Curie Fellow at GEPS – the Globalisation, Education and Social Policies – research centre at the Department of Sociology of the Autonomous University of Barcelona, Spain. Formerly, Camilla was a Lecturer in Comparative and International Education at Teachers College, Columbia University (USA), and a researcher at Humboldt University in Berlin (Germany). Previously, Camilla worked in education at UNESCO headquarters. Her research interests are in international large-scale assessments in lower and middle income contexts, global educational policy, and education privatisations. Her current research project ILAINC, on the privatisation of international education assessment, can be followed on Facebook at 'ILSA Inc. The ILSA industry'. She has published in Journal of Education Policy ; Comparative Education; Globalization, Societies and Education; Compare: A Journal of Comparative and International Education; Critical Studies in Education; and Assessment in Education: Principles, Policy and Practice. Camilla is a director of the Laboratory of International Assessment Studies - <ulink href="http://international-assessments.org">http://international-assessments.org</ulink></p> </aug> <nolink nlid="nl1" bibid="bib15" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib28" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib21" firstref="ref8"></nolink> <nolink nlid="nl4" bibid="bib10" firstref="ref11"></nolink> <nolink nlid="nl5" bibid="bib18" firstref="ref12"></nolink> <nolink nlid="nl6" bibid="bib19" firstref="ref13"></nolink> <nolink nlid="nl7" bibid="bib16" firstref="ref16"></nolink> <nolink nlid="nl8" bibid="bib11" firstref="ref19"></nolink> <nolink nlid="nl9" bibid="bib27" firstref="ref20"></nolink> <nolink nlid="nl10" bibid="bib13" firstref="ref21"></nolink> <nolink nlid="nl11" bibid="bib22" firstref="ref24"></nolink> <nolink nlid="nl12" bibid="bib25" firstref="ref29"></nolink> <nolink nlid="nl13" bibid="bib26" firstref="ref30"></nolink> <nolink nlid="nl14" bibid="bib17" firstref="ref34"></nolink> <nolink nlid="nl15" bibid="bib29" firstref="ref35"></nolink> <nolink nlid="nl16" bibid="bib20" firstref="ref36"></nolink> <nolink nlid="nl17" bibid="bib23" firstref="ref38"></nolink> <nolink nlid="nl18" bibid="bib14" firstref="ref39"></nolink> <nolink nlid="nl19" bibid="bib24" firstref="ref41"></nolink> <nolink nlid="nl20" bibid="bib12" firstref="ref47"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1493726
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Treason and Revenge: The Emergence and Continuation of ILSA Contracting
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Camilla+Addey%22">Camilla Addey</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8431-5568">0000-0001-8431-5568</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Critical+Studies+in+Education%22"><i>Critical Studies in Education</i></searchLink>. 2025 66(3):411-428.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 18
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Contracts%22">Contracts</searchLink><br /><searchLink fieldCode="DE" term="%22Networks%22">Networks</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Assessment%22">Educational Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Testing%22">Educational Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Capital%22">Social Capital</searchLink><br /><searchLink fieldCode="DE" term="%22Cultural+Capital%22">Cultural Capital</searchLink><br /><searchLink fieldCode="DE" term="%22Ethnography%22">Ethnography</searchLink><br /><searchLink fieldCode="DE" term="%22Trust+%28Psychology%29%22">Trust (Psychology)</searchLink><br /><searchLink fieldCode="DE" term="%22National+Competency+Tests%22">National Competency Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Communities+of+Practice%22">Communities of Practice</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+History%22">Educational History</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Tests%22">Science Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Achievement%22">Science Achievement</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22National+Assessment+of+Educational+Progress%22">National Assessment of Educational Progress</searchLink><br /><searchLink fieldCode="SU" term="%22Program+for+International+Student+Assessment%22">Program for International Student Assessment</searchLink><br /><searchLink fieldCode="SU" term="%22Trends+in+International+Mathematics+and+Science+Study%22">Trends in International Mathematics and Science Study</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/17508487.2024.2357120
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1750-8487<br />1750-8495
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: OECD and IEA International Large-Scale Assessments (ILSAs) contractors have remained largely invisible despite playing a key role in the making of education assessment data. With the help of Bourdieu's thinking, this paper analyses data generated through document analysis of ILSA Reports and in-depth interviews with staff working on ILSAs at the IEA, the OECD and ILSA contractors. The paper traces education assessment developments in the USA and at the IEA and the OECD since the 1980s, and points to the critical role of Education Testing Services (ETS). Although the selection of ILSA contractors ostensibly occurs through a globally competitive tendering process, this paper reveals the pressures and struggles that emerged when economic capital became available at the IEA and the OECD. The struggles reveal PISA was created as a competing project to TIMSS. The paper demonstrates how the social and cultural capitals that individuals acquired, and in particular trust, shaped who can and who cannot produce ILSA data today. Overall, the paper makes visible the ILSA data chefs and their recipes: ILSAs are not raw, they are cooked under pressure with personal and organizational tensions, struggles, conflicts, and emotional bonds.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1493726
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1493726
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/17508487.2024.2357120
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 18
        StartPage: 411
    Subjects:
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Contracts
        Type: general
      – SubjectFull: Networks
        Type: general
      – SubjectFull: Educational Assessment
        Type: general
      – SubjectFull: Educational Testing
        Type: general
      – SubjectFull: Social Capital
        Type: general
      – SubjectFull: Cultural Capital
        Type: general
      – SubjectFull: Ethnography
        Type: general
      – SubjectFull: Trust (Psychology)
        Type: general
      – SubjectFull: National Competency Tests
        Type: general
      – SubjectFull: Communities of Practice
        Type: general
      – SubjectFull: Educational History
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: Science Tests
        Type: general
      – SubjectFull: Science Achievement
        Type: general
      – SubjectFull: National Assessment of Educational Progress
        Type: general
      – SubjectFull: Program for International Student Assessment
        Type: general
      – SubjectFull: Trends in International Mathematics and Science Study
        Type: general
    Titles:
      – TitleFull: Treason and Revenge: The Emergence and Continuation of ILSA Contracting
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Camilla Addey
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1750-8487
            – Type: issn-electronic
              Value: 1750-8495
          Numbering:
            – Type: volume
              Value: 66
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Critical Studies in Education
              Type: main
ResultId 1