View in EDS HTML Full Text PDF Full Text

Examining How Teachers Judge Student Writing: An Australian Case Study

Saved in:
Bibliographic Details
Title: Examining How Teachers Judge Student Writing: An Australian Case Study
Language: English
Authors: Wyatt-Smith, Claire, Castleton, Geraldine
Source: Journal of Curriculum Studies. Mar-Apr 2005 37(2):131-154.
Availability: Customer Services for Taylor & Francis Group Journals, 325 Chestnut Street, Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420 (Toll Free); Fax: 215-625-8914.
Peer Reviewed: Y
Page Count: 24
Publication Date: 2005
Document Type: Journal Articles
Reports - General
Descriptors: Program Effectiveness, Student Evaluation
ISSN: 0022-0272
Abstract: This paper reports a 3-year (1999-2001) Australian study of teacher judgement of student writing. It analyses teachers' talk to discover how they arrive at such judgements. It focuses on the processes teachers use as they read and appraise student writing, as distinct from judgements recorded as numerical or letter grades. It identifies and discusses a set of data-based indexes the teachers rely on to constitute their judgement. In so doing, the "global" standard-setting of external assessment (judging the quality of student work against stated standards), and the "local" of teacher judgement (based on the richness of what teachers bring to the task) are reconsidered. This study notes how teacher judgement of student coursework may be intertwined with and shaped both by officially authorized curriculum materials, syllabus documents, and assessment practices, and by other essentially private, local ways of knowing.
Abstractor: Author
Number of References: 25
Entry Date: 2005
Access URL: https://taylorandfrancis.metapress.com/link.asp?target=contribution&id=L1F9EKQM9D11D0W0
Accession Number: EJ695104
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEHVzXtnzuAXd-k80sbFurOAAAA4TCB3gYJKoZIhvcNAQcGoIHQMIHNAgEAMIHHBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCUCLj044_zJ4gVS9gIBEICBmc6hhtgDBp14912JK_JNKK6WgG3-UoA6m-UNjZ1HsnmodZjPeqgxCh8AA97J-yCK5hIHR3yhg1ueyQ1rd2liwY8LPmFnM1vGyKx7m94wL8DoHEiE4krSnW1GPyHV2TcEYdlhngYLSgCOaAb0miwGVFQJm4Dq1O_cUbyaOxN0sDsdk-ybn4Ih04Ctzc5kIcNZ3Fgcu1CWzFbcLQ==
Text:
  Availability: 1
  Value: <anid>AN0015544757;b9j01mar.05;2019Feb14.14:17;v2.2.500</anid> <title id="AN0015544757-1">Examining how teachers judge student writing: an Australian case study. </title> <p>This paper reports a 3‐year (1999–2001) Australian study of teacher judgement of student writing. It analyses teachers' talk to discover how they arrive at such judgements. It focuses on the processes teachers use as they read and appraise student writing, as distinct from judgements recorded as numerical or letter grades. It identifies and discusses a set of data‐based indexes the teachers rely on to constitute their judgement. In so doing, the 'global' standard‐setting of external assessment (judging the quality of student work against stated standards), and the 'local' of teacher judgement (based on the richness of what teachers bring to the task) are reconsidered. This study notes how teacher judgement of student coursework may be intertwined with and shaped both by officially authorized curriculum materials, syllabus documents, and assessment practices, and by other essentially private, local ways of knowing.</p> <p>We report on a section of a 3‐year Australian study that examines how teachers arrive at judgements of student writing, including their use of official scoring procedures and other factors. The two‐pronged approach of the study is to investigate, first, the judgement processes that teachers rely on to appraise their own students' writing, and, secondly, the processes adopted when the same teachers judge writing by students unknown to them, although at the same grade/year level in other schools. At issue is the nature of judgement itself—the critical question being whether the judgement processes remain constant or stable in both circumstances. In examining teachers' accounts of judgement processes, we follow Lapadat's ([<reflink idref="bib15" id="ref1">15</reflink>]: 42) interest in 'the social processes of shared experiences and discussion which amplify and make thoughts, behaviours and events meaningful'. This focus provides an opportunity to examine the impact of official evaluative frameworks and scoring procedures as well as the contribution of implicit determinants of teachers' judgements.</p> <hd id="AN0015544757-2">Background to the study</hd> <p>In the last decade, several countries, including the UK, the USA, Canada, Japan, and Scotland, have introduced educational reforms in which testing and defined standards have played a key role (Clarke <emph>et al.</emph>[<reflink idref="bib4" id="ref2">4</reflink>]). These changes have been motivated by several concerns, including the achievement of a country's students relative to those in other countries, differences (sometimes wide) among the academic standards and performance of school students from different cultural and linguistic backgrounds and in different parts of the country, the search for explicit school‐attainment standards, public accountability demonstrated in measurable outcomes, local school management, and parental school choice. In Australia, these concerns exert a powerful influence in shaping policy, especially in regard to literacy—witness renewed governmental interest in testing and measurement against defined standards to secure improved educational outcomes.</p> <p>All Australian Commonwealth (i.e. federal government), State, and Territory Education Ministers have agreed to the national literacy and numeracy goal: 'That every child leaving the primary school should be numerate, and be able to read, write and spell at an appropriate level' (Department of Employment, Education, Training and Youth Affairs [<reflink idref="bib9" id="ref3">9</reflink>]: 9). Linked to this goal is a commitment to the comprehensive assessment of all students as early as possible to identify those at 'educational risk', or at risk of not making satisfactory progress towards this goal. Other key elements include professional development for teachers, recognition of the vital role of the teacher in literacy learning, and the use of minimum standards or benchmarks for evaluating literacy achievement at years 3, 5, and 7. In sum, these initiatives focus attention on judgement, particularly teacher judgement.</p> <p>Relatively little is known about how teachers make judgements of student achievement in specific contexts (Cooksey and Freebody [<reflink idref="bib5" id="ref4">5</reflink>], [<reflink idref="bib6" id="ref5">6</reflink>], [<reflink idref="bib7" id="ref6">7</reflink>], Cooksey <emph>et al.</emph>[<reflink idref="bib8" id="ref7">8</reflink>], Smith [<reflink idref="bib23" id="ref8">23</reflink>], Wyatt‐Smith [<reflink idref="bib24" id="ref9">24</reflink>], Wyatt‐Smith and Pascoe [<reflink idref="bib25" id="ref10">25</reflink>]). Yet, the nature of teacher judgement and accountability are now considered important matters. In an Australian project, for example, Masters ([<reflink idref="bib16" id="ref11">16</reflink>]: 10) documented the performance of year‐3 and year‐5 students in several literacy tasks, and then, facing the issue of 'adequate standards', pointed out that 'the process of deciding minimum acceptable scores always involves professional judgement'. Masters's project established 'cut‐scores' or benchmark standards, using tasks considered representative of minimal capabilities critical to school progress, to determine students who were above, at, or below 'satisfactory levels of performance'. Thus, three crucial judgements were made without explication: what literacy capabilities constitute critical performance levels for subsequent success in schooling; which tasks reflect the contents of the benchmarks; and where the minimal capability cut‐score should be located. The definition of these judgements as professional is offered publicly as an assertion of the suitability or acceptability of the judgements.</p> <p>Here, and elsewhere in educational practice, the issue of judgement is rarely treated as analytically tractable. Consequently, the most interesting aspect of gauging students' performance is masked either by technical procedures that seek to establish reliability <emph>after</emph> the items, the training, and the analytic procedures have been put in place, or by the invocation of professional 'connoisseurship' or insider knowledge (Sadler [<reflink idref="bib21" id="ref12">21</reflink>]). The 'guild' of assessors thus names itself as having access to what is taken to be measurable crucial aspects of performance evaluation, even though it may rely partly on unstated judgement practices and procedures.</p> <p>Although the benefits of standard‐setting are widely recognized, difficulties in understanding teachers' judgements arise at the intersection of stated assessment expectations and unstated procedural rules. Variability from instance‐to‐instance and combinations of variables in judgement remain opaque. Phelps ([<reflink idref="bib18" id="ref13">18</reflink>]) and Wyatt‐Smith ([<reflink idref="bib24" id="ref14">24</reflink>]) point out that teacher judgement represents largely uncharted territory in assessment research. They highlighted the need for systematic research of judgement processes, claiming that such processes do not readily lend themselves to examination, even by the teachers who make the judgements. In this paper, we chart this territory, explicating the processes that practising teachers relied on to arrive at judgements of students' writing quality.</p> <hd id="AN0015544757-3">Study design</hd> <p></p> <hd id="AN0015544757-4">Participants</hd> <p>The study's data include semi‐structured interviews with 18 year‐5 teachers (each interview being approximately 1 hour in duration). In total, we recorded approximately 185 hours of talk, in which 37 teachers presented 'think‐aloud' judgements of student writing samples. In reporting work‐in‐progress, we draw on a sub‐set of the data, presenting an in‐depth analysis of the recorded talk of two teachers, Sue and Val, as they performed 'think‐aloud' judgements of year‐5 writing. The teachers each had approximately 20 years experience teaching in state primary schools in Queensland, Australia, with Val having an additional 3 years experience teaching overseas.</p> <p>At the time of the study (1999–2001), the teachers, with a teacher's aide, were team‐teaching 61 students in their 5<sups>th</sups> year of compulsory schooling (average age 10 years) in a suburban school on the outskirts of a large city. The student population at the school was diverse and included students from several European, Middle Eastern, and Asian countries, as well as Aboriginal and Torres Strait Islander students. The year‐5 class in question reflected this diversity, although there were no students who self‐identified as Aboriginal or Torres Strait Islander.</p> <hd id="AN0015544757-5">Data collection method</hd> <p>The aim of the 'think‐aloud' judgements was to capture <emph>judgement in action</emph>, that is, as judgement occurred (as distinct from how it is recollected at some later moment in time). A related aim was to study how judgement occurred in three different contexts, as outlined below. The think‐aloud method was chosen as a way to capture teachers' thinking aloud what was salient to them as they read student writing for judgement purposes. Of special interest in the talk was the information within and beyond the text that the teachers oriented to as they read successive pieces of student writing, and how they ascribed meaning and value to that writing.</p> <p>Data collection occurred in three stages, each stage representing a particular context for judgement. In the first stage, the teachers were asked to talk about the processes they followed as they read and judged each of 25 pieces of writing produced by their own students. We refer to these judgements as 'in‐context', to indicate how the teachers were familiar with the institutional, curricular, and pedagogical contexts in which the samples had been generated. Secondly, the teachers were asked to judge 25 previously unseen writing samples collected from several schools in the south‐east corner of Queensland. These samples were chosen to represent the range of text types and the range of writing performance found in year 5. These judgements are referred to as 'out‐of‐context', to capture how the teachers had access to a deliberately more limited history of the samples to be judged. The teachers were told whether the piece was a first or final draft, and if the students had been given time to research the topic.</p> <p>Finally, the teachers made judgements on the same 50 samples, on this occasion judging them against the national writing standard or benchmark for year 5. (Australian literacy benchmarks for years 3, 5, and 7 are used in conjunction with state‐based testing programmes to generate data on literacy performance—including reading, writing, and spelling—for system‐recording and funding purposes. In our study, these benchmarks represent a 'system context' for judgement.)[<reflink idref="bib1" id="ref15">1</reflink>] This stage of the study is not reported here. For purposes of analysis, each teacher's talk was audio‐recorded and transcribed in full, using conventional transcription procedures.</p> <p>These stages represented different judgement contexts and, therefore, allowed us to investigate:</p> <p></p> <ulist> <item> 1. whether judgement processes remained constant across contexts and across teachers, and</item> <p></p> <item> 2. if any changes were observed, the nature of those changes.</item> </ulist> <hd id="AN0015544757-6">Analytic procedures</hd> <p>In the context of the work of Garfinkel ([<reflink idref="bib12" id="ref16">12</reflink>]), Baker ([<reflink idref="bib1" id="ref17">1</reflink>]), and Silverman ([<reflink idref="bib22" id="ref18">22</reflink>]), the 'think‐aloud' data were read as interactional data that generated accounts of how the teachers arrived at their judgements. The purpose of the 'think‐alouds', adapted from Miles and Huberman's ([<reflink idref="bib17" id="ref19">17</reflink>]) cognitive‐mapping procedure, was to record judgement in action, that is, as it was being formulated, rather than in talk that sought to recapture judgement from a later perspective. The role of the researcher was primarily to prompt each teacher, where necessary, to make available her thought processes as she made judgements of individual student scripts. The intention was to record and explicate what the teachers, individually and collectively, did as they formulated judgements of quality.</p> <p>Our analytic task was to determine how the participating teachers engaged in acts of judging by focusing on teacher talk that sought to verbalize and explicate judgement processes. In keeping with the emphasis on teachers' actual judgement processes made available through talk, our study is informed by an ethnomethodological interest in members' knowledge of their ordinary, day‐to‐day affairs of their own institutions, where that knowledge serves as part of the same setting to which it brings order (Garfinkel [<reflink idref="bib12" id="ref20">12</reflink>]). According to Garfinkel ([<reflink idref="bib12" id="ref21">12</reflink>]: 1), 'the activities whereby members produce and manage settings of organized everyday affairs are identical with members' procedures for making those settings "accountable" ', and, therefore, available for scrutiny by others. Ethnomethodologists recognize that this 'reflexivity' is both a phenomenon and a feature of all social activity, so that the notion of the reflexive accountability of actions is of fundamental interest. Linked to the reflexive quality of social action is the concept of indexicality, that is the indexical properties of normal language use that are essential features of members' accountability procedures in everyday social activity as they constantly reference their commonsense understandings of social structures and actions. Boden ([<reflink idref="bib3" id="ref22">3</reflink>]) refers to this process as the 'retrospective‐prospective nature of accounts' (pp. 57–58), and argues that talk 'provides the primary medium through which the past is incorporated into present action and each are [<emph>sic</emph>] projected into an evolving, never‐to‐be‐arrived‐at future' (p. 57).</p> <p>A feature of ethnomethodology is the prominence given to the local, moment‐by‐moment determination of meaning in social contexts. Study of local practices and methods throws light on how people achieve rationality, order, and structure to arrive at knowledge in everyday life. An outcome of an ethnomethodological stance is the immutable fact that, regardless of their insight into the matter, social actors are unavoidably engaged, through their own actions, in producing and reproducing the intelligible characteristics of their own circumstances (Hester and Eglin [<reflink idref="bib14" id="ref23">14</reflink>]). In this study, our concern was to capture how teachers routinely work as judges, drawing on 'ways of knowing' (Belenky <emph>et al.</emph>[<reflink idref="bib2" id="ref24">2</reflink>]) that, to the present, have remained private.</p> <p>Given these interests, the corpus of 10 hours of 'think‐aloud' talk of the two teachers was scanned to identify and code recurring features of judgement. Statements or sections of the teachers' talk that exemplified these features were extracted from this corpus to generate a provisional set of judgement indexes. A more detailed examination concentrated on each teacher's 'think‐aloud' talk to determine the function of the proposed indexes in the talk, and, in particular, in judgement processes.</p> <p>Emerging from the data was a picture of the complexity of the 'educational ecology' (Eisner [<reflink idref="bib10" id="ref25">10</reflink>]: 355) in which teachers make judgements about students' performance. There is no simple, linear course that teachers follow to arrive at their judgements. On the contrary, what emerges is a picture of how dynamically networked indexes come into (and out of) play in acts of judgement. In Part A of this paper, we unpack and discuss the indexes active in 'in‐context' judgements, and in Part B we examine those operating in (and omitted from) 'out‐of‐context' judgements. Analyses of how the Australian literacy benchmarks impacted on teacher judgement—a third stage—works at the interface of federal literacy assessment policy and local practice, and will form the basis of another paper.</p> <hd id="AN0015544757-7">Part A: 'In‐context' judgement</hd> <p>The teachers' think‐aloud judgement sessions, although audio‐recorded in separate locations, generated remarkably similar accounts of the nature of judgement and its reliance on indexes to inform the logic of judgement in action. The recorded talk revealed a set of six recurring judgement indexes. These are:</p> <p></p> <ulist> <item> 1. assumed or actual knowledge of the community context in which the school is located;</item> <p></p> <item> 2. teacher experience;</item> <p></p> <item> 3. moderation practices, both planned and incidental;</item> <p></p> <item> 4. assessment criteria and standards;</item> <p></p> <item> 5. first‐hand/in‐class observations of students; and</item> <p></p> <item> 6. knowledge of pedagogy.</item> </ulist> <p>This set of knowledges—conceptualized as indexes for judging—is a construct on our part. For Sue and Val, the indexes were the 'analytic resources' (Baker [<reflink idref="bib1" id="ref26">1</reflink>]: 132) that the teachers relied on to formulate judgements of student writing, and, in general terms, to display their identities as teacher assessors. Each index is discussed below.</p> <hd id="AN0015544757-8">Index 1: Community context</hd> <p>The teachers' talk made clear how they established early in the school year a latent or 'in‐the‐head' standard (Sadler [<reflink idref="bib21" id="ref27">21</reflink>]: 26) for judging, with that standard being locally defined rather than drawn from official curriculum materials, including syllabus documents. The teachers determined their standard by drawing, in part, on their first‐hand observations of students as well as on their knowledge and perceptions of the community surrounding the school. In this way, setting the standard extended to the class and the wider community, with what was to count as appropriate being firmed up and internalized by the teacher over time. Sue spoke of this approach to standard setting, describing it as being reliant on a mix of 'knowing the kids, the general area, and the general feel':</p> <p> <emph>Sue</emph>: There are ... basemarks that you start from, that you think, OK, after being there, I usually give myself to Easter.[<reflink idref="bib2" id="ref28">2</reflink>] ... By Easter you, sort of, know the kids. You know the general area and the general feel, and so then you work out ... what sort of a standard you're going to make. Then within that standard you've got certain children in your class who still aren't going to fit that standard, for a whole lot of different reasons. So those kids have to—you push them along to aim higher, to reach at least the standard that you've made for the school or that particular class.</p> <p>In this extract, the standard is characterized as a baseline that has local (as distinct from system) relevance, although it may not 'fit' or be appropriate for all students. Missing from the extract, and from the body of data as a whole, is any direct reference to official curriculum documents, including syllabus materials, as informing how the baseline is established. This omission is striking, especially in light of the fact that the teachers consistently claimed that their standard informed how they diagnosed student need—children who 'aren't going to fit that standard'.</p> <p>The extract also indicates how teacher observations of the community surrounding the school (index 1), especially SES, play a part in standard setting—'you know the general area and the general feel, and so then you work out ... what sort of a standard you're going to make'. Such observations tend not to be verbalized routinely and, therefore, remain unstated. Nevertheless, they were clearly potent in the initial firming up of the standard, allowing teachers to establish what could reasonably be expected from students in 'this' school (as distinct from the school in the next suburb or elsewhere). Sue elaborated on the recurring issue of SES as shaping teacher expectations:</p> <p> <emph>Sue</emph>: ... you can't expect, or I never expect, that a child who's come from a low socio‐economic school [and] area where I've taught will do as well as a child who's come from a higher‐standard school and area.</p> <p>The connection between school and local area is repeated here in a way that discloses the apparently unproblematic, taken‐for‐granted connection between SES and teacher expectations. The potency of this link has been reported in previous Australian research, including Freebody <emph>et al.</emph> ([<reflink idref="bib11" id="ref29">11</reflink>]). In our study, the link between an assessment standard and SES is potent, especially when it becomes interwoven with talk about family background (as being good/poor), the un/availability of books in the home, parenting practices (including shared reading time), and demonstrated interest and involvement in school learning.</p> <p>The SES/family‐background connection is also evident in the following extract in which 'a good family background' is characterized as being one in which school learning is actively supported:</p> <p> <emph>Val</emph>: ... in schools, depending on where the school is, like low socio‐economic schools, the standard that you expect from the kids there, for me, [is] not nearly as high as the standard that you would expect [from one] who ... comes from a good family background where there's lots of books in the house and the parents have spent, even if they're not spending [it] now, ... a lot of time with the kids, and the parents are at least concerned in some way about the kid's work that they're going to produce. ...</p> <p>In this extract, the teacher accounts for how she establishes the <emph>expected</emph> standard as one that has relevance for the school, and by implication the student cohort, depending on perceived SES. The teacher also sees herself as being licensed to familiarize herself with the school and community contexts, even family backgrounds, as a means of establishing a locally relevant standard.</p> <p>Note again that Sue and Val did <emph>not</emph> indicate that they used official curriculum materials, including syllabus documents, in this process, nor did they mention checking their standard against the national literacy benchmark for year‐5 writing. For them, an externally‐defined, stable standard was not a relevant point of reference; they chose instead to develop a site‐specific, locally‐relevant standard that they expected to change from site‐to‐site, year‐to‐year, and class‐to‐class.</p> <hd id="AN0015544757-9">Index 2: Teacher experience</hd> <p>At no point did Sue or Val call into question the need for re‐establishing the expected standard with each successive cohort of students, nor did they question the appropriateness of relying on a standard that might not have relevance beyond the immediate school and community context in which it was established. The corollary of this was that variability of standard across sites, and even over time within a site, seemed to be taken‐for‐granted or normal:</p> <p> <emph>Sue</emph>: The standard of our class this year is probably lower than the standard for her [Val's] class last year. But that's not to say that next year that in grade 5 you wouldn't be getting a whole group of students through who are going to be above the standard, so you're sort of adjusting as you go along.</p> <p> <emph>Val</emph>: So the skills that are lacking are the skills that you concentrate on to bring those kids up to what you think will be the acceptable standard for that year.</p> <p>Present in this talk is the notion that what counts as the <emph>acceptable standard</emph> for a particular year in a specific classroom remains fluid (as distinct from fixed), capable of shifting in response to different student cohorts. Elsewhere in their talk, the teachers indicated that they could determine the locally relevant and acceptable standard by drawing on the evaluative experience they had accrued over their years of teaching. The teachers talked of knowing what students at year 5, for example, could reasonably be expected to know and do by having worked with successive cohorts of year 5 in different locations. Sue talked of the connection between how she judged and her teaching and evaluative experience:</p> <p> <emph>Sue</emph>: ... you can tell, very easily. I mean, after years of teaching, ... I've been teaching 20 years, you can pick up a lot of things that children will try. I mean, there's another aspect ... of our judgement. The longer you've been at the game, the more you know what children will either try or cover up or get help from or be sneaky, and you know what to expect ... same with projects.</p> <p>In this extract, Sue can be heard identifying her 20 years of teaching and evaluative experience as a resource for professional judgement—this is of particular interest, given the absence of explicitly defined, endorsed writing standards to inform teachers' judgement‐making. Thus, the standard that the teachers firmed up over time at each site remained typically in unarticulated form, and, therefore, was not readily available for scrutiny or inspection, even by the teachers themselves. The teachers' awareness of how their standard could shift over time and over sites made particular demands on them.</p> <hd id="AN0015544757-10">Index 3: Moderation practices</hd> <p>As noted above, Sue and Val worked in a team‐teaching situation, sharing the marking for 61 students. In their talk, both teachers discussed the need to be reliable and consistent in how they judged student writing, referring to both rater consistency over time and inter‐rater consistency. They also spoke of how they met before marking to establish jointly a scoring guide. Val described the processes they followed after this preliminary meeting:</p> <p> <emph>Val</emph>: Sue takes her little pile and goes to her house, and I take my little pile and go to my house. And then, if there's anybody who we're having a real dilemma about, we'll come back and say, 'Have a look at this, you read this and what do you think about this one, you know, is there something there?'</p> <p>In this extract, Val makes the point that Sue and she were sufficiently confident in their relationship as co‐assessors to cross‐mark or exchange graded student papers and discuss the fairness of the grades awarded. This practice was routine, with student papers being accompanied by the relevant scoring guide the teachers had established themselves. The teachers' self‐initiated sharing and discussion of papers—a form of moderation—provides an opening for considering the explicit formal provision made at the state level to secure consistency of teacher judgement in years 1–10.</p> <p>In Queensland, years 1–10 represent a passage of schooling typically distinguished from senior schooling (years 11–12), the latter marked as a 2‐year period during which high‐stakes assessment occurs to determine university eligibility. Although formal moderation procedures, or checks and balances on teacher judgement, are mandated in senior schooling, in years 1–10 no provision is made at the state level to support or check teacher judgement. This omission underscores the fact that consistency of judgement across school sites is not currently an educational priority, putting at risk public confidence in the capacity of schooling to deliver reliable judgements and ensure equity and justice to all students. We return to this matter below.</p> <hd id="AN0015544757-11">Index 4: Assessment criteria and standards</hd> <p>We noted previously that Sue and Val routinely co‐developed a scoring guide for each major writing task. According to the teachers, the statements in the guides did not follow a uniform pattern. The first 'Teacher‐designed scoring guide' (Appendix A) focuses on textual features exclusively, whereas the second 'Teacher‐designed scoring sheet' (Appendix B) is concerned with planning and editing. The teachers also talked of how the development of assessment criteria involved 'a valued process of negotiation', to use Val's words. During the think‐alouds, the teachers talked of the assessment criteria as being designed to capture the anticipated features of student writing, as well as providing a checklist that, in principle, secured consistency of teacher judgements. It is as if the stated criteria provided a means for achieving both accountability and transparency. In short, the criteria were the means of making judgements not only available, but also defensible. The following extract shows, however, that the criteria alone did not wholly account for how judgement occurred, and that teachers routinely took account of other factors, including the nature and extent of assistance provided to the student writer:</p> <p> <emph>Sue</emph>: He has to be helped to get to the standard we asked [for]. ... And he got to that standard, and he got to that standard but he had heaps of help. ... He had teacher's aide help; he had my help. He had not so much help at home, no, but he does spend a lot of time doing the decoration. So, this is another thing that comes into play with assessment—you've got to know the child and what we have to do to build them up.</p> <p>The teacher's talk on this occasion starts with the idea of a fixed standard ('the standard we asked [for]'), and reveals the teacher's assessment decision that the student's work met the requirements of the standard ('He got to that standard'). However, the extract also shows that the teacher's act of judging the writing against the standard was not done in isolation from other considerations. The teacher could and did readily call on her first‐hand knowledge of the assistance that she provided and also of the assistance given to the student writer by the teacher aide and, to a lesser extent, others at home.</p> <p>Thus, in the above extract there is evidence that the teacher did not read students' writing as being ahistorical and de‐contextualized. Instead, as she read, Sue brought together her knowledge of the expected standard and what she had observed about the production history of the writing. Also emerging is the emphasis once again on 'knowing' and 'building' the child, with assessment construed as actively constructing the identity of the student writer.</p> <hd id="AN0015544757-12">Index 5: Observations of the student</hd> <p>Sue and Val repeatedly drew on the notion of 'knowing' and 'coming to know' students individually and collectively (as a class) by tracking their development over time, or to use Val's words on one occasion, 'See ... we know them so well, we can tell what they do from project to project, or piece of writing to piece of writing'. Close analysis of their talk showed how claims to 'know' the student were related closely to how each teacher recollected prior observations of ability, motivation, and personality and how the student applied himself or herself to school learning. These observations included on‐task behaviours, both in the classroom and out of school (if and how homework was completed). These perceptions typically remained latent, and yet exerted a powerful influence on how teacher judgements actually occurred. In the following extract, Thomas is talked about as a student with limited control of written English who needs encouragement:</p> <p> <emph>Val</emph>: ... so ... young Thomas here ... he's tried to use an old worldly sort of a font. ... With Thomas, we know that this is a kid that needs encouragement so [laugh]. So that influences what you're going to do and what you'd write on here.</p> <p>Again referring to the same student, the teacher drew on a second piece of writing, this time factoring in other perceptions of Thomas's personality and on‐task behaviours.</p> <p> <emph>Val</emph>: Well, the thing was with poor old Thomas ... he's one of these kids, he threw his whole heart and soul into this Explorer Project [class assignment on an Australian explorer] ... [H]e was fast, he was active constantly, he did huge amounts of researching, he read lots of stuff, he took mountains of notes and, in the end probably not much saved him. Unfortunately for Thomas ... his, his written presentation is not—I mean he's got lots of things, but his mark is probably indicating how much I knew he put into this ... His fine motor skills, I mean, as you can see by the handwriting, are exceedingly low ... but for him to produce this map was just you know, really good, really good.</p> <p>On revisiting this judgement later in the meeting, the teacher went on to say:</p> <p> <emph>Val</emph>: I guess in a lot of ways I was more lenient. ... If ... he had a just given me this and I had never seen the amount of effort that went into it, if I'd never seen how much of himself he'd poured into this, his mark would probably have been lower.</p> <p>Taken together, these extracts demonstrate how the teacher could contextualize the piece of writing in front of her by recollecting the student's enthusiastic engagement with the task. More importantly, a link is disclosed between the teacher's first‐hand, in‐class observations of the child at work ('he threw his whole heart and soul into this Explorer Project') and her leniency in marking. There is also the suggestion that the teacher wanted to use the mark as a recognition, even a reward, for effort, the judgement being that the work was 'really good, really good' [for him]. The point here is that the teacher's judgement is not informed by a fixed standard of textual quality. Instead, there is a standard that relates directly to Thomas, allowing her to track his individual progress over time and across tasks. The teacher's repeated words about quality—the description of the Explorer Project as 'really good'—pertain less to demonstrated knowledge and linguistic features of the map than to what she 'knew' of the student.</p> <p>In the transcripts of the 'think‐aloud' talk, numerous instances of talk show how the individual teacher's perceptions of student ability, personality, and effort were integral to the making of judgements. It was as if the teachers actively sought to align the writing to be judged with what they had directly observed of the student during class. Also at play was the teachers' sensitivity to ways in which poor grades could impact on student motivation, with judgement being shaped as much by this consideration as by the quality of the writing itself. A critical concern for the teachers was how particular students needed encouragement, with marks being one way of encouraging student effort.</p> <p> <emph>Sue</emph>: We know you've got to also build, because knowing the children enables us to build their confidence, right. So he's weak at language, we know that, so therefore I know how much effort he put in this and he wrote it out—his draft‐out about three times. So when you see a final product, [you] know, being a classroom teacher, [you] see what happens in between, and what he's built up to. And therefore, he got a 'satisfactory' for content, 'cause he's actually got the content, but overall we know what he's like.</p> <p>There is a clear connection here between the teacher's knowledge of an individual student as 'weak at language' and her observation of his effort—'I know how much effort he put in this'. Also evident is the teacher's interest in protecting the student from the negative effects of grading decisions, especially as those decisions could affect self‐esteem, motivation, and the student's relationship with the teacher. Note, however, that the teachers' leniency in judging—'to build their confidence'—was restricted to those students perceived to be underachieving or needing encouragement.</p> <p>Although much talk focused on effort and knowing the student, only limited references were made in the in‐context judgement talk to students' gender. This contrasts with the out‐of‐context judgement talk (part B below). Furthermore, the only reference to ethnicity in the in‐context talk was made in relation to parental pressure that a student was under to produce 'good grades', and to the effect of an individual student's cultural background on his analytic and imaginative capabilities. Consider the following:</p> <p> <emph>Val</emph>: Charlie, ... this child, he's ... Vietnamese background. He's very, very clever, sort of, typically Vietnamese in terms of mathematics. He's very ... you know, analytical. He can think; he thinks about sort of little components and how to do it. ... [S]o everything with Charlie is going to be exceedingly organized but not overly imaginative.</p> <p>On this occasion, the teacher 'read' the student writer as 'typically Vietnamese in terms of mathematics', 'analytical', and 'exceedingly organized', with these features being contrasted with his limited ability to display imagination in his 'Traditional Story' (see Appendix B), even though imagination is not listed as an assessment criterion. Knowledge of the student's cultural background allows Val to attribute to the student certain positive features (organization and analysis), as well as limitations (imagination). Furthermore, it was with this set of expectations, that Charlie was 'going to be exceedingly organized but not overly imaginative', that Val began to appraise his writing. What emerged on this occasion was a picture of judgement as involving trade‐offs or compensations. The perceived limitations in imagination were compensated by strengths in organization.</p> <p>On several other occasions when the teachers examined students' writing, they commented on how strengths in certain features compensated for weaknesses in others, with this trading‐off being integral to how judgements were made. Note that this was done without recourse to any published formula for combining the criteria. Furthermore, these acts occurred without leaving a trace (except in the talk). This finding suggests that further research is needed on the trade‐offs that experienced teachers make as they formulate judgements.</p> <hd id="AN0015544757-13">Index 6: Knowledge of pedagogy</hd> <p>Just as the teachers drew on various observations and perceptions of the student writer, some of them quite detailed, so too they drew on detailed recollections of their own pedagogy, including classroom talk and other interactions, as they read and judged student writing. Repeatedly, their talk made clear how the teachers not only related the piece of writing to the child writer, but also tied the writing and their valuation of it to the teaching and learning contexts and practices in which the writing had its origins and purpose. In the following extract, Val captures these connections by disclosing how, as she judges the writing, she can recollect episodes of 'kidwatching' (Gollasch [<reflink idref="bib13" id="ref30">13</reflink>]):</p> <p> <emph>Val</emph>: You can see it, ... but the feeling that you get about the kid, that's influencing what goes into this is all of the other things that you see every day, ... when you're sitting there watching that kid, or when that kid's coming to your table and he's asking, you know, Does this sentence make sense? Is this sentence right? Then that kid will change that sentence because of some talk that you've had. ... I know that he's used the word 'rotated' in that story because we talked a lot about what rotated was [pause], and I can just about guarantee that every single kid in there has used the word 'rotated'.</p> <p>Although Val says that 'you can see it [in the writing]', her talk reveals that she is the agent who does the seeing, as teacher, and in particular, as authoritative teacher‐of‐writing in the classroom. The disclosure on offer is that Val not only sees the student <emph>in situ</emph>, but also the writing in front of her <emph>in situ</emph>. She can recall how the student asked questions, and how she, as teacher, made suggestions to improve drafts. Also disclosed is Val's awareness of her authority not only in the classroom, but also in shaping how the writing comes to be. Consider, for example, her prediction that she could 'just about guarantee that every single kid in there has used the word "rotated" '.</p> <p>The pedagogy index was particularly salient in that it was regularly updated by what Val referred to above as 'all of the other things that you see every day'. It was all of these other 'things' that enabled the teachers to locate a single piece of writing in relation to the larger body of writing produced by the student, as well as in relation to recollected talk about writing during its development. Once again, Val described how judgement was tied to such recollections:</p> <p> <emph>Val</emph>: We know our group of kids because we've read so much of their work and we've talked to them a lot. You see, it's the talking bit that, little individual chats that you have with kids about certain parts of their story, that make all the difference.</p> <p>On this occasion, Val highlights how, as she appraises student writing, she can recollect 'little individual chats', as well as observations of student reactions to her suggestions. The teachers' talk also shows that they deliberately look to see if the students have initiated their own improvements. In short, what was at play was an unstated set of assessment features relating to recollected classroom interactions and student initiative, a key point being that these were never made public or available to students or parents as factors contributing to judgement.</p> <hd id="AN0015544757-14">How the indexes worked</hd> <p>To this point, we have discussed the teachers' talk as they formulated judgements of their own students' writing. We have shown how actual judgements are constituted by a dynamic mix of indexes that comprise official or public scoring guides and other essentially private, local ways of knowing (Belenky <emph>et al.</emph>[<reflink idref="bib2" id="ref31">2</reflink>]). The indexes were the means teachers used to constitute judgements, and in so doing to capture the social relationships that they (as teacher assessors) shared with their students (as recipients of teacher judgements). The indexes allowed the teachers to account for how judgements are 'made to happen', and in this way they were relied on to impose order on judgement possibilities at a particular point in time. In effect, the indexes were called into play or activated in such a way that they had a point‐in‐time relevance. That is, the set of indexes worked to constitute the judgements as rational and defensible. Figure 1 captures the interplay of indexes, displaying their constitutive function. While figure 1 is discussed in detail below, it is worth noting that it shows judgement as not only being influenced by, but also being constituted through, the operation of the multiple contributions of the set of indexes.</p> <p>Graph: Figure 1. How indexes work to constitute in‐context judgements.</p> <p>Although the set of indexes remained stable across the data (that is, students' writing known to the teachers), the emphasis given to particular indexes, and the ways in which they were combined, were shown to vary not only from teacher‐to‐teacher but also from judgement‐to‐judgement, as the teachers moved from one piece of writing to the next and oriented to different aspects of the text. A related observation is that the teachers could readily bring the available indexes into (and out of) play as they read and appraised writing quality. Through the interplay of available indexes, the teachers were able to connect the student writing in front of them to prior observations of the student and classroom interactions that had shaped the writing. In this way, the indexes had a <emph>retrospective</emph> relevance for the teachers, enabling them to read and value the writing in terms of what it revealed about the student and his or her development as a writer over time and across tasks. Furthermore, we suggest that the judgements had <emph>prospective</emph> relevance in that they had the potential to carry forward to inform teacher observations and interactions with students at some point in time. The notion of judgement as having both retrospective and prospective relevance is displayed in figure 1.</p> <p>Against this backdrop, Part B examines the judgement processes of the same two teachers as they judged the writing of students who were unknown to them but from the same year‐level at other schools.</p> <hd id="AN0015544757-15">Part B: Out‐of‐context judgement</hd> <p>In the second stage of data collection, the teachers were asked to judge 25 samples of unseen authentic pieces of student writing drawn from several other schools. Of special interest is how Sue and Val either tried to call upon the indexes that they used in arriving at in‐context judgements (stage 1 of the study), or experienced difficulty in arriving at judgements when they could not do so. The indexes that were not available in the out‐of‐context judgements are discussed first: Index 1: Community context; Index 5: Observations of the student, and Index 6: Knowledge of pedagogy.</p> <hd id="AN0015544757-16">Index 1: (Not) knowing community context</hd> <p>In their talk about in‐context judgements, the teachers regularly drew on their knowledge of their own classrooms and students, and also on their perceptions of the community surrounding the school, particularly its SES. These perceptions were shown to be important in formulating the standard of writing to be expected in the classroom and in determining the range of writing performance that students demonstrated. In the out‐of‐context judgement setting, the teachers commented on the lack of this knowledge, indicating that it caused them some discomfort and uncertainty in judging:</p> <p> <emph>Sue</emph>: I don't know the school either, and the standard of the whole grade. ... [I]f you've only got your own class, ... you can see ..., from your bottom person to your top person, your range.</p> <p>She then elaborated:</p> <p> <emph>Sue</emph>: [A]lright, say that I was at a lower economic group, ... the children have to achieve to whatever they can achieve to. ... So, therefore, you've got to be able to give some high marks so that children can see within their own class, 'OK, that's good, that's what I'm aiming for'. So, therefore, you couldn't mark a whole grade right down low; you've got to have to have some sort of range.</p> <p>In the following extract, Sue reiterated the difficulty of reaching a judgement without knowledge of the student writer and the pedagogical context, once again drawing on the existing standard within her own classroom:</p> <p> <emph>Sue</emph>: I'm judging from what I think my kids can do here, but then every teacher sets work differently, so it's very hard to do something out of context.</p> <p>and:</p> <p> <emph>Sue</emph>: What's the expectations of the teacher who gave this work? Did they have several days? Did they only do it in a day? Did they do it in a week?</p> <p>In the above extracts, Sue, in her role as judge, can be heard searching for ways to orient to the previously unseen writing. Val's talk similarly indicated uncertainty about how to orient to the writing—how to orient exclusively to the textual (dislocated from the social) world of the classroom and teacher‐student interactions. Both teachers were striving to enact judgement as a deeply social act, as they had done in stage 1 of the study, and experienced both uncertainty and discomfort when they did not have access to the indexes or knowledge necessary to do so. Furthermore, while <emph>not</emph> knowing the institutional and community context for the writing appeared to trigger some uncertainty, greater uncertainty was caused by the teachers' interest, even persistence, in trying to read the student in the writing.</p> <hd id="AN0015544757-17">Index 5: (Not) knowing the student</hd> <p>The lack of knowledge about the student writer of the piece to be judged made the teachers uncertain about the fairness and appropriateness of their judgements. This uncertainty can be traced back to the teachers' demonstrated reliance on recollected observations of 'the kid' and of how their interactions with the student writer had a material impact on the student writing, a point that Val captured:</p> <p> <emph>Val</emph>: [B]ut the feeling that you get about the kid, that's influencing what goes into this is all of the other things that you see every day, you know, when you're sitting there watching that kid, or when that kid's coming to your table, and he's asking, you know, 'Does this sentence make sense?' 'Is this sentence right?' Then that kid will change that sentence because of some talk that you've had. ... Whereas a kid whose piece of writing you just get there, you've got no idea whether that kid's ever had anything to do with, do you know, the teacher.</p> <p>Sue similarly demonstrated the high value she ascribed to having first‐hand observations of the student writer at work, raising again a concern about the accuracy of making out‐of‐context judgements in the absence of this knowledge:</p> <p> <emph>Sue</emph>: Knowing the student does affect your marking scheme, yes, and knowing them also gives them a more accurate, I think it's a more accurate, assessment.</p> <p>Here Sue can be heard defending the suitability, even accuracy, of judgements that factor in knowledge of the student—'knowing them'. The corollary of this was that, for her, the possibility of arriving at an inaccurate judgement occurred when the teacher could not connect the act of judging to such knowledge.</p> <p>The next extract again shows the value Sue places both on first‐hand observations of the writer in the classroom, and on having regular access to the individual student writer so that his or her intent can be determined. In fact, Sue states that she would usually delay arriving at a judgement until she had had that opportunity. In co‐operating with the researcher, however, she did arrive at a grading decision, but qualified it by noting that the student in question could be worthy of a different grade on the basis of potential noted by the teacher in the writing:</p> <p> <emph>Sue</emph>: I'd have to ask. I'd be spending time talking with this one. ... I would be talking a lot to this child and asking them [<emph>sic</emph>] to explain a few things to me that I don't quite understand. ... I'd leave it, I wouldn't, I probably wouldn't mark it until after I've talked to the kid because I wouldn't understand enough about it. If I had to give it some sort of a mark, ... it's got a good amount of content in it. ... It's a Satisfactory Achievement, but it's got the potential to be a lot better than that, too.</p> <p>The need for the teachers to 'understand' the writing not only in terms of authorial intent, but also in relation to how it had been jointly accomplished by student and teacher (sometimes as co‐writer), was a recurring issue in the talk. The teachers talked of classroom writing as a social enterprise, and on this and other occasions in the out‐of‐context stage, they were hesitant in making judgements, making the point that they were ill‐equipped to do so because they had not played a part in drafting the script and did not know its production history. Again, teacher judgements, as applied to classroom writing, appear to be deeply social acts, enmeshed in talk and other interactions.</p> <p>Of further interest in the out‐of‐context judgements is how the teachers actively searched for traces of students' gender in the writing. This feature of the teachers' talk was not apparent in the 'in‐context' setting because all the students were 'known' to the teachers. However, it becomes a notable feature of the teachers' talk in the 'out‐of‐context' judgements. In the following extract, Val at first had difficulty in determining the gender of the student; however, she appeared able to resolve this problem by noting the frequency of 'slang things' in the text, deemed to be a feature of boys' rather than girls' work:</p> <p> <emph>Val</emph>: So he may as well written his own story, or her own story. Ah no, I'd say it's a boy, ... lots and lots of slang things in here, and here he kept not much sentence structure, absolutely no paragraphing. ... I guess, I don't know, need a lot of talking.</p> <p>In other instances, the decision on the gender of the student became contingent on the topic, or the way the student handled the topic:</p> <p> <emph>Sue</emph>: [H]e's been into James Bond, this one.</p> <p>The apparent discomfort the teachers experienced while judging writing by unknown students was intensified by a lack of knowledge about the teaching context in which the writing had its origins.</p> <hd id="AN0015544757-18">Index 6: (Not) knowing the pedagogical context</hd> <p>Another instance of the teachers' search for an index that they appeared to draw on routinely in stage 1 (although unavailable in this out‐of‐context judgement setting) is revealed in the following excerpt:</p> <p> <emph>Sue</emph>: I don't know how much the teacher then expects either ... whether this has just been a short theme or thing, I'm not sure. Whereas we did it over a long time, over quite a few weeks, and our entries were ... longer, so again not knowing the context and how long they had to do this, so if this was only a day's exercises, I'd have to put it up a bit. But then, if it's supposed to be more than that, the exercise, I don't know.</p> <p>Here Sue is heard to experience difficulty reaching a decision without knowing more information about the pedagogical context in which the writing was undertaken. She is trying to discover how long the student had to complete the task, indicating that this information becomes critical in determining the student's competence at the task. Sue demonstrates that, in the absence of this knowledge, she is applying another way of knowing that is available to her, namely recollections of her classroom where Val and she team‐taught the same topic. Because their students had a long period of time to complete the topic, they produced longer (diary) entries. Sue, thus, draws some comparison between the unknown students' work and the expectations operating in her classroom, and finally makes a provisional judgement, indicating that it could be changed in the light of further information. The extract also gives some insight into how Sue arrives at judgements on students' work on the basis of the length of the text. Once again, she indicates how issues of quality are inseparable from issues pertaining to teacher expectations for the task and the classroom conditions in which the writing was produced.</p> <hd id="AN0015544757-19">Using available indexes</hd> <p></p> <hd id="AN0015544757-20">Index 2: Teacher experience</hd> <p>In the absence of indexes relating to community context, the student, and the pedagogical context, the teachers most often resorted to drawing on their experience first as teachers with knowledge of curriculum and second as teachers with evaluative experience of year‐5 students' work to assist them in their decision‐making:</p> <p> <emph>Val</emph>: ... I know that in the grade 5 syllabus there's a good chance that the kid studies something about bushrangers 'cause that's in the grade‐5 syllabus, therefore, he's ... using vocabularies like, ... 'trooper', and 'mounted', and ... 'galloped towards me'.</p> <p>Based on this kind of experience, Sue was able to comment on one child's work:</p> <p> <emph>Sue:</emph> [This is] unusual for a grade‐5 child. [pause] 'The cool breeze is more like a wind and is pushing at my back and the hair on the back of my head is falling onto my face'—that's very unusual writing for a child in grade 5.</p> <p>Sue could appraise the writing as 'unusual for a grade‐5 child' because she had extensive experience of reading a wide range of writing quality produced by students in this year of schooling. Even with this experience, however, both teachers also voiced some unease at judging a series of different text types in the one session, indicating that they routinely marked writing of one type only in a marking session. The point to emerge here was that the teachers were used to judging successive papers in a pile, usually over a period of some 30 minutes to 1 hour, during which their sense of standard tended to firm up. It was as if in the course of reading and appraising one paper after another that they firmed up what they would accept as a grade of A, B, and so on. Val commented on this practice:</p> <p> <emph>Val</emph>: Sometimes you have sort of a 'floaty' mark, ... and when you see more of the same sort of pieces, then you become more definite on what that particular one was.</p> <p>Stated criteria also played an important role in firming up expectations.</p> <hd id="AN0015544757-21">Index 4: Assessment criteria and standards</hd> <p>A key observation is that, in the out‐of‐context setting, talk about assessment criteria came more prominently to the fore, apparently in response to the lack of knowledge of the student, the community, the locally‐expected standard, and the classroom context. Arriving at relevant task criteria, even when judging the work of unknown students, did not pose a difficulty for the teachers in question, as they applied the same criteria to the unknown students' writing as they did to their own students, knowing them to also be in year 5:</p> <p> <emph>Sue</emph>: it's not difficult to come up with criteria, 'cause ... we know what to expect of the children. We seem to have expectations already set in our minds and where we're aiming to get the children at.</p> <p>and:</p> <p> <emph>Val</emph>: [H]ere's where I would look at criteria, having to know the purpose of what they're writing for. ... [I]t's just like an imaginative piece of writing, then OK, I would think that in terms of imagination and things, it's quite good.</p> <p>Although the two teachers had established their own criteria for judging, even in the out‐of‐context stage they were not working with an elaborated set of standards. In short, the features of a grading scale A–E were not defined, with the scoring grid of the type shown in Appendix A serving to report imprecise information about performance on each criterion. This is not to suggest any deficiency in the capabilities of the teachers in their judgement and reporting practices. Instead, it serves to highlight a point made previously, that, in the absence of any official or endorsed standards and criteria for judging writing performance, teachers relied on their tacit or in‐the‐head standards, continuing to value their local, point‐in‐time relevance. A diagrammatic representation of the two available indexes in the out‐of‐context judgements appears ins figure 2. The figure shows how judgement decisions could be made in the absence of those indexes routinely used in in‐context judgements. In this context, however, the purpose was not to inform teaching and learning, and the judgement had no prospective relevance. Moreover, figure 2 shows how the notion of judgement itself is constituted in a different way: judgement is an end‐point, with the judgement act in no way situated in relation to teaching and learning.</p> <p>Graph: Figure 2. How indexes work to constitute out‐of‐context judgement.</p> <hd id="AN0015544757-22">Conclusion</hd> <p>In conclusion, we draw attention to some apparent paradoxes of difference between the 'global' of standard‐setting in, for example, external assessment and the 'local' of teacher judgement based on the richness of what teachers bring to the task. First, we note the fundamental difference between setting and implementing standards <emph>as a public act</emph> (a move to provide official accounts of judgement), and locally accomplished teacher judgement <emph>as an essentially private act</emph>. In highlighting this difference, we do not suggest that teacher judgement should not be subject to scrutiny. However, we make the distinction between the acts of establishing and publishing standards on the one hand, and the capacity of such standards of themselves to regulate how judgement actually occurs on the other. We have examined this difference in this paper, explicating the processes that practising teachers relied on to arrive at judgements of writing quality. We have shown that, even though the participating teachers had a set of standards (their own), they were not limited by these standards. Instead, they actively incorporated other knowledge, relating, for example, to past performance and their 'local' observations of student progress over time and across tasks. The critical issue here relates to the possible scope of the 'global', official accounts of standard‐setting, and teachers' locally accomplished judgements, with the mix of indexes that constituted the local being washed out of publicly available statements of standards.</p> <p>The second difference relates to the purposes of 'global' standard‐setting as distinct from the purposes of teachers' local judgements. There is no doubt that published standards, established for use in external examinations or in benchmarking exercises, find their purpose as accountability measures, working to ensure that schools and teachers are delivering quality education in which the public can have confidence. Such standards can also play a role in tracking national and regional achievement and trends over time, providing important system data on cohorts and sub‐groups within cohorts. In this way, the judgement data that result from applying these standards can be interpreted in relation to various contextual factors such as socio‐economic disadvantage and cultural and linguistic backgrounds. In Australia, for example, the literacy benchmarking data is reported in ways that represent the achievement of sub‐groups of the student cohorts as determined by gender, geographic location, and cultural and linguistic background.[<reflink idref="bib3" id="ref32">3</reflink>] While these purposes and uses for 'global' standard‐setting are important, we note that, for the teachers in this study, primacy was given to tracking individual progress on and across tasks, with each judgement linking past performance to possible futures—surely a key aspect of quality assessment for learning. Furthermore, we have shown the inherent indexicality of the teachers' judgement, making clear how contextual knowledge played a part in formulating judgement itself—as distinct from interpreting judgement after it is formalized. In particular, the teachers' judgements entailed a dynamic process of drawing on and variously combining available indexes. It is these indexes that enabled the teachers to contextualize and (re)situate students' writing in relation to past and projected possibilities for classroom interaction. At issue is how, by pulling on the available indexes, the teachers enacted judgement <emph>as social practice</emph> (see figures 1 and 2).</p> <p>One way to minimize the effects of the paradoxes mentioned to this point is to enhance teachers' professional knowledge about judgement—and their work as judges. In making this suggestion, we are not endorsing the 'rules' of external assessment and the need for bringing teachers in line with them; instead, we are highlighting the need to make traditionally accepted rules and standards more accommodating of the 'local' in the interest of establishing more appropriate and rich indicators of achievement. To this end, we are recommending that teachers be 're‐skilled', so that they may be better prepared to stake out their territory in highly political judgement tasks. We have suggested how, in part, this will involve teachers in deprivatizing judgement in a 'classroom‐oriented' self‐examination that demands both an openness to discovery and rigour. In such an examination, teachers would benefit from collaboratively scrutinizing how they apply formal scoring procedures, and how these procedures intersect with other indexes they rely on to make judgements. Integral to the process would be individual teachers' attempts to confront their own assumptions and values about what constitutes effective judgement in schooling. These changes are of fundamental importance if researchers are to learn more about how judgement actually occurs, bringing a critical eye to bear on the connections between teachers and students, what happens in the classroom world, and how teachers experience judgement in different assessment systems and institutional contexts. A key point is that if researchers were to understand how local judgements are made, they would be able to access a more comprehensive picture of student achievement, including information currently omitted from traditional formulations of standards that rely on numerical scores and verbal descriptors.</p> <hd id="AN0015544757-23">Appendix A: Teacher‐designed scoring guide</hd> <p></p> <hd id="AN0015544757-24">School camp—newspaper report</hd> <p>Overall assessment ___</p> <p>Name: __ Teacher: __</p> <p></p> <p> <ephtml> <table><thead valign="bottom"><tr><td /><td>Criteria</td></tr><tr><td>Comments</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td></tr></thead><tbody><tr><td>Presentation/format</td><td /><td /><td /><td /><td /></tr><tr><td>Heading picture/ad</td><td /><td /><td /><td /><td /></tr><tr><td>Punctuation</td><td /><td /><td /><td /><td /></tr><tr><td>Paragraphing</td><td /><td /><td /><td /><td /></tr><tr><td>Sentence structure</td><td /><td /><td /><td /><td /></tr><tr><td>Tense verbs</td><td /><td /><td /><td /><td /></tr><tr><td>Sequencing</td><td /><td /><td /><td /><td /></tr><tr><td>Intro and conclusion</td><td /><td /><td /><td /><td /></tr><tr><td>Spelling</td><td /><td /><td /><td /><td /></tr><tr><td>Vocabulary</td><td /><td /><td /><td /><td /></tr><tr><td>General comment</td><td /><td /><td /><td /><td /></tr></tbody></table> </ephtml> </p> <hd id="AN0015544757-25">Appendix B</hd> <p></p> <hd id="AN0015544757-26">Teacher‐designed scoring sheet</hd> <p></p> <hd id="AN0015544757-27">Traditional story</hd> <p>Overall assessment ___</p> <p>Name: __ Teacher: __</p> <p></p> <p> <ephtml> <table><thead valign="bottom"><tr><td /><td>Criteria</td></tr><tr><td>Comments</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td></tr></thead><tbody><tr><td>Planning</td><td /><td /><td /><td /><td /></tr><tr><td>Spelling</td><td /><td /><td /><td /><td /></tr><tr><td>Punctuation</td><td /><td /><td /><td /><td /></tr><tr><td>Grammar—sentence‐structure</td><td /><td /><td /><td /><td /></tr><tr><td>Vocabulary</td><td /><td /><td /><td /><td /></tr><tr><td>Sequencing</td><td /><td /><td /><td /><td /></tr><tr><td>Paragraphing</td><td /><td /><td /><td /><td /></tr><tr><td>Editing</td><td /><td /><td /><td /><td /></tr><tr><td>Presentation</td><td /><td /><td /><td /><td /></tr><tr><td>General comment</td><td /><td /><td /><td /><td /></tr></tbody></table> </ephtml> </p> <hd id="AN0015544757-28">Notes</hd> <p>Notes</p> <ref id="AN0015544757-29"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref15" type="bt">1</bibl> <bibtext> For details of the Australian National Plan and the literacy benchmarks, see Department of Employment, Education, Training and Youth Affairs ([9]).</bibtext> </blist> <blist> <bibl id="bib2" idref="ref24" type="bt">2</bibl> <bibtext> The Australian school year runs from the end of January to the second week of December and is divided into four terms of approximately the same duration. Easter occurs at the end of term 1.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref22" type="bt">3</bibl> <bibtext> For this reporting, see Queensland Studies Authority ([19]).</bibtext> </blist> </ref> <ref id="AN0015544757-30"> <title> References </title> <blist> <bibtext> BakerC(1997)Membership categorization and interview accountsIn D. Silverman (ed.),Qualitative Research: Theory, Method and PracticeLondonSage130143</bibtext> </blist> <blist> <bibtext> BelenkyMFClinchyBMGoldbergerNRTaruleJM(1986)Women's Ways of Knowing: The Development of Self, Voice, and MindNew YorkBasic Books</bibtext> </blist> <blist> <bibtext> BodenD(1994)The Business of Talk: Organizations in ActionCambridgePolity Press</bibtext> </blist> <blist> <bibl id="bib4" idref="ref2" type="bt">4</bibl> <bibtext> Clarke, M, Madaus, GF, Horn, CL and Ramos, MA. (2000). Retrospective on educational testing and assessment in the 20th century. Journal of Curriculum Studies, 32(2): 159–181.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref4" type="bt">5</bibl> <bibtext> Cooksey, RW and Freebody, P. (1985). Generalized multivariate lens model analysis for complex human inference tasks. Organizational Behavior and Human Decision Processes, 35(1): 46–72.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref5" type="bt">6</bibl> <bibtext> Cooksey, RW and Freebody, P. (1986). Social judgment theory and cognitive feedback: a general model for analyzing educational policies and decisions. Educational Evaluation and Policy Analysis, 8(1): 17–29.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref6" type="bt">7</bibl> <bibtext> Cooksey, RW and Freebody, P. (1987). Cue subset contributions in the Hierarchical Multivariate Lens Model: judgments of children's reading achievement. Organizational Behavior and Human Decision Processes, 37(2): 115–132.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref7" type="bt">8</bibl> <bibtext> Cooksey, RW, Freebody, P and Davidson, GR. (1986). Teachers' predictions of children's early reading achievement: an application of Social Judgment Theory. American Educational Research Journal, 23(1): 41–64.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref3" type="bt">9</bibl> <bibtext> Department Of Employment, Education, Training And Youth Affairs(1998)Literacy for All: The Challenge for Australian SchoolsAustralian Schooling Monograph Series No. 1CanberraAusAustralian Government Publishing Servicehttp://<ulink href="http://www.dest.gov.au/schools/publications/1998/index.htm">www.dest.gov.au/schools/publications/1998/index.htm</ulink> (visited 28 September 2003)</bibtext> </blist> <blist> <bibtext> Eisner, EW. (2000). Those who ignore the past ...: 12 'easy' lessons for the next millennium. Journal of Curriculum Studies, 32(2): 343–357.</bibtext> </blist> <blist> <bibtext> FreebodyPLudwigCGunnS(1995)Everyday Literacy Practices In and Out of Schools in Low Socio‐economic Urban Communities1Report to the Commonwealth Department of Employment, Education and Training;Centre for Literacy Education Research, Griffith UniversityBrisbaneAustralia</bibtext> </blist> <blist> <bibtext> GarfinkelH(1967)Studies in EthnomethodologyEnglewood CliffsNJPrentice‐Hall</bibtext> </blist> <blist> <bibtext> GollaschK(ed.)(1982)Language and Literacy: The Selected Writings of Kenneth S. GoodmanLondonRoutledge & Kegan Paul</bibtext> </blist> <blist> <bibtext> HesterSEglinP(1997)Culture in Action: Studies in Membership Categorization AnalysisStudies in Ethnomethodology and Conversation Analysis, No. 4LanhamMDInternational Institute for Ethnomethodology and Conversation Analysis and University Press of America</bibtext> </blist> <blist> <bibtext> Lapadat, JC. (2000). Evaluative discourse and achievement motivation: students' perceptions and theories. Language and Education, 4(1): 37–61.</bibtext> </blist> <blist> <bibtext> MastersGN(1997)Literacy Standards in AustraliaReport to the Department of Education Employment Training and Youth AffairsCanberraJ. S. McMillan Printing Group</bibtext> </blist> <blist> <bibtext> MilesMBHubermanAM(1994)Qualitative Data Analysis: An Expanded Sourcebook2nd ednThousand OaksCASage</bibtext> </blist> <blist> <bibtext> PhelpsLW(1989)Images of student writing: the deep structure of teacher responseIn C. M. Anson (ed.),Writing and Response: Theory, Practice, and ResearchUrbanaILNational Council of Teachers of English3767</bibtext> </blist> <blist> <bibtext> Queensland Studies Authority(2003)<ulink href="http://www.qsa.qld.edu.au">http://www.qsa.qld.edu.au</ulink> (visited 30 August 2004)</bibtext> </blist> <blist> <bibtext> SadlerDR(1986)Subjectivity, objectivity, and teachers' qualitative judgmentsDiscussion paper,Assessment Unit, Board of Secondary School StudiesBrisbaneAustraliahttp://<ulink href="http://www.qsa.qld.edu.au/yrs11%5f12/assessment/discussionpapers.html">www.qsa.qld.edu.au/yrs11%5f12/assessment/discussionpapers.html</ulink> (visited 28 September 2003)</bibtext> </blist> <blist> <bibtext> Sadler, DR. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2): 119–144.</bibtext> </blist> <blist> <bibtext> SilvermanD(1997)Introducing qualitative researchIn D. Silverman (ed.),Qualitative Research: Theory, Method and PracticeLondonSage17</bibtext> </blist> <blist> <bibtext> SmithCM(1995)Teachers' reading practices in the secondary school writing classroom: a reappraisal of the nature and function of pre‐specified assessment criteriaDoctoral dissertation,University of QueenslandAustralia</bibtext> </blist> <blist> <bibtext> Wyatt‐Smith, CM. (1999). Reading for assessment: how teachers ascribe meaning and value to student writing. Assessment in Education: Principles, Policy and Practice, 6(2): 195–223.</bibtext> </blist> <blist> <bibtext> Wyatt‐Smith, CM and Pascoe, J. (2000). Teacher indexes of year 5 writing performance. Literacy Learning: The Middle Years, 8(2): 23–33.</bibtext> </blist> </ref> <aug> <p>By CLAIRE WYATT–SMITH and GERALDINE CASTLETON</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib15" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib23" firstref="ref8"></nolink> <nolink nlid="nl3" bibid="bib24" firstref="ref9"></nolink> <nolink nlid="nl4" bibid="bib25" firstref="ref10"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref11"></nolink> <nolink nlid="nl6" bibid="bib21" firstref="ref12"></nolink> <nolink nlid="nl7" bibid="bib18" firstref="ref13"></nolink> <nolink nlid="nl8" bibid="bib12" firstref="ref16"></nolink> <nolink nlid="nl9" bibid="bib22" firstref="ref18"></nolink> <nolink nlid="nl10" bibid="bib17" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib14" firstref="ref23"></nolink> <nolink nlid="nl12" bibid="bib10" firstref="ref25"></nolink> <nolink nlid="nl13" bibid="bib11" firstref="ref29"></nolink> <nolink nlid="nl14" bibid="bib13" firstref="ref30"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ695104
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Examining How Teachers Judge Student Writing: An Australian Case Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wyatt-Smith%2C+Claire%22">Wyatt-Smith, Claire</searchLink><br /><searchLink fieldCode="AR" term="%22Castleton%2C+Geraldine%22">Castleton, Geraldine</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Curriculum+Studies%22"><i>Journal of Curriculum Studies</i></searchLink>. Mar-Apr 2005 37(2):131-154.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Customer Services for Taylor & Francis Group Journals, 325 Chestnut Street, Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420 (Toll Free); Fax: 215-625-8914.
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2005
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - General
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0272
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper reports a 3-year (1999-2001) Australian study of teacher judgement of student writing. It analyses teachers' talk to discover how they arrive at such judgements. It focuses on the processes teachers use as they read and appraise student writing, as distinct from judgements recorded as numerical or letter grades. It identifies and discusses a set of data-based indexes the teachers rely on to constitute their judgement. In so doing, the "global" standard-setting of external assessment (judging the quality of student work against stated standards), and the "local" of teacher judgement (based on the richness of what teachers bring to the task) are reconsidered. This study notes how teacher judgement of student coursework may be intertwined with and shaped both by officially authorized curriculum materials, syllabus documents, and assessment practices, and by other essentially private, local ways of knowing.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: Author
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 25
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2005
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://taylorandfrancis.metapress.com/link.asp?target=contribution&id=L1F9EKQM9D11D0W0" linkWindow="_blank">http://taylorandfrancis.metapress.com/link.asp?target=contribution&id=L1F9EKQM9D11D0W0</link>
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ695104
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ695104
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 131
    Subjects:
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
    Titles:
      – TitleFull: Examining How Teachers Judge Student Writing: An Australian Case Study
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wyatt-Smith, Claire
      – PersonEntity:
          Name:
            NameFull: Castleton, Geraldine
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2005
          Identifiers:
            – Type: issn-print
              Value: 0022-0272
          Numbering:
            – Type: volume
              Value: 37
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Journal of Curriculum Studies
              Type: main
ResultId 1