Disciplinary Differences and Other Variations in Assessment Cultures in Higher Education: Exploring Variability and Inconsistencies in One University in England
Saved in:
| Title: | Disciplinary Differences and Other Variations in Assessment Cultures in Higher Education: Exploring Variability and Inconsistencies in One University in England |
|---|---|
| Language: | English |
| Authors: | Ylonen, Annamari (ORCID |
| Source: | Assessment & Evaluation in Higher Education. 2018 43(6):1009-1017. |
| Availability: | Taylor & Francis. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 9 |
| Publication Date: | 2018 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education |
| Descriptors: | Comparative Analysis, Higher Education, Foreign Countries, Summative Evaluation, Semi Structured Interviews, Formative Evaluation, Undergraduate Students, Grades (Scholastic), Intellectual Disciplines |
| Geographic Terms: | United Kingdom (England) |
| DOI: | 10.1080/02602938.2018.1425369 |
| ISSN: | 0260-2938 |
| Abstract: | This article argues that differing disciplinary assessment cultures are likely to be an important factor in explaining differences in student marks and grades both within and between higher education institutions. Using institution-wide data on undergraduate student marks over the last five years in one UK higher education institution we demonstrate variability in the distribution of marks in terms of the 'distance travelled'. This issue was further explored via interviews with senior teaching-active staff. We suggest that the distribution of marks is likely to reflect different disciplinary assessment cultures as well as complexity in the process of marking and assessment. These findings signify that it will be highly challenging, if not impossible, to establish nationally comparable learning gain measures using student mark data because of the underlying inconsistencies in the process of awarding marks. In the current higher education context, with the ongoing implementation of the Teaching Excellence Framework, it remains important to debate and further investigate these issues with all stakeholders, including students. |
| Abstractor: | As Provided |
| Number of References: | 26 |
| Entry Date: | 2018 |
| Accession Number: | EJ1184291 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGPoi00nICX0NMqLjV9TNYZAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDKPMN7l9oarUQWtEEwIBEICBm-2qOGqieAxpdutl7U3EPlcHZdEPpvP-PFxEvee0Fmr8pc81eoiUAN_qDdSYbQRRxjJTgCBw-4gxzl2rtzTLvm6XXOx8-nar3pgTUbdnPFCrArVoYRTsi6KwEpjsll7xkdlldRULmruzy6y0YqH47esWgU5a257Xylui1lQsFUNVh4npaaa4JXOi_0z6gYhXIbAGWmApaBNQylHB Text: Availability: 1 Value: <anid>AN0130606749;eva01sep.18;2019May22.10:29;v2.2.500</anid> <title id="AN0130606749-1">Disciplinary differences and other variations in assessment cultures in higher education: exploring variability and inconsistencies in one university in England </title> <p>This article argues that differing disciplinary assessment cultures are likely to be an important factor in explaining differences in student marks and grades both within and between higher education institutions. Using institution-wide data on undergraduate student marks over the last five years in one UK higher education institution we demonstrate variability in the distribution of marks in terms of the 'distance travelled'. This issue was further explored via interviews with senior teaching-active staff. We suggest that the distribution of marks is likely to reflect different disciplinary assessment cultures as well as complexity in the process of marking and assessment. These findings signify that it will be highly challenging, if not impossible, to establish nationally comparable learning gain measures using student mark data because of the underlying inconsistencies in the process of awarding marks. In the current higher education context, with the ongoing implementation of the Teaching Excellence Framework, it remains important to debate and further investigate these issues with all stakeholders, including students.</p> <p>Keywords: Assessment; grades; degree classification; learning gain</p> <hd id="AN0130606749-2">Introduction</hd> <p>Various summative and formative assessment approaches are used to guide teaching and learning processes in university education. The debate about the benefits and disadvantages of assessment practices in higher education, and the perceived fairness and unfairness of different assessment approaches, is nothing new. According to Knight and Yorke ([<reflink idref="bib18" id="ref1">18</reflink>], 73) 'in practice, assessment techniques are often chosen as the least bad way of resolving a number of competing contingencies', which include issues relating to time, technology and student expectations for example.</p> <p>A commonly used system of classification of disciplines in higher education is based on the work of Biglan ([<reflink idref="bib6" id="ref2">6</reflink>], [<reflink idref="bib7" id="ref3">7</reflink>] and Becher ([<reflink idref="bib3" id="ref4">3</reflink>]). This system, or the Biglan–Becher typology of disciplines (Neumann [<reflink idref="bib20" id="ref5">20</reflink>]), divides disciplines into four distinct categories: hard pure (e.g. physics), soft pure (e.g. anthropology), hard applied (e.g. engineering) and soft applied (e.g. education). Each category relies on different ways of assessing student learning and often use different teaching approaches because the nature of the underlying knowledge fields are vastly different, as are the curricula (Neumann, Parry, and Becher [<reflink idref="bib21" id="ref6">21</reflink>]). While hard pure subjects tend to have curricula that are cumulative, relying on a gradual accumulation of facts and theories, soft pure subjects tend to rely on more holistic curricula that emphasise the existence of uncertainty alongside increasing levels of insight and deeper understanding (Becher [<reflink idref="bib4" id="ref7">4</reflink>]). Applied disciplines have been suggested to broadly correspond with either hard or soft, but with an increased emphasis on practical applications of knowledge. In terms of assessment methods, hard disciplines commonly utilise examinations and tests, while soft disciplines rely more on essay type assessment to assess students' critical thinking skills and ability to apply knowledge in different contexts (Neumann [<reflink idref="bib20" id="ref8">20</reflink>]). In applied fields 'students are ultimately judged in terms of their readiness to embark on a professional career closely related to their disciplinary base' (Neumann, Parry, and Becher [<reflink idref="bib21" id="ref9">21</reflink>], 409).</p> <p>This basic classification system is useful in understanding how assessment cultures in higher education differ based on the underlying nature of disciplines. However, given the rise of inter-disciplinary subjects and specialisms, it is clear that there can be overlaps as well as borderline cases, and that the reality can often be more nuanced (Becher and Trowler [<reflink idref="bib5" id="ref10">5</reflink>]; Jessop and Maleckar [<reflink idref="bib16" id="ref11">16</reflink>]). Disciplinary classification has therefore become more complex and ambiguous (Jessop and Maleckar [<reflink idref="bib16" id="ref12">16</reflink>]).</p> <p>There is evidence to show that students do better at courses which rely more on coursework assessment and worse on courses that rely on examination (Knight and Yorke [<reflink idref="bib18" id="ref13">18</reflink>]; Gibbs [<reflink idref="bib12" id="ref14">12</reflink>]). Furthermore, students prefer coursework-based assessment, which they see as being fairer than examinations (Gibbs [<reflink idref="bib12" id="ref15">12</reflink>]). It is clear, as Gibbs and Dunbar-Goddet ([<reflink idref="bib13" id="ref16">13</reflink>]) have suggested, that assessment environments at programme-level can vary substantially and that they operate in different ways between institutions. The question this raises is to what extent there is also variability within an institution. Are some subjects, by their nature, more suited to coursework rather than examinations, and vice versa? Could it be, for example, that in some science disciplines, say in chemistry or biology, examination type assessment would be the best way to measure what students have learned? Could it be that many social science disciplines, such as education and sociology, are more suitable for coursework-based assessment?</p> <p>An audit of eight UK universities comprising 18 different degree programmes found that there were significant differences in assessment practices between the broad disciplinary areas of humanities, sciences and professional degree courses such as teaching (Jessop and Maleckar [<reflink idref="bib16" id="ref17">16</reflink>]). Students in science disciplines have to undergo the highest level of both summative and formative assessment, and they also have the highest level of examinations as compared to students in humanities and on professional degree courses (ibid.). Indeed, some authors have even suggested that the measurement of students' achievement is easier and more reliable in science subjects rather than in the area of social sciences, arts and humanities, which tend to lean more on markers' subjective judgments (Knight [<reflink idref="bib17" id="ref18">17</reflink>]). These are the kinds of questions that underlie the investigation in this paper, in an attempt to understand different assessment cultures and what this means in terms of student marks and grades within one particular higher education institution.</p> <p>Some authors have argued that reliability of assessment in higher education is generally not high (Baume, Yorke, and Coffey [<reflink idref="bib2" id="ref19">2</reflink>]). As marking entails a largely unavoidable element of subjectivity, this means that reliability can be affected. Bloxham et al. ([<reflink idref="bib9" id="ref20">9</reflink>]) have suggested that renewed thinking is needed about reliability, fairness and standards in higher education assessment. They go on to argue that it is unlikely that grading consensus will be achieved by the use of such measures as criteria, rubrics, moderation and standardising grade distributions. In order to improve the existing situation what is needed is the development of community processes that aim to develop shared understanding of assessment standards (Bloxham et al. [<reflink idref="bib9" id="ref21">9</reflink>]). However, even this would not alter that, as assessment is a highly complex process, variability and inconsistencies are always like to persist.</p> <hd id="AN0130606749-3">The project, approach and methods</hd> <p>The area of student marks and grades constitutes one of three strands in a research and evaluation project funded by the Higher Education Funding Council for England (HEFCE), focusing on Piloting Measures of Learning Gain. The project began in 2015 and finished at the end of 2017. Findings from the other two strands of the wider project, self-efficacy and concept inventories, are reported and discussed elsewhere. This paper focuses solely on the issue of summative assessment cultures in higher education through student grades and marks.</p> <p>The University which is studied in this paper is a medium-sized research-intensive university in England. It has 16,000 students taught on a green-field residential campus in disciplines across four faculties: arts and humanities; medicine and health sciences; science; social sciences.</p> <p>In higher education across the Organisation for Economic Co-operation and Development (OECD) countries the concept of learning gain has become increasingly prominent in debates about educational provision and achievement. The OECD, which represents 32 more economically-developed nations, has been developing comparative measures of student learning outcomes across different national higher education systems (Tremblay, Lalancette, and Roseveare [<reflink idref="bib25" id="ref22">25</reflink>]). In England, interest in learning gain has been heightened as a result of a major research initiative on learning gain, which was launched nationally in 2015 and is still ongoing. In the US there have been renewed concerns about student learning outcomes and how progress should be measured. Arum and Roksa ([<reflink idref="bib1" id="ref23">1</reflink>]) found that as many as 45% of US college students demonstrated no significant improvement in key academic skills, such as critical thinking, complex reasoning and writing, during their first two years at college.</p> <p>The concept of learning gain can be understood in two principal ways – either as 'distance travelled' based on the difference in student performance between two points in their studies, which could be the start and end of a course or programme, or as 'value-added', based on the comparison between actual performance achieved and predicted performance at the start of studies. More specifically, 'distance travelled' could be interpreted as 'the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time. This allows for a comparison of academic abilities and how participation in higher education has contributed to such intellectual development' (McGrath et al. [<reflink idref="bib19" id="ref24">19</reflink>]; xi). The value-added measure is a complex composite indicator, which is based on statistical analysis of the likelihood of a student or cohort of students outperforming what might be expected of them from a wider population data. It is used as an indicator in one of the most widely used national university league tables in the UK – the Guardian University Guide (Guardian University Guide [<reflink idref="bib14" id="ref25">14</reflink>]).</p> <p>The student marks strand of the Learning Gain project focused on examining how student progress in their learning is expressed in the award of marks. A percentage system is used for the award of marks on taught programmes at the university examined in this paper. Marks are awarded for individual pieces of assessment aggregated first at module level and then by year, with a set algorithm for the conversion into a final percentage which feeds into a calculation of Higher Degree Classification (HDC) against the conventional groups of first class, upper second class, lower second class, third class and fail. In addition, a Grade Point Average (GPA) calculation has recently been added to the student final degree transcript as a supplement to the HDC. It seems unlikely that in the near future the sector will abandon this method of expressing a final degree outcome in favour of GPA, despite the Higher Education Academy's extensive project on GPA (Higher Education Academy [<reflink idref="bib15" id="ref26">15</reflink>]).</p> <p>In addition to the analysis of student marks to be explored in this paper, we carried out six semi-structured interviews covering a range of topics related to the assessment and marking process to provide supplementary data. The main purpose of the interviews was to begin the process of interpreting the findings from the quantitative data (see Figure 1), as the picture which emerged from the statistical analysis was unexpected as well as puzzling. The participants were chosen by using purposive sampling (experienced academics in senior positions in different disciplines in the university), and we are therefore aware that the views represented may not be representative. The participants interviewed, who had all worked in higher education environment and teaching for a considerable length of time (between 10 and 20 years), were from biology, linguistics, business studies, speech and language therapy and environmental sciences. Five interviewees were females and one was male. The interviews lasted on average 20 min each. The interviews were transcribed verbatim and were analysed thematically.</p> <p>Graph: Figure 1. Average difference between final award mark and stage 1 mark across the four main faculties.</p> <hd id="AN0130606749-4">The process of awarding marks</hd> <p>The percentage system of expressing student progress and achievement is common throughout UK higher education. There are, however, many differences in the way in which marks are awarded as well as their overall impact on an individual student's outcome. The process of awarding marks in its current form at the university unfolds in the following way:</p> <p></p> <ulist> <item> (i) An individual academic awards a provisional first mark to a piece of work, on a scale of 1–100. The university provides a generic 'Senate Scale' to guide judgements and to provide a core reference point for the award of marks.</item> <p></p> <item> (ii) The mark, before the student sees it, may be adjusted as a result of moderation or double marking, in accordance with the university's policy.</item> <p></p> <item> (iii) The mark is then transferred to the student record system and the student is given a provisional mark. System software calculates the overall module mark based on the marks of each individual component, weighted appropriately.</item> <p></p> <item> (iv) Marks are approved on a yearly basis by the Boards of Examiners.</item> <p></p> <item> (v) At the end of a student's period of study, an algorithm and set of university-wide rules are used to convert the marks into an overall percentage, normally using 40% of the Stage Two weighted year average and 60% of the Stage Three weighted year average. This classification mark directly maps to a Higher Degree Classification and Grade Point Average. In addition, external examiners from another university sit of the Board of Examiners for each course. Their role is to act as a moderating influence on the award of marks and give external views.</item> </ulist> <p>It is perhaps tempting to see this process as a systematic way of converting student performance into an outcome that can be expressed to all, usually for the purposes of comparison between students by employers and by those assessing suitability for further study. However, there are a number of reasons why this system could be regarded as problematic for the purposes of comparing student outcomes, both within and across institutions. We will briefly discuss some of these difficulties below.</p> <p>Higher education institutions take different approaches to algorithms to calculate outcomes, and regulations vary on issues such as whether a student is required to pass all their modules. Even within institutions, the range of different types of assessment undertaken by students varies. With mark schemes and guidance, there are significant differences in the numbers of and types of assessments undertaken by students on courses as diverse as nursing and history (see, e.g. Gibbs [<reflink idref="bib12" id="ref27">12</reflink>]; Pokorny [<reflink idref="bib22" id="ref28">22</reflink>]). Furthermore, even with institution-wide scales and frameworks to guide the award of marks for assessments, such as the Senate Scale discussed above, it is inevitable that there will be elements of disciplinary marking cultures which makes the comparison of marks across an institution complex (Sambell, McDowell, and Montgomery [<reflink idref="bib24" id="ref29">24</reflink>]; Bloxham et al. [<reflink idref="bib9" id="ref30">9</reflink>]; Pokorny [<reflink idref="bib22" id="ref31">22</reflink>]). It seems true, as Knight and Yorke ([<reflink idref="bib18" id="ref32">18</reflink>]) have argued, that the degree award algorithms used in the UK higher education sector lack robustness.</p> <p>There have been calls to further investigate alternative classificatory systems for representing achievement, which would better meet the needs of different audiences, including students and employers. Almost 15 years ago one of the recommendations of the Burgess Report (Universities UK [<reflink idref="bib26" id="ref33">26</reflink>], 4) made the argument that 'the existing honours degree classification system has outlived its usefulness and is no longer fit for purpose'. It is evident that very little progress has been made in attempts to develop alternative systems to the existing honours degree classification since 2004. It remains to be seen what, if anything, will happen in terms of the development of substitutes to the prevailing HDC system in the coming years.</p> <p>At the university examined in this paper there are differences between disciplines in the number of compulsory credits and optional credits that students have to complete each academic year. For example, in chemistry at an undergraduate level the students have more compulsory credits and fewer optional credits than do students in humanities. The types of assessments also vary between subjects. In some disciplines most assessment is through coursework and student-led projects, for example dissertations, while some disciplines also have examinations, or utilise a mixture of assessment approaches. It is likely that these variations in assessment practices also have an impact on the distribution of marks.</p> <hd id="AN0130606749-5">Calculating learning gain using student marks</hd> <p>Our approach to using student marks to calculate learning gain compared a standard measure of percentage marks awarded at two points in time. The study looked at undergraduate student marks across all schools of study in the University. This approach created 25 groups, classifying integrated masters courses and degrees with foundation years in science schools separately, and excluding the medical school because it does not award marks at the module level. We compared the average mark per student cohort, first by school and then by route (standard, with foundation year, or with integrated master year). We calculated an average mark using the last five years of student cohorts' marks at the end of Year 1 and compared them to the average mark (calculated in the same way) at the end of Year 3.</p> <p>The university uses the Senate Marking Scale as its core reference point for the award of marks and all summative marks are expressed as a percentage. However, there is a recognition that the Senate Scale is not suitable to cover all subjects or modes of assessment, and that subjective judgements still have to be made in order to apply the standards to individual assessments. The application of the Senate Scale was a significant issue raised in the interviews. Though the generic marking scale is used across all disciplines to provide quality assurance, several interviewees pointed out that they had developed their own marking scale based on the Senate Scale, but more tailored to the particular discipline context. In practice, then, this means that there exist different variations of the generic scale at the university. The nature of subjectivity and the complexity in the process of making academic judgements has been discussed and debated over the years (see e.g. Neumann, Parry, and Becher [<reflink idref="bib21" id="ref34">21</reflink>]; Knight and Yorke [<reflink idref="bib18" id="ref35">18</reflink>]; Bloxham [<reflink idref="bib8" id="ref36">8</reflink>]), without finding a satisfactory resolution. However, with the introduction of the Teaching Excellence Framework (TEF), which is committed to measuring learning gain (Department for Education [<reflink idref="bib11" id="ref37">11</reflink>]), there is a renewed urgency to tackle the issue.</p> <hd id="AN0130606749-6">The pattern of student marks</hd> <p>Figure 1 shows learning gain differences expressed as the difference in average marks. The apparent range of variation in distance-travelled is significant. Expressed as marks, the difference between the cohort with the greatest distance-travelled (average student mark 5.52% higher in final year than first year) and the cohort with the lowest (average student mark 4.58% lower) is over 10%. It should be noted that, when looking at the absolute average, the differences rarely move through Higher Degree Classification (HDC) boundaries. For example, an average mark moving from 62% to 66% still results in an upper second class mark. However, the results are notable for this study, as they present tangible evidence of the difficulties in comparing marks even within an institution. The results have exposed some of the underlying assumptions and cultures upon which marks are awarded. At a time when the Teaching Excellence Framework (TEF) is likely to attempt to make comparisons between subjects and institutions under the Student Outcomes and Learning Gain aspect of quality (Department for Education [<reflink idref="bib11" id="ref38">11</reflink>]), this is particularly concerning for higher education.</p> <p>In parts of the wider education system in the UK there are some contexts in which national datasets of learner achievement are compared. Tests taken by youngsters at the end of Key Stage 2 (KS2), when they are aged 11, and at General Certificate of Secondary Education (GCSE) level at the end of compulsory education are two such examples. Key Stage 2 refers to a period of four years of schooling in government-funded schools in England and Wales when pupils are aged between 7 and 11, while GCSEs are standardised tests in different subject areas of the curriculum, which are normally taken at the age of 16 though they can be taken at any age. Notably, these are based on standardised curricula and centrally designed highly controlled assessment designs and methods. In higher education, the application of subject benchmarks (Quality Assurance Agency for Higher Education [<reflink idref="bib23" id="ref39">23</reflink>]) do not in any way constitute a national curriculum for higher education, as they do not define the subject knowledge in sufficient detail. Herein lies the difficulty in attempting comparison both between intuitions and across subjects. The nature of the UK higher education degree has evolved over time and with very distinct subject practices. Indeed, even the nature of disciplines might be seen as a barrier to comparison, with some subjects being more disposed to right and wrong answers, while others are more nuanced. In this context, attempting to reverse engineer an approach to comparison of the outcomes of the degrees between universities is challenging.</p> <p>That subjects give different marking profiles, with mathematical subjects producing a different (bimodal) distribution of marks when compared to essay based subjects, which tend to be more clustered, was brought up by several of the interviewees. The interviewee from the business school suggested that, in practice, it is relative easy in mathematical subjects to either get very high marks or very low marks because answers to questions are typically either correct or incorrect. He went on to suggest that 'this is a well-known phenomenon in maths – it is really difficult to be average'. In social sciences and humanities, on the other hand, the majority of the students are 'average' in this respect; in essay-based assessment there are often not clear-cut right or wrong answers. Another interviewee from environmental sciences also brought this issue to attention by suggesting that 'if you are doing something which involves sums, you can get 100%. But you probably never get 100% writing an essay. So, I think that there are cultural differences between different disciplines. As much as possible I try to use a wider range of marks. I know for some people everything is between 55 and 65 [percent]... I don't think that this is at all fair so I'm quite happy to give something in the 80s if it deserves it and fail it if it deserves it'.</p> <p>A related issue, the trajectory of marks over a period of study, was discussed by another interviewee from health sciences. This interviewee pointed out that students often have misconceptions about there being 'an upwards trajectory', i.e. students may expect to get higher marks in their second and third year at university compared to their first year. This, as the interviewee suggested, 'isn't always the case because each assignment can actually be asking for different things and be tapping into different areas of knowledge, different areas of skills and therefore the students won't necessarily have this nice straightforward upwards trajectory'. This issue has also been discussed by Knight and Yorke ([<reflink idref="bib18" id="ref40">18</reflink>]), who suggest that progression is not always necessarily positive and linear – but multidimensional and personal.</p> <p>Further analysis needs to be carried out on the data-set, looking at how the differences emerge over the last two years of the degree year on year. In addition, it may be that variation in the distribution of marks between disciplines may cause the differences, and that the utilisation of the median mark rather than mean mark for comparisons of this kind might give a very different picture. For example, many degrees contain a final year capstone module - a dissertation or an in-depth project - which may make a great deal of difference to a final mark because such modules are often heavily weighted. The design of curricula may also play a part, with some subjects building skills and knowledge in the spiral mode, as discussed by Bruner ([<reflink idref="bib10" id="ref41">10</reflink>]), while others have a more linear model, which introduces new knowledge and concepts as the student progresses. The results also show a pattern in some scientific subjects, with all of the cohorts showing a lower average mark in the final year as compared to the first. As there is no evidence of any quality problems associated with these courses, it seems likely that the differences are due to variation in the assessment and marking process in different disciplines. In essence, the way that marks are calculated may be significant in establishing a useful way of comparison, and some modelling of approaches might be a useful next step.</p> <p>The variations between the ways marks are awarded are complex and deeply rooted in subject pedagogy. The nature of subject teaching is highly varied in higher education, with arts and humanities subjects often having a greater focus on independent study and small group teaching through seminars, while the sciences have a focus on large group lectures and smaller group work is usually in, or related to, the laboratory (see e.g. Neumann [<reflink idref="bib20" id="ref42">20</reflink>]; Neumann, Parry, and Becher [<reflink idref="bib21" id="ref43">21</reflink>]). These pedagogical patterns in themselves create different assessment opportunities, and examination of typical assessment patterns in sciences shows there are often more smaller pieces of work based around laboratory work, whereas in the arts and humanities there is often more focus on written pieces of work such as essays.</p> <p>In the interviews, these issues also emerged as important and it was clear to all of the interviewees that the nature of the assessment design varies form course to course, even within the same discipline. One lecturer from environmental sciences alluded to this by saying that 'there is such a diversity in assessments in environmental sciences, some are field-based, there are presentations, and there are portfolios'. In addition, students have to produce different numbers of assessments for modules of the same credit size. One lecturer interviewed from the sciences faculty believed that there was wide disparity of practice across different disciplines and schools, and suggested that 'I have my worries that some schools don't have enough items of assessment and some schools have too many'. This disparity could be down to some schools utilising a more modular design to teaching, potentially leading to over-assessment, whereas in other schools assessment is done at the course level, which could lead to under-assessment.</p> <hd id="AN0130606749-7">Conclusions</hd> <p>We have discussed the issue of student marks and grades in the wider context of measuring learning gain in higher education. We used institution-wide data on student marks in one UK higher education institution to demonstrate variability in the distribution of marks in terms of the 'distance travelled' during undergraduate students' studies. This, we argued, is likely to reflect different disciplinary assessment cultures and complexity in the process of marking and assessment. The interview findings brought into focus various inconsistencies inherent in the assessment process, for example, issues relating to marking not being an exact science, different subjects relying on different assessment methods and a continuing lack of grading consensus. The interview data clearly demonstrated that, while all subjects use a 0–100 percentage scale to award marks at undergraduate level, the practices behind the award of marks are not consistent, even though they are all working within university policies and procedures.</p> <p>We argue that the illusion of comparability that the percentage marking system presents masks very different approaches to teaching and measuring the resulting learning in different subjects. This is a significant issue in comparing student learning between subjects and institutions, and this challenge is at the heart of HEFCE Learning Gain project. Although this paper only discusses the results of this comparison from one institution, studies on measuring learning in higher education suggest this is a general issue throughout the higher education landscape (McGrath et al. [<reflink idref="bib19" id="ref44">19</reflink>]). While the first iteration of the Teaching Excellence Framework (TEF) used only employability measures in its metric to measure learning gain (Department for Education [<reflink idref="bib11" id="ref45">11</reflink>]), there is acknowledgement in the year 2 specification of the need to develop new and additional measures of learning gain (Department for Education [<reflink idref="bib11" id="ref46">11</reflink>]: 20).</p> <p>The findings of this project have implications for any attempted comparison of marks within the institution as well as possible comparisons between other higher education institutions. We argue that it will be highly challenging, if not impossible, to establish nationally comparable learning gain measures using this kind of student mark data, as so many inconsistencies have been identified. These data will therefore have to be treated with extreme caution in terms of attempting to draw links between student marks, learning gain, quality and standards. It is important to be clear about what these data do not show and what you can and cannot demonstrate by using these statistics. While our analysis noted differences between subjects and disciplines, it is important to note that even the greatest differences between the beginning and end points were less than 6 marks, with most 'gains' around 3 marks. In the context of the complexity of the process of marking discussed in the interviews, it is possible to view all the differences as not significant. Elsewhere in the education system, where marks are compared, systems like the new GCSE scale of 1–9 or the Key Stage 2 tests where percentages of children reaching the expected level are compared, differences between institutions are more evident, with the systems of marking in themselves designed to be able to differentiate. As previously stated, reverse engineering compatibility onto established marking practices in higher education is challenging.</p> <p>We will be building on the findings of this research by carrying out further analysis of the student marks data to find out details about progression between the first and second year as well as between the second and third year, which may produce new insight into the patterns of student progress. In addition, we are carrying out more interviews to further explore the issue of assessment cultures. The study has several limitations and care should be taken when generalising the findings. We focused only on a single institution, and though a large data-set was used, this focused only on undergraduate students. The supplementary qualitative data was also small-scale. Nevertheless, we believe that the issues we have discussed are relevant to other higher education institutions. We hope that other institutions will be encouraged to examine their own data sets in the light of our findings and that this will create more open and constructive dialogue, discussion and debate about these important matters between UK higher education institutions, as well as internationally.</p> <p>The issues we have examined in our study are of interest to both academics and university management, as well as to those attempting new methods of comparison of higher education. While the TEF provided a specific context for this sort of investigation in the UK, it may be that elsewhere such matters should also be reviewed. In an increasingly global market for higher education, clarity and fairness in the way degrees are awarded is important to students who may be ever more concerned with value for money. It also needs to be borne in mind that a crucial group of stakeholders are the students whose learning is being measured. For them, the efficacy of the measurement is vital to their onward journey, into work or further study. For our study, students' views on the award of marks for their work was out of scope, but it is hoped that further funding may be secured to examine this. In the spirit of both the Burgess Report (Universities UK [<reflink idref="bib26" id="ref47">26</reflink>]); Gibbs ([<reflink idref="bib12" id="ref48">12</reflink>]), including student stakeholders in future work is vital in developing fair and effective approaches to the assessment of student work in higher education.</p> <hd id="AN0130606749-8">Disclosure statement</hd> <p>No potential conflict of interest was reported by the authors.</p> <hd id="AN0130606749-9">Funding</hd> <p>This work was supported by Higher Education Funding Council for England [grant number R202734].</p> <hd id="AN0130606749-10">Notes on contributors</hd> <p> <bold> <emph>Annamari Ylonen</emph> </bold> is a senior research associate in Widening Participation. Her research interests include social justice as applied to education policy and practice, inclusive learning and teaching, Lesson Study, and teacher professional development.</p> <p> <bold> <emph>Helena Gillespie</emph> </bold> is an academic director of Widening Participation. She is engaged in teaching, research and scholarship about teaching and learning in higher education and technology-enhanced learning.</p> <p> <bold> <emph>Adam Green</emph> </bold> is a management information manager in Finance, Planning and Governance.</p> <ref id="AN0130606749-11"> <title> References </title> <blist> <bibl id="bib1" idref="ref23" type="bt">1</bibl> <bibtext> Arum, R., and J. Roksa. 2011. Academically Adrift: Limited Learning on College Campuses. Chicago, IL : University of Chicago.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref19" type="bt">2</bibl> <bibtext> Baume, D., M. Yorke, and M. Coffey. 2010. " What is Happening When We Assess, and How Can We Use Our Understanding of This to Improve Assessment? " Assessment &amp; Evaluation in Higher Education 29 (4): 451 – 477.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref4" type="bt">3</bibl> <bibtext> Becher, T. 1989. Academic Tribes and Territories. Buckingham : Open University Press.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref7" type="bt">4</bibl> <bibtext> Becher, T. 1994. " The Significance of Disciplinary Differences." Studies in Higher Education 19 (2): 151 – 161. 10.1080/03075079412331382007</bibtext> </blist> <blist> <bibl id="bib5" idref="ref10" type="bt">5</bibl> <bibtext> Becher, T., and P. R. Trowler. 2001. Academic Tribes and Territories. Buckingham : Open University Press.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref2" type="bt">6</bibl> <bibtext> Biglan, A. 1973a. " The Characteristics of Subject Matter in Different Scientific Areas." Journal of Applied Psychology 57 (3): 195 – 203. 10.1037/h0034701</bibtext> </blist> <blist> <bibl id="bib7" idref="ref3" type="bt">7</bibl> <bibtext> Biglan, A. 1973b. " Relationships between Subject Matter Characteristics and the Structure and Output of University Departments." Journal of Applied Psychology 57 (3): 204 – 213. 10.1037/h0034699</bibtext> </blist> <blist> <bibl id="bib8" idref="ref36" type="bt">8</bibl> <bibtext> Bloxham, S. 2009. " Marking and Moderation in the UK: False Assumptions and Wasted Resources." Assessment &amp; Evaluation in Higher Education 34 (2): 209 – 220. 10.1080/02602930801955978</bibtext> </blist> <blist> <bibl id="bib9" idref="ref20" type="bt">9</bibl> <bibtext> Bloxham, S., and B. den-Outer, J. Hudson, and M. Price. 2015. " Let's Stop the Pretence of Consistent Marking: Exploring the Multiple Limitations of Assessment Criteria." Assessment &amp; Evaluation in Higher Education 41 (3): 466 – 481.</bibtext> </blist> <blist> <bibtext> Bruner, J. S. 1963. The Process of Education. New York, NY : Random House.</bibtext> </blist> <blist> <bibtext> Department for Education. 2016. Teaching Excellence Framework, Year 2 Specification. London : DfE.</bibtext> </blist> <blist> <bibtext> Gibbs, G. 2010. Using Assessment to Support Student Learning. Leeds : Leeds Metropolitan University.</bibtext> </blist> <blist> <bibtext> Gibbs, G., and H. Dunbar-Goddet. 2009. " Characterising Programme-Level Assessment Environments That Support Learning." Assessment &amp; Evaluation in Higher Education 34 (4): 481 – 489. 10.1080/02602930802071114</bibtext> </blist> <blist> <bibtext> Guardian University Guide. 2018. <ulink href="http://www.theguardian.com/education/universityguide">http://www.theguardian.com/education/universityguide</ulink>.</bibtext> </blist> <blist> <bibtext> Higher Education Academy. 2015. Grade Point Average: Report of the GPA Pilot Project 2013–14. York : HEA.</bibtext> </blist> <blist> <bibtext> Jessop, T., and B. Maleckar. 2016. " The Influence of Disciplinary Assessment Patterns on Student Learning: A Comparative Study." Studies in Higher Education 41 (4): 696 – 711. 10.1080/03075079.2014.943170</bibtext> </blist> <blist> <bibtext> Knight, P. T. 2006. " The Local Practices of Assessment." Assessment &amp; Evaluation in Higher Education 31 (4): 435 – 452. 10.1080/02602930600679126</bibtext> </blist> <blist> <bibtext> Knight, P. T., and M. Yorke. 2003. Assessment, Learning and Employability. Maidenhead : Open University Press.</bibtext> </blist> <blist> <bibtext> McGrath, C., B. Guerin, E. Harte, M. Frearson, and C. Manville. 2015. Learning Gain in Higher Education. Cambridge : Rand Corporation.</bibtext> </blist> <blist> <bibtext> Neumann, R. 2001. " Disciplinary Differences and University Teaching." Studies in Higher Education 26 (2): 135 – 146. 10.1080/03075070120052071</bibtext> </blist> <blist> <bibtext> Neumann, R., S. Parry, and T. Becher. 2002. " Teaching and Learning in Their Disciplinary Contexts: A Conceptual Analysis." Studies in Higher Education 27 (4): 405 – 417. 10.1080/0307507022000011525</bibtext> </blist> <blist> <bibtext> Pokorny, H. 2016. " Assessment for Learning." In Enhancing teaching practice in Higher Education, edited by H. Pokorny and D. Warren : 69 – 90. London : Sage.</bibtext> </blist> <blist> <bibtext> Quality Assurance Agency for Higher Education. 2017. The UK Quality Code for Higher Education. Accessed June 6, 2017. <ulink href="http://www.qaa.ac.uk/assuring-standards-and-quality/the-quality-code/subject-benchmark-statements">http://www.qaa.ac.uk/assuring-standards-and-quality/the-quality-code/subject-benchmark-statements</ulink></bibtext> </blist> <blist> <bibtext> Sambell, K., L. McDowell, and C. Montgomery. 2013. Assessment for Learning in Higher Education. Abingdon : Routledge.</bibtext> </blist> <blist> <bibtext> Tremblay, K., D. Lalancette, and D. Roseveare. 2012. Assessment of Higher Education Learning Outcomes (AHEHO): Feasibility Study Report Volume 1 – Design and Implementation. Paris : OECD.</bibtext> </blist> <blist> <bibtext> Universities UK. 2004. Measuring and Recording Student Achievement (the Burgess Report). London : UUK.</bibtext> </blist> </ref> <aug> <p>By Annamari Ylonen; Helena Gillespie and Adam Green</p> <p>Reported by Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib18" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib20" firstref="ref5"></nolink> <nolink nlid="nl3" bibid="bib21" firstref="ref6"></nolink> <nolink nlid="nl4" bibid="bib16" firstref="ref11"></nolink> <nolink nlid="nl5" bibid="bib12" firstref="ref14"></nolink> <nolink nlid="nl6" bibid="bib13" firstref="ref16"></nolink> <nolink nlid="nl7" bibid="bib17" firstref="ref18"></nolink> <nolink nlid="nl8" bibid="bib25" firstref="ref22"></nolink> <nolink nlid="nl9" bibid="bib19" firstref="ref24"></nolink> <nolink nlid="nl10" bibid="bib14" firstref="ref25"></nolink> <nolink nlid="nl11" bibid="bib15" firstref="ref26"></nolink> <nolink nlid="nl12" bibid="bib22" firstref="ref28"></nolink> <nolink nlid="nl13" bibid="bib24" firstref="ref29"></nolink> <nolink nlid="nl14" bibid="bib26" firstref="ref33"></nolink> <nolink nlid="nl15" bibid="bib11" firstref="ref37"></nolink> <nolink nlid="nl16" bibid="bib23" firstref="ref39"></nolink> <nolink nlid="nl17" bibid="bib10" firstref="ref41"></nolink> |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1184291 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Disciplinary Differences and Other Variations in Assessment Cultures in Higher Education: Exploring Variability and Inconsistencies in One University in England – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Ylonen%2C+Annamari%22">Ylonen, Annamari</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-6692-7528">0000-0001-6692-7528</externalLink>)<br /><searchLink fieldCode="AR" term="%22Gillespie%2C+Helena%22">Gillespie, Helena</searchLink><br /><searchLink fieldCode="AR" term="%22Green%2C+Adam%22">Green, Adam</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Assessment+%26+Evaluation+in+Higher+Education%22"><i>Assessment & Evaluation in Higher Education</i></searchLink>. 2018 43(6):1009-1017. – Name: Avail Label: Availability Group: Avail Data: Taylor & Francis. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 9 – Name: DatePubCY Label: Publication Date Group: Date Data: 2018 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Research – Name: Audience Label: Education Level Group: Audnce Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink> – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Summative+Evaluation%22">Summative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Semi+Structured+Interviews%22">Semi Structured Interviews</searchLink><br /><searchLink fieldCode="DE" term="%22Formative+Evaluation%22">Formative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Grades+%28Scholastic%29%22">Grades (Scholastic)</searchLink><br /><searchLink fieldCode="DE" term="%22Intellectual+Disciplines%22">Intellectual Disciplines</searchLink> – Name: Subject Label: Geographic Terms Group: Su Data: <searchLink fieldCode="DE" term="%22United+Kingdom+%28England%29%22">United Kingdom (England)</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1080/02602938.2018.1425369 – Name: ISSN Label: ISSN Group: ISSN Data: 0260-2938 – Name: Abstract Label: Abstract Group: Ab Data: This article argues that differing disciplinary assessment cultures are likely to be an important factor in explaining differences in student marks and grades both within and between higher education institutions. Using institution-wide data on undergraduate student marks over the last five years in one UK higher education institution we demonstrate variability in the distribution of marks in terms of the 'distance travelled'. This issue was further explored via interviews with senior teaching-active staff. We suggest that the distribution of marks is likely to reflect different disciplinary assessment cultures as well as complexity in the process of marking and assessment. These findings signify that it will be highly challenging, if not impossible, to establish nationally comparable learning gain measures using student mark data because of the underlying inconsistencies in the process of awarding marks. In the current higher education context, with the ongoing implementation of the Teaching Excellence Framework, it remains important to debate and further investigate these issues with all stakeholders, including students. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: Ref Label: Number of References Group: RefInfo Data: 26 – Name: DateEntry Label: Entry Date Group: Date Data: 2018 – Name: AN Label: Accession Number Group: ID Data: EJ1184291 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1184291 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1080/02602938.2018.1425369 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 9 StartPage: 1009 Subjects: – SubjectFull: Comparative Analysis Type: general – SubjectFull: Higher Education Type: general – SubjectFull: Foreign Countries Type: general – SubjectFull: Summative Evaluation Type: general – SubjectFull: Semi Structured Interviews Type: general – SubjectFull: Formative Evaluation Type: general – SubjectFull: Undergraduate Students Type: general – SubjectFull: Grades (Scholastic) Type: general – SubjectFull: Intellectual Disciplines Type: general – SubjectFull: United Kingdom (England) Type: general Titles: – TitleFull: Disciplinary Differences and Other Variations in Assessment Cultures in Higher Education: Exploring Variability and Inconsistencies in One University in England Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Ylonen, Annamari – PersonEntity: Name: NameFull: Gillespie, Helena – PersonEntity: Name: NameFull: Green, Adam IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2018 Identifiers: – Type: issn-print Value: 0260-2938 Numbering: – Type: volume Value: 43 – Type: issue Value: 6 Titles: – TitleFull: Assessment & Evaluation in Higher Education Type: main |
| ResultId | 1 |
