View in EDS HTML Full Text PDF Full Text

The Focus, Function and Framing of Feedback Information: Linguistic and Content Analysis of In-Text Feedback Comments

Saved in:
Bibliographic Details
Title: The Focus, Function and Framing of Feedback Information: Linguistic and Content Analysis of In-Text Feedback Comments
Language: English
Authors: Derham, Cathrine (ORCID 0000-0002-2014-4496), Balloo, Kieran (ORCID 0000-0002-1745-4653), Winstone, Naomi (ORCID 0000-0001-8157-8274)
Source: Assessment & Evaluation in Higher Education. 2022 47(6):896-909.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 14
Publication Date: 2022
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Feedback (Response), Content Analysis, Essays, Summative Evaluation, Grades (Scholastic), Writing Evaluation, Correlation, Language Usage, Error Correction, Language Styles, Evaluators, Student Attitudes, Emotional Response, Health Sciences, Psychology, Undergraduate Students
DOI: 10.1080/02602938.2021.1969335
ISSN: 0260-2938
1469-297X
Abstract: In-text comments, in the form of annotations on students' work, are a form of feedback information that should guide students to take action. Both the focus of the in-text comments, and the ways in which they are linguistically communicated, have potential to impact upon the way in which they are perceived by students. This study reports on an analysis of 2101 in-text comments added by markers to 60 summative essays from two disciplines. The majority of comments, regardless of the grade awarded, were found to be directed at the task performance, rather than relating to the level of process or self-regulation. Work awarded higher grades received fewer annotations; these essays were found to include more feedback comments expressing a positive tone, with limited opportunities for informing further development. Work awarded lower grades mainly received corrective comments, as well as comments characterised by interrogative language and words expressing risk. It is argued that the linguistic style may influence engagement with in-text comments, impacting upon students' affective and emotional states, and their level of cognitive engagement with the feedback information. Recommendations for markers' practices are identified, to facilitate the opportunities for engagement and action that in-text comments might afford.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1361068
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGmsPfp-SsHhU2ceKEhzufdAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDK0BOyWZABdVoQG_PgIBEICBm-noKU-W-DGGsh1fibJ8IiTCBCYkFRXwikOiwFkKeGxw5om-BSmsQXsVfPelEVlE3Y8XLlbfW1Xu8qA2by4SuBQIPM33dgze40fu3uH2-c-tzyJeqmoR0Wrwsm4Z_FAw7q-aHvxJ8XmbzHEcCEXurmAxcV_sbRdtHl2lr6Y147y-uovgitsyxiql51kHN4MhLY9fIXXDZMM_wWR-
Text:
  Availability: 1
  Value: <anid>AN0158478727;eva01sep.22;2022Aug16.02:30;v2.2.500</anid> <title id="AN0158478727-1">The focus, function and framing of feedback information: linguistic and content analysis of in-text feedback comments </title> <p>In-text comments, in the form of annotations on students' work, are a form of feedback information that should guide students to take action. Both the focus of the in-text comments, and the ways in which they are linguistically communicated, have potential to impact upon the way in which they are perceived by students. This study reports on an analysis of 2101 in-text comments added by markers to 60 summative essays from two disciplines. The majority of comments, regardless of the grade awarded, were found to be directed at the task performance, rather than relating to the level of process or self-regulation. Work awarded higher grades received fewer annotations; these essays were found to include more feedback comments expressing a positive tone, with limited opportunities for informing further development. Work awarded lower grades mainly received corrective comments, as well as comments characterised by interrogative language and words expressing risk. It is argued that the linguistic style may influence engagement with in-text comments, impacting upon students' affective and emotional states, and their level of cognitive engagement with the feedback information. Recommendations for markers' practices are identified, to facilitate the opportunities for engagement and action that in-text comments might afford.</p> <p>Keywords: annotations; written feedback; feedback modalities; linguistic style; self-regulation</p> <hd id="AN0158478727-2">Types of written feedback information</hd> <p>Within higher education, written feedback has become the dominant form of feedback information (Agricola, Prins, and Sluijsmans [<reflink idref="bib1" id="ref1">1</reflink>]). Feedback is referred to as 'a process through which learners make sense of information from various sources and use it to enhance their work or learning strategies' (Carless and Boud [<reflink idref="bib8" id="ref2">8</reflink>], 1315). Carless ([<reflink idref="bib5" id="ref3">5</reflink>]) refers to this as the new paradigm of feedback, which is contrasted with the old paradigm where feedback merely serves to transfer information. Whilst feedback with characteristics of the new paradigm is identified by students as being of utmost importance (Winstone et al. [<reflink idref="bib34" id="ref4">34</reflink>]), it has been well established across the sector that student experience surveys frequently indicate dissatisfaction with assessment and feedback (Pitt and Norton [<reflink idref="bib23" id="ref5">23</reflink>]). Thus, there is a need to continue to review feedback practices and the quality of written feedback. The new paradigm advocates active participation by encouraging students to use feedback, which is essential for self-regulation (Carless [<reflink idref="bib6" id="ref6">6</reflink>]; O'Donovan, Rust, and Price [<reflink idref="bib20" id="ref7">20</reflink>]). This may be more challenging to facilitate within a post-Covid world, if practices continue to involve fewer opportunities for face-to-face interaction and discussion, meaning that written feedback practices will need to be even more effective.</p> <p>Written feedback information is commonly provided as comments added directly onto students' work, which we refer to as in-text comments. This is often used in conjunction with rubric-referenced feedback, but the purpose and function of these different forms of feedback are not always clear. Nordrum, Evans, and Gustafsson ([<reflink idref="bib18" id="ref8">18</reflink>]) identified that students viewed in-text comments and rubrics as having different feedback functions. Students regarded in-text comments as useful for task-related concerns, such as suggesting corrections or highlighting errors. Rubric-referenced feedback was viewed as useful for a more holistic overview and identifying new perspectives or understandings. Dirkx et al. ([<reflink idref="bib9" id="ref9">9</reflink>]) stress the importance of considering the intention of the different types of feedback and the importance of identifying that the focus of each is clear for both markers and students. They argue that written comments using a rubric provide greater potential for more detailed comments at the level of self-regulation than in-text comments, which are more suited to providing information on the task.</p> <p>Feedback delivered as in-text comments can be helpful in providing bite-sized pieces of information, situated in the place to which the comment refers (Rand [<reflink idref="bib25" id="ref10">25</reflink>]; Watkins et al. [<reflink idref="bib30" id="ref11">30</reflink>]). This has the effect of enabling the markers' words to interact directly with the students', potentially influencing students' interaction with the comments and encouraging engagement (Liu [<reflink idref="bib16" id="ref12">16</reflink>]). This is perhaps more useful than longer narrative style feedback, as students are inclined to only read written feedback once and therefore messages need to be powerful and succinct (Rand [<reflink idref="bib25" id="ref13">25</reflink>]).</p> <p>Arts, Jaspers, and Joosten-ten Brinke ([<reflink idref="bib2" id="ref14">2</reflink>]) carried out an analysis of eight pieces of formative work which contained a total of 299 annotated comments, using Hattie and Timperley ([<reflink idref="bib11" id="ref15">11</reflink>]) seminal model of feedback as a coding framework. Hattie and Timperley argue that feedback tends to be directed at different levels: self, task, process and self-regulation. They claimed that personal feedback focused on the self is likely to be too disconnected from the task or performance to impact on learning, and feedback at task level is only effective if it can be used to improve future performance. Feedback about the process and at the level of self-regulation are the most effective forms of feedback, having potential to inform future practice. Arts et al. discovered that most annotated comments in their sample related to the task or were classified as corrective comments. Explanations about how to improve future work were scarce. Dirkx et al. ([<reflink idref="bib9" id="ref16">9</reflink>]) analysis of in-text comments also identified that most were focused upon task-specific corrections and evaluative advice, with directive comments being provided in the rubric-referenced feedback.</p> <p>Whilst rubrics may potentially provide a valuable source of directive comments, Liu ([<reflink idref="bib16" id="ref17">16</reflink>]) has also pointed towards the value of in-text comments for encouraging students' engagement with the feedback. Therefore, it is our contention that in-text comments should be both task and process focused, because of their perceived benefit in terms of increasing engagement, thus enabling students to take action. Whilst previous analyses provide an indication of the focus of feedback comments, they do not provide insight into <emph>how</emph> comments are conveyed. Thus, it is important to also consider the linguistic features used by markers as an important influence on the impact of feedback on learning.</p> <hd id="AN0158478727-3">Language used in written feedback</hd> <p>Both the content of the message and the way in which it is communicated is important, as with the absence of facial and vocal cues written feedback can be misconstrued (Winstone and Carless [<reflink idref="bib33" id="ref18">33</reflink>]). The manner in which the feedback message is conveyed can generate an emotional response which has the potential to affect student engagement and thus influence on-going development (Lipnevich and Smith [<reflink idref="bib15" id="ref19">15</reflink>]). Students are inclined to engage more with positively evaluated feedback and disengage from negative comments (Pitt and Norton [<reflink idref="bib23" id="ref20">23</reflink>]). Negatively-phrased comments may lower self-esteem and perceived self-efficacy, impacting upon the learners' beliefs about themselves, and confidence in their ability and behaviours (Ryan and Henderson [<reflink idref="bib26" id="ref21">26</reflink>]). Boud ([<reflink idref="bib4" id="ref22">4</reflink>]) suggests that critical or judgmental language may provoke resentment and apathy which subsequently impacts upon learning. In addition, feedback information that focuses upon highlighting errors may be regarded as personal criticism, obscuring the value of the feedback message. Students may find it difficult to make a distinction between identifying errors made and feeling like they have failed as an individual (Yorke [<reflink idref="bib35" id="ref23">35</reflink>]).</p> <p>The use of academic language, jargon and vague comments can act as barriers to the interpretation of feedback information (Sutton [<reflink idref="bib28" id="ref24">28</reflink>]), leading students to disregard it and eliminate its potential for learning (Weaver [<reflink idref="bib31" id="ref25">31</reflink>]). Whilst words and phrases which suggest delight may encourage engagement, the use of annotation such as '??' or '!!' may be interpreted as incredulity or irritability on behalf of the marker (Ball [<reflink idref="bib3" id="ref26">3</reflink>]). Although authentic questions posed by the marker have potential to promote internal dialogue and action, interrogatives such as 'Why?' or 'Evidence?', may convey a negative tone and require contextualisation in order to be useful to the student (Ball [<reflink idref="bib3" id="ref27">3</reflink>]).</p> <hd id="AN0158478727-4">Feedback information and attainment level</hd> <p>The number of feedback comments given, depends to an extent on the student's level of achievement (Hyland [<reflink idref="bib14" id="ref28">14</reflink>]). Students performing well often receive fewer comments and less guidance for improvement, perhaps based on the assumption that they will be happy to have performed well and have less issues to 'correct'. Orsmond and Merry ([<reflink idref="bib21" id="ref29">21</reflink>]) compared undergraduates' responses to feedback and found high achieving students placed greater emphasis on self-direction following feedback, believing they could manage with less feedback, as long as they had information about how they had performed which could subsequently be used for their own self-assessment. In contrast, non-high achieving students placed greater emphasis on external regulation from markers and were less likely to generate questions and actions for themselves on the basis of feedback information.</p> <p>In order to optimise the potential of in-text feedback, both the focus and the linguistic features of the message need to be considered. This study analyses written feedback information presented in the form of annotations on students' work, with the aim of providing insight into both the focus and function of the comment, and the framing in terms of how the marker expresses an intention through linguistic features. The following research questions were posed:</p> <p></p> <ulist> <item> What is the focus of in-text feedback comments?</item> <p></p> <item> Does the usage of different types of in-text feedback comments vary according to grade awarded?</item> <p></p> <item> Do the linguistic features of in-text feedback comments vary according to grade awarded?</item> <p></p> <item> Do different types of in-text feedback comments differ in their linguistic features?</item> </ulist> <hd id="AN0158478727-5">Method</hd> <p></p> <hd id="AN0158478727-6">Data source</hd> <p>Favourable ethical opinion was obtained to use anonymised feedback comments for the purpose of this research. In-text comments (<emph>N</emph> = 2101) that were annotated on 60 essays formed the data source for the study. Essays were selected from three health sciences modules and three psychology modules, across each level of undergraduate study (i.e. first year, second year and third/final year), and spread evenly across grade boundaries (i.e. 12 essays graded under the pass threshold of 40%; 12 essays graded 40-49%; 12 essays graded 50-59%; 12 essays graded 60-69%; and 12 essays graded 70% and above).</p> <hd id="AN0158478727-7">Content of in-text feedback comments</hd> <p>In order to analyse the in-text comments, a coding scheme was devised (see Table 1) based upon Hattie and Timperley ([<reflink idref="bib11" id="ref30">11</reflink>]) theoretical model. As this model has been used in previous studies (e.g. Arts, Jaspers, and Joosten-ten Brinke [<reflink idref="bib2" id="ref31">2</reflink>]; Dirkx et al. [<reflink idref="bib9" id="ref32">9</reflink>]), it was considered a useful framework for analysing feedback comments. The category 'feedback on task' was broken down into three sub-aspects: edits, evaluative comments and corrective advice. An initial review of the data suggested that comments related to this category were exceptionally varied and detailed and would have not been captured by classifying this feedback category in a single code. The sub-codes therefore enabled additional depth to the analysis.</p> <p>Table 1. Model of feedback to enhance learning coding scheme.</p> <p> <ephtml> <table><thead><tr><td /><td>Feedback Category Level</td><td>Feedback Category Descriptor (<italic>with examples of in-text feedback comments</italic>)</td></tr></thead><tbody valign="top"><tr><td>Feedback on self-regulation level</td><td char=".">6</td><td>Informative developmental guidance <italic>You'll recall we discussed the strengths and limitations of different methodologies in class. Reconsider your argument for the method you've chosen; consider both its strengths and limitations and thereby provide a stronger rationale.</italic></td></tr><tr><td>Feedback related to the process</td><td char=".">5</td><td>Directive developmental guidance in relation to future work <italic>When referencing, you don't need to include the author's Christian name. Please use this link to access and check referencing guidelines to ensure this is correctly referenced in the future.</italic></td></tr><tr><td>Feedback related to task performance</td><td char=".">4</td><td>Corrective or directive advice related to task <italic>This isn't a correct interpretation of this theoretical model. You need to relate this to your own experience and evidence with reference to the literature.</italic></td></tr><tr><td /><td char=".">3</td><td>Evaluative comment related to task <italic>This is a good example to illustrate the case. You've repeated the same point made earlier in your work.</italic></td></tr><tr><td /><td char=".">2</td><td>Edit <italic>A full stop is required. Omitted word 'care'.</italic></td></tr><tr><td>Feedback on the self or personal level</td><td char=".">1</td><td>Personal Opinion <italic>You have worked hard.</italic></td></tr></tbody></table> </ephtml> </p> <p>In order to ensure the scheme was able to be used to consistently to categorise comments, Lombard, Snyder-Duch, and Campanella Bracken ([<reflink idref="bib17" id="ref33">17</reflink>]) guidance for the assessment and reporting of inter-coder reliabilities in content analyses was followed. After coder training in which all authors and a research assistant independently coded all of the comments on four essays (two from each discipline) against a draft version of the scheme, an appropriate level of agreement was reached. The scheme and coding instructions were refined further, followed by a formal pilot coding stage where 30 randomly selected comments were independently coded by the first author and the research assistant. A Krippendorff's alpha (Hayes and Krippendorff [<reflink idref="bib12" id="ref34">12</reflink>]) coefficient of.83 was obtained, which is seen to be an acceptable level of reliability in most situations (Lombard, Snyder-Duch, and Campanella Bracken [<reflink idref="bib17" id="ref35">17</reflink>]). Clarifications were discussed for any discrepancies and changes were agreed upon. Comments coded during the training and pilot stages were not included in the final sample of comments that were analysed.</p> <p>During the formal reliability checking stage, 10% of the total comments (<emph>n</emph> = 212) were randomly selected to be independently coded by the first author and the research assistant, and a Krippendorff's alpha value of.85 was obtained. Subsequently, the research assistant coded all remaining comments.</p> <hd id="AN0158478727-8">Linguistic features of in-text feedback comments</hd> <p>To assess the linguistic features of in-text comments, we used the Linguistic Inquiry and Word Count (LIWC) text analysis software (Pennebaker et al. [<reflink idref="bib22" id="ref36">22</reflink>]). LIWC returns basic information about text excerpts (e.g. the number of words, number of verbs, etc.) as well as analysing the percentage of words in a text excerpt falling under a range of different linguistic categories. For our analysis, we selected linguistic categories that aligned with aspects of feedback practice thought to have potential impact upon students.</p> <p>Henderson et al. ([<reflink idref="bib13" id="ref37">13</reflink>]) outline different ways in which feedback may have impact on students, and we used these to organise our exploratory linguistic analysis (see Table 2). They argue that feedback information may have relational impacts, influencing students' perceptions of credibility, trust, perceived safety and threat. These impacts can influence students' engagement with feedback (e.g. feedback seeking, interpretation and action). We selected four linguistic categories from LIWC aligning with the relational impacts of feedback: personal pronouns, impersonal pronouns, words associated with power, and words associated with affiliation (Table 2).</p> <p>Table 2. Linguistic categories.</p> <p> <ephtml> <table><thead><tr><td>Impact type</td><td>Linguistic indicators</td><td>Examples</td></tr></thead><tbody valign="top"><tr><td>Relational</td><td>Personal pronouns</td><td>I, you, your, we, our</td></tr><tr><td>Impersonal pronouns</td><td>It, that, this, those</td></tr><tr><td>Power</td><td>Judge, comply, inferior, follow, reject, request, weakness</td></tr><tr><td>Affiliation</td><td>Help, let's own, share, together, we</td></tr><tr><td>Affective/motivational</td><td>Positive emotion</td><td>Best, confident, engaging, excellent, interesting, strong</td></tr><tr><td>Negative emotion</td><td>Bad, careless, confused, difficult, missing, poor, weak</td></tr><tr><td>Reward focus</td><td>Achieve, earned, gain, good, reward, success</td></tr><tr><td>Risk focus</td><td>Beware, careful, caution, doubt, fail, risk</td></tr><tr><td>Achievement</td><td>Ability, attain, advantage, determination, try, effort, excellent, progress</td></tr><tr><td>Cognitive</td><td>Interrogatives</td><td>How, what, why, where, when</td></tr><tr><td>Negations</td><td>Aren't, can't, didn't, haven't, done, shouldn't</td></tr><tr><td>Discrepancies</td><td>Could've, couldn't, lack, inadequate, needed, should, unnecessary</td></tr><tr><td>Differentiation</td><td>But, didn't, either, else, if, otherwise</td></tr><tr><td>Insight</td><td>Aware, choice, complicated, decide, wonder, consider, notice, think, perspective</td></tr><tr><td>Causal language</td><td>Because, create, depend, lead, result, use</td></tr><tr><td>Assent</td><td>Absolutely, agree, ok, yes, alright</td></tr><tr><td>Questions</td><td>?</td></tr><tr><td>Tentativeness</td><td>Almost, appear, guess, hope, if, mainly, maybe, might, quite, sometimes</td></tr><tr><td>Certainty</td><td>Absolutely, always, all, clearly, extremely, must, never, specific</td></tr></tbody></table> </ephtml> </p> <p>Henderson et al. ([<reflink idref="bib13" id="ref38">13</reflink>]) also describe potential affective and motivational impacts, where feedback information has the potential to influence students' affective and motivational states. LIWC includes linguistic categories associated with affective processes. Five categories were selected representing words associated with: positive emotion, negative emotion, reward focus, risk focus and achievement (Table 2).</p> <p>Finally, Henderson et al. ([<reflink idref="bib13" id="ref39">13</reflink>], 27) draw attention to the ways in which feedback information can impact on students' thinking, where comments can influence 'the way in which [students] attend to details, process information, form concepts, and how they store and retrieve memory'. Drawing from the linguistic categories in LIWC associated with cognitive processes, we focused on the extent to which in-text comments contained words representing: interrogatives, negations, discrepancies, differentiation, insight, causal language, assent, questions, tentativeness and certainty (Table 2).</p> <hd id="AN0158478727-9">Results</hd> <p></p> <hd id="AN0158478727-10">What is the focus of in-text feedback comments?</hd> <p>Table 3 shows the usage of in-text comments on each essay in terms of the number of comments used on student work and how they were coded within each category of the coding scheme.</p> <p>Table 3. Usage of in-text feedback comments coded within each category per essay.</p> <p> <ephtml> <table><thead><tr><td>Feedback Category Level</td><td>In-text feedback comment category ​</td><td>Min​</td><td>Max​</td><td><italic>M​</italic></td><td><italic>SD​</italic></td></tr><tr><td><italic>f</italic>​</td><td>%​</td><td><italic>f</italic>​​</td><td>%​</td><td><italic>f</italic>​​</td><td>%​</td><td><italic>f</italic>​​</td><td>%​</td></tr></thead><tbody valign="top"><tr><td>N/A</td><td>Total number comments​</td><td char=".">8​</td><td /><td char=".">119​</td><td /><td char=".">35​</td><td /><td char=".">27​</td><td /></tr><tr><td char=".">6</td><td>Informative developmental guidance​</td><td char=".">0​</td><td char=".">0​</td><td char=".">13​</td><td char=".">25​</td><td char=".">1​</td><td char=".">1​</td><td char=".">2​</td><td char=".">4​</td></tr><tr><td char=".">5</td><td>Directive developmental guidance in relation to future work​</td><td char=".">0​</td><td char=".">0​</td><td char=".">18​</td><td char=".">42​</td><td char=".">4​</td><td char=".">12​</td><td char=".">4​</td><td char=".">9​</td></tr><tr><td char=".">4</td><td>Corrective or directive advice related to task​</td><td char=".">1​</td><td char=".">7​</td><td char=".">68​</td><td char=".">78​</td><td char=".">17​</td><td char=".">44​</td><td char=".">16​</td><td char=".">17​</td></tr><tr><td char=".">3</td><td>Evaluative statement or comment related to task​</td><td char=".">0​</td><td char=".">0​</td><td char=".">47​</td><td char=".">93​</td><td char=".">11​</td><td char=".">37​</td><td char=".">8​</td><td char=".">21​</td></tr><tr><td char=".">2</td><td>Edit​</td><td char=".">0​</td><td char=".">0​</td><td char=".">15​</td><td char=".">30​</td><td char=".">2​</td><td char=".">5​</td><td char=".">3​</td><td char=".">6​</td></tr><tr><td char=".">1</td><td>Personal opinion​</td><td char=".">0​</td><td char=".">0​</td><td char=".">0​</td><td char=".">0​</td><td char=".">0​</td><td char=".">0​</td><td char=".">0​</td><td char=".">0​</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note.</emph> Values have been rounded to the closest whole number.</p> <p>Breaking down Table 3, the number of in-text feedback comments ranged from 8 to 119 annotations (<emph>M</emph> = 35.02, <emph>SD</emph> = 27.22) on each essay. Of these comments, only those coded as 'corrective or directive advice related to task' (feedback category level 4) were present on all essays in the sample. The majority of comments on each essay was coded as 'corrective or directive advice related to task' (on average, 44% of in-text comments on each essay were coded to this category) and 'evaluative statement or comment related to task' (feedback category level 3; an average of 37% of in-text comments on each essay were coded to this category). An average of around 12% of comments on each essay were coded as 'directive developmental guidance in relation to future work' (feedback category level 5). Only an average of 5% of comments on each essay were coded as 'edit' (feedback category level 2) and 1% coded as 'informative developmental guidance' (feedback category level 6). No comments were coded as 'personal opinion' (feedback category level 1).</p> <p>Due to the low number of 'informative developmental guidance' comments (only six essays included comments that were coded to this category), this category was excluded from further analyses. The number of comments coded to each category for each essay formed variables for use in further statistical analyses. Technical details about data cleaning and rationales for using each of the statistical tests can be found in the online supplemental material.</p> <p>Focusing only on feedback categories 2 to 5, a one-way within-subject analysis of variance (ANOVA) using Greenhouse-Geisser corrections confirmed that there was a significant difference in the usage of feedback comment categories across essays, <emph>F</emph>(2.48, 146.25) = 102.07, <emph>p</emph> <.001, η<subs>p</subs><sups>2</sups> =.63. Post hoc pairwise comparisons between all feedback categories were also significant (all <emph>p</emph>s <.001 after Bonferroni adjustments), except for the difference between comments coded as 'evaluative statement or comment related to task' and 'corrective or directive advice related to task', <emph>p</emph> =.313 (after Bonferroni adjustment).</p> <hd id="AN0158478727-11">Does the usage of different types of in-text feedback comments vary according to grade awarde...</hd> <p>To determine whether there were relationships between the usage of comments coded to each category and essay grades awarded, scatterplots were scrutinised for linear relationships and Pearson's correlations were then performed. These correlation analyses were initially run separately for each discipline and for each year of study. However, the patterns of each correlation between feedback categories and grades did not differ across disciplines or year of study (although there were differences in statistical significance of effects due to the reduced sample size), so these were combined into a single data set for the analyses. There was no relationship between the usage of comments coded as 'edit' (level 2) and the grade given to the essay, <emph>r</emph> = −.19, <emph>p</emph> =.148. There was a significant positive relationship between grades and usage of comments coded as 'evaluative statement or comment related to task' (level 3), <emph>r</emph> =.32, <emph>p</emph> =.014. There were significant negative relationships between grades and usage of comments coded as 'corrective or directive advice related to the task' (level 4), <emph>r</emph> = −.44, <emph>p</emph> <.001, or 'directive developmental guidance in relation to future work' (level 5), <emph>r</emph> = −.50, <emph>p</emph> <.001. Finally, there was a significant negative relationship between grades and the total number of in-text feedback comments on essays, <emph>r</emph> = −.29, <emph>p</emph> =.027. Significant correlations with grades are depicted in the scatterplots in Figure 1.</p> <p>PHOTO (COLOR): Figure 1. Scatterplots showing the relationship between grades and usage of feedback comments coded at different levels, and between grades and total number of in-text feedback comments on the essay.</p> <p>A multiple regression analysis was performed with the usage of feedback comment categories as predictors and grades as the outcome variable, to determine whether relationships still existed when controlling for each other in the same model. The 'edit' category was not included, as it did not significantly correlate with grades.</p> <p>The regression model suggests that feedback categories explain 44% of the variance in grades. As Table 4 shows, after controlling for the three feedback comment categories in the same analysis, usage of comments coded as 'evaluative statement or comment related to task' (level 3) significantly positively predicted grades, whereas usage of comments coded as 'corrective or directive advice related to the task' (level 4) or 'directive developmental guidance in relation to future work' (level 5) significantly negatively predicted grades.</p> <p>Table 4. Regression model of the usage of in-text feedback comments coded within each category as predictors of grades.</p> <p> <ephtml> <table><thead><tr><td /><td><italic>B</italic></td><td><italic>SE B</italic></td><td>β</td><td><italic>p</italic></td></tr></thead><tbody valign="top"><tr><td>Constant</td><td char=".">55.11</td><td char=".">6.78</td><td /><td char="."><.001</td></tr><tr><td>Evaluative statement or comment related to task (level 3)</td><td char=".">23.78</td><td char=".">5.90</td><td char=".">.43</td><td char="."><.001</td></tr><tr><td>Corrective or directive advice related to the task (level 4)</td><td>−15.34</td><td char=".">6.10</td><td>-.36</td><td char=".">.015</td></tr><tr><td>Directive developmental guidance in relation to future work (level 5)</td><td>−13.18</td><td char=".">6.47</td><td>-.28</td><td char=".">.047</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Note. R</emph><sups>2</sups> =.44, <emph>F</emph>(<reflink idref="bib3" id="ref40">3</reflink>, 55) = 14.41, <emph>p</emph> <.001.</p> <hd id="AN0158478727-12">Do the linguistic features of in-text feedback comments vary according to the grade awarded?</hd> <p>LIWC software was used to calculate the percentage of words in each in-text feedback comment that formed part of each of the selected linguistic domains in Table 2. These percentages were then averaged for each essay to produce the average percentage of words within each linguistic domain across all comments in that essay. In order to understand whether the language used in in-text feedback comments was related to the grades assigned to the essay on which the comment was present, a series of Pearson's correlation analyses were performed. Table 5 displays correlations between the linguistic features of comments and grades.</p> <p>Table 5. Correlations between the linguistic features of in-text feedback comments and grades.</p> <p> <ephtml> <table><thead><tr><td>Linguistic domain</td><td>Grades [<italic>r</italic>]</td></tr></thead><tbody valign="top"><tr><td>Relational</td></tr><tr><td> Personal pronouns</td><td>-.16 [-.40,.10]</td></tr><tr><td> Impersonal pronouns</td><td>-.20 [-.41,.06]</td></tr><tr><td> Power</td><td char=".">.004 [-.28,.29]</td></tr><tr><td> Affiliation</td><td char=".">.07 [-.16,.33]</td></tr><tr><td>Affective/motivational</td></tr><tr><td> Positive emotion</td><td char=".">.57** [.35,.73]</td></tr><tr><td> Negative emotion</td><td char=".">.11 [-.24,.56]</td></tr><tr><td> Reward focus</td><td char=".">.61** [.41,.76]</td></tr><tr><td> Risk focus</td><td>-.43** [-.57, −.30]</td></tr><tr><td> Achievement</td><td char=".">.24 [-.09,.43]</td></tr><tr><td>Cognitive</td></tr><tr><td> Interrogatives</td><td>-.23 [-.42, −.02]</td></tr><tr><td> Negations</td><td>-.13 [-.39,.15]</td></tr><tr><td> Discrepancies</td><td>-.04 [-.27,.17]</td></tr><tr><td> Differentiation</td><td>-.14 [-.36,.06]</td></tr><tr><td> Insight</td><td>-.13 [-.37,.12]</td></tr><tr><td> Causal language</td><td>-.20 [-.42,.06]</td></tr><tr><td> Assent</td><td>-.19 [-.41,.30]</td></tr><tr><td> Questions</td><td>-.37* [-.53, −.20]</td></tr><tr><td> Tentativeness</td><td>-.17 [-.37,.05]</td></tr><tr><td> Certainty</td><td>-.0003 [-.30,.31]</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note.</emph> *<emph>p</emph> <.01; **<emph>p</emph> <.001. Bias-corrected and accelerated bootstrap 95% CIs reported in parentheses.</p> <p>Significant positive relationships were found between grades and the use of words expressing positive affective/motivational impacts, including positive emotion (e.g. '<bold>Nice</bold> clear start with a <bold>good</bold> overview') and having a reward focus (e.g. '<bold>Great</bold> discussion'). Significant negative relationships were found between grades and words expressing negative affective/motivational impacts, including words with a risk focus (e.g. 'Please be <bold>careful</bold> with your grammar'), and between grades and cognitive impacts, including interrogatives (e.g. '<bold>Why</bold>?'), and the use of questions (e.g. How do you know this<bold>?</bold> Evidence<bold>?</bold>).</p> <hd id="AN0158478727-13">Do different types of in-text feedback comments differ in their linguistic features?</hd> <p>To determine whether comments coded to the different feedback categories in the coding scheme differed in their linguistic features, we examined the average percentage of words in comments coded to each feedback category that formed part of each linguistic domain (we did not examine the 'edit' category, as these comments consisted of corrections to the existing text, meaning that the linguistic features would not make sense outside of the context of the essay on which they were written). Wilcoxon signed-rank tests were used to analyse pairwise comparisons between each of the feedback categories for each linguistic domain (a complete account of these statistical tests can be found in the online supplemental material). Figure 2 displays the linguistic features of comments coded to each feedback category with significant differences summarised.</p> <p>PHOTO (COLOR): Figure 2. Linguistic features of the in-text comments coded to each feedback category. a = significant difference between levels 3 and 4; b = significant difference between levels 3 and 5; c = significant difference between levels 4 and 5.</p> <p>As Figure 2 shows, comments coded as 'evaluative statements or comments related to task' (level 3) are defined by having more words expressing affective/motivational impacts (positive emotion and reward focus), and fewer words expressing either relational impacts (impersonal pronouns) or cognitive impacts (interrogatives, discrepancies, insight, causal language, and questions). Comments coded as 'corrective or directive advice related to the task' (level 4) are defined by the use of more words expressing either relational impacts (impersonal pronouns) or cognitive impacts (interrogatives, discrepancies, insight, causal language, and questions), and fewer comments expressing affective/motivational impacts (achievement and positive emotion). Comments coded as 'directive developmental guidance in relation to future work' (level 5) include more words expressing certain affective/motivational impacts (achievement), but fewer words expressing other affective/motivational impacts (positive emotion). These comments also include more words expressing certain cognitive impacts (discrepancies), but fewer words expressing other cognitive impacts (interrogatives and questions). Finally, level five comments include fewer words expressing relational impacts (impersonal pronouns).</p> <hd id="AN0158478727-14">Discussion and recommendations</hd> <p>Whereas previous research has focused upon the content of in-text comments, this study has also considered the linguistic features of the feedback messages, thereby combining two elements which may impact upon the way in which feedback information is perceived by students. In keeping with other research findings (Nordrum, Evans, and Gustafsson [<reflink idref="bib18" id="ref41">18</reflink>]; Arts, Jaspers, and Joosten-ten Brinke [<reflink idref="bib2" id="ref42">2</reflink>]; Dirkx et al. [<reflink idref="bib9" id="ref43">9</reflink>]), our results indicate that in-text comments are primarily related to task performance, rather than process, and comments at the level of self-regulation were rarely identified. This focus on task is information-centric and reflects old paradigm feedback practice (Carless [<reflink idref="bib5" id="ref44">5</reflink>]). In order to provide feedback with characteristics of the new paradigm, a better balance needs to be achieved between feedback at the level of task, process and self-regulation. This is not to suggest additional feedback is required, but instead that markers should focus upon providing feedback at the level of process and self-regulation.</p> <p>Crafting feedback which enables students to develop their understanding and future work is a more complex process than that which highlights errors (Price et al. [<reflink idref="bib24" id="ref45">24</reflink>]), but benefits may be achieved with a change of focus, to reflect practices associated with deep annotators (Liu [<reflink idref="bib16" id="ref46">16</reflink>]). Deep annotators provide critical and analytical comments, whereas surface annotators tend to reuse information and write comments which lack analytical engagement with the text. Lipnevich and Smith ([<reflink idref="bib15" id="ref47">15</reflink>]) argue that a consequence of surface level feedback is surface level understanding and engagement by the student, thus limiting opportunities to take action.</p> <p>Essays awarded higher grades received fewer comments. The feedback served to affirm the grade and offered opportunities for evaluative judgement (Tai et al. [<reflink idref="bib29" id="ref48">29</reflink>]), but offered little developmental guidance. The lack of directive comments and comments at the level of self-regulation leaves students with less opportunity for on-going development. This is despite the fact that high achieving students are receptive to critical feedback comments, because they help them to attain their high grade goals (Carless [<reflink idref="bib7" id="ref49">7</reflink>]). Not surprisingly the linguistic style for these students reflected a positive affect and was reward focused. This is likely to elicit a positive emotional reaction and students are subsequently more likely to process and make use of the feedback (Lipnevich and Smith [<reflink idref="bib15" id="ref50">15</reflink>]). However, positive comments without directive guidance do not provide advice which makes action possible. All students, including those with high marks, require feedback which enables them to enhance future work. It is therefore recommended that markers refocus and ensure the feedback provided to these students achieves this goal.</p> <p>It is important to consider the linguistic style of comments in order to encourage engagement and action. Corrective advice was often phrased in a style which did not provide decisive direction. Phrases such as '<emph>You might like to...' </emph>suggests optionality and an invitation to respond. '<emph>You should always ensure...' </emph>provides more certainty but should be accompanied by an explanation as to why this direction has been given. Corrective feedback was prevalent in work which received lower grades and featured words with negative connotations such as '<emph>careless' </emph>and '<emph>poor'.</emph> In addition, comments were often risk focused: E.g. '<emph>This is a poor explanation of this concept and you need to be careful with referencing'</emph>. Such feedback may lead to feelings of anxiety, a sense of inadequacy and lower self-esteem (Shields [<reflink idref="bib27" id="ref51">27</reflink>]).</p> <p>Rewording to encourage engagement and more direction is recommended: E.g. '<emph>I notice the explanation here does not include X and Y, which are essential to this concept and should form part of your explanation. You need to include the date for this reference source. Please use this link to access and check referencing guidelines to ensure this is correctly referenced in the future'</emph>. Although it could be argued that this longer, more detailed feedback may add to workload, it is suggested that fewer, more carefully worded comments would improve quality and student satisfaction. As Forsythe and Johnson ([<reflink idref="bib10" id="ref52">10</reflink>], 853) argue, learning from feedback is less effective when students 'spend time monitoring the extent to which they make mistakes because they will have less cognitive resources available to solve problems and questions posed to them'.</p> <p>Whilst written feedback is largely one-directional communication, the presentation of feedback as authentic or exploratory questions has potential to promote internal dialogue and enable action. Nystrand ([<reflink idref="bib19" id="ref53">19</reflink>]) suggests questions are a means of presenting a dialogic approach, which may encourage students to engage with feedback, if they are considered part of an interaction which prompts a response and invites further communication and action. Questions which appear as interrogatives, such as <emph>How?</emph> or <emph>Why?</emph> are unlikely to achieve a meaningful response, particularly when focused at the level of task, where the feedback is of limited value if the student is unlikely to revisit the assignment again. The same applies to the use of annotations such as? ?. To be effective, questions need to be posed at the level of process, containing direction and enabling self-regulation if the new paradigm is to be achieved. In addition, the use of personal pronouns and words expressing an affiliation are encouraged to foster a sense of relationship and encourage dialogue. For example, simply rephrasing '<emph>Please arrange a tutorial prior to submission' </emph>to '<emph>I can help you to develop your work before you resubmit, when we meet for a tutorial' </emph>may be all that is required to encourage engagement.</p> <p>The amount of feedback is arguably less important than the focus and linguistic style, whether this is provided as an in-text comment or within accompanying rubric-related forms. A change in framing feedback comments should not increase academic workload, but instead make the time spent more productive and effective. There is the need for shared understanding between markers and students about the purpose of both formative and summative feedback and both should have a landing place; that is, a future task or assignment, where the comments provided have relevance and thus support students in on-going development of skills or understanding. Winstone and Carless ([<reflink idref="bib33" id="ref54">33</reflink>]) argue that positive outcomes are achieved not through substantial changes requiring extra workload, but through the combined effects of small improvements to feedback that help achieve the paradigm change.</p> <hd id="AN0158478727-15">Limitations and opportunities for future research</hd> <p>The focus of this research was on in-text comments only, without consideration of accompanying rubric-related feedback information. Therefore, it is not known if more detailed, directive feedback at the level of self-regulation was provided in comments within a rubric or as a general overall comment about the work. Dirkx et al. ([<reflink idref="bib9" id="ref55">9</reflink>]) discovered more directive feedback and process-related comments in rubrics than in annotations, but the majority of comments were still task-focused. Thus, it is likely that students are not receiving a balance of feedback at different levels with the use of different modalities, but this would be an area to explore further both in terms of the focus of the comments and the linguistic features.</p> <p>A further limitation is the extent to which we could explore linguistic features, which was bound by the pre-set categories within the text analysis software (Pennebaker et al. [<reflink idref="bib22" id="ref56">22</reflink>]). To overcome this, a broad range of linguistic categories were selected which aligned with key potential areas of the impact: relational, affective and motivational, and cognitive aspects (Henderson et al. [<reflink idref="bib13" id="ref57">13</reflink>]). An area for future research could be to explore how students receive and act upon in-text feedback comments which present different linguistic features.</p> <p>Although we sampled essays across multiple years of students' study, we did not examine how the focus, function and framing of in-text comments might change as students progress through their courses and assignments become more demanding. Therefore, a longitudinal analysis of change in content and linguistic style of in-text comments might enrich any follow-up research.</p> <hd id="AN0158478727-16">Conclusion</hd> <p>Feedback provided as in-text comments is characterised primarily by remarks focused upon the task, without an accompanying sense of direction. This reflects the old paradigm, (Carless [<reflink idref="bib5" id="ref58">5</reflink>]) and has limited potential impact upon future development. The focus of this research reinforces the need to consider linguistic features which may encourage self-regulation. Whilst it is recognised that students need to understand their own role in the feedback process and levels of feedback literacy (Winstone, Balloo, and Carless [<reflink idref="bib32" id="ref59">32</reflink>]), providing feedback which facilitates student development by encouraging engagement, understanding and action, is the first vital step in this process. The way in which feedback is presented will have consequences, and the extent to which it prompts a response is likely to depend upon the affective and cognitive response of the student. Lipnevich and Smith ([<reflink idref="bib15" id="ref60">15</reflink>]) refer to written feedback as the start of a conversation, in that it provides direction and advice to which the student can then respond. It is helpful to consider in-text comments as the beginning of a conversation and thus markers need to be mindful of the way in which they open up that conversation, by providing feedback information that leaves students feeling motivated and able to take action.</p> <p>To achieve a paradigm shift there needs to be a better balance between feedback on task performance, directive developmental guidance, and feedback at the level of self-regulation. Markers can effectively convey their intention to inform, encourage and create a relationship with students through their linguistic style. Comments should be positively framed and supportive, even when there are criticisms of work, in order to enable students the opportunity to engage and take action. The intention and consequences of feedback, regardless of the format in which this is presented, therefore requires consideration. All students need the call to action that feedback can provide, regardless of the quality of their work. Currently, in-text comments do not appear to fulfil the potential to promote self-regulation. In order to maximise their impact, it is argued that feedback practices should be modified to ensure the content and style enables engagement, understanding, motivation and action. Modifications to the focus of comments and linguistic features, ensuring direction through the use of appropriate language, could have a more positive impact upon students' learning by maximising the benefit of in-text comments.</p> <hd id="AN0158478727-17">Acknowledgements</hd> <p>The authors thank María Norman for her assistance with data coding.</p> <hd id="AN0158478727-18">Disclosure statement</hd> <p>No potential conflict of interest was reported by the authors.</p> <ref id="AN0158478727-19"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Supplemental data for this article is available online at https://doi.org/10.1080/02602938.2021.1969335.</bibtext> </blist> </ref> <ref id="AN0158478727-20"> <title> References </title> <blist> <bibtext> Agricola, B. T., F. J. Prins, and D. M. A. Sluijsmans. 2020. " Impact of Feedback Request Forms and Verbal Feedback on Higher Education Students' Feedback Perception, Self-Efficacy, and Motivation." Assessment in Education: Principles, Policy & Practice 27 (1): 6 – 25. doi: 10.1080/0969594X.2019.1688764.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref14" type="bt">2</bibl> <bibtext> Arts, J. G., M. Jaspers, and D. Joosten-ten Brinke. 2016. " A Case Study on Written Comments as a Form of Feedback in Teacher Education: So Much to Gain." European Journal of Teacher Education 39 (2): 159 – 173. doi: 10.1080/02619768.2015.1116513.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref26" type="bt">3</bibl> <bibtext> Ball, E. C. 2010. " Annotation an Effective Device for Student Feedback: A Critical Review of the Literature." Nurse Education in Practice 10 (3): 138 – 143. doi: 10.1016/j.nepr.2009.05.003.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref22" type="bt">4</bibl> <bibtext> Boud, D. 1995. Enhancing Learning through Self-Assessment. Oxon : Routledge.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref3" type="bt">5</bibl> <bibtext> Carless, D. 2015a. Excellence in University Assessment: Learning from Award-Winning Practice. Abingdon, Oxford : Routledge.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref6" type="bt">6</bibl> <bibtext> Carless, D. 2015b. " Exploring Learning-Oriented Assessment Processes." Higher Education 69 (6): 963 – 976. doi: 10.1007/s10734-014-9816-z..</bibtext> </blist> <blist> <bibl id="bib7" idref="ref49" type="bt">7</bibl> <bibtext> Carless, D. 2019. " Learners' Feedback Literacy and the Longer Term: Developing Capacity for Impact." In The Impact of Feedback in Higher Education, edited by M. Henderson, R. Ajjawi, D. Boud, and E. Molloy, 51 – 65. Cham : Palgrave Macmillan. doi: 10.1007/978-3-030-25112-3_4.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref2" type="bt">8</bibl> <bibtext> Carless, D., and D. Boud. 2018. " The Development of Student Feedback Literacy: Enabling Uptake of Feedback." Assessment & Evaluation in Higher Education 43 (8): 1315 – 1325. doi: 10.1080/02602938.2018.1463354.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref9" type="bt">9</bibl> <bibtext> Dirkx, K., D. Joosten-ten Brinke, J. Arts, and M. van Diggelen. 2019. " Text and Rubric-Referenced Feedback: Differences in Focus, Level, and Function." Active Learning in Higher Education : 146978741985520. doi: 10.1177/1469787419855208.</bibtext> </blist> <blist> <bibtext> Forsythe, A., and S. Johnson. 2017. " Thanks, but No-Thanks for the Feedback." Assessment & Evaluation in Higher Education 42 (6): 850 – 859. doi: 10.1080/02602938.2016.1202190.</bibtext> </blist> <blist> <bibtext> Hattie, J., and H. Timperley. 2007. " The Power of Feedback." Review of Educational Research 77 (1): 81 – 112. doi: 10.3102/003465430298487.</bibtext> </blist> <blist> <bibtext> Hayes, A. F., and K. Krippendorff. 2007. " Answering the Call for a Standard Reliability Measure for Coding Data." Communication Methods and Measures 1 (1): 77 – 89. doi: 10.1080/19312450709336664.</bibtext> </blist> <blist> <bibtext> Henderson, M., R. Ajjawi, D. Boud, 2019. and, and E. Molloy. " Identifying Feedback That Has Impact." In The Impact of Feedback in Higher Education, edited by Michael Henderson, R. Ajjawi, D. Boud, and E. Molloy, 15 – 34. Cham : Palgrave Macmillan. doi: 10.1007/978-3-030-25112-3_2.</bibtext> </blist> <blist> <bibtext> Hyland, F. 2001. " Providing Effective Support: Investigating Feedback to Distance Language Learners." Open Learning: The Journal of Open, Distance and e-Learning 16 (3): 233 – 247. doi: 10.1080/02680510120084959.</bibtext> </blist> <blist> <bibtext> Lipnevich, A. A., and J. K. Smith. 2009. " I Really Need Feedback to Learn:' Students' Perspectives on the Effectiveness of the Differential Feedback Messages." Educational Assessment, Evaluation and Accountability 21 (4): 347 – 367. doi: 10.1007/s11092-009-9082-2.</bibtext> </blist> <blist> <bibtext> Liu, K. 2006. " Annotation as an Index to Critical Writing." Urban Education 41 (2): 192 – 207. doi: 10.1177/0042085905282261.</bibtext> </blist> <blist> <bibtext> Lombard, M., J. Snyder-Duch, and C. Campanella Bracken. 2004. "Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects.". <ulink href="http://matthewlombard.com/reliability/">http://matthewlombard.com/reliability/</ulink>.</bibtext> </blist> <blist> <bibtext> Nordrum, L., K. Evans, and M. Gustafsson. 2013. " Comparing Student Learning Experiences of in-Text Commentary and Rubric-Articulated Feedback: Strategies for Formative Assessment." Assessment & Evaluation in Higher Education 38 (8): 919 – 940. doi: 10.1080/02602938.2012.758229.</bibtext> </blist> <blist> <bibtext> Nystrand, M. 1997. Opening Dialogue: Understanding the Dynamics of Language and Learning in the English Classroom. London : Teachers College Press. doi: 10.2307/417942.</bibtext> </blist> <blist> <bibtext> O'Donovan, B., C. Rust, and M. Price. 2016. " A Scholarly Approach to Solving the Feedback Dilemma in Practice." Assessment & Evaluation in Higher Education 41 (6): 938 – 949. doi: 10.1080/02602938.2015.1052774.</bibtext> </blist> <blist> <bibtext> Orsmond, P., and S. Merry. 2013. " The Importance of Self-Assessment in Students' Use of Tutors' Feedback: A Qualitative Study of High and Non-High Achieving Biology Undergraduates." Assessment & Evaluation in Higher Education 38 (6): 737 – 753. doi: 10.1080/02602938.2012.697868.</bibtext> </blist> <blist> <bibtext> Pennebaker, J. W., R. L. Boyd, K. Jordan, and K. Blackburn. 2015. The Development and Psychometric Properties of LIWC2015. Austin, TX : University of Texas at Austin.</bibtext> </blist> <blist> <bibtext> Pitt, E., and L. Norton. 2017. " Now That's the Feedback I Want!' Students' Reactions to Feedback on Graded Work and What They Do with It." Assessment & Evaluation in Higher Education 42 (4): 499 – 516. doi: 10.1080/02602938.2016.1142500.</bibtext> </blist> <blist> <bibtext> Price, M., K. Handley, J. Millar, and B. O'Donovan. 2010. " Feedback: All That Effort, but What is the Effect? " Assessment & Evaluation in Higher Education 35 (3): 277 – 289. doi: 10.1080/02602930903541007.</bibtext> </blist> <blist> <bibtext> Rand, J. 2017. " Misunderstandings and Mismatches: The Collective Disillusionment of Written Summative Assessment Feedback." Research in Education 97 (1): 33 – 48. doi: 10.1177/0034523717697519.</bibtext> </blist> <blist> <bibtext> Ryan, T., and M. Henderson. 2018. " Feeling Feedback: Students' Emotional Responses to Educator Feedback." Assessment & Evaluation in Higher Education 43 (6): 880 – 892. doi: 10.1080/02602938.2017.1416456.</bibtext> </blist> <blist> <bibtext> Shields, S. 2015. " My Work is Bleeding': Exploring Students' Emotional Responses to First-Year Assignment Feedback." Teaching in Higher Education 20 (6): 614 – 624. doi: 10.1080/13562517.2015.1052786.</bibtext> </blist> <blist> <bibtext> Sutton, P. 2012. " Conceptualizing Feedback Literacy: Knowing, Being, and Acting." Innovations in Education and Teaching International 49 (1): 31 – 40. doi: 10.1080/14703297.2012.647781.</bibtext> </blist> <blist> <bibtext> Tai, J., R. Ajjawi, D. Boud, P. Dawson, and E. Panadero. 2018. " Developing Evaluative Judgement: Enabling Students to Make Decisions about the Quality of Work." Higher Education 76 (3): 467 – 481. doi: 10.1007/s10734-017-0220-3.</bibtext> </blist> <blist> <bibtext> Watkins, D., P. Dummer, K. Hawthorne, J. Cousins, C. Emmett, and M. Johnson. 2014. "Healthcare Students' Perceptions of Electronic Feedback through GradeMark®." Journal of Information Technology Education 13 : 27 – 47. <ulink href="http://www.jite.org/documents/Vol13/JITEv13ResearchP027-047Watkins0592.pdf">http://www.jite.org/documents/Vol13/JITEv13ResearchP027-047Watkins0592.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Weaver, M. R. 2006. " Do Students Value Feedback? Student Perceptions of Tutors' Written Responses." Assessment & Evaluation in Higher Education 31 (3): 379 – 394. doi: 10.1080/02602930500353061.</bibtext> </blist> <blist> <bibtext> Winstone, N. E., K. Balloo, and D. Carless. 2020. " Discipline-Specific Feedback Literacies: A Framework for Curriculum Design." Higher Education. doi: 10.1007/s10734-020-00632-0.</bibtext> </blist> <blist> <bibtext> Winstone, N. E., and D. Carless. 2019. Designing Effective Feedback Processes in Higher Education: A Learning-Focused Approach. London : Routledge.</bibtext> </blist> <blist> <bibtext> Winstone, N. E., R. A. Nash, J. Rowntree, and M. Parker. 2017. " It'd Be Useful, but I Wouldn't Use It': Barriers to University Students' Feedback Seeking and Recipience." Studies in Higher Education 42 (11): 2026 – 2041. doi: 10.1080/03075079.2015.1130032.</bibtext> </blist> <blist> <bibtext> Yorke, M. 2003. " Formative Assessment in Higher Education: Moves towards Theory and the Enhancement of Pedagogic Practice." Higher Education 45 (4): 477 – 501. doi: 10.1023/A:1023967026413.</bibtext> </blist> </ref> <aug> <p>By Cathrine Derham; Kieran Balloo and Naomi Winstone</p> <p>Reported by Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib34" firstref="ref4"></nolink> <nolink nlid="nl2" bibid="bib23" firstref="ref5"></nolink> <nolink nlid="nl3" bibid="bib20" firstref="ref7"></nolink> <nolink nlid="nl4" bibid="bib18" firstref="ref8"></nolink> <nolink nlid="nl5" bibid="bib25" firstref="ref10"></nolink> <nolink nlid="nl6" bibid="bib30" firstref="ref11"></nolink> <nolink nlid="nl7" bibid="bib16" firstref="ref12"></nolink> <nolink nlid="nl8" bibid="bib11" firstref="ref15"></nolink> <nolink nlid="nl9" bibid="bib33" firstref="ref18"></nolink> <nolink nlid="nl10" bibid="bib15" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib26" firstref="ref21"></nolink> <nolink nlid="nl12" bibid="bib35" firstref="ref23"></nolink> <nolink nlid="nl13" bibid="bib28" firstref="ref24"></nolink> <nolink nlid="nl14" bibid="bib31" firstref="ref25"></nolink> <nolink nlid="nl15" bibid="bib14" firstref="ref28"></nolink> <nolink nlid="nl16" bibid="bib21" firstref="ref29"></nolink> <nolink nlid="nl17" bibid="bib17" firstref="ref33"></nolink> <nolink nlid="nl18" bibid="bib12" firstref="ref34"></nolink> <nolink nlid="nl19" bibid="bib22" firstref="ref36"></nolink> <nolink nlid="nl20" bibid="bib13" firstref="ref37"></nolink> <nolink nlid="nl21" bibid="bib24" firstref="ref45"></nolink> <nolink nlid="nl22" bibid="bib29" firstref="ref48"></nolink> <nolink nlid="nl23" bibid="bib27" firstref="ref51"></nolink> <nolink nlid="nl24" bibid="bib10" firstref="ref52"></nolink> <nolink nlid="nl25" bibid="bib19" firstref="ref53"></nolink> <nolink nlid="nl26" bibid="bib32" firstref="ref59"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1361068
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Focus, Function and Framing of Feedback Information: Linguistic and Content Analysis of In-Text Feedback Comments
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Derham%2C+Cathrine%22">Derham, Cathrine</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-2014-4496">0000-0002-2014-4496</externalLink>)<br /><searchLink fieldCode="AR" term="%22Balloo%2C+Kieran%22">Balloo, Kieran</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-1745-4653">0000-0002-1745-4653</externalLink>)<br /><searchLink fieldCode="AR" term="%22Winstone%2C+Naomi%22">Winstone, Naomi</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-8157-8274">0000-0001-8157-8274</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Assessment+%26+Evaluation+in+Higher+Education%22"><i>Assessment & Evaluation in Higher Education</i></searchLink>. 2022 47(6):896-909.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 14
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2022
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Content+Analysis%22">Content Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Summative+Evaluation%22">Summative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Grades+%28Scholastic%29%22">Grades (Scholastic)</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Correlation%22">Correlation</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Usage%22">Language Usage</searchLink><br /><searchLink fieldCode="DE" term="%22Error+Correction%22">Error Correction</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Styles%22">Language Styles</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Attitudes%22">Student Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Emotional+Response%22">Emotional Response</searchLink><br /><searchLink fieldCode="DE" term="%22Health+Sciences%22">Health Sciences</searchLink><br /><searchLink fieldCode="DE" term="%22Psychology%22">Psychology</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/02602938.2021.1969335
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0260-2938<br />1469-297X
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In-text comments, in the form of annotations on students' work, are a form of feedback information that should guide students to take action. Both the focus of the in-text comments, and the ways in which they are linguistically communicated, have potential to impact upon the way in which they are perceived by students. This study reports on an analysis of 2101 in-text comments added by markers to 60 summative essays from two disciplines. The majority of comments, regardless of the grade awarded, were found to be directed at the task performance, rather than relating to the level of process or self-regulation. Work awarded higher grades received fewer annotations; these essays were found to include more feedback comments expressing a positive tone, with limited opportunities for informing further development. Work awarded lower grades mainly received corrective comments, as well as comments characterised by interrogative language and words expressing risk. It is argued that the linguistic style may influence engagement with in-text comments, impacting upon students' affective and emotional states, and their level of cognitive engagement with the feedback information. Recommendations for markers' practices are identified, to facilitate the opportunities for engagement and action that in-text comments might afford.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1361068
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1361068
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/02602938.2021.1969335
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 14
        StartPage: 896
    Subjects:
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Content Analysis
        Type: general
      – SubjectFull: Essays
        Type: general
      – SubjectFull: Summative Evaluation
        Type: general
      – SubjectFull: Grades (Scholastic)
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Correlation
        Type: general
      – SubjectFull: Language Usage
        Type: general
      – SubjectFull: Error Correction
        Type: general
      – SubjectFull: Language Styles
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Student Attitudes
        Type: general
      – SubjectFull: Emotional Response
        Type: general
      – SubjectFull: Health Sciences
        Type: general
      – SubjectFull: Psychology
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
    Titles:
      – TitleFull: The Focus, Function and Framing of Feedback Information: Linguistic and Content Analysis of In-Text Feedback Comments
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Derham, Cathrine
      – PersonEntity:
          Name:
            NameFull: Balloo, Kieran
      – PersonEntity:
          Name:
            NameFull: Winstone, Naomi
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2022
          Identifiers:
            – Type: issn-print
              Value: 0260-2938
            – Type: issn-electronic
              Value: 1469-297X
          Numbering:
            – Type: volume
              Value: 47
            – Type: issue
              Value: 6
          Titles:
            – TitleFull: Assessment & Evaluation in Higher Education
              Type: main
ResultId 1