Across Performance Contexts: Using Automated Writing Evaluation to Explore Student Writing
Saved in:
| Title: | Across Performance Contexts: Using Automated Writing Evaluation to Explore Student Writing |
|---|---|
| Language: | English |
| Authors: | McCaffrey, Daniel F., Zhang, Mo, Burstein, Jill |
| Source: | Grantee Submission. 2022 6:167-199. |
| Peer Reviewed: | Y |
| Page Count: | 33 |
| Publication Date: | 2022 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R305A160115 |
| Document Type: | Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Persuasive Discourse, Academic Language, Writing (Composition), Academic Achievement, Writing Skills, Language Usage, Writing Evaluation, Automation, Natural Language Processing, College Students |
| Abstract: | Background: This exploratory writing analytics study uses argumentative writing samples from two performance contexts--standardized writing assessments and university English course writing assignments--to compare: (1) linguistic features in argumentative writing; and (2) relationships between linguistic characteristics and academic performance outcomes. Writing data from this study come from 180 students enrolled at five four-year universities in the United States. Automated writing evaluation (AWE) tools were used to generate linguistic features from students' writing. Literature Review: Few studies have been conducted that use AWE to examine postsecondary writing skill and relationships between writing and broader academic performance outcomes. To address this gap, our study draws upon research on standardized and coursework writing, construct modeling, and AWE feature design. We also draw on related work to demonstrate how AWE can provide insights about linguistic characteristics of students' writing and relationships of that writing to academic performance factors. Research Questions: We examine the feasibility of using AWE to examine writing samples from standardized assessment and coursework contexts to assess how the samples compare to each other and how they are related to measures of students' academic performance. We organize this study around two research questions (RQ): RQ1: What can AWE tell us about variation in student writing samples drawn from two specific performance contexts: standardized argumentative writing versus coursework argumentative writing? RQ2: What are the relationships between writing subconstructs (as captured by AWE) and other measures of students' academic performance? Methodology: As part of a larger program of research (n = 735), study participants (n = 180) were administered two standardized test instruments: the HEIghten Written Communication (WC) test and the HEIghten Critical Thinking test. The HEIghten tests are commercially available tests designed for assessing postsecondary students' skills. Students also submitted writing samples from their coursework writing assignments. We collected institutional data, including SAT total scores, SAT Reading scores, ACT total scores, high school GPAs, and college GPAs. AWE tools were used to generate the 36 writing features--aggregated to create six subconstruct composite scores--that were used to analyze both the HEIghten WC test and the coursework writing samples. To answer RQ1, we used multiple statistical analysis and data visualization methods. The distributions of the AWE subconstruct scores between the HEIghten WC and coursework responses were compared in histograms and boxplots. Paired t tests were conducted for each composite feature to examine whether the means of the composite scores were statistically the same (H0). To answer RQ2, we used the academic outcome variables from the institutional data noted previously. We also examined the correlations of AWE feature scores with the scores students received from the multiple-choice portion of the HEIghten WC test. Results: Findings showed that AWE feature distributions differed between the two performance contexts. This finding suggests that both AWE feature interpretation and the relationship of these features to academic performance factors are associated with the writing context. Findings also suggest that writing data from both contexts offers a complementary picture of students' writing achievement and relationships with performance outcomes. Discussion: Using AWE feature interpretation, this study establishes a foundation for variation in postsecondary student writing and demonstrates how a granular sense of variation can be leveraged in order to understand relationships between writing achievement and broader postsecondary academic performance factors. Conclusions: Study findings have practical implications for how institutions might use writing analytics to obtain detailed information about student writing and, in turn, relate that information to other performance measures to provide relevant support for students. However, study findings must be qualified in terms of study sample, genres examined, standardized criterion measures, and shifting educational pedagogies. |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2022 |
| Access URL: | https://wac.colostate.edu/docs/jwa/vol6/mccaffrey-etal.pdf |
| Accession Number: | ED618142 |
| Database: | ERIC |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED618142 Name: ERIC Full Text Category: fullText Text: Full Text from ERIC |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: ED618142 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Across Performance Contexts: Using Automated Writing Evaluation to Explore Student Writing – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22McCaffrey%2C+Daniel+F%2E%22">McCaffrey, Daniel F.</searchLink><br /><searchLink fieldCode="AR" term="%22Zhang%2C+Mo%22">Zhang, Mo</searchLink><br /><searchLink fieldCode="AR" term="%22Burstein%2C+Jill%22">Burstein, Jill</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Grantee+Submission%22"><i>Grantee Submission</i></searchLink>. 2022 6:167-199. – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 33 – Name: DatePubCY Label: Publication Date Group: Date Data: 2022 – Name: SourceSuprt Label: Sponsoring Agency Group: SrcSuprt Data: Institute of Education Sciences (ED) – Name: NumberContract Label: Contract Number Group: NumCntrct Data: R305A160115 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Reports - Research – Name: Audience Label: Education Level Group: Audnce Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink> – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Persuasive+Discourse%22">Persuasive Discourse</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Language%22">Academic Language</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+%28Composition%29%22">Writing (Composition)</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Skills%22">Writing Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Usage%22">Language Usage</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Background: This exploratory writing analytics study uses argumentative writing samples from two performance contexts--standardized writing assessments and university English course writing assignments--to compare: (1) linguistic features in argumentative writing; and (2) relationships between linguistic characteristics and academic performance outcomes. Writing data from this study come from 180 students enrolled at five four-year universities in the United States. Automated writing evaluation (AWE) tools were used to generate linguistic features from students' writing. Literature Review: Few studies have been conducted that use AWE to examine postsecondary writing skill and relationships between writing and broader academic performance outcomes. To address this gap, our study draws upon research on standardized and coursework writing, construct modeling, and AWE feature design. We also draw on related work to demonstrate how AWE can provide insights about linguistic characteristics of students' writing and relationships of that writing to academic performance factors. Research Questions: We examine the feasibility of using AWE to examine writing samples from standardized assessment and coursework contexts to assess how the samples compare to each other and how they are related to measures of students' academic performance. We organize this study around two research questions (RQ): RQ1: What can AWE tell us about variation in student writing samples drawn from two specific performance contexts: standardized argumentative writing versus coursework argumentative writing? RQ2: What are the relationships between writing subconstructs (as captured by AWE) and other measures of students' academic performance? Methodology: As part of a larger program of research (n = 735), study participants (n = 180) were administered two standardized test instruments: the HEIghten Written Communication (WC) test and the HEIghten Critical Thinking test. The HEIghten tests are commercially available tests designed for assessing postsecondary students' skills. Students also submitted writing samples from their coursework writing assignments. We collected institutional data, including SAT total scores, SAT Reading scores, ACT total scores, high school GPAs, and college GPAs. AWE tools were used to generate the 36 writing features--aggregated to create six subconstruct composite scores--that were used to analyze both the HEIghten WC test and the coursework writing samples. To answer RQ1, we used multiple statistical analysis and data visualization methods. The distributions of the AWE subconstruct scores between the HEIghten WC and coursework responses were compared in histograms and boxplots. Paired t tests were conducted for each composite feature to examine whether the means of the composite scores were statistically the same (H0). To answer RQ2, we used the academic outcome variables from the institutional data noted previously. We also examined the correlations of AWE feature scores with the scores students received from the multiple-choice portion of the HEIghten WC test. Results: Findings showed that AWE feature distributions differed between the two performance contexts. This finding suggests that both AWE feature interpretation and the relationship of these features to academic performance factors are associated with the writing context. Findings also suggest that writing data from both contexts offers a complementary picture of students' writing achievement and relationships with performance outcomes. Discussion: Using AWE feature interpretation, this study establishes a foundation for variation in postsecondary student writing and demonstrates how a granular sense of variation can be leveraged in order to understand relationships between writing achievement and broader postsecondary academic performance factors. Conclusions: Study findings have practical implications for how institutions might use writing analytics to obtain detailed information about student writing and, in turn, relate that information to other performance measures to provide relevant support for students. However, study findings must be qualified in terms of study sample, genres examined, standardized criterion measures, and shifting educational pedagogies. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: CodeSource Label: IES Funded Group: SrcInfo Data: Yes – Name: DateEntry Label: Entry Date Group: Date Data: 2022 – Name: URL Label: Access URL Group: URL Data: <link linkTarget="URL" linkTerm="https://wac.colostate.edu/docs/jwa/vol6/mccaffrey-etal.pdf" linkWindow="_blank">https://wac.colostate.edu/docs/jwa/vol6/mccaffrey-etal.pdf</link> – Name: AN Label: Accession Number Group: ID Data: ED618142 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED618142 |
| RecordInfo | BibRecord: BibEntity: Languages: – Text: English PhysicalDescription: Pagination: PageCount: 33 StartPage: 167 Subjects: – SubjectFull: Persuasive Discourse Type: general – SubjectFull: Academic Language Type: general – SubjectFull: Writing (Composition) Type: general – SubjectFull: Academic Achievement Type: general – SubjectFull: Writing Skills Type: general – SubjectFull: Language Usage Type: general – SubjectFull: Writing Evaluation Type: general – SubjectFull: Automation Type: general – SubjectFull: Natural Language Processing Type: general – SubjectFull: College Students Type: general Titles: – TitleFull: Across Performance Contexts: Using Automated Writing Evaluation to Explore Student Writing Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: McCaffrey, Daniel F. – PersonEntity: Name: NameFull: Zhang, Mo – PersonEntity: Name: NameFull: Burstein, Jill IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2022 Numbering: – Type: volume Value: 6 Titles: – TitleFull: Grantee Submission Type: main |
| ResultId | 1 |
