Utility-Value Score: A Case Study in System Generalization for Writing Analytics
Saved in:
| Title: | Utility-Value Score: A Case Study in System Generalization for Writing Analytics |
|---|---|
| Language: | English |
| Authors: | Beigman Klebanov, Beata, Priniski, Stacy, Burstein, Jill, Gyawali, Binod, Harackiewicz, Judith, Thoman, Dustin |
| Source: | Grantee Submission. 2018 2:314-328. |
| Peer Reviewed: | Y |
| Page Count: | 15 |
| Publication Date: | 2018 |
| Sponsoring Agency: | National Science Foundation (NSF) National Institutes of Health (DHHS) Institute of Education Sciences (ED) |
| Contract Number: | DRL1622991 2R01GM10270305 R305B150003 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Learning Analytics, Context Effect, Automation, Generalization, Value Judgment, College Freshmen, Essays, Writing Evaluation, STEM Education, Relevance (Education), Content Area Writing, Introductory Courses, Psychology, Biology, Student Motivation, Natural Language Processing, Data Collection |
| Geographic Terms: | California (Long Beach), Wisconsin (Madison) |
| Abstract: | Collection and analysis of students' writing samples on a large scale is a part of the research agenda of the emerging writing analytics community that promises to deliver an unprecedented insight into characteristics of student writing. Yet with a large scale often comes variability of contexts in which the samples were produced--different institutions, different purposes of writing, different author demographics, to name just a few possible dimensions of variation. What are the implications of such variation for the ability of automated methods to create indices/features based on the writing samples that would be valid and meaningful? This paper presents a case study in system generalization. Building on a system developed to assess the expression of "utility value" (a social-psychology-based construct) in essays written by first-year biology students at one postsecondary institution, we vary data parameters and observe system performance. From the point of view of social psychology, all these variants represent the same underlying construct (i.e., utility value), and it is thus very tempting to think that an automatically produced "utility-value" score could provide a meaningful analytic, consistently, on a large collection of essays. However, findings from this research show that there are challenges: Some variations are easier to deal with than others, and some components of the automated system generalize better than others. The findings are then discussed both in the context of the case study and more generally. |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2020 |
| Accession Number: | ED605731 |
| Database: | ERIC |
| Abstract: | Collection and analysis of students' writing samples on a large scale is a part of the research agenda of the emerging writing analytics community that promises to deliver an unprecedented insight into characteristics of student writing. Yet with a large scale often comes variability of contexts in which the samples were produced--different institutions, different purposes of writing, different author demographics, to name just a few possible dimensions of variation. What are the implications of such variation for the ability of automated methods to create indices/features based on the writing samples that would be valid and meaningful? This paper presents a case study in system generalization. Building on a system developed to assess the expression of "utility value" (a social-psychology-based construct) in essays written by first-year biology students at one postsecondary institution, we vary data parameters and observe system performance. From the point of view of social psychology, all these variants represent the same underlying construct (i.e., utility value), and it is thus very tempting to think that an automatically produced "utility-value" score could provide a meaningful analytic, consistently, on a large collection of essays. However, findings from this research show that there are challenges: Some variations are easier to deal with than others, and some components of the automated system generalize better than others. The findings are then discussed both in the context of the case study and more generally. |
|---|
