View in EDS

Utility-Value Score: A Case Study in System Generalization for Writing Analytics

Saved in:
Bibliographic Details
Title: Utility-Value Score: A Case Study in System Generalization for Writing Analytics
Language: English
Authors: Beigman Klebanov, Beata, Priniski, Stacy, Burstein, Jill, Gyawali, Binod, Harackiewicz, Judith, Thoman, Dustin
Source: Grantee Submission. 2018 2:314-328.
Peer Reviewed: Y
Page Count: 15
Publication Date: 2018
Sponsoring Agency: National Science Foundation (NSF)
National Institutes of Health (DHHS)
Institute of Education Sciences (ED)
Contract Number: DRL1622991
2R01GM10270305
R305B150003
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Learning Analytics, Context Effect, Automation, Generalization, Value Judgment, College Freshmen, Essays, Writing Evaluation, STEM Education, Relevance (Education), Content Area Writing, Introductory Courses, Psychology, Biology, Student Motivation, Natural Language Processing, Data Collection
Geographic Terms: California (Long Beach), Wisconsin (Madison)
Abstract: Collection and analysis of students' writing samples on a large scale is a part of the research agenda of the emerging writing analytics community that promises to deliver an unprecedented insight into characteristics of student writing. Yet with a large scale often comes variability of contexts in which the samples were produced--different institutions, different purposes of writing, different author demographics, to name just a few possible dimensions of variation. What are the implications of such variation for the ability of automated methods to create indices/features based on the writing samples that would be valid and meaningful? This paper presents a case study in system generalization. Building on a system developed to assess the expression of "utility value" (a social-psychology-based construct) in essays written by first-year biology students at one postsecondary institution, we vary data parameters and observe system performance. From the point of view of social psychology, all these variants represent the same underlying construct (i.e., utility value), and it is thus very tempting to think that an automatically produced "utility-value" score could provide a meaningful analytic, consistently, on a large collection of essays. However, findings from this research show that there are challenges: Some variations are easier to deal with than others, and some components of the automated system generalize better than others. The findings are then discussed both in the context of the case study and more generally.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2020
Accession Number: ED605731
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED605731
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED605731
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Utility-Value Score: A Case Study in System Generalization for Writing Analytics
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Beigman+Klebanov%2C+Beata%22">Beigman Klebanov, Beata</searchLink><br /><searchLink fieldCode="AR" term="%22Priniski%2C+Stacy%22">Priniski, Stacy</searchLink><br /><searchLink fieldCode="AR" term="%22Burstein%2C+Jill%22">Burstein, Jill</searchLink><br /><searchLink fieldCode="AR" term="%22Gyawali%2C+Binod%22">Gyawali, Binod</searchLink><br /><searchLink fieldCode="AR" term="%22Harackiewicz%2C+Judith%22">Harackiewicz, Judith</searchLink><br /><searchLink fieldCode="AR" term="%22Thoman%2C+Dustin%22">Thoman, Dustin</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Grantee+Submission%22"><i>Grantee Submission</i></searchLink>. 2018 2:314-328.
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 15
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2018
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)<br />National Institutes of Health (DHHS)<br />Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: DRL1622991<br />2R01GM10270305<br />R305B150003
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Learning+Analytics%22">Learning Analytics</searchLink><br /><searchLink fieldCode="DE" term="%22Context+Effect%22">Context Effect</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Generalization%22">Generalization</searchLink><br /><searchLink fieldCode="DE" term="%22Value+Judgment%22">Value Judgment</searchLink><br /><searchLink fieldCode="DE" term="%22College+Freshmen%22">College Freshmen</searchLink><br /><searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22Relevance+%28Education%29%22">Relevance (Education)</searchLink><br /><searchLink fieldCode="DE" term="%22Content+Area+Writing%22">Content Area Writing</searchLink><br /><searchLink fieldCode="DE" term="%22Introductory+Courses%22">Introductory Courses</searchLink><br /><searchLink fieldCode="DE" term="%22Psychology%22">Psychology</searchLink><br /><searchLink fieldCode="DE" term="%22Biology%22">Biology</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Motivation%22">Student Motivation</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Collection%22">Data Collection</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22California+%28Long+Beach%29%22">California (Long Beach)</searchLink><br /><searchLink fieldCode="DE" term="%22Wisconsin+%28Madison%29%22">Wisconsin (Madison)</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Collection and analysis of students' writing samples on a large scale is a part of the research agenda of the emerging writing analytics community that promises to deliver an unprecedented insight into characteristics of student writing. Yet with a large scale often comes variability of contexts in which the samples were produced--different institutions, different purposes of writing, different author demographics, to name just a few possible dimensions of variation. What are the implications of such variation for the ability of automated methods to create indices/features based on the writing samples that would be valid and meaningful? This paper presents a case study in system generalization. Building on a system developed to assess the expression of "utility value" (a social-psychology-based construct) in essays written by first-year biology students at one postsecondary institution, we vary data parameters and observe system performance. From the point of view of social psychology, all these variants represent the same underlying construct (i.e., utility value), and it is thus very tempting to think that an automatically produced "utility-value" score could provide a meaningful analytic, consistently, on a large collection of essays. However, findings from this research show that there are challenges: Some variations are easier to deal with than others, and some components of the automated system generalize better than others. The findings are then discussed both in the context of the case study and more generally.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED605731
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED605731
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 15
        StartPage: 314
    Subjects:
      – SubjectFull: Learning Analytics
        Type: general
      – SubjectFull: Context Effect
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Generalization
        Type: general
      – SubjectFull: Value Judgment
        Type: general
      – SubjectFull: College Freshmen
        Type: general
      – SubjectFull: Essays
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: Relevance (Education)
        Type: general
      – SubjectFull: Content Area Writing
        Type: general
      – SubjectFull: Introductory Courses
        Type: general
      – SubjectFull: Psychology
        Type: general
      – SubjectFull: Biology
        Type: general
      – SubjectFull: Student Motivation
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Data Collection
        Type: general
      – SubjectFull: California (Long Beach)
        Type: general
      – SubjectFull: Wisconsin (Madison)
        Type: general
    Titles:
      – TitleFull: Utility-Value Score: A Case Study in System Generalization for Writing Analytics
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Beigman Klebanov, Beata
      – PersonEntity:
          Name:
            NameFull: Priniski, Stacy
      – PersonEntity:
          Name:
            NameFull: Burstein, Jill
      – PersonEntity:
          Name:
            NameFull: Gyawali, Binod
      – PersonEntity:
          Name:
            NameFull: Harackiewicz, Judith
      – PersonEntity:
          Name:
            NameFull: Thoman, Dustin
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2018
          Numbering:
            – Type: volume
              Value: 2
          Titles:
            – TitleFull: Grantee Submission
              Type: main
ResultId 1