View in EDS HTML Full Text PDF Full Text

Exploring Rater Accuracy Using Unfolding Models Combined with Topic Models: Incorporating Supervised Latent Dirichlet Allocation.

Saved in:
Bibliographic Details
Title: Exploring Rater Accuracy Using Unfolding Models Combined with Topic Models: Incorporating Supervised Latent Dirichlet Allocation.
Authors: Wheeler, Jordan M.1 jmwheeler@uga.edu, Engelhard, George1, Wang, Jue2
Source: Measurement. Jan-Mar 2022, Vol. 20 Issue 1, p34-46. 13p.
Subjects: SERVQUAL (Service quality framework), Hyperbolic processes
Abstract: Objectively scoring constructed-response items on educational assessments has long been a challenge due to the use of human raters. Even well-trained raters using a rubric can inaccurately assess essays. Unfolding models measure rater's scoring accuracy by capturing the discrepancy between criterion and operational ratings by placing essays on an unfolding continuum with an ideal-point location. Essay unfolding locations indicate how difficult it is for raters to score an essay accurately. This study aims to explore a substantive interpretation of the unfolding scale based on a supervised Latent Dirichlet Allocation (sLDA) model. We investigate the relationship between latent topics extracted using sLDA and unfolding locations with a sample of essays (n = 100) obtained from an integrated writing assessment. Results show that (a) three latent topics moderately explain (r2 = 0.561) essay locations defined by the unfolding scale and (b) failing to use and/or cite the source articles led to essays that are difficult-to-score accurately. [ABSTRACT FROM AUTHOR]
Copyright of Measurement is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 155053460
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Exploring Rater Accuracy Using Unfolding Models Combined with Topic Models: Incorporating Supervised Latent Dirichlet Allocation.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wheeler%2C+Jordan+M%2E%22">Wheeler, Jordan M.</searchLink><relatesTo>1</relatesTo><i> jmwheeler@uga.edu</i><br /><searchLink fieldCode="AR" term="%22Engelhard%2C+George%22">Engelhard, George</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Wang%2C+Jue%22">Wang, Jue</searchLink><relatesTo>2</relatesTo>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Measurement%22">Measurement</searchLink>. Jan-Mar 2022, Vol. 20 Issue 1, p34-46. 13p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22SERVQUAL+%28Service+quality+framework%29%22">SERVQUAL (Service quality framework)</searchLink><br /><searchLink fieldCode="DE" term="%22Hyperbolic+processes%22">Hyperbolic processes</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objectively scoring constructed-response items on educational assessments has long been a challenge due to the use of human raters. Even well-trained raters using a rubric can inaccurately assess essays. Unfolding models measure rater's scoring accuracy by capturing the discrepancy between criterion and operational ratings by placing essays on an unfolding continuum with an ideal-point location. Essay unfolding locations indicate how difficult it is for raters to score an essay accurately. This study aims to explore a substantive interpretation of the unfolding scale based on a supervised Latent Dirichlet Allocation (sLDA) model. We investigate the relationship between latent topics extracted using sLDA and unfolding locations with a sample of essays (n = 100) obtained from an integrated writing assessment. Results show that (a) three latent topics moderately explain (r2 = 0.561) essay locations defined by the unfolding scale and (b) failing to use and/or cite the source articles led to essays that are difficult-to-score accurately. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Measurement is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=155053460
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/15366367.2021.1915094
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 34
    Subjects:
      – SubjectFull: SERVQUAL (Service quality framework)
        Type: general
      – SubjectFull: Hyperbolic processes
        Type: general
    Titles:
      – TitleFull: Exploring Rater Accuracy Using Unfolding Models Combined with Topic Models: Incorporating Supervised Latent Dirichlet Allocation.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wheeler, Jordan M.
      – PersonEntity:
          Name:
            NameFull: Engelhard, George
      – PersonEntity:
          Name:
            NameFull: Wang, Jue
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Text: Jan-Mar 2022
              Type: published
              Y: 2022
          Identifiers:
            – Type: issn-print
              Value: 15366367
          Numbering:
            – Type: volume
              Value: 20
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Measurement
              Type: main
ResultId 1