View in EDS PDF Full Text

A Rubric for the Detection of Students in Crisis

Saved in:
Bibliographic Details
Title: A Rubric for the Detection of Students in Crisis
Language: English
Authors: Burkhardt, Amy, Lottridge, Susan, Woolf, Sherri
Source: Educational Measurement: Issues and Practice. Sum 2021 40(2):72-80.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 9
Publication Date: 2021
Document Type: Journal Articles
Reports - Evaluative
Descriptors: Scoring Rubrics, Identification, At Risk Students, Standardized Tests, Scoring, Interrater Reliability
DOI: 10.1111/emip.12410
ISSN: 0731-1745
Abstract: For some students, standardized tests serve as a conduit to disclose sensitive issues of harm or distress that may otherwise go unreported. By detecting this writing, known as "crisis papers," testing programs have a unique opportunity to assist in mitigating the risk of harm to these students. The use of machine learning to automatically detect such writing is necessary in the context of online tests and automated scoring. To achieve a detection system that is accurate, humans must first consistently label the data that are used to train the model. This paper argues that the existing guidelines are not sufficient for this task and proposes a three-level rubric to guide the collection of the training data. In showcasing the fundamental machine learning procedures for creating an automatic text classification system, the following evidence emerges as support of the operational use of this rubric. First, hand-scorers largely agree with one another in assigning labels to text according to the rubric. Additionally, when this labeled data are used to train a baseline classifier, the model exhibits promising performance. Recommendations are made for improving the hand-scoring training process, with the ultimate goal of quickly and accurately assisting students in crisis.
Abstractor: As Provided
Entry Date: 2021
Accession Number: EJ1298847
Database: ERIC
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHKG_YkDiwgzETQB4Om6t1fAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJbjX0vzerbmRaLW6AIBEICBmx6QR7HxZ8H1UrWv5DogbEj3SV_kJRbFcOM751WG5nblzwyYEl5jCvgjfMdcggehJEPb_9jwtyfLebqAGu07ZxF9EjPjNgqV4bvIAOzsJkp4BNSsC1lQTtspJtg9f3dX6IuJyweK3pZYCsKgUkX9sVKpJx9_xjDD7K4TfKOdu-NQ3FZIGstmni2KLOSdgAF0xtmW4pygF6-y85Mk
Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1298847
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Rubric for the Detection of Students in Crisis
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Burkhardt%2C+Amy%22">Burkhardt, Amy</searchLink><br /><searchLink fieldCode="AR" term="%22Lottridge%2C+Susan%22">Lottridge, Susan</searchLink><br /><searchLink fieldCode="AR" term="%22Woolf%2C+Sherri%22">Woolf, Sherri</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Measurement%3A+Issues+and+Practice%22"><i>Educational Measurement: Issues and Practice</i></searchLink>. Sum 2021 40(2):72-80.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 9
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2021
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Scoring+Rubrics%22">Scoring Rubrics</searchLink><br /><searchLink fieldCode="DE" term="%22Identification%22">Identification</searchLink><br /><searchLink fieldCode="DE" term="%22At+Risk+Students%22">At Risk Students</searchLink><br /><searchLink fieldCode="DE" term="%22Standardized+Tests%22">Standardized Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/emip.12410
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0731-1745
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: For some students, standardized tests serve as a conduit to disclose sensitive issues of harm or distress that may otherwise go unreported. By detecting this writing, known as "crisis papers," testing programs have a unique opportunity to assist in mitigating the risk of harm to these students. The use of machine learning to automatically detect such writing is necessary in the context of online tests and automated scoring. To achieve a detection system that is accurate, humans must first consistently label the data that are used to train the model. This paper argues that the existing guidelines are not sufficient for this task and proposes a three-level rubric to guide the collection of the training data. In showcasing the fundamental machine learning procedures for creating an automatic text classification system, the following evidence emerges as support of the operational use of this rubric. First, hand-scorers largely agree with one another in assigning labels to text according to the rubric. Additionally, when this labeled data are used to train a baseline classifier, the model exhibits promising performance. Recommendations are made for improving the hand-scoring training process, with the ultimate goal of quickly and accurately assisting students in crisis.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1298847
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1298847
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/emip.12410
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 9
        StartPage: 72
    Subjects:
      – SubjectFull: Scoring Rubrics
        Type: general
      – SubjectFull: Identification
        Type: general
      – SubjectFull: At Risk Students
        Type: general
      – SubjectFull: Standardized Tests
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
    Titles:
      – TitleFull: A Rubric for the Detection of Students in Crisis
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Burkhardt, Amy
      – PersonEntity:
          Name:
            NameFull: Lottridge, Susan
      – PersonEntity:
          Name:
            NameFull: Woolf, Sherri
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-print
              Value: 0731-1745
          Numbering:
            – Type: volume
              Value: 40
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Educational Measurement: Issues and Practice
              Type: main
ResultId 1