View in EDS

GE FRST Evaluation Report: How Well Does a Statistically-Based Natural Language Processing System Score Natural Language Constructed-Responses?

Saved in:
Bibliographic Details
Title: GE FRST Evaluation Report: How Well Does a Statistically-Based Natural Language Processing System Score Natural Language Constructed-Responses?
Language: English
Authors: Burstein, Jill C., Kaplan, Randy M., Educational Testing Service, Princeton, NJ.
Peer Reviewed: N
Page Count: 30
Publication Date: 1995
Document Type: Reports - Evaluative
Descriptors: Computer Assisted Testing, Constructed Response, Cost Effectiveness, Hypothesis Testing, Natural Language Processing, Performance Based Assessment, Reading Comprehension, Scoring, Standardized Tests, Test Construction, Test Items
Abstract: There is a considerable interest at Educational Testing Service (ETS) to include performance-based, natural language constructed-response items on standardized tests. Such items can be developed, but the projected time and costs required to have these items scored by human graders would be prohibitive. In order for ETS to include these types of items on standardized tests, automated scoring systems need to be developed and evaluated. Automated scoring systems could decrease the time and costs required for human graders to score these items. This report details the evaluation of a statistically-based scoring system, the General Electric Free-Response Scoring Tool (GE FRST). GE FRST was designed to score short-answer, constructed-responses of up to 17 words. The report describes how the system performs for responses on three different item types: (1) the formulating-hypotheses item; (2) a paraphrase language proficiency item; and (3) a reading comprehension item. For the sake of efficiency, it is important to evaluate systems on a number of item types to see if the system's scoring method can generalize to a number of item types. An appendix shows learning information abut responses recognized by GE FRST. (Contains 7 figures, 13 tables, and 3 references.) (Author/SLD)
Entry Date: 1996
Accession Number: ED393937
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED393937
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED393937
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: GE FRST Evaluation Report: How Well Does a Statistically-Based Natural Language Processing System Score Natural Language Constructed-Responses?
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Burstein%2C+Jill+C%2E%22">Burstein, Jill C.</searchLink><br /><searchLink fieldCode="AR" term="%22Kaplan%2C+Randy+M%2E%22">Kaplan, Randy M.</searchLink><br /><searchLink fieldCode="AR" term="%22Educational+Testing+Service%2C+Princeton%2C+NJ%2E%22">Educational Testing Service, Princeton, NJ.</searchLink>
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 30
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 1995
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Constructed+Response%22">Constructed Response</searchLink><br /><searchLink fieldCode="DE" term="%22Cost+Effectiveness%22">Cost Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Hypothesis+Testing%22">Hypothesis Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Performance+Based+Assessment%22">Performance Based Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Comprehension%22">Reading Comprehension</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Standardized+Tests%22">Standardized Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: There is a considerable interest at Educational Testing Service (ETS) to include performance-based, natural language constructed-response items on standardized tests. Such items can be developed, but the projected time and costs required to have these items scored by human graders would be prohibitive. In order for ETS to include these types of items on standardized tests, automated scoring systems need to be developed and evaluated. Automated scoring systems could decrease the time and costs required for human graders to score these items. This report details the evaluation of a statistically-based scoring system, the General Electric Free-Response Scoring Tool (GE FRST). GE FRST was designed to score short-answer, constructed-responses of up to 17 words. The report describes how the system performs for responses on three different item types: (1) the formulating-hypotheses item; (2) a paraphrase language proficiency item; and (3) a reading comprehension item. For the sake of efficiency, it is important to evaluate systems on a number of item types to see if the system's scoring method can generalize to a number of item types. An appendix shows learning information abut responses recognized by GE FRST. (Contains 7 figures, 13 tables, and 3 references.) (Author/SLD)
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 1996
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED393937
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED393937
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 30
    Subjects:
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Constructed Response
        Type: general
      – SubjectFull: Cost Effectiveness
        Type: general
      – SubjectFull: Hypothesis Testing
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Performance Based Assessment
        Type: general
      – SubjectFull: Reading Comprehension
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Standardized Tests
        Type: general
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Test Items
        Type: general
    Titles:
      – TitleFull: GE FRST Evaluation Report: How Well Does a Statistically-Based Natural Language Processing System Score Natural Language Constructed-Responses?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Educational Testing Service, Princeton, NJ.
      – PersonEntity:
          Name:
            NameFull: Burstein, Jill C.
      – PersonEntity:
          Name:
            NameFull: Kaplan, Randy M.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 1995
ResultId 1