View in EDS

Beyond Essay Length: Evaluating e-rater[R]'s Performance on TOEFL[R] Essays. Research Reports. Report 73. RR-04-04

Saved in:
Bibliographic Details
Title: Beyond Essay Length: Evaluating e-rater[R]'s Performance on TOEFL[R] Essays. Research Reports. Report 73. RR-04-04
Language: English
Authors: Chodorow, Martin, Burstein, Jill
Source: Educational Testing Service. 2004.
Availability: Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site: http://www.ets.org.
Peer Reviewed: N
Page Count: 48
Publication Date: 2004
Document Type: Numerical/Quantitative Data
Reports - Research
Tests/Questionnaires
Descriptors: Essays, Test Scoring Machines, English (Second Language), Student Evaluation, Scores, Spanish, Semitic Languages, Japanese, Writing Evaluation
Abstract: This study examines the relation between essay length and holistic scores assigned to Test of English as a Foreign Language[TM] (TOEFL[R]) essays by e-rater[R], the automated essay scoring system developed by ETS. Results show that an early version of the system, e-rater99, accounted for little variance in human reader scores beyond that which could be predicted by essay length. A later version of the system, e-rater01, performs significantly better than its predecessor and is less dependent on length due to its greater reliance on measures of topical content and of complexity and diversity of vocabulary. Essay length was also examined as a possible explanation for differences in scores among examinees with native languages of Spanish, Arabic, and Japanese. Human readers and e-rater01 show the same pattern of differences for these groups, even when effects of length are controlled. Appended are: (1) TOEFL Writing Scoring Guide; and (2) Confusion Matrices for Essay Scores Combined across Mixed Cross-validation Sets for Seven Prompts. (Contains 18 tables, 3 figures, and 5 endnotes.)
Abstractor: Author
Number of References: 18
Entry Date: 2006
Accession Number: ED492918
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED492918
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED492918
AccessLevel: 3
PubType: Electronic Resource
PubTypeId: electronicResource
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Beyond Essay Length: Evaluating e-rater[R]'s Performance on TOEFL[R] Essays. Research Reports. Report 73. RR-04-04
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Chodorow%2C+Martin%22">Chodorow, Martin</searchLink><br /><searchLink fieldCode="AR" term="%22Burstein%2C+Jill%22">Burstein, Jill</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Testing+Service%22"><i>Educational Testing Service</i></searchLink>. 2004.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site: http://www.ets.org.
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 48
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2004
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Numerical/Quantitative Data<br />Reports - Research<br />Tests/Questionnaires
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Scoring+Machines%22">Test Scoring Machines</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Spanish%22">Spanish</searchLink><br /><searchLink fieldCode="DE" term="%22Semitic+Languages%22">Semitic Languages</searchLink><br /><searchLink fieldCode="DE" term="%22Japanese%22">Japanese</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study examines the relation between essay length and holistic scores assigned to Test of English as a Foreign Language[TM] (TOEFL[R]) essays by e-rater[R], the automated essay scoring system developed by ETS. Results show that an early version of the system, e-rater99, accounted for little variance in human reader scores beyond that which could be predicted by essay length. A later version of the system, e-rater01, performs significantly better than its predecessor and is less dependent on length due to its greater reliance on measures of topical content and of complexity and diversity of vocabulary. Essay length was also examined as a possible explanation for differences in scores among examinees with native languages of Spanish, Arabic, and Japanese. Human readers and e-rater01 show the same pattern of differences for these groups, even when effects of length are controlled. Appended are: (1) TOEFL Writing Scoring Guide; and (2) Confusion Matrices for Essay Scores Combined across Mixed Cross-validation Sets for Seven Prompts. (Contains 18 tables, 3 figures, and 5 endnotes.)
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: Author
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 18
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2006
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED492918
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED492918
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 48
    Subjects:
      – SubjectFull: Essays
        Type: general
      – SubjectFull: Test Scoring Machines
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Spanish
        Type: general
      – SubjectFull: Semitic Languages
        Type: general
      – SubjectFull: Japanese
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
    Titles:
      – TitleFull: Beyond Essay Length: Evaluating e-rater[R]'s Performance on TOEFL[R] Essays. Research Reports. Report 73. RR-04-04
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Chodorow, Martin
      – PersonEntity:
          Name:
            NameFull: Burstein, Jill
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 02
              Type: published
              Y: 2004
          Titles:
            – TitleFull: Educational Testing Service
              Type: main
ResultId 1