Enriching Automated Essay Scoring Using Discourse Marking.
Saved in:
| Title: | Enriching Automated Essay Scoring Using Discourse Marking. |
|---|---|
| Language: | English |
| Authors: | Burstein, Jill, Kukich, Karen, Wolff, Susanne, Lu, Chi, Chodorow, Martin |
| Peer Reviewed: | N |
| Page Count: | 9 |
| Publication Date: | 2001 |
| Document Type: | Reports - Evaluative |
| Descriptors: | College Students, Essays, Higher Education, Scoring, Test Scoring Machines |
| Abstract: | Electronic Essay Rater (e-rater) is a prototype automated essay scoring system built at Educational Testing Service that uses discourse marking in addition to syntactic information and topical content vector analyses to assign essay scores automatically. This paper gives a general description of e-rater as a whole, but its emphasis is on the importance of discourse marking and argument partitioning for annotating the argument structure of an essay. Two content vector analysis programs used to predict scores, EssayContent and ArgContent, have been compared. This paper reports on the overall evaluation results from e-rater's scoring performance on 13 sets of essay data from the writing assessment of the Graduate Management Admissions Test and 2 sets of essay data from the Test of Written English. EssayContent assigns scores to essays by using a standard cosine correlation that treats the essay as a "bag of words," in that it does not consider word order. ArgContent uses a novel content vector analysis approach for score assignment based on the individual arguments in an essay. The average agreement between ArgContent scores and human rater scores found in previous research was 82% as compared to 69% agreement between EssayContent and the human raters. These results suggest that discourse marking enriches e-raters' scoring capability. When e-rater uses its whole set of predictive features, agreement with human rater scores ranges from 87% to 94% across the 15 sets of essay responses used in this study. (Contains 3 tables and 16 references.) (Author/SLD) |
| Entry Date: | 2002 |
| Accession Number: | ED458267 |
| Database: | ERIC |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED458267 Name: ERIC Full Text Category: fullText Text: Full Text from ERIC |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: ED458267 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Enriching Automated Essay Scoring Using Discourse Marking. – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Burstein%2C+Jill%22">Burstein, Jill</searchLink><br /><searchLink fieldCode="AR" term="%22Kukich%2C+Karen%22">Kukich, Karen</searchLink><br /><searchLink fieldCode="AR" term="%22Wolff%2C+Susanne%22">Wolff, Susanne</searchLink><br /><searchLink fieldCode="AR" term="%22Lu%2C+Chi%22">Lu, Chi</searchLink><br /><searchLink fieldCode="AR" term="%22Chodorow%2C+Martin%22">Chodorow, Martin</searchLink> – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: N – Name: Pages Label: Page Count Group: Src Data: 9 – Name: DatePubCY Label: Publication Date Group: Date Data: 2001 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Reports - Evaluative – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Scoring+Machines%22">Test Scoring Machines</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Electronic Essay Rater (e-rater) is a prototype automated essay scoring system built at Educational Testing Service that uses discourse marking in addition to syntactic information and topical content vector analyses to assign essay scores automatically. This paper gives a general description of e-rater as a whole, but its emphasis is on the importance of discourse marking and argument partitioning for annotating the argument structure of an essay. Two content vector analysis programs used to predict scores, EssayContent and ArgContent, have been compared. This paper reports on the overall evaluation results from e-rater's scoring performance on 13 sets of essay data from the writing assessment of the Graduate Management Admissions Test and 2 sets of essay data from the Test of Written English. EssayContent assigns scores to essays by using a standard cosine correlation that treats the essay as a "bag of words," in that it does not consider word order. ArgContent uses a novel content vector analysis approach for score assignment based on the individual arguments in an essay. The average agreement between ArgContent scores and human rater scores found in previous research was 82% as compared to 69% agreement between EssayContent and the human raters. These results suggest that discourse marking enriches e-raters' scoring capability. When e-rater uses its whole set of predictive features, agreement with human rater scores ranges from 87% to 94% across the 15 sets of essay responses used in this study. (Contains 3 tables and 16 references.) (Author/SLD) – Name: DateEntry Label: Entry Date Group: Date Data: 2002 – Name: AN Label: Accession Number Group: ID Data: ED458267 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED458267 |
| RecordInfo | BibRecord: BibEntity: Languages: – Text: English PhysicalDescription: Pagination: PageCount: 9 Subjects: – SubjectFull: College Students Type: general – SubjectFull: Essays Type: general – SubjectFull: Higher Education Type: general – SubjectFull: Scoring Type: general – SubjectFull: Test Scoring Machines Type: general Titles: – TitleFull: Enriching Automated Essay Scoring Using Discourse Marking. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Burstein, Jill – PersonEntity: Name: NameFull: Kukich, Karen – PersonEntity: Name: NameFull: Wolff, Susanne – PersonEntity: Name: NameFull: Lu, Chi – PersonEntity: Name: NameFull: Chodorow, Martin IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2001 |
| ResultId | 1 |
