Computerized Scoring of Essays for Analytical Writing Assessments: Evaluating Score Validity.
Saved in:
| Title: | Computerized Scoring of Essays for Analytical Writing Assessments: Evaluating Score Validity. |
|---|---|
| Language: | English |
| Authors: | Kelly, P. Adam |
| Peer Reviewed: | N |
| Page Count: | 43 |
| Publication Date: | 2001 |
| Sponsoring Agency: | Educational Testing Service, Princeton, NJ. Graduate Record Examination Board Program. |
| Document Type: | Reports - Research Speeches/Meeting Papers |
| Descriptors: | Computer Assisted Testing, Elementary Secondary Education, Essays, Higher Education, Scores, Scoring, Test Scoring Machines, Validity, Writing Evaluation |
| Abstract: | The purpose of this research was to establish, within the constraints of the methods presented, whether the computer is capable of scoring essays in much the same way that human experts rate essays. The investigation attempted to establish what was actually going on within the computer and within the mind of the rater and to describe the degree to which these processes equated. Revealing this parallelism depended on the careful assessment of the "intrinsic" aspects of validity as proposed by S. Messick (1995) of computerized essay scores. The focus was "e-rater" (TM), a computer-based essay scoring system developed by the Educational Testing Service. The study used 1,794 existing Graduate Record Examination Writing Assessment essays written and scored during recent test administration. Factor analysis and the advice of expert raters were used to guide the deconstruction of essay scoring models into subscore models corresponding to writing characteristics within the essay. The writing characteristics identified in this process were used as the basis for developing characteristic-specific scoring rubrics to be used by expert raters. Fresh essay samples were scored by expert rates, both holistically and characteristic-by-characteristic. The same essay samples were assigned both holistic scores and character-wise subscores by the computer scoring models. The degree of convergent validity of scores was evidenced by the proportion of agreement and strength of pairwise correlation among scores and subscores. The statistics derived in this study suggest that simpler e-rater models might do just as well at agreeing with the scores of expert rates, although the proportion of total variance in the expert rater scores explained by the e-rater scores might decrease from an already modest level. (Contains 7 tables and 30 references.) (Author/SLD) |
| Entry Date: | 2002 |
| Accession Number: | ED458296 |
| Database: | ERIC |
| Abstract: | The purpose of this research was to establish, within the constraints of the methods presented, whether the computer is capable of scoring essays in much the same way that human experts rate essays. The investigation attempted to establish what was actually going on within the computer and within the mind of the rater and to describe the degree to which these processes equated. Revealing this parallelism depended on the careful assessment of the "intrinsic" aspects of validity as proposed by S. Messick (1995) of computerized essay scores. The focus was "e-rater" (TM), a computer-based essay scoring system developed by the Educational Testing Service. The study used 1,794 existing Graduate Record Examination Writing Assessment essays written and scored during recent test administration. Factor analysis and the advice of expert raters were used to guide the deconstruction of essay scoring models into subscore models corresponding to writing characteristics within the essay. The writing characteristics identified in this process were used as the basis for developing characteristic-specific scoring rubrics to be used by expert raters. Fresh essay samples were scored by expert rates, both holistically and characteristic-by-characteristic. The same essay samples were assigned both holistic scores and character-wise subscores by the computer scoring models. The degree of convergent validity of scores was evidenced by the proportion of agreement and strength of pairwise correlation among scores and subscores. The statistics derived in this study suggest that simpler e-rater models might do just as well at agreeing with the scores of expert rates, although the proportion of total variance in the expert rater scores explained by the e-rater scores might decrease from an already modest level. (Contains 7 tables and 30 references.) (Author/SLD) |
|---|
