Examining Rater Effects in Assessing Learners' Speech and Writing: Insights from ICNALE
Saved in:
| Title: | Examining Rater Effects in Assessing Learners' Speech and Writing: Insights from ICNALE |
|---|---|
| Language: | English |
| Authors: | Hyunwoo Kim |
| Source: | English Teaching. 2026 81(1):59-80. |
| Availability: | Korea Association of Teachers of English. 6105 English Education Department, Chinju National University of Education, 369beon-gil 3, Jinyangho-ro, Jinju, Gyeongsangnam-do, 52673, Republic of Korea. Tel: +82-42-629-7381; Fax: +82-42-629-7320; e-mail: katejournal29@gmail.com; Web site: https://journal.kate.or.kr/ |
| Peer Reviewed: | Y |
| Page Count: | 22 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Writing Evaluation, Speech Evaluation, Interrater Reliability, Bias, Scoring, English Teachers, English (Second Language), Experience, Evaluators, College Students, Essays, Foreign Countries |
| Geographic Terms: | Asia |
| ISSN: | 1017-7108 2671-9312 |
| Abstract: | This study investigated rater effects in the ICNALE Global Rating Archives (GRA), which contain ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria. After cleaning the raw dataset, two separate three-facet partial credit models were fitted to the speech and essay datasets to examine rater consistency, and a bias analysis was conducted to investigate the relationship between raters' prior rating experience and severity. The results indicated that six raters in the speech ratings exhibited overly inconsistent rating patterns, but no rater in the essay ratings did. It was also found that raters' prior rating experience was significantly associated with the complexity, involvement, and accuracy criteria in the speech ratings and with the complexity criterion in the essay ratings. However, the corresponding effect sizes for these interactions were trivial, indicating limited practical impact. The educational implication is that rigorous rater training could enhance rater consistency among both pre-service and less-experienced in-service English teachers in speaking and writing assessment. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1504902 |
| Database: | ERIC |
| Abstract: | This study investigated rater effects in the ICNALE Global Rating Archives (GRA), which contain ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria. After cleaning the raw dataset, two separate three-facet partial credit models were fitted to the speech and essay datasets to examine rater consistency, and a bias analysis was conducted to investigate the relationship between raters' prior rating experience and severity. The results indicated that six raters in the speech ratings exhibited overly inconsistent rating patterns, but no rater in the essay ratings did. It was also found that raters' prior rating experience was significantly associated with the complexity, involvement, and accuracy criteria in the speech ratings and with the complexity criterion in the essay ratings. However, the corresponding effect sizes for these interactions were trivial, indicating limited practical impact. The educational implication is that rigorous rater training could enhance rater consistency among both pre-service and less-experienced in-service English teachers in speaking and writing assessment. |
|---|---|
| ISSN: | 1017-7108 2671-9312 |
