Human Ratings and Complexity, Accuracy, and Fluency (CAF) Indices: A Correlational Study of a Standardised Monologic English-Speaking Test in China
Saved in:
| Title: | Human Ratings and Complexity, Accuracy, and Fluency (CAF) Indices: A Correlational Study of a Standardised Monologic English-Speaking Test in China |
|---|---|
| Language: | English |
| Authors: | Hengzhi Hu (ORCID |
| Source: | SAGE Open. 2025 15(2). |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 13 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Foreign Countries, Speech Tests, English (Second Language), Accuracy, Language Fluency, Standardized Tests, Difficulty Level, Objective Tests, Examiners, College Students |
| Geographic Terms: | China |
| DOI: | 10.1177/21582440251343944 |
| ISSN: | 2158-2440 |
| Abstract: | Foreign language (L2) learners' speaking proficiency is often quantified using two dimensions: intuitive human ratings and analytical, linguistic complexity, accuracy, and fluency (CAF) indices. While previous research and assessment practices have predominantly focused on either the subjective approach to L2 speaking or the objective one, it is essential to establish an association between these two seemingly contradictory assessment methods to enhance and promote more credible assessment judgements. To this end, 160 recordings from a monologic task of a standardised English test in China were analysed to quantify CAF in the present study, and the scores were then compared with human ratings given by qualified examiners. Correlation and regression analyses demonstrated that human ratings were positively correlated with speaking fluency, with the number of pauses produced by a candidate being the most significant predictor of human-judged scores. Speaking complexity also positively predicted human ratings, with examiners tending to focus more on grammatical complexity than lexical complexity. In contrast, no correlations were found between human ratings and speaking accuracy. The findings of this study reinforce the possibility of "halo" effects on human raters in L2 assessment and suggest that rater training should focus on helping examiners recognise and mitigate such potential effects. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1477239 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Foreign language (L2) learners' speaking proficiency is often quantified using two dimensions: intuitive human ratings and analytical, linguistic complexity, accuracy, and fluency (CAF) indices. While previous research and assessment practices have predominantly focused on either the subjective approach to L2 speaking or the objective one, it is essential to establish an association between these two seemingly contradictory assessment methods to enhance and promote more credible assessment judgements. To this end, 160 recordings from a monologic task of a standardised English test in China were analysed to quantify CAF in the present study, and the scores were then compared with human ratings given by qualified examiners. Correlation and regression analyses demonstrated that human ratings were positively correlated with speaking fluency, with the number of pauses produced by a candidate being the most significant predictor of human-judged scores. Speaking complexity also positively predicted human ratings, with examiners tending to focus more on grammatical complexity than lexical complexity. In contrast, no correlations were found between human ratings and speaking accuracy. The findings of this study reinforce the possibility of "halo" effects on human raters in L2 assessment and suggest that rater training should focus on helping examiners recognise and mitigate such potential effects. |
|---|---|
| ISSN: | 2158-2440 |
| DOI: | 10.1177/21582440251343944 |
