Evaluating the impact of nonverbal behavior on language ability ratings.
Saved in:
| Title: | Evaluating the impact of nonverbal behavior on language ability ratings. |
|---|---|
| Authors: | Burton, J. Dylan1,2 jdburton@gsu.edu |
| Source: | Language Testing. Oct2024, Vol. 41 Issue 4, p729-758. 30p. |
| Subject Terms: | *Nonverbal ability, *Language ability, *Competency-based teacher education, Semantics, Grammar |
| Abstract: | Nonverbal behavior can impact language proficiency scores in speaking tests, but there is little empirical information of the size or consistency of its effects or whether language proficiency may be a moderating variable. In this study, 100 novice raters watched and scored 30 recordings of test takers taking an international, high stakes proficiency test. The speech samples were each 2 minutes long and ranged in proficiency levels. The raters scored each sample on fluency, vocabulary, grammar, and comprehensibility using 7-point semantic differential scales. Nonverbal behavior was extracted using an automated machine learning software called iMotions, and data was analyzed with ordinal mixed effects regression. Results showed that attentional variance predicted fluency, vocabulary, and grammar scores, but only when accounting for proficiency. Higher standard deviations of attention corresponded with lower scores for the lower-proficiency group, but not the mid/higher-proficiency group. Comprehensibility scores were only predicted by mean valence when proficiency was an interaction term. Higher mean valence, or positive emotional behavior, corresponded with higher scores in the lower-proficiency group, but not the mid/higher-proficiency group. Effect sizes for these predictors were quite small, with small amounts of variance explained. These results have implications for construct representation and test fairness. [ABSTRACT FROM AUTHOR] |
| Copyright of Language Testing is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 180428207 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Evaluating the impact of nonverbal behavior on language ability ratings. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Burton%2C+J%2E+Dylan%22">Burton, J. Dylan</searchLink><relatesTo>1,2</relatesTo><i> jdburton@gsu.edu</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Language+Testing%22">Language Testing</searchLink>. Oct2024, Vol. 41 Issue 4, p729-758. 30p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Nonverbal+ability%22">Nonverbal ability</searchLink><br />*<searchLink fieldCode="DE" term="%22Language+ability%22">Language ability</searchLink><br />*<searchLink fieldCode="DE" term="%22Competency-based+teacher+education%22">Competency-based teacher education</searchLink><br /><searchLink fieldCode="DE" term="%22Semantics%22">Semantics</searchLink><br /><searchLink fieldCode="DE" term="%22Grammar%22">Grammar</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Nonverbal behavior can impact language proficiency scores in speaking tests, but there is little empirical information of the size or consistency of its effects or whether language proficiency may be a moderating variable. In this study, 100 novice raters watched and scored 30 recordings of test takers taking an international, high stakes proficiency test. The speech samples were each 2 minutes long and ranged in proficiency levels. The raters scored each sample on fluency, vocabulary, grammar, and comprehensibility using 7-point semantic differential scales. Nonverbal behavior was extracted using an automated machine learning software called iMotions, and data was analyzed with ordinal mixed effects regression. Results showed that attentional variance predicted fluency, vocabulary, and grammar scores, but only when accounting for proficiency. Higher standard deviations of attention corresponded with lower scores for the lower-proficiency group, but not the mid/higher-proficiency group. Comprehensibility scores were only predicted by mean valence when proficiency was an interaction term. Higher mean valence, or positive emotional behavior, corresponded with higher scores in the lower-proficiency group, but not the mid/higher-proficiency group. Effect sizes for these predictors were quite small, with small amounts of variance explained. These results have implications for construct representation and test fairness. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Language Testing is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=180428207 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1177/02655322241255709 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 30 StartPage: 729 Subjects: – SubjectFull: Nonverbal ability Type: general – SubjectFull: Language ability Type: general – SubjectFull: Competency-based teacher education Type: general – SubjectFull: Semantics Type: general – SubjectFull: Grammar Type: general Titles: – TitleFull: Evaluating the impact of nonverbal behavior on language ability ratings. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Burton, J. Dylan IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 10 Text: Oct2024 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 02655322 Numbering: – Type: volume Value: 41 – Type: issue Value: 4 Titles: – TitleFull: Language Testing Type: main |
| ResultId | 1 |
