Examining the Accuracy of a Conversation-Based Assessment in Interpreting English Learners' Written Responses. Research Report. ETS RR-21-03
Saved in:
| Title: | Examining the Accuracy of a Conversation-Based Assessment in Interpreting English Learners' Written Responses. Research Report. ETS RR-21-03 |
|---|---|
| Language: | English |
| Authors: | Lopez, Alexis A., Guzman-Orth, Danielle, Zapata-Rivera, Diego, Forsyth, Carolyn M., Luce, Christine |
| Source: | ETS Research Report Series. Dec 2021. |
| Availability: | Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets |
| Peer Reviewed: | Y |
| Page Count: | 15 |
| Publication Date: | 2021 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Junior High Schools Middle Schools Secondary Education |
| Descriptors: | Accuracy, English Language Learners, Language Proficiency, Language Tests, Communicative Competence (Languages), Test Format, Mathematics Tests, Middle School Students, Task Analysis, Computer Assisted Testing, Peer Relationship, Teacher Student Relationship, Written Language, Responses, Test Items, Item Analysis, English (Second Language), Second Language Learning |
| ISSN: | 2330-8516 |
| Abstract: | Substantial progress has been made toward applying technology enhanced conversation-based assessments (CBAs) to measure the English-language proficiency of English learners (ELs). CBAs are conversation-based systems that use conversations among computer-animated agents and a test taker. We expanded the design and capability of prior conversation-based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 responses for the 82 students). The study findings helped us to understand the factors that affect the accuracy of the CBA system's interpretations and shed light on how to improve CBA systems that incorporate scaffolding. |
| Abstractor: | As Provided |
| Entry Date: | 2022 |
| Accession Number: | EJ1340994 |
| Database: | ERIC |
| Abstract: | Substantial progress has been made toward applying technology enhanced conversation-based assessments (CBAs) to measure the English-language proficiency of English learners (ELs). CBAs are conversation-based systems that use conversations among computer-animated agents and a test taker. We expanded the design and capability of prior conversation-based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 responses for the 82 students). The study findings helped us to understand the factors that affect the accuracy of the CBA system's interpretations and shed light on how to improve CBA systems that incorporate scaffolding. |
|---|---|
| ISSN: | 2330-8516 |
