View in EDS HTML Full Text PDF Full Text

Addressing Bias in Spoken Language Systems Used in the Development and Implementation of Automated Child Language-Based Assessment

Saved in:
Bibliographic Details
Title: Addressing Bias in Spoken Language Systems Used in the Development and Implementation of Automated Child Language-Based Assessment
Language: English
Authors: Alison L. Bailey (ORCID 0000-0001-9303-8304), Alexander Johnson, Natarajan Balaji Shankar, Hariram Veeramani, Julie A. Washington, Abeer Alwan
Source: Journal of Educational Measurement. 2026 63(1).
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 30
Publication Date: 2026
Sponsoring Agency: National Science Foundation (NSF)
Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) (DHHS/NIH)
Contract Number: 2202585
P01HD070837
Document Type: Journal Articles
Reports - Research
Descriptors: Bias, Oral Language, Automation, Child Language, Speech Communication, Scoring, Accuracy, Natural Language Processing, Algorithms, Measurement, Language Variation, Black Dialects, Suprasegmentals, Pronunciation, Language Usage, Grammar, Language Impairments, Reading Difficulties
DOI: 10.1111/jedm.12435
ISSN: 0022-0655
1745-3984
Abstract: This article addresses bias in Spoken Language Systems (SLS) that involve both Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) and reports experiments to improve the performance of SLS for automated language and literacy-related assessments with students who are under served in the U.S. educational system. We frame bias in SLS in terms of testing fairness and validity, stemming in part from the exclusion of sufficiently large training datasets in varieties of English other than General American English (GAE). We adopt an Interpretation/Use Argument approach to validity focused on clarity of constructs and scoring accuracy. While SLS use ASR to automatically transcribe students' utterances, and apply NLP algorithms to ASR transcripts to measure students' speech samples, it is well-documented in studies with adults that ASR is typically more problematic for African American English (AAE) speakers than for other groups due to differences in prosody, pronunciation, word usage, and grammar. We utilized child speech and text corpora to improve algorithms that score oral task responses for child AAE speakers and, in some experiments, children with oral language and reading difficulties. Favorable results provide impetus and possible solutions for fair and inclusive assessments for diverse student groups in the future.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1501375
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFcE4Jxq_DgwpUfpzYr_YBEAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDC8cFeGIXNif9GNgyQIBEICBm1KFJLVXKQzUntP0i_yGLJ5dpgLQLKi47URfAd5cQbmNkC51uDxXHFOdTq8pfRZpqfs-0BM8kYQ2y3yEwnraWJNE2_SC4jJzcWaNzVHN0Ia7BtPWHGA18nQEUH4Ilw7dtQuiSPz82YlaWUcqf8Syc2OA5fuZPICJhWklQttDY75egZLxaZkjX9_BA16Q-Btgn_0Vyo8pSCsTB88E
Text:
  Availability: 1
  Value: <anid>AN0192629999;mea01mar.26;2026Apr01.06:22;v2.2.500</anid> <title id="AN0192629999-1">Addressing Bias in Spoken Language Systems Used in the Development and Implementation of Automated Child Language‐Based Assessment </title> <sbt id="AN0192629999-2">Introduction</sbt> <p>This article addresses bias in Spoken Language Systems (SLS) that involve both Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) and reports experiments to improve the performance of SLS for automated language and literacy‐related assessments with students who are under served in the U.S. educational system. We frame bias in SLS in terms of testing fairness and validity, stemming in part from the exclusion of sufficiently large training datasets in varieties of English other than General American English (GAE). We adopt an Interpretation/Use Argument approach to validity focused on clarity of constructs and scoring accuracy. While SLS use ASR to automatically transcribe students' utterances, and apply NLP algorithms to ASR transcripts to measure students' speech samples, it is well‐documented in studies with adults that ASR is typically more problematic for African American English (AAE) speakers than for other groups due to differences in prosody, pronunciation, word usage, and grammar. We utilized child speech and text corpora to improve algorithms that score oral task responses for child AAE speakers and, in some experiments, children with oral language and reading difficulties. Favorable results provide impetus and possible solutions for fair and inclusive assessments for diverse student groups in the future.</p> <p>Interactive Spoken Language Systems (SLS) that involve both Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) offer the possibility of a powerful tool for PreK‐12 educational assessment, freeing up teachers' time and engaging students in personalized and autonomous learning (e.g., Bryant et al., [<reflink idref="bib10" id="ref1">10</reflink>]). (See Appendix for glossary of key terms and list of acronyms.) Such automated systems can also enable more frequent formative and classroom summative assessment because they offer teachers the ease of management and ability to provide students with more timely scores or feedback on their performances (e.g., Bailey et al., [<reflink idref="bib4" id="ref2">4</reflink>]). Thus, overall, SLS can facilitate more effective (i.e., efficient, current, and accurate) learning and assessment. ASR technology, for example, can be used to automate the transcription of students' oral responses to spoken prompts or reading stimuli, and then downstream language modeling can be used to extract statistics from the transcript that relate to student knowledge and skills in a target area (e.g., use of key vocabulary, syntactic complexity of responses, accuracy of oral reading, completeness of target content knowledge, etc.).</p> <p>These processes have already been implemented to an extent in systems such as the <emph>Google Read Along</emph> application that uses speech recognition to listen to children as they read aloud and to offer them feedback (Google, [<reflink idref="bib27" id="ref3">27</reflink>]). ETS has also deployed SLS that use ASR and NLP to automatically grade participant responses during the <emph>Test of English for Teaching</emph> (TEFT, Zechner et al., [<reflink idref="bib92" id="ref4">92</reflink>]). Researchers have also attempted to improve the effectiveness of educational SLS with, for example, pronunciation training (Sancinetti et al., [<reflink idref="bib68" id="ref5">68</reflink>]) and automatic essay scoring (Ke et al., [<reflink idref="bib39" id="ref6">39</reflink>]; Uto et al., [<reflink idref="bib77" id="ref7">77</reflink>]). These systems typically operate by learning a best‐fit mapping between selected characteristics of a student response and a human‐annotated ground truth label (i.e., a known target, as in the score given by a human scorer). This mapping can then be extrapolated to new samples for making inferences about them. However, excluding segments of the student population from participation in the collection of large training datasets for algorithms used in these systems raises a concern for algorithmic bias in educational assessment that can lead to inaccuracies in evaluating student performances when SLS are part of the assessment design, implementation and scoring. Exclusion of particular groups of people has been a key concern of fairness in SLS (Rajan et al., [<reflink idref="bib63" id="ref8">63</reflink>]), and in the educational context, it can ultimately contribute to a lack of assessment fairness and threaten the validity of an assessment. Fairness in this assessment context is concerned with the accurate representation of the speech of test‐takers so that their test performances (e.g., scores) that rely on speech and/or the transcripts produced from speech can be interpreted meaningfully. Negative bias is created when SLS lack information from specific groups of intended test‐takers, resulting in deficit‐based or erroneous interpretations.</p> <p>The integration of technologies in early schooling contexts in particular, needs more extensive exploration (e.g., Neumann & Neumann, [<reflink idref="bib55" id="ref9">55</reflink>]), not least because assessments based on recent technological advancements may not validly measure the recorded responses of young students whose speech may not conform to General American English (GAE) commonly encountered in mainstream classrooms and in curricular materials, such as speech from speakers of underrepresented languages or varieties of a language like African American English (AAE) and speech from speakers with language‐related disabilities (e.g., language impairment and reading difficulties). GAE is most used in formal education, commerce, and media in the United States (Crystal, [<reflink idref="bib14" id="ref10">14</reflink>]) and is the variety of English that has been regularized through print. AAE is a well‐documented, rule‐governed dialect used by many African Americans (e.g., Green, [<reflink idref="bib28" id="ref11">28</reflink>]).</p> <p>Currently, AAE is underrepresented in many speech and text corpora (i.e., large databases of samples of naturally occurring language) that are used to train even adult SLS. As a result, many commercially available speech recognition systems trained on these corpora perform worse for AAE than for GAE (Koenecke et al., [<reflink idref="bib41" id="ref12">41</reflink>]). This poses a challenge for creating SLS that perform equitably across speaker demographics, even with limited training data for some populations. This article focuses primarily on challenges and innovations in creating systems to conduct automatic spoken language assessments for child speakers of AAE with whom even less research has been conducted. Additionally, in some experiments, we report results with the speech of students who are less skilled readers or have a language impairment and/or reading disability and whose speech or oral reading productions have also been scarce in corpora and may provide an added challenge to SLS.</p> <hd id="AN0192629999-3">Approaches to Fairness and Validity in Spoken Language Systems</hd> <p>In this article, we situate the use of SLS within the broader frameworks of assessment validity and the equitable assessment of diverse students, closely evaluating the adequacy of techniques that include speech samples containing differences between the speech of linguistically underrepresented students and GAE—the variety of English that has most commonly served as training data for algorithms in automated assessment (see Prinos et al., [<reflink idref="bib60" id="ref13">60</reflink>] for a recent review of fairness and ASR applications more generally).</p> <p>Assessment validity is "...the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests" (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, [<reflink idref="bib1" id="ref14">1</reflink>], p. 11). We adopt an Interpretation/Use Argument (Kane, [<reflink idref="bib37" id="ref15">37</reflink>]) approach to validity. According to Chapelle (2012), "The interpretive argument with the supporting evidence is the validity argument." (p.19). This approach also builds on the Assessment Use Argument that Bachman ([<reflink idref="bib2" id="ref16">2</reflink>]) proposed for addressing the validity of language assessments. Typically, a coherent validity argument examines assumptions at every level of assessment development, gathering evidence that those assumptions are sound so that the elicited responses from students and the score inferences made on those responses are credible and ultimately useful to teachers and their students. More specifically for the SLS work described here, Bejar ([<reflink idref="bib7" id="ref17">7</reflink>]) aligns assessment validity with the quality control that must go into the design of automated scoring systems, evidence extraction (e.g., features that are properties of student responses) and evidence synthesis (e.g., formulating a score).</p> <p>Recently, Randall and colleagues (Randall et al., [<reflink idref="bib65" id="ref18">65</reflink>]; Randall, [<reflink idref="bib66" id="ref19">66</reflink>]) have called for a justice‐oriented, antiracist approach to assessment validation at every phase of the assessment development process and in test specifications (e.g., construct definition, item creation, scoring criteria, selection of bias and sensitivity panel membership, etc.). This approach compels test developers to critically interrogate their processes and reflect on key aspects that may introduce or perpetuate inequities (e.g., "<emph>What groups may be advantaged and disadvantaged by the administration of this assessment? In the short term? The long term?</emph>" (Randall et al., [<reflink idref="bib65" id="ref20">65</reflink>] p. 6). Lack of representation of speech from students across racial/ethnic and linguistic lines in the large datasets that sample speech for training SLS is an important, if relatively recent, feature of assessment development that poses a major threat to claims of validity in the Interpretation/Use Arguments of automated assessments. Redressing this situation is galvanized by the kinds of questions posed by a justice‐oriented, antiracist approach to assessment validation (e.g., Evans & Taylor, [<reflink idref="bib19" id="ref21">19</reflink>]).</p> <p>The problem of a lack of fairness, however, may not only be viewed as a threat to the validity of assessments that rely on SLS but may require a fairness argument in addition (Huggins‐Manley et al., [<reflink idref="bib32" id="ref22">32</reflink>]) if validity is understood, as it frequently is, to mean comparability of score inferences across different groups of test‐takers (e.g., Xi, [<reflink idref="bib89" id="ref23">89</reflink>]). Indeed, comparability may not always be fair (Randall et al., [<reflink idref="bib65" id="ref24">65</reflink>]). In this instance, a fair application of SLS requires training with different speech datasets and an understanding that students who are AAE speakers will produce <emph>different</emph> linguistic responses to assessment tasks. As Huggins‐Manley et al. ([<reflink idref="bib32" id="ref25">32</reflink>], p.367) stress, "We desire to have valid inferences from a validity argument and simultaneously make claims about fairness that can go beyond comparable score inferences from the assessment."</p> <p>Fairness in this context constitutes ensuring (<reflink idref="bib1" id="ref26">1</reflink>) the clarity of the constructs being assessed with AAE child speakers and students who are less skilled readers or have a language impairment and/or reading disability, and (<reflink idref="bib2" id="ref27">2</reflink>) the accuracy of scoring to these constructs. However, evaluating clarity and accuracy must occur at two separate levels given that the automated version of the oral narrative tasks studied here relies on clarity and accuracy at both a prerequisite transcription stage when the speech stream of children's task responses is converted to text (created by ASR) and a task scoring stage (created by NLP). Consequently, while we define the constructs to be measured by the automated tasks in further detail below, to contribute to the Interpretation/Use Argument, we continue to evaluate construct clarity and accuracy after each stage of the reported SLS development.</p> <hd id="AN0192629999-4">Clarity of constructs</hd> <p>Universal Design for Assessment (UDA) is applied using the same principles as Universal Design for Learning that allow for the widest possible inclusion of students in education (Thompson et al., [<reflink idref="bib75" id="ref28">75</reflink>]). By designing assessments to include students from a wide range of backgrounds and abilities from the start of an assessment's development, claims about what the performances of all participating students mean can still be made. A key element to UDA is "precisely defined constructs" (Thompson et al., [<reflink idref="bib75" id="ref29">75</reflink>], p. 7). The constructs being measured by the tasks in the experiments we report in this article are multifaceted. Ostensibly, the tasks elicit oral narrative productions: a story retelling in response to a story being told to the child test‐taker and a picture description in response to being shown a single picture prompt.</p> <p>Narrative productions are intended to measure a child's ability to use language in a functional way to convey a story's or picture's setting, characters, and key story elements. Narratives reveal information about a child's vocabulary skills (e.g., names for objects in the stories or picture) grammar usage (e.g., tense consistency), grammatical accuracy (e.g., noun‐verb agreement), and coherence (e.g., ordered information). Moreover, narration is an extended discourse genre that requires the integration of many additional underlying language skills, including the comprehension and production of a wide vocabulary (beyond target object names), comprehension and production of a wide range of syntactic structures and morphological forms (beyond tense and grammatical accuracy) and phonological processing and production (e.g., pronunciation, prosody). Consequently, narrative productions have intended scorable (construct relevant) features, as well as attendant underlying language features that are construct irrelevant in this assessment context.</p> <hd id="AN0192629999-5">Accuracy of construct scoring</hd> <p>Both the target narrative features to be scored, as well as the additional underlying features that narrative production requires, can all be subject to dialectal variation. Narratives are strongly influenced by different cultural emphases and values (e.g., McCabe et al., [<reflink idref="bib53" id="ref30">53</reflink>]). AAE‐speaking children's narrative productions may differ from those of children with different dialects, including GAE (Gardner‐Neblett et al., [<reflink idref="bib24" id="ref31">24</reflink>]). Their narratives must be scored accurately for construct relevant features but the likelihood of this may decrease if these features spoken in AAE dialect remain largely absent in current SLS due to a lack of relevant training corpora.</p> <p>Specifically, the questions we ask address both the fairness and the validity arguments in this automated assessment context, namely (<reflink idref="bib1" id="ref32">1</reflink>) How well do SLS perform in educational applications with limited amounts of publicly available in‐domain data on the speech of AAE speaking students and, in some experiments, the speech of students with language and reading difficulties? and (<reflink idref="bib2" id="ref33">2</reflink>) Do SLS introduce a significant amount of algorithmic bias into educational assessment tasks?</p> <p>The next section presents background information on the educational and social justice impetus for this research, as well as known challenges in spoken language understanding technology for low‐resource tasks (i.e., tasks for which machine‐learning‐based systems cannot achieve state‐of‐the‐art performance as a consequence of the limited "in‐domain" data available). We then present a novel framework for performing automatic assessments of children's language and reading‐related abilities and report results of experiments testing on children's speech from both speakers of AAE and nonspeakers of AAE who speak Southern American English dialect and, additionally, in some instances with speakers who present with language and reading difficulties. We discuss lessons learned particularly the importance of clarity of the assessment constructs and accuracy of scoring to those constructs to ensure fairness and validity in this assessment context, and conclude with suggestions for future directions.</p> <hd id="AN0192629999-6">Background</hd> <p>As a result of a nationwide shortage of well‐qualified teachers (Nguyen et al., [<reflink idref="bib56" id="ref34">56</reflink>]) and a mismatch between the racial composition of the student population and available teachers (Redding, [<reflink idref="bib67" id="ref35">67</reflink>]), African American children may be taught by teachers who are ill‐equipped to provide effective language and literacy instruction and assessment. Representation of students' race or ethnicity in the teaching faculty who educate them is a predictor of student academic achievement (Egalite et al., [<reflink idref="bib18" id="ref36">18</reflink>]). In addition, a lack of knowledge of the specific linguistic features of AAE, even among teachers who are familiar with this cultural language system from daily exposure, may render educators less effective in supporting and accurately assessing the language and literacy development of African American students (Washington & Iruka, [<reflink idref="bib83" id="ref37">83</reflink>]).</p> <p>These factors alone or in combination may contribute to the disparity that is found in reading levels between African American elementary students in low socioeconomic status schools and their white counterparts (Downer et al., [<reflink idref="bib17" id="ref38">17</reflink>]). Furthermore, teachers may not take advantage of students' language learning at home and, when students are evaluated, they may be rated below their actual reading abilities (Washington et al., [<reflink idref="bib81" id="ref39">81</reflink>]). SLS may help to alleviate these issues, as teachers use technology programmed with the necessary cultural and linguistic expertise to save time and increase accuracy when providing students with language‐ and literacy‐related feedback on their performances, particularly in a formative assessment context (i.e., use of assessment for informing on‐going learning and instruction, Wiliam, [<reflink idref="bib87" id="ref40">87</reflink>]). Despite the need for such systems, one inherent challenge in designing SLS for equitable education is that populations who are underserved in schools are typically also underrepresented in digital technologies.</p> <hd id="AN0192629999-7">AAE Usage</hd> <p>AAE is one of the most studied varieties of English in both children and adults (Lanehart & Malik, [<reflink idref="bib46" id="ref41">46</reflink>]; Oetting, [<reflink idref="bib57" id="ref42">57</reflink>]). However, it has been relatively ignored in the area of SLS. Johnson ([<reflink idref="bib34" id="ref43">34</reflink>]) points out that while all language domains are influenced by AAE, it most significantly presents differences in rules for production of phonology, morphology and syntax. Usage of AAE can be placed on a low‐ to high‐density dialect continuum with children who are high‐density users of AAE more likely to be growing up in material poverty (Washington & Craig, [<reflink idref="bib82" id="ref44">82</reflink>]). High‐density usage among students from low‐income backgrounds is likely related to isolation and widespread school segregation. Children in the Southern United States additionally use particularly high levels of the regional variety of English, which combines with AAE to result in oral language that differs significantly from the print form of GAE (Oetting & Pruitt, [<reflink idref="bib58" id="ref45">58</reflink>]).</p> <p>We know significantly less about the characteristics of AAE in connected text and discourse, including how connected speech impacts phonological features for intelligibility with school‐age children, than we do about selected grammatical features of AAE and the phonological development and disorders of much younger AAE speakers (Stockman et al., [<reflink idref="bib73" id="ref46">73</reflink>]). Only a small number of studies have examined the "rhythm" of AAE (Thomas, [<reflink idref="bib74" id="ref47">74</reflink>]). These are the prosodic and other suprasegmental features (e.g., stress, pitch, and length of sounds) of African American child speech that can influence the intended meaning of utterances, as well as intelligibility for non‐AAE speakers. These features are important to document for ASR to recognize the speech of a large segment of African American child speakers. In addition, some phonological and morphosyntactic features of AAE vary systematically from GAE (e.g., Washington & Seidenberg, [<reflink idref="bib84" id="ref48">84</reflink>]). These may obscure intelligibility and include the deletion of final consonants and morphological markers, which may influence the perception of word boundaries by listeners, as well as changes in vowel sounds, such as the monophthongization of diphthongs (e.g., "ahs" instead of "ice"). Such features can characterize both AAE and Southern American English (Kretzschmar, [<reflink idref="bib44" id="ref49">44</reflink>]).</p> <hd id="AN0192629999-8">Literacy Development and Assessment with AAE‐Speaking Students</hd> <p>Dialect influences on the development of African American students literacy skills have been well documented (e.g., Craig et al. [<reflink idref="bib13" id="ref50">13</reflink>]; Puranik et al., [<reflink idref="bib61" id="ref51">61</reflink>]). Young students who are high‐density users of AAE are found to more often struggle with development of early literacy skills than children who are lower‐density users of AAE (Washington et al., [<reflink idref="bib81" id="ref52">81</reflink>]). The linguistic difference or <emph>distance</emph> between AAE and printed forms of GAE appears to contribute to slower rates of development in both reading and writing for African American students (Brown et al., [<reflink idref="bib9" id="ref53">9</reflink>]).</p> <p>For children who use AAE, the assessment context also presents a special challenge. Decades of research have demonstrated that assessment instruments routinely underestimate the language and literacy knowledge of African American children who use AAE (e.g., Washington & Iruka, [<reflink idref="bib83" id="ref54">83</reflink>]). When African American children are assessed, they hear items presented in GAE, but often respond to these items in AAE. These responses may be scored as incorrect or inaccurate, reducing the African American child's overall score. In clinical contexts, this can result in children being identified with having language impairments where none exist (Seymour, [<reflink idref="bib70" id="ref55">70</reflink>]).</p> <hd id="AN0192629999-9">Automatic Speech Recognition for Spoken Language Assessments</hd> <p>As automated speech‐to‐text tools gain wider use in literacy assessments, children with different varieties of spoken English or atypicalities (e.g., language impairments) are increasingly at risk due to biases in these technologies trained primarily on speakers of GAE. ASR technology is critical for converting speech into text, with some of the state‐of‐the‐art systems in this field including Wav2Vec 2.0 (Baevski et al., [<reflink idref="bib3" id="ref56">3</reflink>]), HuBERT (W.‐N. Hsu et al., [<reflink idref="bib30" id="ref57">30</reflink>]), and Whisper (Radford et al., [<reflink idref="bib62" id="ref58">62</reflink>]). The operational framework of the mentioned neural network‐based systems fundamentally involves a conversion of speech waveforms into a transcription of the utterance. The neural networks consist of several "layers," or series of mathematical operations through which the input is processed. Initially, raw audio is passed through a series of convolutional layers, where typically intricate low‐level acoustic features are extracted. Following this stage, the encoder layers most often use "self‐attention" to focus on the most important parts of the input, regardless of their position, and "feed‐forward layers" to refine this information into meaningful representations. This helps the system understand complex input more effectively. This encoding process is crucial for capturing the hierarchical information embedded in the audio signals. Finally, these are mapped to a series of logits, or outputs, that encode the probability of occurrence of any individual "token" in the utterance, where the token can be an individual character, a phoneme, or an entire word depending on the ASR system chosen. The resulting logits are then decoded using a language model, which predicts the next word in a sequence based on the preceding few words by leveraging patterns learned from textual data, while also considering the probabilities of word occurrences based on the history of prior decoded words to construct the hypothesized transcriptions.</p> <p>The differences between the aforementioned systems lead to differing performances in low‐resource settings. Wav2Vec 2.0, the oldest among these, leverages self‐supervised learning and contrastive learning methods to directly map an input audio waveform to text. HuBERT utilizes a k‐means based clustering algorithm to model the self‐supervised learning objective as a masked prediction of hidden units. This connects to the challenges of handling dialects, accents and atypicalities in expected language features because HuBERT's clustering algorithm helps the model learn patterns from speech without relying on labeled data. By focusing on a self‐supervised approach, it can better capture such diverse speech variations, making it a relevant method to explore in the context of this article. Through training on nearly 680,000 hours of data (far higher than other systems), Whisper was able to achieve impressive performance using an encoder‐decoder architecture.</p> <p>While these systems represent significant strides in ASR, the bias in their training data leads to less than desirable performance in low‐resource settings. Of particular note is the relatively high transcription error rate seen in dialectal (Koenecke et al., [<reflink idref="bib41" id="ref59">41</reflink>]) and children's (Fan et al., [<reflink idref="bib21" id="ref60">21</reflink>]) speech. This must be given particular consideration while designing spoken language assessment systems, as incorrect transcriptions could lead to a cascading effect on the performance of other parts of any proposed framework or assessment development. It is thus essential to design SLS that are robust to transcription errors.</p> <hd id="AN0192629999-10">Machine Learning for Automatic Assessment Scoring</hd> <p>One challenge in using automated systems for language assessment that are impacted by transcription errors is that they distort the meaning of the responses. One possible strategy to alleviate the impact of transcription errors involves employing systems that comprehensively assess the meaning of the entire utterance to predict scores for a language assessment. This can be accomplished by leveraging Large Language Models (LLMs), which fundamentally entail the acquisition of "embeddings"—high‐dimensional vector representations of the input text. Many neural network‐based systems for processing speech and text utilize the BERT (Bidirectional Encoder Representations from Transformers) architecture (Devlin et al., [<reflink idref="bib16" id="ref61">16</reflink>]). BERT utilizes bidirectional context, enabling it to consider both the preceding and following words when understanding the meaning of a particular word in a sentence. This bidirectional approach allows for the capture of rich contextual information producing highly contextualized word embeddings. BERT was chosen for our experiments because it excels at leveraging context to interpret meaning, even in noisy or error‐prone inputs like ASR transcriptions. This approach pretrains on a vast corpus of text using masked language modeling in which select words are first randomly masked. The context of the surrounding words is used by the system to then predict these words. This, coupled with an auxiliary task for next sentence prediction, allows embeddings obtained from BERT to have great predictive power in downstream tasks (applications or specific tasks that use the output or results of the initial system).</p> <p>Since the emergence of BERT, several related models have been routinely used in NLP. DistilBERT (Sanh et al., [<reflink idref="bib69" id="ref62">69</reflink>]) is trained by distilling the BERT model to emphasize parameter reduction. ALBERT (A Lite BERT) (Lan et al., [<reflink idref="bib45" id="ref63">45</reflink>]) further refined this approach by combining parameter reduction with a new task involving sentence order prediction. XLNet (Yang et al., [<reflink idref="bib90" id="ref64">90</reflink>]) introduced an autoregressive mechanism to capture bidirectional context. BART (Lewis et al., [<reflink idref="bib50" id="ref65">50</reflink>]) combines elements of autoencoding and autoregressive modeling and is trained by reordering sentences with missing or scrambled words, which helps it become good at tasks like generating and correcting text. RoBERTa (Robustly optimized BERT approach) (Liu et al., [<reflink idref="bib51" id="ref66">51</reflink>]) optimized BERT's training methodology by dynamically masking and updating token sequences. Each of these models works differently to improve how they understand language. DistilBERT is a smaller and faster version of BERT, designed to use fewer resources. ALBERT makes it even more efficient by reducing its size and adding a new way to understand sentence order. XLNet takes a different approach by looking at the order of words in a way that helps it understand context better. RoBERTa improves how BERT is trained by shuffling and updating the words it focuses on during learning.</p> <p>New research on automated spoken language assessments has contributed valuable insights into the complexities of evaluating children's oral language productions. Wong et al. ([<reflink idref="bib88" id="ref67">88</reflink>]) have investigated the efficacy of multitask learning as a means to overcome the scarcity of data in automatic oral English proficiency assessment for Mandarin speakers. This approach, focused on expanding the applicability of assessments, underscores the importance of addressing linguistic diversity in testing scenarios. In addition, Baumann et al. ([<reflink idref="bib6" id="ref68">6</reflink>]) compared the performance of Wav2Vec2.0 (Baevski et al., [<reflink idref="bib3" id="ref69">3</reflink>]) and Kaldi time delay neural network‐based (Povey et al., [<reflink idref="bib59" id="ref70">59</reflink>]) grapheme embeddings as features for evaluating children's phonological working memory for nonwords (i.e., nonsense words or pseudowords that have no meaning in the language but abide by a language's phonotactic rules and are pronounceable). Similarly, Getman et al. ([<reflink idref="bib25" id="ref71">25</reflink>]) used hidden states from Wav2Vec2.0 to predict mispronunciations and abnormalities in children's speech.</p> <p>Several studies of essay scoring have also analyzed the performance of existing language models for automatic scoring. Studies have used Natural Language Understanding (NLU) to score written essays for narrative language proficiency (Ramesh & Sanampudi, [<reflink idref="bib64" id="ref72">64</reflink>]; Shibata & Uto, [<reflink idref="bib71" id="ref73">71</reflink>]). Notably, B. W. Lee et al. ([<reflink idref="bib47" id="ref74">47</reflink>]) combined handcrafted linguistic features (manually designed by experts to capture specific linguistic properties), which capture advanced semantics, with soft label predictions or probabilistic outputs representing the likelihood of various labels from RoBERTa (Liu et al., [<reflink idref="bib51" id="ref75">51</reflink>]) in a hybrid model, which achieves state‐of‐the‐art‐performance readability score classification. However, further work is still needed to make such systems more robust to differences in children's speech and texts across dialects.</p> <hd id="AN0192629999-11">Challenges for Children's ASR</hd> <p>The challenges in machine learning‐based children's speech recognition are well known (Das et al., [<reflink idref="bib15" id="ref76">15</reflink>]). Most neural network‐based ASR systems are trained using adult speech, leading to three main issues when drawing inferences from children's speech data: (<reflink idref="bib1" id="ref77">1</reflink>) young children's speech often contains production errors (substituted speech sounds, for example) and variability in how they produce speech sounds due to the fact that they have yet to master the complex articulatory gestures needed to produce conventional or adult‐like English speech sounds (Koenig et al., [<reflink idref="bib42" id="ref78">42</reflink>]). Consequently, here is both higher inter‐ and intraspeaker variability in child speech when compared to adults, (<reflink idref="bib2" id="ref79">2</reflink>) the frequency range of children's voices is much higher than that of adults, making them less compatible with systems trained on adult's speech (S. Lee et al., [<reflink idref="bib49" id="ref80">49</reflink>]), and (<reflink idref="bib3" id="ref81">3</reflink>) ASR systems trained to recognize the words of an adult's vocabulary will likely have bias toward interpreting the child's words as ones more commonly used by adults (S. Lee et al., [<reflink idref="bib48" id="ref82">48</reflink>]). Therefore, in order to more effectively create an ASR system for children, the system needs to be trained using child‐specific speech data. However, available child speech data are extremely limited compared to adult speech data. Although most high‐performing ASR systems are trained on at least 1,000 hours of annotated speech data, practical considerations have made it difficult to obtain this large amount of speech data from children (Johnson et al., [<reflink idref="bib35" id="ref83">35</reflink>]). While some corpora like the Center for Spoken Language Understanding (CSLU) Kids' Speech corpus (Shobaki et al., [<reflink idref="bib72" id="ref84">72</reflink>]) and My Science Tutor database (Ward et al., [<reflink idref="bib80" id="ref85">80</reflink>]) are available, none of these databases come close to containing 1,000 hours of speech.</p> <p>The lack of a large database of child speech has prompted researchers to use more indirect methods of training ASR systems for children's speech. One common method is domain adaptation. Domain adaptation is the process of utilizing data from a given source distribution in training a machine learning algorithm to perform inference with a different target distribution (Ben‐David et al., [<reflink idref="bib8" id="ref86">8</reflink>]). In children's ASR, a common approach is to use a combination of adults' data and children's data in training a neural network. To train a neural network to deal with variability in children's speech, researchers pretrain the network on adults' speech and then fine‐tuned the network with a smaller amount of children's speech to teach it child speech patterns (Fan et al., [<reflink idref="bib20" id="ref87">20</reflink>]). Data augmentation methods, or methods that generate artificial training data from real samples, can be applied to add more generality to the training set (Ko et al., [<reflink idref="bib40" id="ref88">40</reflink>]).</p> <hd id="AN0192629999-12">Challenges in ASR for Children with Non‐GAE Dialects</hd> <p>The problems with children's ASR compound for children whose English differs from GAE. Current state‐of‐the‐art ASR systems are known to perform poorly for adult speech when the training domain is of one dialect and the testing domain is of another dialect (Livescu & Glass, [<reflink idref="bib52" id="ref89">52</reflink>]; Mirsamadi & Hansen, [<reflink idref="bib54" id="ref90">54</reflink>]; Wei et al., [<reflink idref="bib86" id="ref91">86</reflink>]). While recent advancements in ASR for cross‐dialect domain adaptation, particularly between dialects of native and nonnative speakers, have yielded improvements in ASR performance for these cases, the quality of the system is still degraded as compared to in‐domain tasks (Korzekwa et al., [<reflink idref="bib43" id="ref92">43</reflink>]). These results, along with the previously mentioned challenges inherent in children's ASR development, highlight the needs for more training data of target dialects and more robust domain adaptation techniques for ASR systems designed for assessment applications with racially and linguistically diverse groups of children.</p> <hd id="AN0192629999-13">ASR for AAE speakers</hd> <p>Despite the growing body of research in ASR for nonnative and accented speech, similar approaches for AAE speech remain understudied. Recent work by Koenecke et al. ([<reflink idref="bib41" id="ref93">41</reflink>]) demonstrates that the current commercially available ASR systems developed by Apple, IBM, Google, Amazon, and Microsoft all perform significantly worse for adult speakers of AAE than for GAE speakers. The authors show that, for all these systems, the acoustic model is not adequately trained to recognize phonological characteristics of AAE (i.e., differences in pronunciation and rhythm), and the language model is not able to handle common morphological and syntactic patterns of AAE (i.e., differences in grammar and word usage). However, this knowledge has not been applied to ASR for adult or child speech. This is despite the continued need for more inclusive technology, and this need is particularly pressing with educational technology (e.g., Kazimzade et al., [<reflink idref="bib38" id="ref94">38</reflink>]). Considerations for AAE must be integrated into ASR systems before speech‐based technology can be fairly and effectively implemented in classrooms.</p> <hd id="AN0192629999-14">Data</hd> <p>The speech sample data were sourced from the Atlanta, Georgia Corpus. Two other publicly available databases, the Weebit Corpus and VIST Human Evaluation Data (VHED), were used as well. This was necessary due to the limited availability of data from children attempting oral narrative tasks. To address this gap, we selected datasets with reading and difficulty levels similar to those of the target demographic, ensuring the data adequately represented the task demands and enhanced the robustness of the proposed systems.</p> <hd id="AN0192629999-15">Atlanta, Georgia Corpus</hd> <p>The current study uses audio recordings of 3rd‐4th and 7th‐8th grade students who were speakers of AAE or a mix of characteristics of GAE and Southern American English dialect spoken in the Atlanta, Georgia area (Fisher et al., [<reflink idref="bib22" id="ref95">22</reflink>]). Data for the current study comprise speech samples from the corpus of children's verbatim responses to two oral narrative tasks from the norm‐referenced <emph>Test of Narrative Language</emph> (TNL; Gillam & Pearson, [<reflink idref="bib26" id="ref96">26</reflink>]). The tasks were designed to measure students' story comprehension and production through a story retelling and a picture stimulus to generate a narrated description. The complete TNL can be used as an early screening tool for speech‐language impairments and language‐based learning disabilities, including reading. Students whose language samples are in the corpus were originally part of a study of a reading intervention and initially selected based on referral by teachers who identified children as exhibiting reading difficulties or as candidates for a control group with no impairment. These referrals were confirmed via one of three standardized reading measures (e.g., <emph>Test of Word Reading Efficiency</emph>—Second Edition, Torgeson et al., [<reflink idref="bib76" id="ref97">76</reflink>]).</p> <p>We utilized this corpus because it provides different oral language tasks that (<reflink idref="bib1" id="ref98">1</reflink>) are representative of actual tasks that learning specialists might use, (<reflink idref="bib2" id="ref99">2</reflink>) elicited different types of narration (the story retelling requires narrative comprehension and production skills and the picture description requires story generation skills) from the participants to test SLS performance across these conditions, and (<reflink idref="bib3" id="ref100">3</reflink>) were likely to have elicited AAE and other dialectal speech such as Southern American English through the naturalistic extended discourse that the narrative genre can afford. Due to the small sample size of 7th‐8th graders (<emph>n</emph> = 30), no comparative analysis with the larger 3rd‐4th grade cohort was conducted but these students' data were retained to maximize the number of speech samples required for the experiments.</p> <p>In the story‐retelling task, 184 students first heard a story read aloud to them by the test administrator. The students were then asked to retell the story that was recorded and later scored on their ability to use a set of predetermined keywords from the original story‐retelling rubric. The keywords contain story elements (e.g., character names, locations, times, action verbs, etc.) that must be retold in the same verb tense and order to receive credit in scoring the task (see also Israelsen‐Augenstein et al., [<reflink idref="bib33" id="ref101">33</reflink>]). Each child's raw score was an integer value between 0 and the total number of keywords correct.</p> <p>In the picture description task, 191 children were shown an image containing a character and several elements to describe. The children were prompted to generate a story about the image, making their story as complete as possible. The children's responses were recorded and later scored based on rubrics that allocate points for including key elements, such as creating and consistently using character names, creating logically sequenced events for the story, and using grammatically correct utterances. Each child's raw score was an integer value between 0 and the total number of points awarded for inclusion of elements.</p> <p>The assessments were individually administered and audio recorded at a child's school by a trained member of the original project staff according to the TNL testing manual's standard protocols. Audio was recorded in stereo at a sampling rate of 48 kHz. All recordings were resampled to mono audio with a sampling rate of 16kHz for experimentation in the current study. The average total length of the two task responses combined was about 5 minutes, resulting in approximately 16 total hours of speech. Although not necessary for the TNL administration or our subsequent SLS training procedures, the original project team had additionally transcribed each audio recording that the current study can use as ground truth transcriptions. The current study also utilizes scores for the recorded task performances that had been independently scored adhering to the testing manual's standard protocol "by a speech‐language pathologist and a second trained project staff member or graduate student. If disagreements on specific items arose, the two scorers discussed differences and came to consensus" (Fisher et al., [<reflink idref="bib22" id="ref102">22</reflink>], p. 1132). The scorers were speakers of Southern American English. While they did not speak AAE, they were familiar with the variety used in the Atlanta, Georgia area.</p> <p>The corpus also contained demographic metadata on students for (<reflink idref="bib1" id="ref103">1</reflink>) Type of Disability: This demographic characteristic is categorized as (i) Control (i.e., no disability/impairment), (ii) Reading Disability (RD) only (i.e., a student has a reading disability such as dyslexia that does not occur with or as a secondary effect of a primary language impairment or other condition (e.g., autism spectrum disorder), or (iii) Reading Disability with a Language Impairment (LI) (i.e., a reading disability that co‐occurs with a primary language impairment), (<reflink idref="bib2" id="ref104">2</reflink>) Reading Status: This demographic characteristic is categorized as either (i) Less‐Skilled reader (i.e., the student is evaluated to read at a level below their appropriate grade level), or (ii) Skilled reader (i.e., the student reads at or above their appropriate grade level), and (<reflink idref="bib3" id="ref105">3</reflink>) Dialect: This demographic characteristic is categorized as either (i) AAE, or (ii) Non‐AAE (i.e., mix of characteristics of GAE and Southern American English native to the Atlanta, Georgia area) identified according to procedures in Koenecke et al. ([<reflink idref="bib41" id="ref106">41</reflink>]).</p> <hd id="AN0192629999-16">WeeBit Corpus</hd> <p>The WeeBit Corpus contains short news article‐style texts used for children's reading comprehension tasks (Vajjala & Meurers, [<reflink idref="bib78" id="ref107">78</reflink>]). Each text is labeled on a difficulty scale of 1 and 5, with difficulty 1 meant for neurotypically developing children ages 7‐8 and difficulty 5 meant for neurotypically developing children ages 14‐16. By having the network simultaneously learn to both predict difficulty levels of children's texts and scores for narrative language proficiency on transcripts from the oral story‐retelling task in the Atlanta, Georgia Corpus, a multitask learning framework is created in which the system must learn to combine both knowledge of age‐typical reading texts (WeeBit) and knowledge of oral language abilities.</p> <p>We motivate this approach with research showing that children's reading abilities are correlated with their oral language proficiency. This is due, as Huettig and Pickering ([<reflink idref="bib31" id="ref108">31</reflink>]) argue, to shared underlying processes such as predictive dependencies and lexical and syntactic representations used in both speech and text comprehension. This is particularly true if the oral language domain selected shares literate language structures as narrative discourse does. For example, African American children's oral narrative skills at age four mediate the link between language and early literacy outcomes at age five (Gardner‐Neblett & Iruka, [<reflink idref="bib23" id="ref109">23</reflink>]). Therefore, the strategy is to combine oral language and texts to make the system predict assessment scores.</p> <hd id="AN0192629999-17">VHED Dataset</hd> <p>The VIST Human Evaluation Data (VHED) dataset (C.‐Y Hsu et al., [<reflink idref="bib29" id="ref110">29</reflink>]) is a composite of multiple audiovisual story‐telling datasets with a wide representation of story tasks. This dataset contains image captions corresponding to sequences of images from short stories. The captions are also labeled by human annotators with an average quality ranking of the overall story on a scale of 1 to 5. The intention in using the VHED dataset is to implicitly teach the model to score story quality in addition to being trained on in‐domain data from the picture description task used to generate stories in the Atlanta, Georgia Corpus.</p> <hd id="AN0192629999-18">Description of Experiments</hd> <p>This section describes the frameworks and experiments used for predicting children's scores of narrative skills from the storyretelling (Johnson et al., [<reflink idref="bib36" id="ref111">36</reflink>]) and picture description (Veeramani et al., [<reflink idref="bib79" id="ref112">79</reflink>]) speech samples in the Atlanta, Georgia Corpus. For Stage 1 of the experiments, different ASR systems were evaluated to select the system best suited for generating accurate transcriptions of the speech samples. At Stage 2, two distinct NLP approaches—one rubric‐based (i.e., String‐Search Rubric Matching‐based Scoring or SSRM) and one machine learning‐based (i.e., Neural Linguistic Feature‐based Scoring or NLF)—were applied to the ASR‐generated transcripts to predict scores for the two tasks. These stages reflect a systematic approach to optimizing transcription quality and scoring accuracy for both tasks.</p> <hd id="AN0192629999-19">Story‐Retelling Scoring Framework</hd> <p></p> <hd id="AN0192629999-20">String‐search rubric matching‐based scoring (SSRM)</hd> <p>A preliminary approach (shown in Figure 1a) involves fuzzy string matching (85% similarity) in ASR transcripts to identify close matches with the assigned points for keywords in the human scoring of the story‐retelling task. This approach does not rely on a machine learning model but instead uses a rule‐based method to compare text similarity and serves as a baseline approach by which to compare the later machine learning method. Points are awarded in the SSRM‐based scoring if a word in the transcript has a Levenshtein edit distance (Yujian & Bo, [<reflink idref="bib91" id="ref113">91</reflink>]) within 15% of the word length of a story telling task keyword needed for scoring the task. This process can be viewed as a form of entity matching, where the keywords serve as the ground truth elements required for scoring the task.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01mar26/jedm12435-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12435-fig-0001.jpg" title="1 Overview of (a) the SSRM approach and (b) the NLF approach to scoring the story‐retelling task. Source: Johnson et al. ([36]), used by permission of the International Speech and Communication Association." /> </p> <p></p> <hd id="AN0192629999-22">Neural linguistic feature‐based scoring (NLF)</hd> <p>The SSRM approach to automating scoring, as mentioned above, has a limitation as it only considers close matches to keywords in transcripts without considering their order of appearance, or whether they refer to specific characters in the story. To address this, a neural network‐based approach is adopted with a 45‐15‐40 "train‐val‐test" split that divided the Atlanta, Georgia data into three parts: one for training the model (train, 45%), one for tuning and validating the model's performance during development (val, 15%), and one for testing how well the model performs on unseen data (test, 40%). With this data, several methods used in readability assessment by B. W. Lee et al. ([<reflink idref="bib47" id="ref114">47</reflink>]) are employed. Several LLMs such as BERT (Devlin et al., [<reflink idref="bib16" id="ref115">16</reflink>]), RoBERTa (Liu et al., [<reflink idref="bib51" id="ref116">51</reflink>]), BART (Lewis et al., [<reflink idref="bib50" id="ref117">50</reflink>]), and XLNET (Yang et al., [<reflink idref="bib90" id="ref118">90</reflink>]) are then trained on a combination of the WeeBit Corpus and the story‐retelling task training data to generate soft labels that are generated by using the trained LLMs to predict probabilities for specific tasks based on input text, rather than assigning fixed, discrete labels.</p> <p>The optimal subset of 255 linguistic features for the WeeBit corpus from B. W. Lee et al. ([<reflink idref="bib47" id="ref119">47</reflink>]) is combined with additional features from their study's "Discourse," "Semantic," and "Traditional" categories. These features capture various measures of essay quality, such as the density of predicted entities in the text and lexical difficulty of words present. The handcrafted features are then concatenated with soft labels from the LLMs and input into an XGBoost (Chen et al., [<reflink idref="bib12" id="ref120">12</reflink>]) classifier which helps by leveraging both types of features to make more accurate predictions for the scoring task (as shown in Figure 1b). The accuracy of this method for transcripts generated from different ASR systems is presented in the Results and Discussion section.</p> <hd id="AN0192629999-23">Exploring evaluation of system performance fairness</hd> <p>To address the importance of fairness, with regard to how well the system matches human‐labeled scores for the story‐retelling task, the model's performance is further analyzed across the three demographic categories: Reading Disability/Language Impairment, Reading Status, and AAE dialect or not. The breakdown of the system performance across these three different demographic contexts is reported in Table 3 of the Results and Discussion section.</p> <hd id="AN0192629999-24">Picture Description Scoring Framework</hd> <p>As shown in Figure 2, a LLM was trained to take ASR transcripts and human scoring of the 191 speech samples from the picture description task of the Atlanta, Georgia Corpus and to then automatically score the task, similar to the approach taken for the story‐retelling speech samples. Similar preprocessing steps to the story‐retelling task were conducted, with the sample recordings first downsampled to 16kHz to reduce the data size, which simplifies processing while retaining sufficient audio quality for analysis, and then transcribed using two different ASR systems (Whisper and HuBERT). These transcripts are then fed into a LLM to generate soft labels, which can then be used to classify a student's score.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01mar26/jedm12435-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12435-fig-0002.jpg" title="2 The framework of a LLM (e.g., BERT) trained to predict corresponding scores for an ASR transcript of spoken responses to the picture description task. Note: Example word errors are shown in the ASR transcript excerpt. Source: Adapted from Veeramani et al. ([79]), used by permission of the International Speech and Communication Association.(Picture plate for illustration purposes only. Source: unlicensed stock image, Google)." /> </p> <p></p> <p>To measure performance degradation due to ASR error, the classification from the ground truth is also performed to quantify the deviation in performance from an ideal transcription (i.e., human annotated transcription with 100% word accuracy, to quantify the deviation in performance). Each student's raw assessment score is discretized into one of five labels corresponding to a continuous grade of 0%‐20%, 20%‐40%, 40%‐60%, 60%‐80%, or 80%‐100% (with the data distribution for the individual labels to be 7%, 19%, 38%, 32%, and 4%, respectively) before training the LLM to predict the student's score range from the transcription text. Story coherence and grammatical correctness are among the primary considerations in human scoring of the narrative tasks. We therefore conducted experiments with BERT (Devlin et al., [<reflink idref="bib16" id="ref121">16</reflink>]), ALBERT (Lan et al., [<reflink idref="bib45" id="ref122">45</reflink>]), DistilBERT (Sanh et al., [<reflink idref="bib69" id="ref123">69</reflink>]), and XLNET (Yang et al., [<reflink idref="bib90" id="ref124">90</reflink>]), as these models have been proven to be beneficial in various text classification tasks. Seventy‐five percent of the picture description speech samples were randomly assigned for the training of these systems, with the remaining 25% used for testing.</p> <p>In addition to the picture description speech samples, the system is also jointly trained with text from the VHED dataset (C.‐Y. Hsu et al., [<reflink idref="bib29" id="ref125">29</reflink>]) to augment the size of the training corpus. Eighty percent of the VHED dataset is used in training the language model to simultaneously predict scores of this narrative task and average quality rankings of the VHED text samples, and the performance is reported in Table 4 in the next section.</p> <hd id="AN0192629999-26">Results and Discussion</hd> <p></p> <hd id="AN0192629999-27">Performance of SSRM and NLF Frameworks for Story Retelling</hd> <p>Table 1 shows the Word Error Rates (WER) of four different ASR systems used for first generating transcripts in comparison to the ground truth transcripts on the story‐retelling task. WER is operationalized as the sum of the substitutions, deletions, and insertions that are necessary to align the ASR transcription with the ground truth transcript, divided by the total number of words in the ground truth transcript.</p> <p>1 Table ASR Word Error Rate (WER) for the SSRM Approach for Transcripts Generated by Each System for the Story‐Retelling Task</p> <p> <ephtml> <table><thead><tr><th>Transcripts</th><th>WER</th></tr></thead><tbody><tr><td>Wav2Vec2</td><td>37.0%</td></tr><tr><td>HuBert</td><td>46.7%</td></tr><tr><td>HuBert Large</td><td>43.9%</td></tr><tr><td>Whisper</td><td>26.8%</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Source</emph>: Adapted from Johnson et al. ([<reflink idref="bib36" id="ref126">36</reflink>]), used by permission of the International Speech and Communication Association.</p> <p>We observe that ASR transcripts generated from Whisper appear to have the lowest WER, when compared to the ground truth transcripts. When applying the initial SSRM framework, we observe that the Whisper transcripts have an 86.4% accuracy in correctly detecting and transcribing the test keywords needed for scoring the task.</p> <p>Evidence for fairness and validity at Stage 1 focuses on ensuring accurate transcription of children's oral narratives by evaluating the WER across demographic groups to assess transcription accuracy. Fairness is supported by analyzing whether transcription errors disproportionately affect construct‐relevant elements, such as key narrative details, while validity is demonstrated by ensuring that the transcriptions accurately capture the linguistic features necessary for downstream scoring. An evaluation of the baseline approach using an entity‐matching method across demographics shows that raw transcriptions deviate from human annotators at this stage. While Whisper achieves the lowest WER, we note that there is no universally accepted benchmark for what constitutes "good enough" agreement with human transcribers, particularly for AAE‐speaking children.</p> <p>Table 2 shows the performance, in terms of classification accuracy, of the NLF method while using only the LLM with a final classification layer (no linguistic features) after being trained on the WeeBit corpus and training set of either the ground truth or the four different ASR‐generated transcriptions of the student responses to the story‐retelling task. The final classification layer combines inputs (e.g., LLM soft labels) to predict task scores. The exclusion of linguistic features underscores reliance on learned representations from the LLM, contrasting with approaches integrating handcrafted features.</p> <p>2 Table The Classification Accuracy of Each LLM (without Linguistic Features) Considered When Predicting Scores Using Ground Truth and Four ASR‐Generated Transcripts for the Story‐Retelling Task</p> <p> <ephtml> <table><thead><tr><th>Transcripts</th><th>BERT</th><th>ROBERTA</th><th>BART</th><th>XLNET</th></tr></thead><tbody><tr><td>Ground Truth</td><td>96%</td><td>91%</td><td>80%</td><td>84%</td></tr><tr><td>Wav2Vec2</td><td>85%</td><td>87%</td><td>70%</td><td>85%</td></tr><tr><td>HuBert</td><td>89%</td><td>83%</td><td>71%</td><td>80%</td></tr><tr><td>HuBert Large</td><td>95%</td><td>86%</td><td>75%</td><td>80%</td></tr><tr><td>Whisper</td><td>96%</td><td>90%</td><td>78%</td><td>82%</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Source</emph>: Adapted from Johnson et al. ([<reflink idref="bib36" id="ref127">36</reflink>]), used by permission of the International Speech and Communication Association.</p> <p>Table 2 shows that BERT performs equally well with either the Whisper or HuBert Large ASR transcriptions of the story‐retelling data. We found that there is a higher semantic similarity of the Whisper transcripts (compared to the other ASR systems) with the ground truth transcripts (i.e., if a human can denote a child's speech then so can Whisper), which meant these transcripts were used for further analyses across different demographic categories.</p> <p>The student speech samples are divided on the basis of the three different demographic classifications, and the performance of the best system for each group is shown in Table 3. We break down the performance of the Whisper ASR system, including WER and classification accuracy, on both the ground truth transcripts and Whisper ASR transcripts. This analysis is conducted across three demographic characteristic groupings for the students in the Atlanta, Georgia Corpus, namely the Type of Disability, Reading Status, and Dialect categories. We assess variations in system performance within these different categorizations to evaluate the fairness of the framework for applications to speech samples with diverse students.</p> <p>3 Table Classification Accuracy for SSRM Scoring and NLF Scoring Approaches across Student Demographics for Ground Truth and Whisper Transcripts for the Story‐Retelling Task</p> <p> <ephtml> <table><thead><tr><th /><th /><th>String‐Search Rubric Matching (SSRM) Approach</th><th>Neural Linguistic Feature (NLF) Approach</th></tr><tr valign="bottom"><th>Demographic Group</th><th align="center">WER</th><th align="center"># of students</th><th align="center">GT Acc.</th><th align="center">Whisper Acc.</th><th align="center">GT Acc.</th><th align="center">Whisper Acc.</th></tr></thead><tbody><tr><td align="center">Type of Disability: Reading Disability/Language Impairment</td></tr><tr><td>Control</td><td align="center">21.1%</td><td align="center">32</td><td align="center">87.5%</td><td align="center">87.5%</td><td align="center">99.9%</td><td align="center">99.9%</td></tr><tr><td>RD only</td><td align="center">26.8%</td><td align="center">60</td><td align="center">88.3%</td><td align="center">86.6%</td><td align="center">87.5%</td><td align="center">87.5%</td></tr><tr><td>RD+LI</td><td align="center">34.5%</td><td align="center">27</td><td align="center">88.8%</td><td align="center">85.0%</td><td align="center">87.8%</td><td align="center">85.0%</td></tr><tr><td align="center">Reading Status</td></tr><tr><td>Less‐Skilled</td><td align="center">27.0%</td><td align="center">142</td><td align="center">88.0%</td><td align="center">85.9%</td><td align="center">95.0%</td><td align="center">89.5%</td></tr><tr><td>Skilled</td><td align="center">21.1%</td><td align="center">32</td><td align="center">87.5%</td><td align="center">87.5%</td><td align="center">97.5%</td><td align="center">96.9%</td></tr><tr><td align="center">Dialect</td></tr><tr><td>AAE</td><td align="center">26.8%</td><td align="center">116</td><td align="center">87.9%</td><td align="center">84.4%</td><td align="center">94.0%</td><td align="center">94.1%</td></tr><tr><td>Non‐AAE</td><td align="center">25.4%</td><td align="center">68</td><td align="center">88.2%</td><td align="center">89.7%</td><td align="center">99.2%</td><td align="center">99.2%</td></tr></tbody></table> </ephtml> </p> <ulist> <item>3 <emph>Note</emph>: The number of students in the Type of Disability and Reading Status categories (<emph>n</emph> = 119 and <emph>n</emph> = 174, respectively) excludes children with ADHD and other disorders who could not be readily assessed, or who could not be placed into either the Less‐Skilled or Skilled Reading Status categories in the Fisher et al. ([<reflink idref="bib22" id="ref128">22</reflink>]) study.</item> <item>4 Abbreviations: WER, Word Error Rate; GT Acc., Ground Truth Accuracy; Whisper Acc., Whisper Accuracy; RD, Reading Disability; LI, Language Impairment.</item> <item>5 <emph>Source</emph>: Adapted from Johnson et al. ([<reflink idref="bib36" id="ref129">36</reflink>]), used by permission of the International Speech and Communication Association.</item> </ulist> <p>Table 3 shows that the NLF approach outperforms the SSRM approach, showcasing the superiority of machine learning over the rubric‐based baseline. The combined system (BERT soft labels + handcrafted linguistic features + XGBoost) achieves a high 98.5% classification accuracy on the Whisper ASR transcripts averaged across all demographics. In contrast, the rubric‐based approach reaches only about 86.4% accuracy, with marginal improvement even using ground truth transcripts. The performance difference highlights that the system captures grammar and logic rules, not assessed by a simple fuzzy string search employed in the rubric‐based method. The results for the different demographic categories in Table 3 imply that the method performs fairly across Types of Disability, Reading Status, and Dialect.</p> <p>The control group mentioned in Table 3 specifically refers to students without a disability. This distinction is important because, in the analysis of Reading Status and Dialect categories, all students are included regardless of disability. This partially explains the elevated WER observed for these demographic groupings compared to the control group.</p> <p>In the Type of Disability demographic category, the NLF approach consistently equals or surpasses the SSRM approach across all metrics. Nevertheless, the rubric‐based approach demonstrates a higher level of fairness across the groups in this category, as evidenced by ASR transcript scores that deviate by less than 3% absolute value from control students (i.e., no disability) to those in the Reading Disability and Language Impairment group. While the NLF method achieves nearly perfect accuracy for the control case, its performance declines for students in the Reading Disability Only and the Reading Disability and Language Impairment groups who may exhibit nonstandard language errors not present in the WeeBit corpus. The intricate nature and differing severities of these language disorders can contribute to significant variability in the narrative language abilities of students (e.g., Wellman et al, [<reflink idref="bib85" id="ref130">85</reflink>]). This indicates a necessity for additional data to model children's language in the presence of reading disabilities and language impairments to enhance performance more equitably across these demographics.</p> <p>A similar trend is observed in the Reading Status demographic category. However, the NLF approach performs better than the SSRM approach across both Less‐Skilled and Skilled Reading Status groups. The almost 6% drop in performance of this system from the ground truth transcripts to the Whisper transcripts for the Less‐Skilled Reading Status group may mean that further ASR improvements are needed for the speech differences that these children display.</p> <p>Relatively unbiased performance is observed across AAE and non‐AAE speech for the system. Despite the relatively high WER for the children's speech samples, the application of additional language modeling for automatic assessment scoring from these transcripts proves advantageous. To strengthen the Interpretation/Use Argument, we can explicitly relate WER and scoring accuracy to the assessment construct, which measures students' oral narrative skills, such as coherence and linguistic structure. Construct‐relevant details, like key narrative elements, are critical to scoring validity, and transcription errors affecting these details can compromise the assessment. Our entity recognition‐based approach (i.e., SSRM) quantifies this impact by identifying how transcription errors affect key construct‐relevant features. Despite these inaccuracies, language models help mitigate WER effects by leveraging context, supporting the validity argument and ensuring the system aligns with the constructs being measured.</p> <p>However, further research is needed to examine the relationship between AAE, WER of the ASR system used, and assessment scoring performance. Although Koenecke et al. ([<reflink idref="bib41" id="ref131">41</reflink>]) highlight that numerous commercially available ASR systems exhibit inferior performance for U.S. East Coast AAE speakers compared to California GAE, limited efforts have been made to assess ASR performance for AAE in comparison to other American English varieties, such as Georgia's Southern American English. The elevated WERs observed for both Georgia dialects, as indicated in Table 3 (AAE and non‐AAE), underscore the necessity for additional research aimed at comprehending and enhancing ASR capabilities for various forms of children's regional varieties of spoken English.</p> <p>At Stage 2, fairness and validity are assessed by examining the system's ability to generate scores that align with human‐labeled ground truth scores. Fairness is demonstrated by verifying that the scoring criteria are applied equitably across demographic groups and do not introduce bias. Validity is established by ensuring that the scoring reflects construct‐relevant features of the oral narratives, such as story coherence, grammar, and vocabulary, while disregarding construct‐irrelevant features like minor pronunciation errors or disfluencies. An evaluation of the NLF scoring approach indicates that disparities that arise from entity matching‐based scoring are minimized across demographics.</p> <hd id="AN0192629999-28">Performance of Framework for Picture Description</hd> <p>For the picture description task, the classification accuracy of different LLMs on two ASR transcripts (Whisper or HuBert, found most accurate in the story‐retelling task) are shown in Table 4. It can be seen that the choice of LLM, as well as the quality of the ASR transcripts play a role in system performance for picture descriptions. We observe that similar to the system designed for story retelling, the ASR transcripts generated by Whisper have a low WER, and thus we feed these into the LLM for soft‐label generation. We also note that BERT achieves high accuracy in predicting the overall scores in the picture description task, outperforming other models such as XLNet, ALBERT and DistilBERT.</p> <p>4 Table The Classification Accuracy of Each LLM Considered When Predicting Scores Using Ground Truth and Two ASR‐Generated Transcripts for the Picture Description Task</p> <p> <ephtml> <table><thead><tr><th>Transcripts</th><th>WER</th><th>BERT</th><th>ALBERT</th><th>DistilBERT</th><th>XLNET</th></tr></thead><tbody><tr><td>Ground Truth</td><td>‐</td><td>98.0</td><td>95.5</td><td>84.0</td><td>92.0</td></tr><tr><td>HuBERT Large</td><td>33.5</td><td>96.0</td><td>87.5</td><td>83.0</td><td>91.0</td></tr><tr><td>Whisper</td><td>22.4</td><td>96.5</td><td>95.5</td><td>84.0</td><td>91.3</td></tr></tbody></table> </ephtml> </p> <ulist> <item>6 <emph>Note</emph>: WER compares the system's hypothesized transcription to the ground truth reference. The WER column is absent for the ground truth as it serves as the reference.</item> <item>7 <emph>Source</emph>: Adapted from Veeramani et al. ([<reflink idref="bib79" id="ref132">79</reflink>]), used by permission of the International Speech and Communication Association.</item> </ulist> <hd id="AN0192629999-29">Limitations</hd> <p>We acknowledge several limitations of these experiments. First, because all the students whose speech samples comprise the Atlanta, Georgia Corpus were from a single school district, we only experiment on a limited range of dialects and factors that impact children's linguistic development rather than additional key factors (e.g., socioeconomic status, amount of exposure to dialect and other languages, etc.). Second, an additional source of error beyond that of a lack of AAE training data, is the potential for the task scorers to be non‐AAE speakers. These scorers may be less familiar with the linguistic features of AAE than AAE speakers thus impacting the accuracy of the human‐scoring of the tasks later used as ground truth scores in the experiments. Third, our analysis is restricted to responses from a specific set of story retellings and picture descriptions elicited in a standardized assessment context. Additional work is needed to test how the framework generalizes to a wider range of prompts and oral narrative elicitation contexts, as well as different spoken language tasks and assessments that students encounter in language and literacy instruction, such as expository discourse tasks and oral reading tests.</p> <hd id="AN0192629999-30">Conclusions and Future Work</hd> <p>This article summarizes progress toward combating algorithmic bias in automatic SLS for language assessment. Given a relatively small amount of speech training data from target populations, the system learns to automatically score oral responses well across the students' dialects and reading status, and in the presence of a language impairment and/or reading disability in several cases. Overall, the approach provides a promising framework for equitably assessing students' oral narrative abilities. The favorable results across different demographics and two tasks that elicit narrative comprehension and production skills point to the system being able to give increased performance and specificity (e.g., scoring students in 10 score levels instead of 5 or outputting reasons for why score points were deducted) as more training data become available.</p> <p>In the evaluation of story retelling and picture description, the neural network‐based approach (i.e., NLF Scoring) achieves high accuracy (ranging from 85% to 99.9%) across all students, even when specifically evaluated on students in a marginalized demographic, considering factors such as AAE use, reading disability, reading disability with language impairment, and lower reading ability status. Returning to the Interpretation/Use Argument approach to validity, we can consider if this provides sufficient agreement to be confident that the automated scoring is meaningful and useful to prospective teachers. There are currently no minimum thresholds to evaluate the strength of the models for use as feedback to improve the SLS or to report to users as confidence measures. However, we see this as an important next step before any operational version that should include classroom teachers trying out the SLS to first obtain their user experiences. Such work can include teacher feedback on how consistently the system assesses different student groups relative to their own independent assessments and knowledge of student performances in the classroom. Teachers as users with experience working with AAE‐speaking students and students with language and reading difficulties should guide evaluations of the system in terms of its acceptable accuracy and appropriate use (Johnson et al., [<reflink idref="bib35" id="ref133">35</reflink>]). Reported issues and patterns of deviance can then be used to improve on accuracy rates. Although we interpret the SLS as working as well as human scoring, this is within a constrained domain. We would caution that student scores obtained with these automated systems, even when made operational, be used as just one of many different inputs to inform classroom instruction.</p> <p>At a time when educational assessment is being scrutinized by families and educators alike as a result of the social, racial and/or educational inequities faced by African American students and other underrepresented groups of students, many of which were exacerbated by the unequal educational impacts of the COVID‐19 pandemic (e.g., Bailey et al, [<reflink idref="bib5" id="ref134">5</reflink>]), researchers working across engineering and education disciplines can adopt a justice‐oriented framework (e.g., Randall et al., [<reflink idref="bib65" id="ref135">65</reflink>]) to help guide automated assessment design and validity and fairness research efforts at every stage of the development process. SLS may even serve to more systematically include previously underrepresented groups of students if sufficient data is generated through well‐defined speech sampling. In addition, the process should include scorers from underrepresented groups such as AAE speakers, wherever possible, in assigning human‐labeled ground truth scores.</p> <p>Future steps can include expanding this approach to other types of language and oral reading assessments and improving the investigated ASR systems for better recognition and inclusion of diverse children's speech (Johnson et al., [<reflink idref="bib36" id="ref136">36</reflink>]). Of particular interest is enhancing the performance of these systems in the presence of disfluencies or additional dialectal speech to strengthen the effectiveness of the frameworks among a wider demographic. Additional avenues of approach include creating detailed systems trained to predict characteristics of the student response individually (Johnson, [<reflink idref="bib34" id="ref137">34</reflink>]) and forecast individual item‐response pairs that can influence a student's score. To accomplish this, we call on the education and technology communities to invest in databases that can support this research and application with protected classes of children from underrepresented racial/ethnic groups and children with disabilities. All these efforts will be consequential for continued fair and valid interpretations of automated assessment such that teachers of racially and linguistically diverse students can interpret results with greater confidence that the assessment tools they are using to understand language and literacy‐related skills take into account their students' language‐related backgrounds.</p> <hd id="AN0192629999-31">Acknowledgments</hd> <p>This research was supported in part by the National Science Foundation under Award Number 2202585 to A. Alwan (PI) and A.L. Bailey (Co‐PI). We thank Dr. Robin Morris for access to the Atlanta, Georgia Corpus and for feedback on aspects of this article. Collection of the Corpus data was supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health under Award Number P01HD070837 to R. Morris. The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the National Science Foundation or the National Institutes of Health.</p> <p>Portions of the background and technical sections are based on the second author's doctoral dissertation. The empirical findings reported here were presented at Interspeech, 2023 and the Ninth Workshop on Speech and Language Technology in Education (SLaTE), 2023.</p> <p>We thank the Special Issue editors, two anonymous reviewers, and Nick Ziolkowski for their insightful comments and helpful feedback.</p> <hd id="AN0192629999-32">Appendix</hd> <p></p> <hd id="AN0192629999-33">Glossary of Key Terms and List of Acronyms</hd> <p>AAE: African American English.</p> <p>Automatic Speech Recognition (ASR): Machine‐learning technology used to render speech into text.</p> <p>Fine‐tuning: Adjusting a pretrained model on specific task data to improve its performance for that task.</p> <p>Downsampling: Reducing the size of audio data to make it more manageable for processing.</p> <p>GAE: General American English.</p> <p>In‐domain data: Data that align with the specific characteristics of the task for which a system is being developed or tested.</p> <p>Levenshtein Distance: A metric that measures the number of edits (insertions, deletions, or substitutions) required to transform one word or phrase into another.</p> <p>Large Language Model (LLM): A type of machine learning model trained on vast amounts of text data to understand and generate human‐like language.</p> <p>Natural Language Processing (NLP): Machine‐learning technology used to interpret language.</p> <p>NLF Scoring: Neural Linguistic Feature‐based Scoring.</p> <p>NLU: Natural Language Understanding.</p> <p>Soft Labels: Probabilistic outputs from a model that indicate the likelihood of different possible answers rather than a single fixed label.</p> <p>Spoken Language Systems (SLS): Computer system comprising both ASR and NLP to understand and respond to user input.</p> <p>SSRM Scoring: String‐Search Rubric Matching‐based Scoring.</p> <p>UDA: Universal Design for Assessment.</p> <p>WER: Word Error Rate.</p> <ref id="AN0192629999-34"> <title> References </title> <blist> <bibl id="bib1" idref="ref14" type="bt">1</bibl> <bibtext> American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref16" type="bt">2</bibl> <bibtext> Bachman, L.F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1–34.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref56" type="bt">3</bibl> <bibtext> Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). Wav2vec 2.0: A framework for self‐supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, & H. Lin (Eds.), Advances in neural information processing systems (pp. 12449–12460, Vol. 33). Curran Associates, Inc.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref2" type="bt">4</bibl> <bibtext> Bailey, A.L., Blackstock‐Bernstein, A., Ryan, E., & Pitsoulakis, D. (2016). Data mining with natural language processing and corpus linguistics: Unlocking access to school children's language in diverse contexts to improve instructional and assessment practices. In S. El Atia, O. Zaïane & D. Ipperciel. (Eds.) Data mining and learning analytics in educational research. (pp. 255–275). Wiley‐Blackwell.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref134" type="bt">5</bibl> <bibtext> Bailey, A. L., Martínez, J. F., Oranje, A., & Faulkner‐Bond, M. (2022). Introduction to twin pandemics: How a global health crisis and persistent racial injustices are impacting educational assessment. Educational Assessment, 27(2), 93–97.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref68" type="bt">6</bibl> <bibtext> Baumann, I., Wagner, D., Bayerl, S., & Bocklet, T. (2022). Non‐words pronunciation classification in language development tests for preschool children. In Proceedings of Interspeech 2022. International Speech and Communication Association.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref17" type="bt">7</bibl> <bibtext> Bejar, I.I. (2011) A validity‐based approach to quality control and assurance of automated scoring. Assessment in Education: Principles, Policy & Practice, 18(3), 319–341.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref86" type="bt">8</bibl> <bibtext> Ben‐David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Vaughan, J. W. (2010). A theory of learning from different domains. Machine Learning, 79, 151–175.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref53" type="bt">9</bibl> <bibtext> Brown, M. C., Sibley, D. E., Washington, J. A., Rogers, T. T., Edwards, J. R., MacDonald, M. C., & Seidenberg, M. S. (2015). Impact of dialect use on a basic component of learning to read. Frontiers in Psychology, 6, 1–17.</bibtext> </blist> <blist> <bibtext> Bryant, J.Heitz, C.Sanghvi, S. & Wagle, D. (2020). How Artificial Intelligence will Impact K‐12 Teachers. McKinsey & Company: New York. Retrieved from https://<ulink href="http://www.datalogicsindia.com/static/media/How‐artificial‐intelligence‐will‐impact‐K‐12‐teachers.c768da79c41e9893a67b.pdf">www.datalogicsindia.com/static/media/How‐artificial‐intelligence‐will‐impact‐K‐12‐teachers.c768da79c41e9893a67b.pdf</ulink></bibtext> </blist> <blist> <bibtext> Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple.... Language Testing, 29(1), 19–27.</bibtext> </blist> <blist> <bibtext> Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, I., Zhou, T., et al. (2015). Xgboost: Extreme gradient boosting. R package version 0.4‐2, 1(4), 1–4.</bibtext> </blist> <blist> <bibtext> Craig, H. K., Zhang, L., Hensel, S. L., & Quinn, E. J. (2009). African American English– speaking students: An examination of the relationship between dialect shifting and reading outcomes. Journal of Speech, Language, and Hearing Research, 52, 839–855.</bibtext> </blist> <blist> <bibtext> Crystal, D. (2005). The stories of English. Abrams.</bibtext> </blist> <blist> <bibtext> Das, S., Nix, D., & Picheny, M. (1998). Improvements in children's speech recognition performance. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, (Cat. No. 98CH36181) (Vol. 1, pp. 433–436). IEEE (formerly Institute of Electrical and Electronics Engineers).</bibtext> </blist> <blist> <bibtext> Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: pre‐training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, Minneapolis, MN, June 2–7, 2019 (Vol. 1, pp. 4171–4186). Association for Computational Linguistics. Doi.org/10.18653/v1/n19‐1423</bibtext> </blist> <blist> <bibtext> Downer, J. T., Goble, P., Myers, S. S., & Pianta, R. C. (2016). Teacher‐child racial/ethnic match within pre‐kindergarten classrooms and children's early school adjustment. Early Childhood Research Quarterly, 37, 26–38.</bibtext> </blist> <blist> <bibtext> Egalite, A. J., Kisida, B., & Winters, M. A. (2015). Representation in the classroom: The effect of own‐race teachers on student achievement. Economics of Education Review, 45, 44–52.</bibtext> </blist> <blist> <bibtext> Evans, C. & Taylor, C. (Eds.) (2025). Culturally responsive assessment in classrooms and large‐scale contexts: Theory, research, and practice. Routledge.</bibtext> </blist> <blist> <bibtext> Fan, R., Afshan, A., & Alwan, A. (2021). BI‐APC: Bidirectional autoregressive predictive coding for unsupervised pre‐training and its application to children's ASR. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7023–7027). IEEE (formerly Institute of Electrical and Electronics Engineers).</bibtext> </blist> <blist> <bibtext> Fan, R., Zhu, Y., Wang, J., & Alwan, A. (2022). Towards better domain adaptation for self‐supervised models: A case study of child ASR. IEEE Journal of Selected Topics in Signal Processing, 16(6), 1242–1252.</bibtext> </blist> <blist> <bibtext> Fisher, E. L., Barton‐Hulsey, A., Walters, C., Sevcik, R. A., & Morris, R. (2019). Executive functioning and narrative language in children with dyslexia. American Journal of Speech‐Language Pathology, 28(3), 1127–1138.</bibtext> </blist> <blist> <bibtext> Gardner‐Neblett, N., & Iruka, I. U. (2015). Oral narrative skills: Explaining the language‐emergent literacy link by race/ethnicity and SES. Developmental Psychology, 51(7), 889–904.</bibtext> </blist> <blist> <bibtext> Nicole Gardner‐Neblett, Elizabeth P. Pungello, Iheoma U. Iruka (2012). Oral narrative skills: Implications for the reading development of African American children. Child Development Perspectives, 6(3), 218–224.</bibtext> </blist> <blist> <bibtext> Getman, Y., Al‐Ghezi, R., Voskoboinik, K., Grosz, T., Kurimo, M., Salvi, G., Svendsen, T.,´ & Strombergsson, S. (2022). wav2vec2‐based Speech Rating System for Children¨ with Speech Sound Disorder. In Proceedings of Interspeech 2022 (pp. 3618–3622). doi.org/10.21437/Interspeech.2022‐10103</bibtext> </blist> <blist> <bibtext> Gillam, R. B., & Pearson, N. A. (2004). Test of Narrative Language (TNL). Pro‐Ed.</bibtext> </blist> <blist> <bibtext> Google. (2023). Read along by Google. Retrieved from play.google.com/store/apps/details?id=com.google.android.apps.%20seekh&hl=enUS&gl=US</bibtext> </blist> <blist> <bibtext> Green, L. J. (2002). African American English: A linguistic introduction. Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Hsu, C.‐Y., et al. (2022). Learning to rank visual stories from human ranking data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers) (pp. 6365–6378). Association for Computational Linguistics.</bibtext> </blist> <blist> <bibtext> Hsu, W.‐N., Bolte, B., Tsai, Y.‐H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self‐supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech and Language Proceedings, 29, 3451–3460. https://doi.org/10.1109/TASLP.2021.3122291</bibtext> </blist> <blist> <bibtext> Huettig, F., & Pickering, M. J. (2019). Literacy advantages beyond reading: Prediction of spoken language. Trends in Cognitive Sciences, 23(6), 464–475.</bibtext> </blist> <blist> <bibtext> A. Corinne Huggins‐Manley, Brandon M. Booth, Sidney K. D'mello (2022). Toward argument‐based fairness with an application to AI‐enhanced educational assessments. Journal of Educational Measurement, 59(3), 362–388.</bibtext> </blist> <blist> <bibtext> Israelsen‐Augenstein, M., Fox, C., Gillam, S. L., Holbrook, S., & Gillam, R. (2022). Monitoring indicators of scholarly language: A progress monitoring tool for documenting changes in narrative complexity over time. Frontiers in Education, 7, 918127. https://doi.org/10.3389/feduc.2022.918127.</bibtext> </blist> <blist> <bibtext> Johnson, A. (2024). Towards inclusive low‐resource speech technologies: A case study of educational systems for African American English‐speaking children. Unpublished doctoral dissertation, University of California, Los Angeles. ProQuest Dissertations & Theses. Retrieved from https://<ulink href="http://www.proquest.com/dissertations‐theses/towards‐inclusive‐low‐resource‐speech/docview/2968778385/se‐2">www.proquest.com/dissertations‐theses/towards‐inclusive‐low‐resource‐speech/docview/2968778385/se‐2</ulink></bibtext> </blist> <blist> <bibtext> Johnson, A., Washington, J., Morris, R., Ostendorf, M., Bailey, A., & Alwan, A. (2022). Towards effective speech‐based AI in the classroom: The case of AAE‐speaking children. In Thirty‐sixth Conference on Neural Information Processing Systems (NeurIPs) Workshop, (NSF). Retrieved from https://par.nsf.gov/servlets/purl/10426921</bibtext> </blist> <blist> <bibtext> Johnson, A., Veeramani, H., Natarajan, B., & Alwan, A. (2023). An equitable framework for automatically assessing children's oral narrative language abilities. In Proceedings of Interspeech, 2023. International Speech and Communication Association. Doi: 10.21437/Interspeech.2023‐1257</bibtext> </blist> <blist> <bibtext> Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.</bibtext> </blist> <blist> <bibtext> Kazimzade, G., Patzer, Y., Pinkwart, N. (2019). Artificial Intelligence in education meets inclusive educational technology—The technical state‐of‐the‐art and possible directions. In J. Knox, Y. Wang, & M. Gallagher (Eds.). Artificial intelligence and inclusive education. Perspectives on rethinking and reforming education (pp. 61–73). Springer.</bibtext> </blist> <blist> <bibtext> Ke, X., Zeng, Y., & Luo, H. (2016). Autoscoring essays based on complex networks. Journal of Educational Measurement, 53(4), 478–497.</bibtext> </blist> <blist> <bibtext> Ko, T., Peddinti, V., Povey, D., & Khudanpur, S. (2015). Audio augmentation for speech recognition. In Sixteenth Annual Conference of the International Speech Communication Association.</bibtext> </blist> <blist> <bibtext> Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020). Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14), 7684–7689.</bibtext> </blist> <blist> <bibtext> Koenig, L. L., Lucero, J. C., & Perlman, E. (2008). Speech production variability in fricatives of children and adults: Results of functional data analysis. The Journal of the Acoustical Society of America, 124(5), 3158–3170.</bibtext> </blist> <blist> <bibtext> Korzekwa, D., Lorenzo‐Trueba, J., Zaporowski, S., Calamaro, S., Drugman, T., & Kostek, B. (2021). Mispronunciation detection in non‐native (l2) English with uncertainty modeling. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7738–7742).</bibtext> </blist> <blist> <bibtext> Kretzschmar, W. A. (2015). African American Voices in Atlanta. In S. Lanehart (Ed.). The Oxford handbook of African American language (pp. 219–235). Oxford University Press.</bibtext> </blist> <blist> <bibtext> Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). AL‐BERT: A lite BERT for self‐supervised learning of language representations. In Eighth International Conference on Learning Representations, 2020. The International Conference on Learning Representations. Openreview.net/forum?id=H1eA7AEtvS</bibtext> </blist> <blist> <bibtext> Lanehart, S., & Malik, A. M. (2015). Language use in African American communities. In S. Lanehart (Ed.). The Oxford handbook of African American language (pp. 1–19). Oxford University Press.</bibtext> </blist> <blist> <bibtext> Lee, B. W., Jang, Y. S., & Lee, J. (2021). Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 10669–10686). Association for Computational Linguistics. Doi: doi.org/10.18653/v1/2021.emnlp‐main.834</bibtext> </blist> <blist> <bibtext> Lee, S., Potamianos, A., & Narayanan, S. (1997). Analysis of children's speech: Duration, pitch and formants. In Fifth European Conference on Speech Communication and Technology. European Speech Communication Association.</bibtext> </blist> <blist> <bibtext> Lee, S., Potamianos, A., & Narayanan, S. (1999). Acoustics of children's speech: Developmental changes of temporal and spectral parameters. The Journal of the Acoustical Society of America, 105(3), 1455–1468.</bibtext> </blist> <blist> <bibtext> Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019). Bart: Denoising sequence‐to‐sequence pre‐training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics. ArXiv preprint ArXiv:1910.13461.</bibtext> </blist> <blist> <bibtext> Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. ArXiv preprint ArXiv:1907.11692.</bibtext> </blist> <blist> <bibtext> Livescu, K., & Glass, J. (2000). Lexical modeling of non‐native speech for automatic speech recognition. In 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100) (Vol. 3, pp. 1683–1686). IEEE (formerly Institute of Electrical and Electronics Engineers).</bibtext> </blist> <blist> <bibtext> McCabe, A., Bailey, A. L., & Melzi, G. (Eds.). (2008). Spanish‐language narration and literacy: Culture, cognition, and emotion. Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Mirsamadi, S., & Hansen, J. H. (2017). On multi‐domain training and adaptation of end‐to‐end RNN acoustic models for distant speech recognition. In Proceedings of Interspeech (pp. 404–408).</bibtext> </blist> <blist> <bibtext> Neumann, M.M, & Neumann, D.L. (2015). The use of touch screen tablets at home and preschool to foster emergent literacy. Journal of Early Childhood Literacy, 17(2), 203–220.</bibtext> </blist> <blist> <bibtext> Nguyen, T. D., Lam, C. B., & Bruno, P. (2022). Is there a national teacher shortage? A systematic examination of reports of teacher shortages in the United States. Annenberg Institute at Brown University. Retrieved from https://files.eric.ed.gov/fulltext/ED629162.pdf</bibtext> </blist> <blist> <bibtext> Oetting, J. B. (2015). Some similarities and differences between African American English and Southern White English in children. In S. Lanehart (Ed). The Oxford handbook of African American language (pp. 212–225). Oxford University Press.</bibtext> </blist> <blist> <bibtext> Oetting, J. B., & Pruitt, S. (2005). Southern African‐American English use across groups. Journal of Multilingual Communication Disorders, 3(2), 136–144.</bibtext> </blist> <blist> <bibtext> Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P., et al. (2011). The Kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding. IEEE (formerly Institute of Electrical and Electronics Engineers).</bibtext> </blist> <blist> <bibtext> Prinos, K., Patwari, N., & Power, C.A. (2024). Speaking of accent: A content analysis of accent misconceptions in ASR research. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 24). Association for Computing Machinery. https://doi.org/10.1145/3630106.3658969</bibtext> </blist> <blist> <bibtext> Cynthia Puranik, Lee Branum‐Martin, Julie A. Washington (2020). The relation between dialect density and the codevelopment of writing and reading in African American children. Child Development, 91(4), e866–e882.</bibtext> </blist> <blist> <bibtext> Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large‐scale weak supervision. ArXiv preprint ArXiv:2212.04356.</bibtext> </blist> <blist> <bibtext> Rajan, S. S., Udeshi, S., & Chattopadhyay, S. (2022, March). Aequevox: Automated fairness testing of speech recognition systems. In International conference on fundamental approaches to software engineering (pp. 245–267). Springer International Publishing.</bibtext> </blist> <blist> <bibtext> Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: A systematic literature review. Artificial Intelligence Review, 55(3), 2495–2527.</bibtext> </blist> <blist> <bibtext> Randall, J., Slomp, D., Poe, M., & Oliveri, M. E. (2022). Disrupting white supremacy in assessment: Toward a justice‐oriented, antiracist validity framework. Educational Assessment, 27(2), 170–178.</bibtext> </blist> <blist> <bibtext> Randall, J. (2023). It ain't near ʻbout fair: Re‐envisioning the bias and sensitivity review process from a justice‐oriented antiracist perspective. Educational Assessment, 28(2), 68–82.</bibtext> </blist> <blist> <bibtext> Redding, C. (2019). A teacher like me: A review of the effect of student–teacher racial/ethnic matching on teacher perceptions of students and student academic and behavioral outcomes. Review of Educational Research, 89(4), 499–535.</bibtext> </blist> <blist> <bibtext> Sancinetti, M., Vidal, J., Bonomi, C., & Ferrer, L. (2022). A transfer learning approach for pronunciation scoring. In 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6812–6816). IEEE (formerly Institute of Electrical and Electronics Engineers).</bibtext> </blist> <blist> <bibtext> Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). Distilbert, a distilled version of BERT: Smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. <ulink href="http://arxiv.org/abs/1910.01108">http://arxiv.org/abs/1910.01108</ulink></bibtext> </blist> <blist> <bibtext> Seymour, H. N. (2004). The challenge of language assessment for African American English speaking children: A historical perspective. Seminars in Speech and Language, 25(1), 3–12.</bibtext> </blist> <blist> <bibtext> Shibata, T., & Uto, M. (2022). Analytic automated essay scoring based on deep neural networks integrating multidimensional item response theory. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 2917–2926). International Committee on Computational Linguistics.</bibtext> </blist> <blist> <bibtext> Shobaki, K., Hosom, J.‐P., & Cole, R. (2007). Center for Spoken Language Understanding (CSLU): Kids' Speech Version 1.1. Philadelphia: Linguistic Data Consortium. https://doi.org/10.35111/q5tn‐8096</bibtext> </blist> <blist> <bibtext> Stockman, I. J., Newkirk‐Turner, B. L., Swartzlander, E., & Morris, L. R. (2016). Comparison of African American children's performances on a minimal competence core for morphosyntax and the Index of Productive Syntax. American Journal of Speech‐Language Pathology, 25(1), 80–96.</bibtext> </blist> <blist> <bibtext> Thomas, E. R. (2015). Prosodic features of African American English. In S. Lanehart (Ed.).The Oxford handbook of African American language (pp. 420–438). Oxford University Press.</bibtext> </blist> <blist> <bibtext> Thompson, S. J., Johnstone, C. J., & Thurlow, M. L. (2002). Universal design applied to large scale assessments (Synthesis Report 44). University of Minnesota, National Center on Educational Outcomes. <ulink href="http://education.umn.edu/NCEO/OnlinePubs/Synthesis44.html">http://education.umn.edu/NCEO/OnlinePubs/Synthesis44.html</ulink></bibtext> </blist> <blist> <bibtext> Torgeson, J., Wagner, R., & Carol, R. (2011). Test of word reading efficiency—Second edition (TOWRE–2). Pro‐Ed.</bibtext> </blist> <blist> <bibtext> Uto, M., Xie, Y., & Ueno, M. (2020). Neural automated essay scoring incorporating handcrafted features. In Proceedings of the 28th International Conference on Computational Linguistics (pp. 6077–6088). International Committee on Computational Linguistics. https://doi.org/10.18653/v1/2020.coling‐main.535</bibtext> </blist> <blist> <bibtext> Vajjala, S., & Meurers, D. (2012). On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP (pp. 163–173). Association for Computational Linguistics.</bibtext> </blist> <blist> <bibtext> Veeramani, H., Shankar, B.N., Johnson, A., & Alwan, A. (2023) Towards automatically assessing children's oral picture description tasks. In Proceedings of the Ninth Workshop on Speech and Language Technology in Education (SLaTE). Retrieved from https://www.isca‐archive.org/slate_2023/veeramani23_slate.pdf</bibtext> </blist> <blist> <bibtext> Ward, W., Cole, R., & Pradhan, S. (2019). My Science Tutor and the MyST corpus. BoulderLearn. Inc.</bibtext> </blist> <blist> <bibtext> Washington, J. A., Branum‐Martin, L., Sun, C., & Lee‐James, R. (2018). The impact of dialect density on the growth of language and reading in African American children. Language, Speech, and Hearing Services in Schools, 49(2), 232–247.</bibtext> </blist> <blist> <bibtext> Washington, J. A., & Craig, H. K. (1994). Dialectal forms during discourse of poor, urban, African American preschoolers. Journal of Speech, Language, and Hearing Research, 37(4), 816–823.</bibtext> </blist> <blist> <bibtext> Washington, J. A., & Iruka, I. U. (2025). Linguistic justice: Addressing linguistic variation of Black children in teaching and learning. Linguistics and Education, 85, 101382.</bibtext> </blist> <blist> <bibtext> Washington, J. A., & Seidenberg, M. S. (2021). Teaching reading to African American children: When home and school language differ. American Educator, 45(2), 26–33.</bibtext> </blist> <blist> <bibtext> Wellman, R. L., Lewis, B. A., Freebairn, L. A., Avrich, A. A., Hansen, A. J., & Stein, C. M. (2011). Narrative ability of children with speech sound disorders and the prediction of later literacy skills. Language, Speech, and Hearing Services in Schools, 42(4), 561–579.</bibtext> </blist> <blist> <bibtext> Wei, H., Pu, Y., & Yang, J. (2010). Non‐native speech recognition based on speaker adaptation. Sixth International Conference on Natural Computation, 4, 2024–2027.</bibtext> </blist> <blist> <bibtext> Wiliam, D. (2011). What is assessment for learning?Studies in Educational Evaluation, 37(1), 3–14.</bibtext> </blist> <blist> <bibtext> Wong, J. H. M., Zhang, H., & Chen, N. (2022). Variations of multi‐task learning for spoken language assessment. In Proceedings of Interspeech 2022 (pp. 4456–4460). International Speech and Communication Association. https://doi.org/10.21437/Interspeech.2022‐28</bibtext> </blist> <blist> <bibtext> Xi, X. (2010). How do we go about investigating test fairness?Language Testing, 27(2), 147–170.</bibtext> </blist> <blist> <bibtext> Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019). XLnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche‐Buc, E. Fox, & R. Garnett (Eds.). Advances in neural information processing systems (pp. 3384–3394). Curran Associates, Inc.</bibtext> </blist> <blist> <bibtext> Yujian, L., & Bo, L. (2007). A normalized Levenshtein distance metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6), 1091–1095.</bibtext> </blist> <blist> <bibtext> Zechner, K., Chen, L., Davis, L., Evanini, K., Lee, C. M., Leong, C. W., Wang, X., & Yoon, S.‐Y. (2015). Automated scoring of speaking tasks in the Test of English for Teaching (TEFT™). ETS Research Report Series, 2015(2), 1–17.</bibtext> </blist> </ref> <aug> <p>By Alison L. Bailey; Alexander Johnson; Natarajan Balaji Shankar; Hariram Veeramani; Julie A. Washington and Abeer Alwan</p> <p>Reported by Author; Author; Author; Author; Author; Author</p> <p></p> <p>Alison L. Bailey is Professor of Education at the University of California, Los Angeles, 405 Hilgard Ave. Moore Hall, Los Angeles, CA 90095; abailey@gseis.ucla.edu. Alison's research interests center on developing language learning progressions with multilingual learners and supporting teachers' language pedagogy and assessment practices.</p> <p>Alexander "AJ" Johnson is Adjunct Professor of Computer Science at New York University, 251 Mercer St, New York, NY 10012; ajohnson49@g.ucla.edu. AJ's research interests include speech recognition and natural language processing for low resource domains such as accented speech and African American English.</p> <p>Natarajan Balaji Shankar is a PhD student in the Department of Electrical and Computer Engineering, University of California, Los Angeles, 405 Hilgard Ave., Los Angeles, CA 90095; balaji1312@ucla.edu. Balaji's research focuses on speech processing, with an emphasis on low‐resource child speech.</p> <p>Hariram Veeramani is an affiliated researcher in the Department of Electrical and Computer Engineering, University of California, Los Angeles, 405 Hilgard Ave., Los Angeles, CA 90095; hariram@g.ucla.edu. Hari's research areas include Natural Language Understanding, multimodal learning and information retrieval.</p> <p>Julie A. Washington is Professor of Education at the University of California, Irvine, 2068 Education, Irvine, CA 92697; julie.washington@uci.edu. Julie researches the intersection of language, literacy, and poverty in African American children and examines cultural dialects, early literacy skills, language development, and disorders.</p> <p>Abeer Alwan is Distinguished Professor in the Department of Electrical and Computer Engineering, University of California, Los Angeles, 405 Hilgard Ave., 66‐147E EIV, Los Angeles, CA 90095; alwan@ee.ucla.edu. Abeer's research interests include low resource Automatic Speech Recognition, especially child speech.</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib27" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib92" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib68" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib39" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib77" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib63" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib55" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib14" firstref="ref10"></nolink> <nolink nlid="nl10" bibid="bib28" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib41" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib60" firstref="ref13"></nolink> <nolink nlid="nl13" bibid="bib37" firstref="ref15"></nolink> <nolink nlid="nl14" bibid="bib65" firstref="ref18"></nolink> <nolink nlid="nl15" bibid="bib66" firstref="ref19"></nolink> <nolink nlid="nl16" bibid="bib19" firstref="ref21"></nolink> <nolink nlid="nl17" bibid="bib32" firstref="ref22"></nolink> <nolink nlid="nl18" bibid="bib89" firstref="ref23"></nolink> <nolink nlid="nl19" bibid="bib75" firstref="ref28"></nolink> <nolink nlid="nl20" bibid="bib53" firstref="ref30"></nolink> <nolink nlid="nl21" bibid="bib24" firstref="ref31"></nolink> <nolink nlid="nl22" bibid="bib56" firstref="ref34"></nolink> <nolink nlid="nl23" bibid="bib67" firstref="ref35"></nolink> <nolink nlid="nl24" bibid="bib18" firstref="ref36"></nolink> <nolink nlid="nl25" bibid="bib83" firstref="ref37"></nolink> <nolink nlid="nl26" bibid="bib17" firstref="ref38"></nolink> <nolink nlid="nl27" bibid="bib81" firstref="ref39"></nolink> <nolink nlid="nl28" bibid="bib87" firstref="ref40"></nolink> <nolink nlid="nl29" bibid="bib46" firstref="ref41"></nolink> <nolink nlid="nl30" bibid="bib57" firstref="ref42"></nolink> <nolink nlid="nl31" bibid="bib34" firstref="ref43"></nolink> <nolink nlid="nl32" bibid="bib82" firstref="ref44"></nolink> <nolink nlid="nl33" bibid="bib58" firstref="ref45"></nolink> <nolink nlid="nl34" bibid="bib73" firstref="ref46"></nolink> <nolink nlid="nl35" bibid="bib74" firstref="ref47"></nolink> <nolink nlid="nl36" bibid="bib84" firstref="ref48"></nolink> <nolink nlid="nl37" bibid="bib44" firstref="ref49"></nolink> <nolink nlid="nl38" bibid="bib13" firstref="ref50"></nolink> <nolink nlid="nl39" bibid="bib61" firstref="ref51"></nolink> <nolink nlid="nl40" bibid="bib70" firstref="ref55"></nolink> <nolink nlid="nl41" bibid="bib30" firstref="ref57"></nolink> <nolink nlid="nl42" bibid="bib62" firstref="ref58"></nolink> <nolink nlid="nl43" bibid="bib21" firstref="ref60"></nolink> <nolink nlid="nl44" bibid="bib16" firstref="ref61"></nolink> <nolink nlid="nl45" bibid="bib69" firstref="ref62"></nolink> <nolink nlid="nl46" bibid="bib45" firstref="ref63"></nolink> <nolink nlid="nl47" bibid="bib90" firstref="ref64"></nolink> <nolink nlid="nl48" bibid="bib50" firstref="ref65"></nolink> <nolink nlid="nl49" bibid="bib51" firstref="ref66"></nolink> <nolink nlid="nl50" bibid="bib88" firstref="ref67"></nolink> <nolink nlid="nl51" bibid="bib59" firstref="ref70"></nolink> <nolink nlid="nl52" bibid="bib25" firstref="ref71"></nolink> <nolink nlid="nl53" bibid="bib64" firstref="ref72"></nolink> <nolink nlid="nl54" bibid="bib71" firstref="ref73"></nolink> <nolink nlid="nl55" bibid="bib47" firstref="ref74"></nolink> <nolink nlid="nl56" bibid="bib15" firstref="ref76"></nolink> <nolink nlid="nl57" bibid="bib42" firstref="ref78"></nolink> <nolink nlid="nl58" bibid="bib49" firstref="ref80"></nolink> <nolink nlid="nl59" bibid="bib48" firstref="ref82"></nolink> <nolink nlid="nl60" bibid="bib35" firstref="ref83"></nolink> <nolink nlid="nl61" bibid="bib72" firstref="ref84"></nolink> <nolink nlid="nl62" bibid="bib80" firstref="ref85"></nolink> <nolink nlid="nl63" bibid="bib20" firstref="ref87"></nolink> <nolink nlid="nl64" bibid="bib40" firstref="ref88"></nolink> <nolink nlid="nl65" bibid="bib52" firstref="ref89"></nolink> <nolink nlid="nl66" bibid="bib54" firstref="ref90"></nolink> <nolink nlid="nl67" bibid="bib86" firstref="ref91"></nolink> <nolink nlid="nl68" bibid="bib43" firstref="ref92"></nolink> <nolink nlid="nl69" bibid="bib38" firstref="ref94"></nolink> <nolink nlid="nl70" bibid="bib22" firstref="ref95"></nolink> <nolink nlid="nl71" bibid="bib26" firstref="ref96"></nolink> <nolink nlid="nl72" bibid="bib76" firstref="ref97"></nolink> <nolink nlid="nl73" bibid="bib33" firstref="ref101"></nolink> <nolink nlid="nl74" bibid="bib78" firstref="ref107"></nolink> <nolink nlid="nl75" bibid="bib31" firstref="ref108"></nolink> <nolink nlid="nl76" bibid="bib23" firstref="ref109"></nolink> <nolink nlid="nl77" bibid="bib29" firstref="ref110"></nolink> <nolink nlid="nl78" bibid="bib36" firstref="ref111"></nolink> <nolink nlid="nl79" bibid="bib79" firstref="ref112"></nolink> <nolink nlid="nl80" bibid="bib91" firstref="ref113"></nolink> <nolink nlid="nl81" bibid="bib12" firstref="ref120"></nolink> <nolink nlid="nl82" bibid="bib85" firstref="ref130"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1501375
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Addressing Bias in Spoken Language Systems Used in the Development and Implementation of Automated Child Language-Based Assessment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Alison+L%2E+Bailey%22">Alison L. Bailey</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9303-8304">0000-0001-9303-8304</externalLink>)<br /><searchLink fieldCode="AR" term="%22Alexander+Johnson%22">Alexander Johnson</searchLink><br /><searchLink fieldCode="AR" term="%22Natarajan+Balaji+Shankar%22">Natarajan Balaji Shankar</searchLink><br /><searchLink fieldCode="AR" term="%22Hariram+Veeramani%22">Hariram Veeramani</searchLink><br /><searchLink fieldCode="AR" term="%22Julie+A%2E+Washington%22">Julie A. Washington</searchLink><br /><searchLink fieldCode="AR" term="%22Abeer+Alwan%22">Abeer Alwan</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2026 63(1).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 30
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)<br />Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) (DHHS/NIH)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 2202585<br />P01HD070837
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Oral+Language%22">Oral Language</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Child+Language%22">Child Language</searchLink><br /><searchLink fieldCode="DE" term="%22Speech+Communication%22">Speech Communication</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Algorithms%22">Algorithms</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement%22">Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Variation%22">Language Variation</searchLink><br /><searchLink fieldCode="DE" term="%22Black+Dialects%22">Black Dialects</searchLink><br /><searchLink fieldCode="DE" term="%22Suprasegmentals%22">Suprasegmentals</searchLink><br /><searchLink fieldCode="DE" term="%22Pronunciation%22">Pronunciation</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Usage%22">Language Usage</searchLink><br /><searchLink fieldCode="DE" term="%22Grammar%22">Grammar</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Impairments%22">Language Impairments</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Difficulties%22">Reading Difficulties</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.12435
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This article addresses bias in Spoken Language Systems (SLS) that involve both Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) and reports experiments to improve the performance of SLS for automated language and literacy-related assessments with students who are under served in the U.S. educational system. We frame bias in SLS in terms of testing fairness and validity, stemming in part from the exclusion of sufficiently large training datasets in varieties of English other than General American English (GAE). We adopt an Interpretation/Use Argument approach to validity focused on clarity of constructs and scoring accuracy. While SLS use ASR to automatically transcribe students' utterances, and apply NLP algorithms to ASR transcripts to measure students' speech samples, it is well-documented in studies with adults that ASR is typically more problematic for African American English (AAE) speakers than for other groups due to differences in prosody, pronunciation, word usage, and grammar. We utilized child speech and text corpora to improve algorithms that score oral task responses for child AAE speakers and, in some experiments, children with oral language and reading difficulties. Favorable results provide impetus and possible solutions for fair and inclusive assessments for diverse student groups in the future.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1501375
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1501375
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.12435
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 30
    Subjects:
      – SubjectFull: Bias
        Type: general
      – SubjectFull: Oral Language
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Child Language
        Type: general
      – SubjectFull: Speech Communication
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Algorithms
        Type: general
      – SubjectFull: Measurement
        Type: general
      – SubjectFull: Language Variation
        Type: general
      – SubjectFull: Black Dialects
        Type: general
      – SubjectFull: Suprasegmentals
        Type: general
      – SubjectFull: Pronunciation
        Type: general
      – SubjectFull: Language Usage
        Type: general
      – SubjectFull: Grammar
        Type: general
      – SubjectFull: Language Impairments
        Type: general
      – SubjectFull: Reading Difficulties
        Type: general
    Titles:
      – TitleFull: Addressing Bias in Spoken Language Systems Used in the Development and Implementation of Automated Child Language-Based Assessment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Alison L. Bailey
      – PersonEntity:
          Name:
            NameFull: Alexander Johnson
      – PersonEntity:
          Name:
            NameFull: Natarajan Balaji Shankar
      – PersonEntity:
          Name:
            NameFull: Hariram Veeramani
      – PersonEntity:
          Name:
            NameFull: Julie A. Washington
      – PersonEntity:
          Name:
            NameFull: Abeer Alwan
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 63
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1