View in EDS HTML Full Text PDF Full Text

Comparing L2 Intelligibility for Learners of French: Automatic Speech Recognition versus Human Listeners

Saved in:
Bibliographic Details
Title: Comparing L2 Intelligibility for Learners of French: Automatic Speech Recognition versus Human Listeners
Language: English
Authors: Elena Shimanskaya (ORCID 0000-0002-4534-6772)
Source: Foreign Language Annals. 2025 58(2):438-457.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 20
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Comparative Analysis, French, Second Language Learning, Second Language Instruction, Accuracy, Intelligibility, Correlation, Technology Integration, Native Language, Audio Equipment, Speech Communication, Transcripts (Written Records), Evaluators, Oral Reading, Language Teachers
DOI: 10.1111/flan.12807
ISSN: 0015-718X
1944-9720
Abstract: In this study, I compare the accuracy of automatic speech recognition (ASR) transcription against two measures of intelligibility provided by human listeners. The data came from readings of five texts recorded by 15 language learners of French. Human understanding was gauged by (i) asking a group of 36 naïve first language (L1) speakers of French recruited from the general population to provide word-for-word transcriptions of the recordings, and (ii) asking two experienced French teachers (L1 speakers of French) to rate the recordings for intelligibility. The results reveal that ASR transcription accuracy was lower than that of human transcribers. However, all three measures, namely ASR transcription accuracy, human transcription accuracy, and human ratings of intelligibility, were correlated. Implications of these results for using the technology in foreign language teaching and learning are considered in the discussion.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1477307
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwG2qf9m9v_ZDzfNGMu1IHsTAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDKfHMQXnK_qjZlH3PgIBEICBm_jfF2tp40plJ7AqEiW05anUQUeY_Kh7S_isym1H1t07LIP8ykesxZffE8vVxmq2zZQSV1fKj0TbGbY-FsYcLvgc9P2LrdHkHW4cetZITSk0zdRwQ0Rt11kd4hFrsjT1C6AcniXo-Kts6jqYPqdc3AHJ__SODnrsB6zqHslWZ91N1hH4-zy62bFo1k5Qs5kS2HDsuKc_C9bFKHxK
Text:
  Availability: 1
  Value: <anid>AN0186745419;fla01jul.25;2025Jul22.02:37;v2.2.500</anid> <title id="AN0186745419-1">Comparing L2 intelligibility for learners of French: Automatic speech recognition versus human listeners </title> <p>In this study, I compare the accuracy of automatic speech recognition (ASR) transcription against two measures of intelligibility provided by human listeners. The data came from readings of five texts recorded by 15 language learners of French. Human understanding was gauged by (i) asking a group of 36 naïve first language (L1) speakers of French recruited from the general population to provide word‐for‐word transcriptions of the recordings, and (ii) asking two experienced French teachers (L1 speakers of French) to rate the recordings for intelligibility. The results reveal that ASR transcription accuracy was lower than that of human transcribers. However, all three measures, namely ASR transcription accuracy, human transcription accuracy, and human ratings of intelligibility, were correlated. Implications of these results for using the technology in foreign language teaching and learning are considered in the discussion.</p> <p>Keywords: automatic speech recognition; French; intelligibility; phonetics/phonology/pronunciation; technology</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/FLA/01jul25/flan12807-gra-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="flan12807-gra-0001.jpg" title="." /> </p> <p></p> <hd id="AN0186745419-3">INTRODUCTION</hd> <p>In the last couple of decades, automatic speech recognition (ASR) has sparked substantial interest in the second language (L2) learning and teaching community on the part of researchers (Chen et al., [<reflink idref="bib5" id="ref1">5</reflink>]; Inceoglu et al., [<reflink idref="bib12" id="ref2">12</reflink>]; Liakin et al., [<reflink idref="bib20" id="ref3">20</reflink>], [<reflink idref="bib21" id="ref4">21</reflink>], [<reflink idref="bib22" id="ref5">22</reflink>]; McCrocklin, [<reflink idref="bib23" id="ref6">23</reflink>], [<reflink idref="bib24" id="ref7">24</reflink>]; Mroz [<reflink idref="bib26" id="ref8">26</reflink>], [<reflink idref="bib27" id="ref9">27</reflink>]), educators, and curriculum designers. To give just one example of the latter, the "innovative, multimodal French OER" textbook #OnYGo (Blattner et al., [<reflink idref="bib2" id="ref10">2</reflink>]) includes a section <emph>On prononce</emph> (<emph>We pronounce</emph>) where the authors suggest that learners try reading a text out loud and transcribing it using ASR. Learners are then encouraged to compare the original text with the transcription and analyze their errors. This approach assumes that ASR transcription accurately captures the acoustic features of speech and closely mirrors human understanding. It attributes any deviations from the target to nontarget‐like pronunciation, while correct transcriptions are considered indicative of target‐like pronunciation. However, as discussed in the following section, ASR only approximates human speech interpretation. It processes sound waves digitally and incorporates limited non‐acoustic information to generate transcriptions. Therefore, studying ASR accuracy in comparison to human performance is essential for making informed pedagogical decisions.</p> <p>Interest in ASR is understandable since the technology offers several advantages to L2 users, such as opportunities to practice L2 and increased learner autonomy (Liakin et al., [<reflink idref="bib22" id="ref11">22</reflink>]). Moreover, recent improvements in ASR's accuracy increase its appeal for L2 teaching and learning. Microsoft research reports that, for English conversational data, their automated system achieved accuracy rates of 89%–94.2%, depending on the type of speech, compared to professional human transcribers' accuracy rate of 88.7%–94.1% (Xiong et al., [<reflink idref="bib39" id="ref12">39</reflink>]). In spite of these recent advances, the technology is not perfect when processing native speech (Thymé‐Gobbel & Jankowski, [<reflink idref="bib37" id="ref13">37</reflink>]). ASR is even less accurate when it comes to transcribing accented or L2 speech (Jurafsky & Martin, [<reflink idref="bib14" id="ref14">14</reflink>]; Liakin et al., [<reflink idref="bib22" id="ref15">22</reflink>]), which can lead to learners' frustration with the technology (Chen et al., [<reflink idref="bib5" id="ref16">5</reflink>]; McCrocklin, [<reflink idref="bib23" id="ref17">23</reflink>]). Moreover, when considering human listeners, there is evidence suggesting that we can expect different levels of understanding depending on how experienced they are with L2 speech (Kennedy & Trofimovich, [<reflink idref="bib16" id="ref18">16</reflink>]) and, maybe, how sympathetic they are to the L2 speaker. According to the 2012 ACTFL Proficiency guidelines (ACTFL, [<reflink idref="bib1" id="ref19">1</reflink>]), speakers rated at the ACTFL Novice level can be understood by sympathetic listeners accustomed to nonnative speech, while advanced speakers can be understood even by those who are not used to nonnative speech. Thus, to make the best use of ASR in L2 instruction, it is essential to not only study its accuracy and compare it to human performance but also to control for human listeners' prior exposure to L2 speech. The current study attempts to fill this gap by looking at transcription accuracy of three ASR engines (Google, Microsoft Word, and Notes) and comparing it to human listeners with different levels of expertise in interpreting L2 speech.</p> <hd id="AN0186745419-4">LITERATURE REVIEW</hd> <p></p> <hd id="AN0186745419-5">Basic principles of ASR</hd> <p>This section begins with a brief introduction into how modern ASR engines convert speech into text. ASR extracts information from the sound wave, such as amplitude, digitizes it into a sequence of numbers, and maps these numeric representations to letters or wordpieces, which are then combined to form words (Jurafsky & Martin, [<reflink idref="bib14" id="ref20">14</reflink>]). Generally speaking, these processes rely on the estimated probability of a letter based on the numeric representation of the sound and the probability of a word based on the letters. Modern ASR programs use sophisticated language models, such as neural networks, to calculate the probability of a certain segment (e.g., word) based on the segments that occur before and, possibly, after. When Microsoft's speech and dialog research group announced the ASR accuracy rates cited in the introduction (Xiong et al., [<reflink idref="bib39" id="ref21">39</reflink>]), they explained that this result was achieved by "using a series of improvements to [our] neural net‐based acoustic and language models" and by "using the entire history of a dialog session to predict what is likely to come next, effectively allowing the model to adapt to the topic and local context of a conversation" (Huang, [<reflink idref="bib10" id="ref22">10</reflink>], paragraphs 4 and 5). Therefore, it is important to recognize that modern ASR engines use not only the acoustic information to interpret speech, but also the probabilities of letters or words co‐occurring based on the information that the engine has learnt from large linguistic samples.</p> <p>In recent years, ASR technology has undergone substantial advancements. Deng and Huang ([<reflink idref="bib6" id="ref23">6</reflink>]) determined that, on average, since 1988, ASR has been shown to experience an annual error‐reduction rate of 10%. The most recent increase in ASR's accuracy was due to three factors: faster computation, improved algorithms, and the abundance of linguistic data that can be used for training the systems (Thymé‐Gobbel & Jankows, [<reflink idref="bib37" id="ref24">37</reflink>]). ASR in English is more accurate than most other languages precisely because there is a lot of training data available (Thymé‐Gobbel & Jankowski, [<reflink idref="bib37" id="ref25">37</reflink>]). Current ASR engines often entertain multiple competing hypotheses about what is being said while they arrive at the final transcription. On occasion, users can see these hypotheses compete on the screen in real time, as the initially displayed text is replaced with a revised version as the utterance progresses. Many applications today offer some voice functionality; however, most of them are built using some of the few major ASR systems: Google, Microsoft, IBM, Amazon, Facebook, and Apple (Thymé‐Gobbel & Jankowski, [<reflink idref="bib37" id="ref26">37</reflink>]). For example, the web application iSpraak (García et al., 2020) allows teachers to input a short text for learners to read aloud. The app then transcribes the spoken text, compares the transcription with the original, and calculates a match score. Words that do not match the original are then presented in isolation. The application leverages Web Speech API, which deploys different ASR engines depending on the browser being used (personal communication with Dr. Dan Nickolai).</p> <p>Before investigating the accuracy of modern ASR in transcribing L2 speech, it is important to consider how ASR transcription approximates human ability to interpret speech. Even though these processes have a lot of similarities, they are not identical. To give one example of such differences, human listeners' "understanding" of L2 speech can be assessed in different ways, such as rating the ease of understanding a speech sample (a measure of comprehensibility) or providing a transcription (a measure of intelligibility). Notably, the results of a rating task could be different from the results of a transcription task. In contrast, most commercially available ASR systems do not provide a measure of task difficulty. Depending on the type of ASR, the only way to measure "understanding" could be the engine's accuracy in choosing one of the predetermined options (e.g., <emph>yes</emph> vs. <emph>no</emph>) or the accuracy of its transcription.</p> <p>Foreign accent could affect both how well humans understand speech and the accuracy of ASR transcription. Based on what is known about the impact of L2 accent on comprehensibility, many L2 practitioners today may agree that the goal of L2 instruction in the domain of pronunciation is not to eliminate foreign accent, but to make sure that a learner is "understood" by other speakers of the language, as suggested for example by the Intelligibility Principle in pronunciation instruction (Levis, [<reflink idref="bib19" id="ref27">19</reflink>]). According to experts, ASR accuracy can be influenced by several factors, including accent, vocabulary size (e.g., recognizing numbers vs. transcribing human conversations), type of speech (e.g., reading aloud vs. spontaneous conversation), and presence of background noise (Jurafsky & Martin, [<reflink idref="bib14" id="ref28">14</reflink>]). Since in L2 classrooms ASR is primarily used as an approximation of how well humans can understand L2 speakers, the following section offers a quick overview of L2 research with human listeners, and how their "understanding" of L2 speech has been conceptualized and measured.</p> <hd id="AN0186745419-6">Previous studies of L2 comprehensibility, intelligibility, and accentedeness: Human listeners</hd> <p>Most L2 research with human listeners, starting with seminal work by Munro and Derwing ([<reflink idref="bib28" id="ref29">28</reflink>]), distinguish among the three dimensions of intelligibility, comprehensibility, and accentedeness (Bergeron & Trofimovich, [<reflink idref="bib4" id="ref30">4</reflink>]; Derwing & Munro, [<reflink idref="bib7" id="ref31">7</reflink>]; Huensch & Nagle, [<reflink idref="bib11" id="ref32">11</reflink>]; Munro & Derwing, [<reflink idref="bib29" id="ref33">29</reflink>]; Nagle et al., [<reflink idref="bib31" id="ref34">31</reflink>]). Intelligibility represents the actual understanding of what is being said, while comprehensibility refers to "listeners' perceived ease or difficulty in understanding a speakers' L2 speech" (Bergeron & Trofimovich, [<reflink idref="bib4" id="ref35">4</reflink>], p. 548). When it comes to ASR, intelligibility is the only variable that can be studied using automatic transcription. However, it is important to note that although several studies have demonstrated that the three measures are correlated (Derwing & Munro, [<reflink idref="bib7" id="ref36">7</reflink>]; Munro & Derwing, [<reflink idref="bib28" id="ref37">28</reflink>]), it has also been shown that human listeners are able to provide fairly accurate transcriptions of speech that they consider to be difficult to understand (Derwing & Munro, [<reflink idref="bib7" id="ref38">7</reflink>]; Munro & Derwing, [<reflink idref="bib28" id="ref39">28</reflink>]). It is also worth noting that many studies in this area rely on spontaneous speech, which complicates the task of separating phonetic factors from lexical and grammatical ones (Bergeron & Trofimovich, [<reflink idref="bib4" id="ref40">4</reflink>]; Trofimovich & Isaacs, [<reflink idref="bib38" id="ref41">38</reflink>]).</p> <p>Turning to the effects of listener background on ratings of intelligibility, comprehensibility, and accentedness, previous studies have yielded mixed results regarding the impact of familiarity with L2 speech in general or with a specific accent in particular. On the one hand, several studies suggest that listener background has minimal impact on ratings of accentedness and comprehensibility. For instance, high interrater reliability (Munro & Derwing, [<reflink idref="bib28" id="ref42">28</reflink>]) and comparable ratings of accentedness and comprehensibility provided by experienced versus novice raters (Isaacs & Thomson, [<reflink idref="bib13" id="ref43">13</reflink>]) indicate consistency across different listener groups. Along the same lines, Bergeron & Trofimovich ([<reflink idref="bib4" id="ref44">4</reflink>]) found no differences in ratings of accentedness and comprehensibility based on listeners (non)‐familiarity with the first language (L1) of the speakers. Huang ([<reflink idref="bib9" id="ref45">9</reflink>]) observed that listeners' ratings of accentedness remained consistent regardless of whether listeners were teachers familiar with the accent, non‐teachers familiar with it, and non‐teachers unfamiliar with it. On the other hand, some evidence suggests that listener experience can influence L2 intelligibility and accentedness judgments. In their study, Kennedy and Trofimovich ([<reflink idref="bib16" id="ref46">16</reflink>]) demonstrated that listener experience was a significant factor in predicting transcription accuracy of L2 English speech samples while Kornder and Mennen ([<reflink idref="bib18" id="ref47">18</reflink>]) uncovered that listeners' linguistic background (bilingual vs. monolingual) can have an effect when they are asked to judge speech as (non‐)native‐like. To summarize, most studies did not find differences in ratings of accentedness and comprehensibility of L2 speech depending on listeners' background, although there might be some evidence suggesting that those who have experience with L2 speech are able to provide more accurate transcriptions.</p> <hd id="AN0186745419-7">Previous studies of L2 intelligibility: ASR</hd> <p>In contrast to human listeners, ASR systems can only evaluate intelligibility through transcription accuracy, offering no insight into speech comprehensibility or accentedness. Three studies are particularly informative in understanding how ASR transcription accuracy for L2 speech compares to that of human listeners. Inceoglu and colleagues (<reflink idref="bib2" id="ref48">2</reflink>) collected ASR transcriptions of isolated words and short sentences produced by four Taiwanese EFL learners and compared them with those provided by 12 L1 speakers of English. The researchers established that, for isolated words, ASR and human accuracy rates were comparable, 40.81% and 38.62%, but that the correlations between ASR and human transcription accuracy were statistically significant for only two out of four EFL speakers. However, for sentences, human transcriptions were more accurate (83.88%) than ASR (75.52%). Similarly to isolated words, the correlations between ASR and human accuracy rates were statistically significant for only two out of four EFL speakers. However, this study had a few limitations, including the size of the speech sample and transcribers' homogeneous profile in terms of familiarity with L2 speech. First, the analysis was based on 48 isolated words and 24 short sentences produced by four speakers, and only one version of each word/sentence was retained for transcription. Second, the authors acknowledged that all 12 L1 speakers of English who provided the transcriptions reported a very frequent contact with accented English.</p> <p>McCrocklin and Edalatishams ([<reflink idref="bib25" id="ref49">25</reflink>]) also used Google's ASR engine and compared the accuracy of its transcription to that of human transcribers. The speakers in the study came from three groups: L1 speakers of English, L1 Spanish, and L1 Chinese‐speaking learners of English. The researchers took two sentences recorded by each speaker and obtained ASR and human transcriptions. When analyzed as one group, transcription accuracy was similar for human transcribers (92.98%) and ASR (93.31%), and there was a strong and statistically significant correlation between the two. However, when the results from Chinese and Spanish L1 groups were analyzed separately, human accuracy rate for the Spanish group (93%) was not correlated with the accuracy of ASR (92.7%). The same correlation for the Chinese group was strong and statistically significant (89% human vs. 91% ASR accuracy). These results were comparable to those obtained by Inceoglu et al. because they also showed that human and ASR transcription might not be universally correlated for all participants; in this case, the results varied based on participants' L1. Unlike Inceoglu et al., McCrocklin and Edalatishams did not collect data on their listeners' degree of familiarity with accented speech, which means that the study cannot inform us about the effects of human listener background on transcription accuracy.</p> <p>When it comes to studying ASR in L2 contexts, English turns out to be the most commonly studied language. In their overview of 26 studies involving ASR technology in language learning, Shadiev and Liu ([<reflink idref="bib35" id="ref50">35</reflink>]) reported that 17 studies focused on English with Dutch, French, and Spanish being the only other languages studied. The two studies discussed so far in this section analyzed ASR transcriptions accuracy for L2 English. However, studying ASR accuracy for a variety of L2s is important since languages vary with respect to the reliability of connections between spelling and sounds. Languages can be categorized as transparent or opaque based on the consistency of sound‐to‐letter relationships, with English and French usually classified as opaque and Italian serving as an example of a transparent language (Mussar et al., [<reflink idref="bib30" id="ref51">30</reflink>]). Unlike English, French phonology relies heavily on grammar (e.g., adjectival gender agreement) and on syntax (e.g., liaison) (Hannahs, [<reflink idref="bib8" id="ref52">8</reflink>]). Another notable feature of French is its high prevalence of silent‐letter endings, often conveying grammatical or semantic information in writing. Indeed, 56% of words found in children's books in France include such silent‐letter endings (Mussar et al., [<reflink idref="bib30" id="ref53">30</reflink>]). This interplay between morphology and silent letters in French is well illustrated with nominal pluralization: while English nominal plural marker is almost always pronounced (<emph>boy</emph> [bɔɪ] vs. <emph>boys</emph> [bɔɪ<bold>z</bold>]), the corresponding French marker is almost always silent (<emph>garçon</emph> [gaʀsɔ̃] vs. <emph>garçons</emph> [gaʀsɔ̃]). The study discussed next shifts the focus to French, offering valuable insights into ASR accuracy in a non‐English context.</p> <p>Mroz ([<reflink idref="bib27" id="ref54">27</reflink>]) investigated how L2 intelligibility is affected by the use of ASR throughout a semester‐long course. The basis for the analysis was the recordings of four texts read by 26 college students. Intelligibility ratings came from two sources: ASR transcriptions and human raters. Two L1 speakers of French, one accustomed to L2 speech and one unaccustomed, were recruited for the study. The researcher determined that asking humans to transcribe the same four texts 26 times would result in a high degree of familiarity with each text and compromise the study's reliability. Instead, the L1 speakers were given a printed copy of each text and were asked to circle the words where their transcription would not match what they heard if they were asked to give a verbatim transcription. Of particular interest to the current study is that Mroz calculated Pearson correlations between ASR transcription accuracy and human ratings. She found a strong positive correlation, meaning that ASR transcriptions were more accurate for participants whose speech was also rated as more intelligible by humans.</p> <p>Unlike in the studies by Inceoglu et al. ([<reflink idref="bib12" id="ref55">12</reflink>]) and McCrocklin and Edalatishams ([<reflink idref="bib25" id="ref56">25</reflink>]), human listeners did not provide transcriptions in Mroz's (2020) study, so any comparisons should be made with caution. However, it appears that the accuracy of ASR transcription was correlated with human ratings of intelligibility in Mroz's study, while the results of Inceoglu et al. and McCrocklin and Edalatishams were more mixed. None of the three studies investigated how listeners' backgrounds might affect their findings. Inceoglu et al. recruited only one type of listener, Mroz included both a trained rater and an unaccustomed listener with minimal training in speech rating, and McCrocklin and Edalatishams did not collect this information.</p> <p>The current study aims to fill the following gaps in the literature. First, the study contributes to our understanding of ASR accuracy in a language other than English by studying L2 French. French is a particularly interesting language in this respect because even though the correlations between spelling and phonemes are fairly reliable, the language contains a lot of homophones and a lot of grammatical markers present in spelling that are either homophonous or silent in oral speech (see the plural example above). Thus, French spelling conventions are closely connected to grammatical constraints and/or context, meaning that mismatches between the original text and its transcription are not necessarily the result of mispronunciation of individual phonemes. Second, the study is designed to produce reliable results by analyzing transcription accuracy scores based on multiple ASR engines, multiple speech samples per speaker, and multiple recordings of the same text. Third, human listeners' backgrounds are controlled by recruiting two types of listeners, expert raters with extensive teaching experience and naïve listeners recruited from the general population. The study was designed to answer the following two research questions:</p> <p></p> <ulist> <item> 1. How does transcription accuracy differ between ASR and naïve human transcribers, evaluated using exact and phonemic match criteria?</item> <p></p> <item> 2. How does transcription accuracy provided by ASR and naïve human transcribers correlate with intelligibility ratings provided by expert raters?</item> </ulist> <hd id="AN0186745419-8">METHODS</hd> <p></p> <hd id="AN0186745419-9">Participants</hd> <p>Three groups of participants were recruited for the study: L2 speakers of French (<emph>n</emph> = 15), French L1 transcribers from the general population (<emph>n</emph> = 36), and French L1 raters with extensive experience teaching French as a foreign language (<emph>n</emph> = 2). The study received Institutional Review Board approval under protocols 1896694‐1 and 1839330‐3.</p> <hd id="AN0186745419-10">L2 speakers of French</hd> <p>Fifteen adult learners of French provided their recorded readings of five different texts (<emph>Mean Age</emph> = 23, <emph>SD</emph> = 5.6, <emph>Range</emph> 19–41; 6 females, 1 nonbinary), leading to a total of 75 recordings. These participants were enrolled in an in‐person 300‐level French phonetics course at a large U.S. (or American) university and submitted the recordings as part of their assessments in the course (25% of the final grade). They were graded on pronunciation errors, fluency, and intonation. The recordings were done at home, and no model was provided. At the end of the semester, students were informed about the study's objectives and how their recordings would be used. They were also given the opportunity to opt out. None of the participants chose to opt out, and all agreed to complete a brief anonymous autobiographical questionnaire.</p> <p>In the questionnaire, 10 participants reported no knowledge of languages other than English and French. One participant reported Cantonese as their L1, and one self‐identified as a heritage speaker of Tagalog. One participant reported some knowledge of Spanish, and one participant was studying both Spanish and Italian. Most participants (<emph>n</emph> = 14) had taken other university upper‐level French courses (<emph>Mean</emph> = 3.5 courses, <emph>Range</emph> 0–7 courses). Participants had no interactions with ASR as part of the phonetics course. Two participants indicated that they had experience with ASR technology (Duolingo, Rosetta Stone). One participant reported using Google Translate, but no experience with ASR.</p> <p>Alongside the 75 student recordings, five additional recordings were added to the sample for analysis, bringing the total to 80. One recording of each text provided in the textbook on French phonetics served as a model (Kamoun & Ripaud, [<reflink idref="bib15" id="ref57">15</reflink>]). These five recordings were used as reference points in the study and were transcribed/rated by the L1 speakers (see below). However, they were not included into any descriptive or inferential statistics.</p> <hd id="AN0186745419-11">L1 French naïve listeners</hd> <p>Thirty‐six transcribers were recruited through prolific.org and received a small financial compensation ($10) for completing a transcription task of the recordings described above. A total of 35 transcribers in this group reported being monolingual L1 speakers of French (<emph>Mean Age</emph> = 28, <emph>SD</emph> = 7, <emph>Range</emph> 21–49; 15 females, 1 no gender reported). One transcriber reported being trilingual French‐Portuguese‐English. Two transcribers reported that Belgium was their country of residence; the remaining 34 transcribers were residing in France at the time of the study. When it comes to transcribers' knowledge of foreign languages, 33 reported having intermediate or advanced proficiency in English. Two transcribers did not report any knowledge of languages other than French, and one reported intermediate proficiency in Spanish. Transcribers also reported some knowledge of Arabic, Chinese, German, Italian, Japanese, Occitan, Portuguese, and Swedish.</p> <p>Although the study description on prolific.org specified that participants should not have experience with L2 speakers of French, seven transcribers reported being or having been in regular contact with L2 speakers of French (yes/no question). Four out of seven explained that their contact was due to friends or family members being L2 speakers of French. Two transcribers reported that their contact was due to staying abroad (6 months in Quebec, 2 months in California). One transcriber explained that they taught French as a foreign language for a year. Transcribers were also asked about the extent of their interaction with L2 speakers: 22 reported that they were not used to interacting with L2 speakers, 10 reported little familiarity, and four reported being very familiar with L2 speech. It should be pointed out that, out of the four transcribers who reported a high degree of familiarity with L2 speech, two answered negatively to the yes/no question about whether they were in contact with L2 speakers of French. As the anonymous reviewer noted, such a mixed profile is to be expected in multicultural countries like France and Belgium, and likely throughout the Francophone world, where many individuals have encountered L2 speakers of French or have been exposed to accented L2 French through the media. This group, in which 81% reported no regular contact with L2 speakers and only 11% reported a high degree of familiarity with L2 speech, reasonably represents the type of interlocutors an L2 learner might encounter when speaking the language in a Francophone country outside the classroom.</p> <hd id="AN0186745419-12">L1 French expert raters</hd> <p>Two L1 speakers with extensive experience teaching French in the United States completed a rating task (via Qualtrics) modeled after the rating task in Mroz ([<reflink idref="bib27" id="ref58">27</reflink>]). Raters were recruited through the researcher's professional connections and compensated for their time ($50). One rater had been teaching French in the U.S. at the University level for 3 years; the other rater had been teaching both university and K‐12 students in the United States for 10 years. Both raters reported being very accustomed to L2 speech and having high English proficiency, equivalent to Superior ACTFL/C2 CEFR. Neither reported knowing languages other than French and English. One rater reported experience with transcribing, while the other reported no such experience.</p> <hd id="AN0186745419-13">Materials</hd> <p>The texts analyzed in this study were excerpts from five francophone writers: Assja Djebar (48 words), Alfred de Musset (56 words), Raymond Queneau (39 words), Boris Vian (51 words), Philippe Delerm (88 words) (Kamoun & Ripaud, [<reflink idref="bib15" id="ref59">15</reflink>]). Each recording was first trimmed to the beginning and end of the sound wave of the text. Any vocalizations prior and following the actual texts (e.g., clearing one's voice, <emph>Bonjour</emph>/<emph>Hello</emph>) were deleted. None of the recordings contained any comments or digressions after the reading started. Five seconds of silence were added to the beginning and end of each recording to create a more naturally sounding audio file. The researcher listened to the resulting audio files to verify that the loudness and quality of the audio were satisfactory. Individual segments that had noticeably lower volume than the rest of the recording were normalized using the Normalize peak amplitude function to −3.0 dB. All the modifications were performed in Audacity. The resulting files were used for both ASR and human transcription as well as for the rating task.</p> <hd id="AN0186745419-14">PROCEDURES</hd> <p></p> <hd id="AN0186745419-15">ASR transcriptions</hd> <p>Each recording was transcribed by each of the three ASR engines once: in an online Google document (Chrome browser), in Microsoft Word, and in Pages. French was chosen as the language for all ASR applications, and all notifications were disabled to avoid any interference during the transcription process. After each recording was transcribed, the resulting texts were recorded in a separate document, and Word/Pages document was closed/the browser was refreshed. Each transcription was then input into an excel spreadsheet by the researcher word‐by‐word. In cases when words were misspelled (e.g., <emph>posé</emph> for <emph>posée</emph>), misidentified (<emph>Claire</emph> for <emph>clerc</emph>) or missing, the researcher used her best judgment to find the correspondence between the transcription and the original text (see Figure 1). As noted by the anonymous reviewer, this coding procedure did not distinguish between instances where words were transcribed with different spellings from the original and cases where words were entirely omitted from the transcription. Future research could benefit from addressing this distinction; however, it was beyond the scope of the present study. Two scores were calculated for each word of each transcription: Exact Match and Phonemic Match.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/FLA/01jul25/flan12807-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="flan12807-fig-0001.jpg" title="1 Sample of ASR transcription coding (columns: A – original; FH – ASR transcription, FI – Exact Match, FJ – Phonemic Match). ASR, automatic speech recognition." /> </p> <p></p> <hd id="AN0186745419-17">Exact match</hd> <p>Excel's "IF" function was used to assign a score of 1 to each word in the transcription that matched the word in the original text exactly. All the words in the transcription that did not match the original text exactly received the score of 0.</p> <hd id="AN0186745419-18">Phonemic match</hd> <p>The researcher considered each transcribed word that received a score of 0 during automatic scoring (Exact Match above) to determine whether any of these words were phonemic matches to the target word. First, any misspelled words received a score of 1 as long as the misspelling did not alter the word's pronunciation (e.g., <emph>access</emph> vs. <emph>accès</emph>, <emph>alteration</emph> for <emph>altération</emph>). Second, transcriptions that differed from the original in how morphological markers were spelled received a score of 1 for Phonemic Match as long as the difference in spelling did not alter the word's pronunciation. For example, the plural marker –s in French is almost always silent; imperfect past endings –ais, – ait, –aient are all pronounced as /ɛ/ or /e/; 2nd person plural present tense ending –ez, past participle ending –é, and infinitive ending –er are all pronounced as /e/. Finally, homophones (e.g. <emph>ces</emph> vs. <emph>ses</emph>, <emph>Mark</emph> vs. <emph>marques</emph>) and strings with nontarget‐like word segmentation (e.g. <emph>elle annule</emph> vs. <emph>et l'annule</emph>, <emph>quoi qu'elle</emph> vs. <emph>quoiqu'elle</emph>) received the score of 1 for Phonemic Match, again provided the alternative spelling was phonemically equivalent to the original. Any word or string that was not phonemically equivalent to the original, received the score of 0, for example <emph>plaigne</emph>, <emph>plaignent</emph>, <emph>plaignait</emph> for <emph>plaignît</emph>; <emph>méchant c'était</emph> for <emph>méchanceté</emph>; <emph>sans patienter</emph> for <emph>s'impatienter</emph>. A sample of how this coding procedure was implemented is provided in Figure 1.</p> <hd id="AN0186745419-19">Human transcriptions: Naïve transcribers</hd> <p>L1 speakers of French recruited from the general population were asked to transcribe four or five recordings (via Qualtrics). The number of recordings depended on the randomization list. All but one randomization list contained five recordings. After completing a short anonymous auto‐biographical questionnaire, participants were instructed not to listen to the recordings multiple times unless absolutely necessary. It was decided to give transcribers the opportunity to pause and replay the recordings, provided the participants in Huensch and Nagle's ([<reflink idref="bib11" id="ref60">11</reflink>]) study requested to be able to listen multiple times, to pause, and to not have a time limitation. If they were uncertain, they were instructed to write down what they thought they heard. If they were unable to understand at all, they were asked not to write anything. Finally, participants were instructed not to use punctuation or capitalization, but to use accent marks and hyphens if they thought they were necessary. A recording of a short phrase uttered by an L1 speaker and written on the screen in standard French alphabet with no punctuation or capitalization was provided as an example before participants started transcribing. Randomization lists were set up so that each transcriber heard each text only once. Three transcribers out of 36 heard the same speaker twice. This procedure allowed the researcher to obtain two transcriptions (from two different human transcribers) of each recorded text.</p> <p>The scoring procedure was the same as that for ASR transcriptions. The researcher found the most likely correspondence between each transcribed and target word and calculated Exact and Phonemic Match scores for each word. The only type of error observed with human transcribers but not with ASR was the occasional replacement of target words with synonyms (e.g., <emph>traits</emph> for <emph>marques</emph>, <emph>gens</emph> for <emph>hommes</emph>) or changes in the word order of constituents (<emph>au traits sur le visage</emph> for <emph>sur le visage des marques</emph>). These were considered not phonemically equivalent and received a score of 0 for both Exact and Phonemic match. All other patterns of nontarget‐like transcriptions were the same and were treated the same for human and ASR transcriptions.</p> <hd id="AN0186745419-20">Human ratings: Expert raters</hd> <p>Two L1 speakers of French with extensive teaching experience rated the same speech samples for intelligibility. The task was conducted online. Each recording was displayed alongside its corresponding text, with the text broken down into individual words. Raters could select any word in the text by clicking on them, including multiple words within the same text. Raters were instructed to listen to each recording once, with the option to pause, and select all the words where their transcription would differ from the words provided on the screen (see Figure 2). The recordings and texts were organized into 18 pages, with each page containing four or five recordings. All texts on a given page were unique and read by different speakers. As an example, a short phrase spoken by an L2 French speaker was provided, along with the corresponding target sentence. In this example, one mispronounced word was highlighted ([tut] instead of [tu]). The task concluded with a short auto‐biographical questionnaire.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/FLA/01jul25/flan12807-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="flan12807-fig-0002.jpg" title="2 Sample layout of the rating task (expert raters)." /> </p> <p></p> <hd id="AN0186745419-22">Analysis</hd> <p>Individual scores for each word per participant, text, and condition (ASR/naïve transcribers/expert raters) were analyzed in R. For ASR transcriptions, Exact and Phonemic Match scores for individual words were added per each participant and per text. The scores from the three different ASR engines were averaged to create one aggregate ASR score for each participant and each text. Finally, since every text contained a different number of words, ASR transcription accuracy rate was calculated by dividing the score for each text/participant by the number of words in each text to allow comparisons across texts. Similar to ASR, for human transcriptions, Exact and Phonemic scores for individual words were added per participant and per text. Since two transcriptions were obtained from human transcribers for each recording, the aggregate score was the average of these two scores. The human transcription accuracy score was calculated by dividing each text score by the number of words in each text. Unlike the transcription task, which yielded two scores for each recording (Exact vs. Phonemic Match), the rating task produced only one score per recording per rater. Interrater reliability, assessed using the two‐way intraclass correlation coefficient of absolute agreement (ICC), demonstrated excellent reliability (ICC = 0.955, <emph>p</emph> < .001, CI: 0.882–0.983) (Koo & Li, [<reflink idref="bib17" id="ref61">17</reflink>]). Scores obtained from the two raters were averaged for each text and divided by the number of words per text.</p> <hd id="AN0186745419-23">RESULTS</hd> <p>Figure 3 presents mean accuracy scores for ASR and human transcription (naïve listeners) by Exact versus Phonemic Match for individual participants. The box plots reflect descriptive statistics for L2 speakers only, excluding the L1 model. L1 model (▲) results were added to the plot after the descriptive statistics (median, quartiles) had been calculated to allow a visual comparison between L2 participants and the L1 model. Both ASR and human transcriptions of the L1 model were highly accurate. For ASR, the accuracy rate was 0.82 for Exact Match and 0.91 for Phonemic Match; for human transcription, the accuracy rate was 0.82 for Exact Match and 0.86 for Phonemic Match.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/FLA/01jul25/flan12807-fig-0003.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="flan12807-fig-0003.jpg" title="3 Mean transcription accuracy by participant: ASR versus naïve human listeners. ASR, automatic speech recognition." /> </p> <p></p> <p>Looking at the overall results for L2 speakers presented in Figure 3, one can observe two trends in the data. First, transcription accuracy was higher for Phonemic versus Exact Match scores; and this was true for ASR (Exact Match 0.46, Phonemic Match 0.51) and human transcription (Exact Match 0.60, Phonemic Match 0.64). In other words, both ASR and human transcriptions were more accurate when spelling variations were taken into consideration. Second, Figure 3 clearly illustrates that, on average, humans provided transcriptions that were slightly more accurate than those provided by ASR. This was true for both Exact and Phonemic Match scores.</p> <p>Transcription accuracy scores were analyzed using mixed models with lme4 (Bates et al., [<reflink idref="bib3" id="ref62">3</reflink>]), psych (Revelle, [<reflink idref="bib34" id="ref63">34</reflink>]), and afex packages (Singmann et al., [<reflink idref="bib36" id="ref64">36</reflink>]) in the R environment (R Core Team, [<reflink idref="bib33" id="ref65">33</reflink>]). The model did not include intelligibility ratings. Condition (ASR vs. Naïve Listeners) and Match (Exact vs. Phonemic) were taken as fixed factors. The model also tested the interaction of the fixed factors (Condition*Match). Since the study design included repeated‐measures, participants and texts were added as random effects. Initially, the model included by‐text and by‐participant varying intercepts, as well as by‐text and by‐participant varying slopes for Condition. It was decided not to model the effects of Match for individual participants and texts because Phonemic Match scores were always the same or higher than Exact Match scores. This model converged but gave a warning that the random effects were very small. Therefore, the final model only included by‐text and by‐participant varying intercepts:</p> <p>ScoreTranscriber ~ Match * Condition + (1 | Text) + (1 | Participant).</p> <p>The model revealed main effects of Condition (<emph>χ</emph><sups>2</sups>(<reflink idref="bib1" id="ref66">1</reflink>) = 99.11, <emph>p</emph> < .001) and Match (<emph>χ</emph><sups>2</sups>(<reflink idref="bib1" id="ref67">1</reflink>) = 13.17, <emph>p</emph> < .001) and no interaction between the two (<emph>χ</emph><sups>2</sups>(<reflink idref="bib1" id="ref68">1</reflink>) = 0.18, <emph>p</emph> = .669). Thus, the difference in transcription accuracy between ASR and human transcribers was statistically significant. The model also confirmed that Phonemic Match accuracy scores were higher than Exact Match scores for both ASR and human transcribers.</p> <p>To gauge any differences between human transcription accuracy provided by ASR/naïve listeners and human intelligibility ratings provided by expert raters, Figure 4 combines the results of these two tasks, namely the transcription task and the rating task. The figure makes it clear that the expert raters indicated higher intelligibility than the accuracy rate of the transcriptions provided by the human naïve transcribers and by ASR.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/FLA/01jul25/flan12807-fig-0004.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="flan12807-fig-0004.jpg" title="4 Mean transcription accuracy (naïve human transcribers, ASR) versus mean intelligibility ratings (expert human raters). ASR, automatic speech recognition." /> </p> <p></p> <p>The correlation between ASR transcription accuracy scores, human transcription accuracy scores, and human ratings were computed using the <emph>corr.test</emph> function (psych package). The results are reported in Table 1 and visually illustrated in Figure 5. All three types of intelligibility measures were strongly correlated, and the correlations were significant (<emph>p</emph> < .001, "holm" adjustment for multiple tests).</p> <p>1 TABLE Pearson correlation results between ASR transcription accuracy, naïve human transcription accuracy, and expert human intelligibility ratings.</p> <p> <ephtml> <table><thead valign="bottom"><tr valign="bottom"><th /><th>ASR</th><th>Expert raters</th><th>Naïve transcribers</th></tr></thead><tbody valign="top"><tr><td>ASR</td><td>1.00</td><td>0.73</td><td>0.86</td></tr><tr><td>Expert raters</td><td /><td>1.00</td><td>0.80</td></tr><tr><td>Naïve transcribers</td><td /><td /><td>1.00</td></tr></tbody></table> </ephtml> </p> <p>1 Abbreviation: ASR, automatic speech recognition.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/FLA/01jul25/flan12807-fig-0005.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="flan12807-fig-0005.jpg" title="5 Scatterplots of mean transcription accuracy/ratings by participant: (a) naïve human transcribers versus automatic speech recognition (ASR), (b) expert raters versus ASR, and (c) naïve human transcribers versus expert raters." /> </p> <p></p> <hd id="AN0186745419-27">DISCUSSION</hd> <p>The study compared the transcription accuracy of ASR systems and human transcribers for five texts read by 15 L2 learners of French, focusing on the differences between ASR and naïve human transcribers. To achieve this, transcribers were recruited from the general population, with their familiarity with L2 speech documented; fewer than 20% reported regular contact with L2 speakers of French. On average, human transcriptions were more accurate than ASR results, a contrast that was confirmed statistically.</p> <p>These findings align with prior research by Inceoglu et al. ([<reflink idref="bib12" id="ref69">12</reflink>]) on L2 English speech, which also demonstrated that human transcriptions of sentences were more accurate than those of ASR systems. However, unlike Inceoglu et al.'s study, which involved transcribers with frequent exposure to accented speech, the current study extends these results to naïve transcribers from the general population. Additionally, the present analysis is based on longer speech samples from a larger pool of speakers, providing potentially more reliable results. The findings also align with those of McCrocklin and Edalatishams ([<reflink idref="bib25" id="ref70">25</reflink>]), who reported a statistically significant correlation between ASR and human transcription accuracy for Spanish and Chinese L2 speakers of English. However, the current study reveals a much more pronounced difference, with human transcriptions outperforming ASR by 13%–14%, compared to less than a 1% difference in McCrocklin and Edalatishams' study.</p> <p>In addition to the overall comparison of ASR and human transcription accuracy, the research further examined results using exact and phonemic match criteria. Each transcription was evaluated twice: once for exact matches between the original text and the transcription, and again for phonemic matches where spelling differed but pronunciation was equivalent. Statistical analysis revealed that both human and ASR transcriptions were more accurate when spelling variations were taken into account. This finding underscores the importance of addressing spelling‐pronunciation mismatches in French and potentially other languages, where certain spelling differences might not impact pronunciation. As a novel contribution, this insight has not been widely explored and holds significant pedagogical implications for the effective integration of ASR technology into L2 teaching and learning, as discussed in the next section.</p> <p>The study also aimed to examine whether ASR and human transcription accuracy correlate with human intelligibility ratings, as both transcription and ratings are commonly used to measure intelligibility in research. For instance, Mroz ([<reflink idref="bib27" id="ref71">27</reflink>]) found a correlation between intelligibility ratings and ASR transcription accuracy for L2 French speech. In the current study, the intelligibility ratings provided by two experienced L2 teachers correlated with both ASR and human transcriptions provided by naïve listeners, suggesting that both transcription and ratings can serve as measures of intelligibility.</p> <p>While the results of the transcription and rating tasks were not statistically compared for methodological reasons, they indicate that, across speakers, ratings tend to align with transcription accuracy, as evidenced by the significant correlation among the three measures. However, ratings generally yielded higher intelligibility scores than transcription accuracy. Due to the study design, it is unclear whether this discrepancy was due to the raters' extensive experience with L2 speech, which may have improved their understanding, or a task effect, as the presence of the target text might have influenced their perception of intelligibility.</p> <p>Nonetheless, the findings from the rating task offer valuable insights, particularly for the design of L2 assessments. They suggest that even experienced and highly trained teachers might overestimate intelligibility due to their expertise or familiarity with the topic at hand. This highlights a potential limitation of oral assessment tasks, which are often constrained to specific topics, potentially aiding teachers in understanding L2 learners despite pronunciation errors.</p> <p>Finally, one of the key gaps identified in the literature review was the limited number of research studies investigating ASR accuracy in languages other than English. This study provides a comparative perspective on ASR accuracy rates for L2 English, such as Inceoglu et al. ([<reflink idref="bib12" id="ref72">12</reflink>]), who reported 75.5% accuracy, and McCrocklin and Edalatishams ([<reflink idref="bib25" id="ref73">25</reflink>]), who found 93.3% accuracy. In contrast, the ASR transcription accuracy for L2 French speech in the current study was considerably lower, averaging 46%–51%. However, any direct comparisons should be approached with caution, as differences in participant L2 proficiency across studies likely affect the results.</p> <hd id="AN0186745419-28">Pedagogical implications</hd> <p>This section highlights two key implications of the present finding for those interested in using ASR in teaching foreign languages and offers a recommendation to enhance its value for learning. First, while the difference between ASR and human transcription accuracy was statistically significant, it does not automatically mean that ASR cannot be effectively used in L2 classrooms. For instance, as proposed by the authors of $OnYGo, students can generate ASR transcriptions of their readings and compare them to the original text. It is important, however, for both learners and instructors to be aware that ASR‐generated transcriptions may be slightly less accurate than human‐generated ones. Nonetheless, the 13%–14% difference in accuracy might be an acceptable trade‐off for the advantages that ASR offers, such as fostering learner autonomy and providing additional opportunities to practice and test pronunciation in a low‐stakes environment.</p> <p>Second, the observed correlation among all three measures suggests that ASR transcription can serve as a valuable tool in gauging the relative intelligibility of different speech samples. Namely, if one L2 speaker achieves a more accurate ASR transcription than another, that speaker is likely to be better understood by human listeners as well. In classroom settings, for example, learners could use this feedback by comparing their ASR results to the class average. However, instructors and learners should exercise caution when interpreting ASR transcription as an indicator of target‐like versus nontarget‐like pronunciation.</p> <p>This caution is warranted for two reasons. First, as discussed in the literature review, ASR systems rely on both acoustic and non‐acoustic information to interpret speech, meaning factors beyond pronunciation can affect transcription accuracy. Second, this study showed that ASR transcriptions are more accurate when mismatches in spelling that do not result in phonemic differences are accounted for. For instance, discrepancies in ASR output could result from homonyms, grammatical errors introduced by the ASR system, or incorrect tense usage rather than actual pronunciation errors. To ensure that learners receive meaningful feedback from ASR, as suggested in tasks like the one from #OnYGo, they need training to distinguish between transcription differences that signal potential pronunciation issues and those that might be reflective of the current limitations of the technology. This need for training is not unique to ASR. Similar recommendations are made by O'Neill ([<reflink idref="bib32" id="ref74">32</reflink>]) regarding the integration of automatic translation tools into L2 teaching.</p> <p>Teachers planning to use ASR in foreign language classes could develop training materials to help learners understand the tool's strengths and limitations. A useful starting point could be introducing homophones using examples from learners' L1, such as <emph>there/their</emph> for English speakers, to illustrate how ASR determines spelling based on context rather than pronunciation. This activity can be extended to the target language, focusing on common challenges, like plural markings (e.g., <emph>garçon</emph> vs. <emph>garçons</emph>) or verb endings in French (<emph>parler</emph>, <emph>parlez</emph>, <emph>parlé</emph>, <emph>parlais</emph>), with instructors preparing in advance to identify language‐specific issues. Such scaffolding can potentially enhance ASR's value as a tool for improving L2 pronunciation.</p> <hd id="AN0186745419-29">Limitations and future research</hd> <p>As with any study, the current study has a number of limitations. As discussed above, the difference in the tasks completed by naïve listeners recruited from the general population and expert listeners with extensive teaching experience did not allow for a direct comparison between these two groups. In future studies, administering the same task (transcription or rating task) to all participant groups could allow for more direct comparisons based on listeners' experience with L2 speech. Second, the current study did not include a measure of L2 proficiency, a factor that might influence ASR transcription accuracy. Third, no phonemic analysis of the speech samples was conducted, as it was done, for example, by Trofimovich and Isaacs ([<reflink idref="bib38" id="ref75">38</reflink>]). Thus, I was only able to compare the results of the different tasks with each other, without knowing whether the transcriptions reflected actual pronunciation errors.</p> <p>The current study also raises some interesting questions that can be pursued further. For example, future studies could compare accuracy rates across different ASR engines. While this was beyond the scope of the current project, such research would be theoretically valuable and particularly beneficial for L2 instructors. Certainly, conducting studies with learners of various L2s could further our understanding of how the technology performs cross‐linguistically. For some languages, for instance, a language with opaque sound‐to‐letter correspondences like Italian, matching ASR transcription with the original text might be a better approximation of learners' pronunciation accuracy, whereas in French, some mismatches between the transcription and the original can be the result of grammatical or syntactic constraints. It is also crucial to understand how different types of speech, such as reading versus spontaneous conversation, affect ASR transcription accuracy for L2 speakers, as L2 speech often includes more lexical and grammatical errors than L1 speech. These errors, that can influence human comprehensibility (Derwing & Munro, [<reflink idref="bib7" id="ref76">7</reflink>]; Huensch & Nagle, [<reflink idref="bib11" id="ref77">11</reflink>]; Munro & Derwing, [<reflink idref="bib28" id="ref78">28</reflink>]), may have an even greater impact on ASR, as modern engines rely on contextual information to interpret speech. Thus, future research could explore ASR accuracy with spontaneous L2 speech and compare it to results from more controlled tasks, such as reading, to better understand these dynamics.</p> <hd id="AN0186745419-30">CONCLUSION</hd> <p>This study examined the transcription accuracy of recordings by 15 L2 French speakers using three ASR engines, naïve human transcribers, and expert raters, with scores based on exact matches and phonemic equivalence to the original texts. The results show that even though the accuracy of human transcription was higher than that of ASR, all three measures were correlated. These findings suggest that ASR transcription can provide a good approximation of intelligibility but also underscore the importance of training students in interpreting ASR output to add value to their learning and to build students' confidence.</p> <hd id="AN0186745419-31">ACKNOWLEDGMENTS</hd> <p>I am very grateful to the two raters whose work made this study possible.</p> <p>GRAPH: Supportinh material</p> <ref id="AN0186745419-32"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref19" type="bt">1</bibl> <bibtext> ACTFL. (2012). ACTFL proficiency guidelines 2012. https://<ulink href="http://www.actfl.org/uploads/files/general/ACTFLProficiencyGuidelines2012.pdf">www.actfl.org/uploads/files/general/ACTFLProficiencyGuidelines2012.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib2" idref="ref10" type="bt">2</bibl> <bibtext> Blattner, G., Dalola, A., Roulon, S. (2023). #OnYGo. https://onygofrench.com</bibtext> </blist> <blist> <bibl id="bib3" idref="ref62" type="bt">3</bibl> <bibtext> Bates, D., Maechler, M., Bolker, B., Walker, S., Christensen, R. H. B., Singmann, H., Dai, B., Scheipl, F., & Grothendieck, G. (2015). Package 'lme4'. R Foundation for Statistical Computing, Vienna, Austria.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref30" type="bt">4</bibl> <bibtext> Bergeron, A., & Trofimovich, P. (2017). Linguistic dimensions of accentedness and comprehensibility: Exploring task and listener effects in second language French. Foreign Language Annals, 50 (3), 547 – 566. https://doi.org/10.1111/flan.12285</bibtext> </blist> <blist> <bibl id="bib5" idref="ref1" type="bt">5</bibl> <bibtext> Chen, W.‐H., Inceoglu, S., & Lim, H. (2020). Using ASR to improve Taiwanese EFL learners' pronunciation: Learning outcomes and learners' perceptions. In O. Kang, S. Staples, K. Yaw, & K. Hirschi (Eds.), Proceedings of the 11th Pronunciation in Second Language Learning and Teaching Conference (pp. 37 – 48). Northern Arizona University.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref23" type="bt">6</bibl> <bibtext> Deng, L., & Huang, X. (2004). Challenges in adopting speech recognition. Communications of the ACM, 47 (1), 69 – 75. https://doi.org/10.1145/962081.962108</bibtext> </blist> <blist> <bibl id="bib7" idref="ref31" type="bt">7</bibl> <bibtext> Derwing, T. M., & Munro, M. J. (1997). Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19 (1), 1 – 16. https://doi.org/10.1017/S0272263197001010</bibtext> </blist> <blist> <bibl id="bib8" idref="ref52" type="bt">8</bibl> <bibtext> Hannahs, S. J. (2008). French phonology and L2 acquisition. In D. Ayoun (Ed.), French applied linguistics (pp. 50 – 74). John Benjamins Publishing Company. https://doi.org/10.1075/lllt.16.07han</bibtext> </blist> <blist> <bibl id="bib9" idref="ref45" type="bt">9</bibl> <bibtext> Huang, B. H. (2013). The effects of accent familiarity and language teaching experience on raters' judgments of non‐native speech. System, 41 (3), 770 – 785. https://doi.org/10.1016/j.system.2013.07.009</bibtext> </blist> <blist> <bibtext> Huang, X. (2017). Microsoft researchers achieve new conversational speech recognition milestone. Microsoft Research Blog. https://<ulink href="http://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/">www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/</ulink></bibtext> </blist> <blist> <bibtext> Huensch, A., & Nagle, C. (2021). The effect of speaker proficiency on intelligibility, comprehensibility, and accentedness in L2 Spanish: A conceptual replication and extension of Munro and Derwing (1995a). Language Learning, 71 (3), 626 – 668. https://doi.org/10.1111/lang.12451</bibtext> </blist> <blist> <bibtext> Inceoglu, S., Chen, W. H., & Lim, H. (2023). Assessment of L2 intelligibility: Comparing L1 listeners and automatic speech recognition. ReCALL, 35 (1), 89 – 104. https://doi.org/10.1017/S0958344022000192</bibtext> </blist> <blist> <bibtext> Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10 (2), 135 – 159. https://doi.org/10.1080/15434303.2013.769545</bibtext> </blist> <blist> <bibtext> Jurafsky, D., & Martin, J. H. (2024). Speech and Language Processing. Third Edition Draft. https://web.stanford.edu/~jurafsky/slp3/</bibtext> </blist> <blist> <bibtext> Kamoun, C., & Ripaud, D. (2016). Phonétique essentielle du français (A1/A2). Didier.</bibtext> </blist> <blist> <bibtext> Kennedy, S., & Trofimovich, P. (2008). Intelligibility, comprehensibility, and accentedness of L2 speech: The role of listener experience and semantic context. The Canadian Modern Language Review, 64 (3), 459 – 489. https://doi.org/10.3138/cmlr.64.3.459</bibtext> </blist> <blist> <bibtext> Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15 (2), 155 – 163. https://doi.org/10.1016/j.jcm.2016.02.012</bibtext> </blist> <blist> <bibtext> Kornder, L., & Mennen, I. (2021). Listeners' linguistic experience affects the degree of perceived nativeness of first language pronunciation. Frontiers in Psychology, 12, 717615. https://doi.org/10.3389/fpsyg.2021.717615</bibtext> </blist> <blist> <bibtext> Levis, J. M. (2005). Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 39 (3), 369 – 377. https://doi.org/10.2307/3588485</bibtext> </blist> <blist> <bibtext> Liakin, D., Cardoso, W., & Liakina, N. (2013). Mobile speech recognition software: A tool for teaching second language pronunciation. OLBI Working Papers, 5, 85 – 99. https://doi.org/10.18192/olbiwp.v5i0.1120</bibtext> </blist> <blist> <bibtext> Liakin, D., Cardoso, W., & Liakina, N. (2015). Learning L2 pronunciation with a mobile speech recognizer: French/y/. Calico Journal, 32 (1), 1 – 25. https://doi.org/10.1558/cj.v32i1.25962</bibtext> </blist> <blist> <bibtext> Liakin, D., Cardoso, W., & Liakina, N. (2017). Mobilizing instruction in a second‐language context: learners' perceptions of two speech technologies. Languages, 2 (3), 11. https://doi.org/10.3390/languages2030011</bibtext> </blist> <blist> <bibtext> McCrocklin, S.M. (2016). Pronunciation learner autonomy: The potential of automatic speech recognition. System, 57, 25 – 42. https://doi.org/10.1016/j.system.2015.12.013</bibtext> </blist> <blist> <bibtext> McCrocklin, S. (2019). ASR‐based dictation practice for second language pronunciation improvement. Journal of Second Language Pronunciation, 5 (1), 98 – 118. https://doi.org/10.1075/jslp.16034.mcc</bibtext> </blist> <blist> <bibtext> McCrocklin, S., & Edalatishams, I. (2020). Revisiting popular speech recognition software for ESL speech. TESOL Quarterly, 54 (4), 1086 – 1097. https://doi.org/10.1002/tesq.3006</bibtext> </blist> <blist> <bibtext> Mroz, A. (2018). Seeing how people hear you: French learners experiencing intelligibility through automatic speech recognition. Foreign Language Annals, 51 (3), 617 – 637. https://doi.org/10.1111/flan.12348</bibtext> </blist> <blist> <bibtext> Mroz, A. (2020). Aiming for advanced intelligibility and proficiency using mobile ASR. Journal of Second Language Pronunciation, 6 (1), 12 – 38. https://doi.org/10.1075/jslp.18030.mro</bibtext> </blist> <blist> <bibtext> Munro, M. J., & Derwing, T. M. (1995). Foreign accent, comprehensibility, and intelligibility in the speech of second language learners. Language Learning, 45 (1), 73 – 97. https://doi.org/10.1111/j.1467-1770.1995.tb00963.x</bibtext> </blist> <blist> <bibtext> Munro, M. J., & Derwing, T. M. (1999). Foreign accent, comprehensibility, and intelligibility in the speech of second language learners. Language Learning, 49, 285 – 310. https://doi.org/10.1111/j.1467-1770.1995.tb00963.x</bibtext> </blist> <blist> <bibtext> Mussar, R., Sénéchal, M., & Rey, V. (2020). The development of morphological knowledge and spelling in French. Frontiers in Psychology, 11, 146. https://doi.org/10.3389/fpsyg.2020.00146</bibtext> </blist> <blist> <bibtext> Nagle, C. L., Trofimovich, P., Grantham O'brien, M., & Kennedy, S. (2022). Comprehensible to whom? Examining rater, speaker, and interlocutor perspectives on comprehensibility in an interactive context. The Modern Language Journal, 106 (4), 675 – 693. https://doi.org/10.1111/modl.12809</bibtext> </blist> <blist> <bibtext> O'Neill, E. M. (2019). Online translator, dictionary, and search engine use among L2 students. CALL‐EJ: Computer‐Assisted Language Learning–Electronic Journal, 20 (1), 154 – 177.</bibtext> </blist> <blist> <bibtext> R Core Team. (2020). R: A language and environment for statistical computing, Vienna, Austria. https://<ulink href="http://www.R-project.org/">www.R-project.org/</ulink></bibtext> </blist> <blist> <bibtext> Revelle, W. (2024). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois. R package version 2.4.6, https://CRAN.R-project.org/package=psych</bibtext> </blist> <blist> <bibtext> Shadiev, R., & Liu, J. (2023). Review of research on applications of speech recognition technology to assist language learning. ReCALL, 35 (1), 74 – 88. https://doi.org/10.1017/S095834402200012X</bibtext> </blist> <blist> <bibtext> Singmann, H., Bolker, B., Wesfall, J., & Aust, F. (2016). Afex: Analysis of factorial experiments. R package version 0.16‐1. Available online at https://CRAN.R-project.org/package=afex</bibtext> </blist> <blist> <bibtext> Thymé‐Gobbel, A., & Jankowski, C. (2021). Using context and data to create smarter conversations. In Mastering Voice Interfaces: Creating Great Voice Apps for Real Users (pp. 495 – 544). Apress. https://doi.org/10.1007/978-1-4842-7005-9_14</bibtext> </blist> <blist> <bibtext> Trofimovich, P., & Isaacs, T. (2012). Disentangling accent from comprehensibility. Bilingualism: Language and Cognition, 15 (4), 905 – 916. https://doi.org/10.1017/S1366728912000168</bibtext> </blist> <blist> <bibtext> Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2016). Achieving human parity in conversational speech recognition. arXiv preprint arXiv:1610.05256. https://doi.org/10.48550/arXiv.1610.05256</bibtext> </blist> </ref> <aug> <p>By Elena Shimanskaya</p> <p>Reported by Author</p> </aug> <nolink nlid="nl1" bibid="bib12" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib20" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib21" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib22" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib23" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib24" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib26" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib27" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib39" firstref="ref12"></nolink> <nolink nlid="nl10" bibid="bib37" firstref="ref13"></nolink> <nolink nlid="nl11" bibid="bib14" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib16" firstref="ref18"></nolink> <nolink nlid="nl13" bibid="bib10" firstref="ref22"></nolink> <nolink nlid="nl14" bibid="bib19" firstref="ref27"></nolink> <nolink nlid="nl15" bibid="bib28" firstref="ref29"></nolink> <nolink nlid="nl16" bibid="bib11" firstref="ref32"></nolink> <nolink nlid="nl17" bibid="bib29" firstref="ref33"></nolink> <nolink nlid="nl18" bibid="bib31" firstref="ref34"></nolink> <nolink nlid="nl19" bibid="bib38" firstref="ref41"></nolink> <nolink nlid="nl20" bibid="bib13" firstref="ref43"></nolink> <nolink nlid="nl21" bibid="bib18" firstref="ref47"></nolink> <nolink nlid="nl22" bibid="bib25" firstref="ref49"></nolink> <nolink nlid="nl23" bibid="bib35" firstref="ref50"></nolink> <nolink nlid="nl24" bibid="bib30" firstref="ref51"></nolink> <nolink nlid="nl25" bibid="bib15" firstref="ref57"></nolink> <nolink nlid="nl26" bibid="bib17" firstref="ref61"></nolink> <nolink nlid="nl27" bibid="bib34" firstref="ref63"></nolink> <nolink nlid="nl28" bibid="bib36" firstref="ref64"></nolink> <nolink nlid="nl29" bibid="bib33" firstref="ref65"></nolink> <nolink nlid="nl30" bibid="bib32" firstref="ref74"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1477307
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Comparing L2 Intelligibility for Learners of French: Automatic Speech Recognition versus Human Listeners
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Elena+Shimanskaya%22">Elena Shimanskaya</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4534-6772">0000-0002-4534-6772</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Foreign+Language+Annals%22"><i>Foreign Language Annals</i></searchLink>. 2025 58(2):438-457.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 20
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22French%22">French</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Instruction%22">Second Language Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Intelligibility%22">Intelligibility</searchLink><br /><searchLink fieldCode="DE" term="%22Correlation%22">Correlation</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Integration%22">Technology Integration</searchLink><br /><searchLink fieldCode="DE" term="%22Native+Language%22">Native Language</searchLink><br /><searchLink fieldCode="DE" term="%22Audio+Equipment%22">Audio Equipment</searchLink><br /><searchLink fieldCode="DE" term="%22Speech+Communication%22">Speech Communication</searchLink><br /><searchLink fieldCode="DE" term="%22Transcripts+%28Written+Records%29%22">Transcripts (Written Records)</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Oral+Reading%22">Oral Reading</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Teachers%22">Language Teachers</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/flan.12807
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0015-718X<br />1944-9720
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In this study, I compare the accuracy of automatic speech recognition (ASR) transcription against two measures of intelligibility provided by human listeners. The data came from readings of five texts recorded by 15 language learners of French. Human understanding was gauged by (i) asking a group of 36 naïve first language (L1) speakers of French recruited from the general population to provide word-for-word transcriptions of the recordings, and (ii) asking two experienced French teachers (L1 speakers of French) to rate the recordings for intelligibility. The results reveal that ASR transcription accuracy was lower than that of human transcribers. However, all three measures, namely ASR transcription accuracy, human transcription accuracy, and human ratings of intelligibility, were correlated. Implications of these results for using the technology in foreign language teaching and learning are considered in the discussion.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1477307
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1477307
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/flan.12807
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 20
        StartPage: 438
    Subjects:
      – SubjectFull: Comparative Analysis
        Type: general
      – SubjectFull: French
        Type: general
      – SubjectFull: Second Language Learning
        Type: general
      – SubjectFull: Second Language Instruction
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Intelligibility
        Type: general
      – SubjectFull: Correlation
        Type: general
      – SubjectFull: Technology Integration
        Type: general
      – SubjectFull: Native Language
        Type: general
      – SubjectFull: Audio Equipment
        Type: general
      – SubjectFull: Speech Communication
        Type: general
      – SubjectFull: Transcripts (Written Records)
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Oral Reading
        Type: general
      – SubjectFull: Language Teachers
        Type: general
    Titles:
      – TitleFull: Comparing L2 Intelligibility for Learners of French: Automatic Speech Recognition versus Human Listeners
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Elena Shimanskaya
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0015-718X
            – Type: issn-electronic
              Value: 1944-9720
          Numbering:
            – Type: volume
              Value: 58
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Foreign Language Annals
              Type: main
ResultId 1