The Measurement of Implicit and Explicit Knowledge
Saved in:
| Title: | The Measurement of Implicit and Explicit Knowledge |
|---|---|
| Language: | English |
| Authors: | Ellis, Rod, Roever, Carsten |
| Source: | Language Learning Journal. 2021 49(2):160-175. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 16 |
| Publication Date: | 2021 |
| Document Type: | Journal Articles Information Analyses |
| Descriptors: | Grammar, Second Language Learning, Metalinguistics, Psycholinguistics, Language Tests, Oral Language, Knowledge Level, Comparative Analysis, Taxonomy, Decision Making, Criticism, Test Content, Test Construction, Pragmatics, Language Processing, Factor Analysis, Error Correction, Imitation, Test Format |
| DOI: | 10.1080/09571736.2018.1504229 |
| ISSN: | 0957-1736 |
| Abstract: | This article presents a review of research that has investigated ways of measuring implicit and explicit knowledge of a second language (L2), focusing on grammar. It begins by defining implicit and explicit knowledge in terms of a distinguishing set of criteria. Two ways of investigating implicit knowledge are discussed -- through experimental studies of implicit learning and by means of factor-analytic studies. This provides the basis for a taxonomy of tests designed to provide separate measures of the two types of knowledge. Proposals for oral production tests, comprehension tests, judgements tests, tests of metalinguistic knowledge and tests derived from psycholinguistic research are examined and critiqued. The article concludes by suggesting there is a need for tests of L2 pragmatic knowledge to complement those available for grammar and offers a number of suggestions for the design of such tests. |
| Abstractor: | As Provided |
| Entry Date: | 2021 |
| Accession Number: | EJ1292126 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwH-BzHz6A8JFWc6aiEaSetKAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDIYfXE1S6EAafnouGgIBEICBm2jQQEJS7avpULkefp0QOtTAcIwXpPoDHXuA60nD3lGBSJzzN4V5HiGsLVOxM5mbU2wN6PpMJdHsPe6SV7d6Fsg8zCd5hc70bm_4np9HNUdFw7FKhwFE78g2GjVM-3V9SkcCzQztfuPbNHGWnARWP14YUGxiqAgYVi8glkZwX6ZD3pCeGwNs3wMtYoQCMlv4UNQPppTBiCLmFggK Text: Availability: 1 Value: <anid>AN0149596508;sdq01apr.21;2021Apr03.03:09;v2.2.500</anid> <title id="AN0149596508-1">The measurement of implicit and explicit knowledge </title> <p>This article presents a review of research that has investigated ways of measuring implicit and explicit knowledge of a second language (L2), focusing on grammar. It begins by defining implicit and explicit knowledge in terms of a distinguishing set of criteria. Two ways of investigating implicit knowledge are discussed – through experimental studies of implicit learning and by means of factor-analytic studies. This provides the basis for a taxonomy of tests designed to provide separate measures of the two types of knowledge. Proposals for oral production tests, comprehension tests, judgements tests, tests of metalinguistic knowledge and tests derived from psycholinguistic research are examined and critiqued. The article concludes by suggesting there is a need for tests of L2 pragmatic knowledge to complement those available for grammar and offers a number of suggestions for the design of such tests.</p> <p>Keywords: Measuring implicit/explicit L2 knowledge; grammar; pragmatics</p> <hd id="AN0149596508-2">Introduction</hd> <p>A prevailing problem facing SLA researchers is how to obtain evidence that a person's knowledge of a second language is of the implicit rather than explicit kind. It is not possible to examine implicit knowledge directly as there is no direct window into people's minds to see how their knowledge is represented or what kind of knowledge they utilise when they perform a task (although advances in neurolinguistics show promise in this direction). Instead, inferences about the type of knowledge involved have to be drawn by examining a person's linguistic behaviour. This requires making predictions about what kinds of behaviour are most likely to constitute evidence of a learner's implicit and explicit knowledge and then developing validity arguments to support the theoretical premises of these predictions.</p> <p>The measurement of implicit knowledge is of importance for at least three areas of enquiry. SLA researchers (e.g. Williams [<reflink idref="bib40" id="ref1">40</reflink>]; Godfroid [<reflink idref="bib18" id="ref2">18</reflink>]) are interested in establishing to what extent learners (especially adult learners) are capable of implicit learning (i.e. learning without intention and awareness) and to this end they need to be able to assess whether the learning that results from exposure to specific linguistic features results in implicit knowledge. Language instructors are also keen to know whether instruction directed at specific linguistic features leads to implicit or just explicit knowledge of these features and also whether some types of instruction are more likely to achieve this than others. Finally, language testers may also wish to know what aspects of a learner's language proficiency their tests measure. In short, understanding how best to measure implicit knowledge is of enormous significance to the central areas of enquiry in applied linguistics.</p> <p>My goal in this article is to examine the various tests and the measures derived from them that have been used by researchers to assess the extent to which L2 learners' possess implicit knowledge of a target language. Given that the research to date has focused more or less exclusively on grammar, I will restrict my review to tests of grammatical knowledge. However, I will also point to ways (and difficulties) of utilising similar tests for measuring learners' pragmatic knowledge. I begin with a definition of implicit knowledge.</p> <hd id="AN0149596508-3">Defining implicit knowledge (Ellis 2005)</hd> <p>Definitions of implicit knowledge are based on identifying the characteristics that distinguish implicit from explicit knowledge. In Ellis ([<reflink idref="bib12" id="ref3">12</reflink>], [<reflink idref="bib13" id="ref4">13</reflink>]), I proposed seven key characteristics that differentiate the two types of knowledge. Table 1 from Ellis ([<reflink idref="bib12" id="ref5">12</reflink>]) provides a summary of the assumptions I made. Of these characteristics, some are clearly more central to defining implicit knowledge than others and thus of greater importance for designing tests. The key characteristics are (<reflink idref="bib1" id="ref6">1</reflink>) awareness, (<reflink idref="bib2" id="ref7">2</reflink>) accessibility and (<reflink idref="bib3" id="ref8">3</reflink>) self-report. That is, implicit knowledge is most clearly defined as knowledge that the learner has no subjective awareness of, can access for spontaneous language use through automatic processing, and is unable to verbalise.</p> <p>Table 1. Key characteristics of implicit and explicit knowledge.</p> <p> <ephtml> &lt;table&gt;&lt;thead valign="bottom"&gt;&lt;tr&gt;&lt;td&gt;Characteristics&lt;/td&gt;&lt;td&gt;Implicit knowledge&lt;/td&gt;&lt;td&gt;Explicit knowledge&lt;/td&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Awareness&lt;/td&gt;&lt;td&gt;Intuitive awareness of linguistic norms&lt;/td&gt;&lt;td&gt;Conscious awareness of linguistic norms&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Type of knowledge&lt;/td&gt;&lt;td&gt;Procedural knowledge of rules and fragments&lt;/td&gt;&lt;td&gt;Declarative knowledge of grammatical rules and fragments&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Systematicity&lt;/td&gt;&lt;td&gt;Variable but systematic knowledge&lt;/td&gt;&lt;td&gt;Anomalous and inconsistent knowledge&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Accessibility&lt;/td&gt;&lt;td&gt;Access to knowledge by means of automatic processing&lt;/td&gt;&lt;td&gt;Access to knowledge by means of controlled processing&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Use of L2 knowledge&lt;/td&gt;&lt;td&gt;Access to knowledge during fluent performance&lt;/td&gt;&lt;td&gt;Access to knowledge during planning difficulty&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Self-report&lt;/td&gt;&lt;td&gt;Non-verbalisable&lt;/td&gt;&lt;td&gt;Verbalisable&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>Some researchers (e.g. Dienes and Perner [<reflink idref="bib7" id="ref9">7</reflink>]) have viewed the distinction between implicit and explicit knowledge as continuous rather than dichotomous. In Ellis ([<reflink idref="bib11" id="ref10">11</reflink>]), however, I rejected this position arguing that from a connectionist account of language implicit knowledge consists of an elaborated, statistically determined network of weighted associations, making it difficult to see how knowledge could be more or less implicit/explicit. Neurolinguistic studies also point to a clear separation of the two types of knowledge (see Paradis [<reflink idref="bib29" id="ref11">29</reflink>]; Ullman [<reflink idref="bib38" id="ref12">38</reflink>]).</p> <p>There is, however, another possibility that carries greater conviction. This disputes characteristic (<reflink idref="bib2" id="ref13">2</reflink>) by proposing that accessibility may not clearly distinguish the two types of knowledge. I and other researchers have argued that automatic processing is a defining characteristic of implicit knowledge and, therefore, that any language behaviour involving automatic processing affords evidence of the learner's implicit knowledge. This assumption has underscored the development of tests of implicit knowledge as reported in a number of studies (e.g. Ellis [<reflink idref="bib12" id="ref14">12</reflink>], [<reflink idref="bib13" id="ref15">13</reflink>]; Gutiérrez [<reflink idref="bib20" id="ref16">20</reflink>]; Spada, Shiu and Tomita [<reflink idref="bib34" id="ref17">34</reflink>]; Zhang [<reflink idref="bib41" id="ref18">41</reflink>]). However, it has also been challenged by DeKeyser ([<reflink idref="bib6" id="ref19">6</reflink>]), who pointed out that explicit knowledge can be proceduralised and automatised through practice allowing for its use spontaneously. Suzuki and DeKeyser ([<reflink idref="bib36" id="ref20">36</reflink>]) suggested that a distinction needs to be made between 'explicit knowledge' and 'automatic explicit knowledge' and that the latter is functionally equivalent to implicit knowledge although still distinct from it. From this perspective, then, time-pressured tests cannot provide indisputable evidence of implicit knowledge as learners may be able to utilise their automatised explicit knowledge. In other words, to test implicit knowledge it is necessary to base measurement on one or both of the two other defining characteristics of implicit knowledge – awareness and self-report. Later we will consider studies that have attempted this.</p> <p>Discussions of implicit knowledge have focused on grammar (and to a lesser extent phonology and vocabulary). There is an most complete absence of any consideration of implicit pragmatic knowledge probably because the study of interlanguage pragmatics has been more concerned with the social and sociolinguistics dimensions of language use than the psycholinguistic. An exception is Taguchi ([<reflink idref="bib37" id="ref21">37</reflink>]) who drew on Bialystok's two dimensional model of language proficiency to distinguish what she called <emph>pragmatic knowledge</emph> (i.e. the ability to comprehend and produce speech intentions) and <emph>processing fluency</emph> in comprehension and production. In other words, Taguchi proposed distinguishing pragmatic knowledge in terms of learners accessibility to the linguistic forms needed to decode and encode pragmatic intentions. Although Taguchi did not use the terms implicit and explicit knowledge it is clear that her model of pragmatic knowledge addresses this distinction. The model, however, relies solely on the accessibility criterion, which as noted above may not be sufficient. However, the other two characteristics are equally applicable to pragmatic knowledge; learners may or may not be aware of the social meanings they convey through their pragmalinguistic choices and they may or may not be able to verbalise the choices they made.</p> <p>Whether measuring grammatical or pragmatic knowledge, developing tests that distinguish implicit and explicit knowledge is difficult. As de Jong ([<reflink idref="bib4" id="ref22">4</reflink>]) noted:</p> <p>Testing whether learning is implicit or explicit is very difficult, because there are no clear boundaries between implicit and explicit processes and nearly all cognitive processes have both implicit and explicit aspects. This means that implicit learning should not be ruled out as soon as awareness has been established, nor should implicit learning only be assumed when there is no awareness at all of the learning process or product. The same argument holds for implicit and explicit knowledge, which can (and often do) co-exist and operate simultaneously. (p. 7)</p> <p>Most L2 learners possess both types of knowledge so determining which type of knowledge they deploy on particular occasions is problematic. At best, tests can only hope to bias learners to the use of one or the other type of knowledge.</p> <hd id="AN0149596508-4">Two approaches for investigating tests of implicit knowledge</hd> <p>Some of the most interesting work on measuring implicit knowledge comes from studies that have investigated whether learning in the absence of conscious awareness is possible for adults (e.g. Williams [<reflink idref="bib40" id="ref23">40</reflink>]; Rebuschat et al. [<reflink idref="bib32" id="ref24">32</reflink>]; Godfroid [<reflink idref="bib18" id="ref25">18</reflink>]; Kerz, Wiechmann and Riedel [<reflink idref="bib21" id="ref26">21</reflink>]). To demonstrate whether implicit learning is possible it is necessary to test whether the learning that takes place results in implicit or explicit knowledge. The second approach involves administering a battery of tests that have been theorised to afford relatively separate measurers of implicit and explicit knowledge and investigating whether they in fact do so by factor-analysing scores from the test (e.g. Ellis [<reflink idref="bib12" id="ref27">12</reflink>]; Zhang [<reflink idref="bib41" id="ref28">41</reflink>]; Kim and Nam [<reflink idref="bib22" id="ref29">22</reflink>]; Suzuki [<reflink idref="bib35" id="ref30">35</reflink>]). The aim here is to show that scores from the tests load on distinctive factors as predicted.</p> <hd id="AN0149596508-5">Studies of implicit learning</hd> <p>Researchers interested in investigating implicit language learning have adopted an experimental approach often involving an artificial language to ensure that the learners had no prior knowledge of a target feature. Learners are first exposed to multiple exemplars of a specific grammatical structure in a training phase of a study. This is followed by a testing phase. A variety of test types have been used (e.g. a grammaticality judgement test (GJT), forced-choice test and a fill-in-the gap test). By themselves, these tests are not able to show whether learners have acquired explicit or implicit knowledge. But in addition, information is collected through self-report to determine whether the learners had consciously registered (i.e. become aware of) the target structure during the training or testing phases of the study. For example, Rebuschat et al. ([<reflink idref="bib32" id="ref31">32</reflink>]) asked the participants in their study to complete a forced-choice test production test and to provide confidence ratings and source attributions for responses to each item in the test. The participants indicated how confident they were about their responses by choosing from four options (not confident at all – just guessing; somewhat confident, very confident, 100% confident). Source attributions were gathered by requiring the participants to state whether they guessed, relied on intuition, memory or rule knowledge. These two kinds of self-report provided data about the participants' judgement knowledge and structural knowledge. Evidence for implicit learning was held to have occurred when the participants indicated no confidence in the choices they made in the test and were just guessing but nevertheless scored above chance.</p> <p>This approach suffers from a number of problems especially if the tests are of the kind that are likely to tap explicit knowledge. It relies on participants reporting their confidence levels and source attributions honestly. But there is no way of telling if a test-taker based confidence ratings on his/her implicit or explicit knowledge. Similarly, a test-taker may report he/she relied on intuition but in fact may have also referred to a rule. As Rebuschat ([<reflink idref="bib31" id="ref32">31</reflink>]) pointed out, learners often possess both explicit and implicit knowledge of the same grammatical feature and may utilise one or the other or both when subjectively rating test items. Nevertheless, confidence ratings and source attributions can help to check whether tests designed to assess implicit or explicit knowledge actually functioned as intended.</p> <hd id="AN0149596508-6">Factor-analytic studies</hd> <p>The first of a series of factor-analytic studies investigating the assessment of implicit/explicit was Ellis ([<reflink idref="bib12" id="ref33">12</reflink>]). This study set out to validate a set of tests designed to provide relatively separate measures of implicit and explicit knowledge of language. The tests focused on knowledge of 17 English grammatical structures selected to represent different levels of learning difficulty and to include both morphological features (e.g. 3rd person -s) and syntactic structures (e.g. dative alternation). The tests differed in terms of four criteria that were theorised to distinguish the two types of knowledge: (<reflink idref="bib1" id="ref34">1</reflink>) degree of awareness, (<reflink idref="bib2" id="ref35">2</reflink>) time available, (<reflink idref="bib3" id="ref36">3</reflink>) focus of attention and (<reflink idref="bib4" id="ref37">4</reflink>) utility of meta-language. Table 2 shows how the tests in the battery mapped onto these four design features. Three tests (an oral elicited imitation test (EIT), an oral production test and a timed GJT) were hypothesised to measure implicit knowledge and an untimed GJT and metalinguistic knowledge test (MKT) to measure explicit knowledge. The tests were administered to a sample of mixed proficiency adults ESL learners in New Zealand. An exploratory factor analysis produced two factors with scores from the two sets of tests loading more or less as hypothesised. A subsequent confirmatory factor analysis (Loewen and Ellis [<reflink idref="bib24" id="ref38">24</reflink>]) tested a model based on the implicit/explicit distinction and a second model based on the oral/written distinction and showed that only the former constituted a satisfactory fit.[<reflink idref="bib1" id="ref39">1</reflink>]</p> <p>Table 2. Design features of the tests in the test battery (from Ellis et al. [<reflink idref="bib14" id="ref40">14</reflink>]: 47).</p> <p> <ephtml> &lt;table&gt;&lt;thead valign="bottom"&gt;&lt;tr&gt;&lt;td&gt;Criteria&lt;/td&gt;&lt;td&gt;oral imitation&lt;/td&gt;&lt;td&gt;Oral narrative&lt;/td&gt;&lt;td&gt;Timed GJT&lt;/td&gt;&lt;td&gt;Untimed GJT&lt;/td&gt;&lt;td&gt;Meta-language&lt;/td&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Degree of awareness&lt;/td&gt;&lt;td&gt;Feel&lt;/td&gt;&lt;td&gt;Feel&lt;/td&gt;&lt;td&gt;Feel&lt;/td&gt;&lt;td&gt;Rule&lt;/td&gt;&lt;td&gt;Rule&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Time available&lt;/td&gt;&lt;td&gt;Pressured&lt;/td&gt;&lt;td&gt;Pressured&lt;/td&gt;&lt;td&gt;Pressured&lt;/td&gt;&lt;td&gt;Unpressured&lt;/td&gt;&lt;td&gt;Unpressured&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Focus of attention&lt;/td&gt;&lt;td&gt;Meaning&lt;/td&gt;&lt;td&gt;Meaning&lt;/td&gt;&lt;td&gt;Form&lt;/td&gt;&lt;td&gt;Form&lt;/td&gt;&lt;td&gt;Form&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Utility of knowledge of meta-language&lt;/td&gt;&lt;td&gt;No&lt;/td&gt;&lt;td&gt;No&lt;/td&gt;&lt;td&gt;No&lt;/td&gt;&lt;td&gt;Yes&lt;/td&gt;&lt;td&gt;Yes&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>Further studies by Bowles ([<reflink idref="bib2" id="ref41">2</reflink>]) and Zhang ([<reflink idref="bib41" id="ref42">41</reflink>]) on very different populations of learners (i.e. classroom and heritage learners of Spanish in the US in Bowles and Chinese university students in Zhang) have confirmed the results obtained in Loewen and Ellis ([<reflink idref="bib24" id="ref43">24</reflink>]). Other studies (e.g. Spada, Shiu and Tomita [<reflink idref="bib34" id="ref44">34</reflink>]; Suzuki and DeKeyser [<reflink idref="bib36" id="ref45">36</reflink>]), however, have produced different results, leading in particular to questions regarding the validity of the elicited imitation test as a measure of implicit knowledge (as opposed to a measure of automatised explicit knowledge) along with proposals for alternative tests of implicit knowledge borrowed from the psycholinguistic literature. The doubts regarding the validity of the elicited imitation test and also the timed GJT that were raised by these studies are considered below.</p> <p>The strength of the factor-analytic approach is that it serves to test theory-driven hypotheses about the type of knowledge that different tests measure. The starting point is the definition of implicit/explicit knowledge (see preceding section). This then serves to identify the key characteristics of tests hypothesised to provide relatively distinct measures of the two types of knowledge. The tests are designed accordingly, administered to a sample of L2 learners and the scores submitted to a confirmatory factor analysis to determine whether in fact the tests distinguish the knowledge types as intended. This approach, which to date has been used exclusively for tests of grammar, is the obvious way to tackle the development of tests of pragmatic knowledge.</p> <hd id="AN0149596508-7">A taxonomy of tests of implicit and explicit knowledge</hd> <p>There is now a range of tests that have designed to measure implicit and explicit knowledge. Table 3 lists the various tests along with key variants in the basic types. It also gives examples of studies that have investigated these types. These test types differ in a fundamental way. Whereas production and comprehension tests can claim to be authentic in the sense that they involve how language is used in normal communication, the other tests (the judgement, metalinguistic knowledge and psycholinguistic tests) are clearly lacking in authenticity as language users do not normally have to judge the grammaticality of sentences, demonstrate knowledge of meta-language or read sentences as they appear word by word on a computer screen in the course of their everyday use of language. The authenticity of the tests may not be important if the purpose is to investigate theoretical issues about language acquisition in experimental research but face validity does become important if the tests are to serve as proficiency tests.</p> <p>Table 3. A taxonomy of tests of implicit/explicit knowledge.</p> <p> <ephtml> &lt;table&gt;&lt;thead valign="bottom"&gt;&lt;tr&gt;&lt;td&gt;Type of test&lt;/td&gt;&lt;td&gt;Versions&lt;/td&gt;&lt;td&gt;Example&lt;/td&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Oral production tests&lt;/td&gt;&lt;td&gt;&lt;list list-type="Bullet"&gt;&lt;list-item&gt;&lt;p&gt;Free production *&lt;/p&gt;&lt;/list-item&gt;&lt;list-item&gt;&lt;p&gt;Controlled production **&lt;/p&gt;&lt;/list-item&gt;&lt;list-item&gt;&lt;p&gt;Elicited imitation *&lt;/p&gt;&lt;/list-item&gt;&lt;list-item&gt;&lt;p&gt;Error correction **&lt;/p&gt;&lt;/list-item&gt;&lt;/list&gt;&lt;/td&gt;&lt;td&gt;Ellis (&lt;xref ref-type="bibr" rid="bibr12"&gt;2005&lt;/xref&gt;); Spada, Shiu and Tomita (&lt;xref ref-type="bibr" rid="bibr34"&gt;2015&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Macrory and Stone (&lt;xref ref-type="bibr" rid="bibr26"&gt;2000&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Ellis (&lt;xref ref-type="bibr" rid="bibr12"&gt;2005&lt;/xref&gt;); Zhang (&lt;xref ref-type="bibr" rid="bibr41"&gt;2015&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Spada, Shiu and Tomita (&lt;xref ref-type="bibr" rid="bibr34"&gt;2015&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Comprehension&lt;/td&gt;&lt;td&gt;&lt;list list-type="Bullet"&gt;&lt;list-item&gt;&lt;p&gt;Picture-matching listening test *&lt;/p&gt;&lt;/list-item&gt;&lt;/list&gt;&lt;/td&gt;&lt;td&gt;De Jong (&lt;xref ref-type="bibr" rid="bibr4"&gt;2005a&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Judgement tests&lt;/td&gt;&lt;td&gt;&lt;list list-type="Bullet"&gt;&lt;list-item&gt;&lt;p&gt;Timed * vs untimed **&lt;/p&gt;&lt;/list-item&gt;&lt;list-item&gt;&lt;p&gt;Aural * vs written **&lt;/p&gt;&lt;/list-item&gt;&lt;/list&gt;&lt;/td&gt;&lt;td&gt;Ellis (&lt;xref ref-type="bibr" rid="bibr12"&gt;2005&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Kim and Nam (&lt;xref ref-type="bibr" rid="bibr22"&gt;2017&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Metalinguistic knowledge tests&lt;/td&gt;&lt;td&gt;&lt;list list-type="Bullet"&gt;&lt;list-item&gt;&lt;p&gt;Receptive knowledge **&lt;/p&gt;&lt;/list-item&gt;&lt;list-item&gt;&lt;p&gt;Productive knowledge **&lt;/p&gt;&lt;/list-item&gt;&lt;/list&gt;&lt;/td&gt;&lt;td&gt;Ellis (&lt;xref ref-type="bibr" rid="bibr12"&gt;2005&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Psycholinguistic tests&lt;/td&gt;&lt;td&gt;&lt;list list-type="Bullet"&gt;&lt;list-item&gt;&lt;p&gt;Word monitoring test *&lt;/p&gt;&lt;/list-item&gt;&lt;list-item&gt;&lt;p&gt;Self-paced reading test *&lt;/p&gt;&lt;/list-item&gt;&lt;list-item&gt;&lt;p&gt;Visual word task *&lt;/p&gt;&lt;/list-item&gt;&lt;/list&gt;&lt;/td&gt;&lt;td&gt;Suzuki and DeKeyser (&lt;xref ref-type="bibr" rid="bibr36"&gt;2015&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Vafaee, Suzuki and Kachisnke (&lt;xref ref-type="bibr" rid="bibr39"&gt;2017&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Suzuki (&lt;xref ref-type="bibr" rid="bibr35"&gt;2017&lt;/xref&gt;)&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>The tests marked with a * in Table 3 were designed as potential measures of implicit knowledge while those marked with a ** were intended to measure explicit knowledge. Thus tests of implicit knowledge include free production, elicited imitation, picture-matching comprehension tests, timed and aural judgement tests and psycholinguistic tests. There are some notable differences in these tests. Some of them (free production, elicited imitation and picture-matching) stipulate that learners should focus on meaning rather than form. However, the judgement tests require learners to focus on form and attempt to elicit the use of implicit knowledge by creating processing pressure either through an aural presentation of the sentence stimuli or through time restrictions. However, as noted above, simply ensuring that test performance is pressured cannot guarantee that learners draw on their implicit knowledge as they may have been able to access automatised explicit knowledge. The psycholinguistic tests adopt a very different approach. They require a focus on the meaning of stimuli but measure implicit knowledge in terms of learners' sensitivity to grammatical violations through a comparison of response times to grammatical and ungrammatical sentences, assuming that learners will take longer to process ungrammatical than grammatical forms.</p> <p>In the sections that follow I will provide a commentary on each of the different tests and also consider to what extent it might be feasible to use each to assess implicit pragmatic knowledge.</p> <hd id="AN0149596508-8">Tests involving oral production</hd> <p></p> <hd id="AN0149596508-9">Free oral production tests</hd> <p>A key characteristic of a free production test is that learners are not made aware of what linguistic features the test has been designed to measure (i.e. learners are required to focus solely on meaning). A free oral production test also requires real-time language processing so learners need to draw on their automatised knowledge. Finally, learners are unlikely to draw on their knowledge of meta-language. In short, free production tests correspond quite closely to how language is used in everyday communication and satisfy Ellis' ([<reflink idref="bib12" id="ref46">12</reflink>]) four criteria for tests of implicit knowledge (see Table 2).</p> <p>Ellis ([<reflink idref="bib12" id="ref47">12</reflink>]) included an oral narrative task in his battery of tests. Learners were asked to read a story through twice and then retell it orally. To encourage them to speak spontaneously they were told they only had 3 minutes. Scores on eight grammar structures were calculated using obligatory occasion analysis. In the confirmatory factor analysis of the battery of tests, oral narrative scores loaded on the implicit factor but the loading was weaker than for both the EIT and the timed GJT, possibly because these other tests measured knowledge of all 18 grammatical structures. The oral narrative scores also correlated weakly but significantly with scores on the Untimed GJT and the MKT. Clearly, it was not a 'pure' measure of implicit knowledge.</p> <p>Bowles ([<reflink idref="bib2" id="ref48">2</reflink>]) study of L2 Spanish also reported that scores from the oral narrative test loaded on the implicit factor in a study that used the same range of tests as in Ellis ([<reflink idref="bib12" id="ref49">12</reflink>]). However, other follow-up factor-analytic studies (e.g. Suzuki and DeKeyser [<reflink idref="bib36" id="ref50">36</reflink>]; Zhang [<reflink idref="bib41" id="ref51">41</reflink>]) did not include a free oral production test in their test battery probably because administering such a test is very time-consuming and because of the difficulty of ensuring that production of the target structures is task-essential (see Loschky and Bley-Vroman [<reflink idref="bib25" id="ref52">25</reflink>]). Evidence of this latter problem comes from Spada, Shiu and Tomita ([<reflink idref="bib34" id="ref53">34</reflink>]) who included a picture-cued story-telling task. Even though learners were given the key words for each picture, they failed to consistently use the target structure (passive forms) and scores on this test were very low. Spada, Shiu and Tomita concluded that the 'specific design features of the oral production task exclude it as a reliable measure of implicit knowledge' ([<reflink idref="bib34" id="ref54">34</reflink>]: 740).</p> <p>Despite the problems with oral production tasks, they have high face validity as a language testing device. Also, as Ellis' and Bowles' studies showed, they have construct validity as tests of implicit knowledge. Clearly, though they do not prevent learners from accessing their explicit knowledge especially if this is of the automatised kind. Role-play tests – a popular way of testing pragmatic ability – have similar design features to the oral narrative tests in Ellis and Bowles studies. Perhaps, though, learners are even more likely in a role-play to draw on whatever explicit pragmatic knowledge they possess as the specification of the situation included in such a task sensitises learners to attend consciously to using language in a socially appropriate way.</p> <hd id="AN0149596508-10">Controlled production tests</hd> <p>Arguably the difference between a controlled and free production test is continuous rather than dichotomous – that is, tests can constrain the extent to which learners are directed to attend to specific grammatical forms to a greater or lesser extent. Spada, Shiu and Tomita's ([<reflink idref="bib34" id="ref55">34</reflink>]) picture-cued story-telling task lies somewhere in the middle of this continuum. Ellis' ([<reflink idref="bib12" id="ref56">12</reflink>]) Oral narrative task lies towards the free production end of the continuum.</p> <p>Tests at the controlled end of the continuum are what Rebuschat ([<reflink idref="bib31" id="ref57">31</reflink>]) called direct tests. That is they explicitly require students to make use of their grammatical knowledge to complete the test – for example, a fill-in-the gap test where learners are asked to complete sentences using specific linguistic forms. Kerz, Wiechmann and Riedel ([<reflink idref="bib21" id="ref58">21</reflink>]) used such a test to measure whether implicit learning had taken place but they also included confidence ratings to establish whether the zero-order correlation criterion (Dienes et al. [<reflink idref="bib8" id="ref59">8</reflink>]) had been met (see above). They also asked participants to report on whether they had identified the target feature and to describe what they had noticed. Without the subjective measures and self-report information such a test cannot be used to measure implicit knowledge. Godfroid ([<reflink idref="bib18" id="ref60">18</reflink>]) included a controlled production test involving pictures and word prompts but used it as a measure of explicit knowledge.</p> <p>A controlled production test encourages a high degree of linguistic awareness, a focus on form and the use of metalinguistic knowledge. As such it is more likely to tap explicit knowledge especially if there is no time pressure. If time pressure is exerted – for example, by presenting the stimuli aurally and by setting a time limit for responding to each stimulus – learners are perhaps more likely to draw on their implicit knowledge if they have it. However, such test conditions would not preclude the use of automatised explicit knowledge. Thus, a controlled production test cannot serve as a satisfactory measure of either type of knowledge.</p> <p>The discourse completion test (DCT), popular in interlanguage pragmatic research, is a controlled production test. The written form of this test most likely taps meta-pragmatic (i.e. explicit knowledge) as learners are encouraged to consciously think about what they would say in the situations given to them (see Golato [<reflink idref="bib19" id="ref61">19</reflink>]). However, introducing processing pressure by requiring an aural rather than a written response and by setting a time limit might show whether the learner has access to automatised knowledge (implicit or explicit). This is doubtful, though. Enochs and Yoshitake-Strain ([<reflink idref="bib15" id="ref62">15</reflink>]) factor-analysed a battery of tests of pragmatic competence, including a role play, a written DCT, an aural DCT and a multiple choice DCT. They found that scores all loaded on the same factor, suggesting that the tests were all measuring the same type of knowledge – probably meta-pragmatic.</p> <hd id="AN0149596508-11">Elicited imitation test</hd> <p>It is useful to distinguish two uses of this test. It has been used as a measure of global language proficiency or 'L2 processing efficiency' (Gaillard and Tremblay [<reflink idref="bib17" id="ref63">17</reflink>]), in which case the assumption is that test-takers draw on an amalgam of implicit and explicit knowledge. It has also been used in SLA research as a measure of implicit knowledge. Common to its use for both of these purposes, the test requires learners to listen to a set of sentences that are sufficiently long to prevent rote recall and then to reproduce then orally. Thus, the test involves both input- and output-processing – listening comprehension and oral production. However, measures derived from the test are based entirely on learner production. As there are differences in the specific features of tests intended to assess general language proficiency and implicit knowledge, I will consider them separately.</p> <p>As a test of global language proficiency, Ortega et al.'s ([<reflink idref="bib28" id="ref64">28</reflink>]) Spanish EIT has served as a model for the design of EITs for a variety of languages and for validation studies such as Gaillard and Tremblay ([<reflink idref="bib17" id="ref65">17</reflink>]) and Bowden ([<reflink idref="bib1" id="ref66">1</reflink>]). The EIT includes sentences that cover a wide variety of grammatical structures and differ in sentence length (from 7 to 17 syllables in Ortega et al.'s test). All the sentences are grammatical. Learners are given a fixed time to repeat each sentence (e.g. 2.5 seconds in Bowden's study) and are not required to perform any response other than repeating the sentences. Ortega ([<reflink idref="bib27" id="ref67">27</reflink>]) developed a rating scale for evaluating learners' performance on an EIT. This does not focus on the accuracy with which specific grammatical structures are performed but on the quantity and quality of the idea units that a learner includes in his/her repetition of a sentence (e.g. 'perfect repetition' = 4; 'more than half the content preserved; slight changes in content that make the content inexact' = 2; 'silence, unintelligible content, or only one content word' = 0). Studies have shown that the test has high reliability, is strongly correlated with other standardised measures of oral proficiency as well as with learners' self-assessment scores, and effectively discriminates between learners who vary in language learning experience (Bowden [<reflink idref="bib1" id="ref68">1</reflink>]). Some doubts about its practicality exist, however, as rating learners' performances on each sentence is time-consuming.</p> <p>The use of the EIT as a measure of implicit L2 knowledge began with Ellis' ([<reflink idref="bib12" id="ref69">12</reflink>]) study. Seventeen structures were embedded in sentences that expressed beliefs about something (e.g. 'New Zealand is greener and more beautiful than most countries'). Some of the sentences were grammatical and some ungrammatical but test-takers were not told this. The rationale for including ungrammatical sentences was that learners draw on their long-term memory of the target structures and thus automatically correct the errors without necessarily noticing them. Studies have shown that native speakers do this. The instructions given to learners were to (<reflink idref="bib1" id="ref70">1</reflink>) listen to each sentence, (<reflink idref="bib2" id="ref71">2</reflink>) indicate whether they agreed or disagreed with the proposition it encoded and (<reflink idref="bib3" id="ref72">3</reflink>) repeat the sentence in correct English. The sentences were scored dichotomously in terms of whether a learner had produced the target structure in a sentence correctly or not. If learners paraphrased a sentence avoiding use of the target structure they were scored 0. This resulted in a total accuracy score, a grammatical sentences score, an ungrammatical sentences score for each learner and also for scores for each separate structure. Total scores on the EIT loaded strongly (.87) on the implicit factor in Loewen and Ellis' ([<reflink idref="bib24" id="ref73">24</reflink>]) confirmatory factor analysis. It constituted the best measure of implicit knowledge. This finding has been replicated in other factor-analytic studies.</p> <p>But does the EIT measure implicit knowledge? Erlam ([<reflink idref="bib16" id="ref74">16</reflink>]) argued that to function as a test of implicit knowledge the EIT needs to (<reflink idref="bib1" id="ref75">1</reflink>) require a primary focus on meaning, (<reflink idref="bib2" id="ref76">2</reflink>) include a delay between presentation of the stimuli and repetition and (<reflink idref="bib3" id="ref77">3</reflink>) be time-pressured. She argued that the test in Ellis ([<reflink idref="bib12" id="ref78">12</reflink>]) satisfied these conditions. The fact that the learners often corrected the ungrammatical sentences, that there was no correlation between the length of the sentences and accuracy in reproducing the target structures, and that EIT scores were strongly correlated with scores from other tests involving online language use all supported the claim that it measured implicit knowledge. Nevertheless, as noted previously, this claim has been challenged on the grounds that the EIT cannot distinguish between implicit knowledge and well-automatised explicit knowledge. This challenge arises from studies that have shown that scores from established psycholinguistic measures of implicit knowledge such as the word monitoring task do not load on the same factor as EIT scores, suggesting the need to distinguish three types of linguistic knowledge – implicit knowledge, automatised explicit knowledge and non-automatised explicit knowledge. These studies will be considered later in the section dealing psycholinguistic measures.</p> <p>A number of modifications to the EIT might arguably enhance its construct validity as a test of implicit knowledge. Spada, Shiu and Tomita ([<reflink idref="bib34" id="ref79">34</reflink>]) substituted the belief statements used in Ellis ([<reflink idref="bib12" id="ref80">12</reflink>]) with truth-value statements. This has the advantage of showing clearly whether learners have processed the sentence stimuli for meaning. In Ellis' EIT, learners had to respond 'yes', 'no' or 'not sure' with no way of ensuring that their responses actually reflected their beliefs. But if they have to respond 'true' or 'false' to factual statements it is possible to see if they have understood a sentence and therefore to eliminate those sentences that they failed to understand from the scoring. Kim and Nam ([<reflink idref="bib22" id="ref81">22</reflink>]) addressed another possible limitation of the Ellis EIT, which did not impose a strict time limit for the learners' repetition of the stimulus sentences. Kim and Nam had learners complete the EIT under time pressure (allowing just 20% more time than native speakers required) and an unpressured condition. They reported that the time-pressured version afforded a stronger measure of implicit knowledge.[<reflink idref="bib2" id="ref82">2</reflink>] These modifications increase the need for automatic processing.</p> <p>Another interesting possibility comes from the use of an EIT for dynamic assessment – a form of assessment supported by sociocultural theory (Lantolf and Poehner [<reflink idref="bib23" id="ref83">23</reflink>]). Its aim is to overcome the dualism of 'assessment' and 'instruction' by investigating what L2 learners can accomplish both without and with assistance. van Compernolle and Zhang ([<reflink idref="bib3" id="ref84">3</reflink>]) describe a study where learners were asked to repeat each sentence in an EIT and, if they were unable to do so, were offered graduated assistance. The study did not set out to investigate implicit and explicit knowledge but it suggests an interesting way of doing this within a dynamic assessment framework. If learners can repeat a sentence spontaneously and independently this might constitute evidence of implicit knowledge (or automated explicit knowledge); if, however, they require assistance but then succeed in reproducing the sentence this might be indicative of explicit knowledge. If they fail to reproduce the sentence even with help this would demonstrate they had neither implicit nor explicit knowledge.</p> <p>Irrespective of whether the EIT measures implicit knowledge or automatised explicit knowledge, it has obvious potential as a measure of learners' ability to use a range of grammatical structures without the need for controlled processing. For this reason, it is of potential interest for the measurement of pragmatic competence. It is, however, not easy to see how it can be adapted to focus on pragmatic aspects of language. In a grammatical EIT, the grammaticality of the stimulus sentences depends solely on the linguistic context provided by each sentence. But this is not possible in a pragmatic EIT as appropriateness depends on the situational, not the linguistic, context. Thus to include both pragmatically appropriate and inappropriate stimuli it would be necessary to specify the situational context for each utterance. This may be possible, especially if the situational context is described in the learners' L1. Alternatively, an EIT might focus solely on learners' pragmalinguistic knowledge by assessing the accuracy with which they are able to reproduce sentences containing different realisations of the same speech act (e.g. direct and indirect forms of requests). This, however, would tell us nothing about their sociopragmatic knowledge. Another problem in designing an EIT to assess pragmatic knowledge is that it may not be possible to replicate the dual-tasking element of grammatical EITs by asking learners to make belief or truth-value judgements before reproducing the sentences. Asking learners to judge whether a sentence is appropriate or inappropriate (in relation to the situational context provided) will draw attention to its pragmatic acceptability and encourage conscious attention to the linguistic forms needed to reproduce or correct it. Using an EIT as a measure of implicit pragmatic knowledge will clearly need careful consideration of design issues.</p> <hd id="AN0149596508-12">Error correction tests</hd> <p>An error correction test was not included in the Ellis ([<reflink idref="bib12" id="ref85">12</reflink>]) test battery or in most of the follow-up studies. Spada, Shiu and Tomita ([<reflink idref="bib34" id="ref86">34</reflink>]), however, did include one. In addition to requiring learners to correct the errors in sentences, their test asked them to identify the errors first and to explain them after correcting them. Such a test involves a high degree of awareness, is unpressured, focuses attention explicitly on form and involves the use of meta-language. As such, according to the design features that informed the Ellis battery (see Table 2), it clearly functions as a measure of learners' explicit knowledge. Support for this claim comes from Spada, Shiu and Tomita's study, they found that the three scores based on the error correction test along with scores on a written GJT loaded on the same factor. Further support comes from studies such as Shintani and Ellis ([<reflink idref="bib33" id="ref87">33</reflink>]). This study reported that performance on an error correction test following form-focused instruction deteriorated markedly over time. Shintani and Ellis suggested that this was because an epiphenomenon of explicit knowledge is that learners can easily forget it. There is, of course, the possibility of asking learners to rate their confidence and indicate the knowledge source they used when correcting sentences. As with other form-focused methods of assessment these subjective measures might be used to derive measures of implicit knowledge. However, I know of no attempt to do this with error correction tests.</p> <p>It should be easy to develop pragmatic tests where learners are asked to rewrite situationally embedded inappropriate utterances in appropriate language. Such tests, however, would serve primarily as measures of explicit pragmatic knowledge.</p> <hd id="AN0149596508-13">Comprehension tests</hd> <p>de Jong ([<reflink idref="bib4" id="ref88">4</reflink>]) pointed out the importance of investigating receptive knowledge as well as productive knowledge when investigating learners' implicit and explicit knowledge. He argued that overall the literature points to the receptive representation of linguistic knowledge preceding the productive representation but with some shared representation developing over time. Somewhat surprisingly, however, there have been few attempts to investigate receptive implicit knowledge of L2 grammar.</p> <p>In his own study, de Jong ([<reflink idref="bib5" id="ref89">5</reflink>]) investigated the effects of listening training on the acquisition of a grammatical feature in an artificial language and included both receptive and productive tests designed to measure implicit and explicit knowledge. In the training task, learners heard a sentence and indicated which picture matched the meaning of the sentence by pressing a key on a computer keyboard. To test receptive knowledge de Jong used a self-paced listening task (described below in the section dealing with psycholinguist tests). He also administered a testing version of the picture-matching task and recorded response times. Longer reaction times to the sentences containing the grammatical target were taken as evidence of slower processing. However, de Jong admitted that these tests could not distinguish between the rapid processing of explicit knowledge and implicit knowledge. de Jong also asked the learners to complete a questionnaire where they were asked to describe the rule that was the target of the study and reported that half the participants produced at least a half correct description of the rule, suggesting that in fact, for many of the learners, the tests may have measured explicit knowledge.</p> <p>Qin and van Compernolle ([<reflink idref="bib30" id="ref90">30</reflink>]) describe an interesting dynamic assessment study involving pragmatic implicature. Learners were presented with a situation for a specific speech act along with the utterance performing the act and then asked to decide what the meaning of the utterance was by answering a multiple choice question (five choices). If they chose the correct answer they moved on to the next item in the test. If they chose a wrong answer they received graduated feedback in the form of prompts. An example of an item from the test and the prompts can be found in the Appendix. As suggested above, correct responses provided without assistance and following prompting might be taken as signs of implicit knowledge and explicit knowledge respectively. Qin and Compernolle's test was written and unpressured; an aural, time-pressured test would have stronger validity if the aim was to distinguish implicit and explicit knowledge.</p> <p>Listening comprehension tests have a clear advantage over any kind of production test as they can be designed to require rapid online processing – a key requirement for testing implicit knowledge. In a production test, even a speeded one, learners still have some latitude for controlled processing. A time-pressured picture-matching test could be designed to investigate learners' receptive sociopgragmatic knowledge; learners could be shown pictures with empty speech bubbles and then asked to pick the picture that best matched the utterance they listened to. For example, for the utterance 'Excuse me but would you mind closing the window?' learners would select from pictures showing a boy speaking to another boy, a boy speaking to his mother and a boy speaking to an old man. Time-pressuring the learners' response would help to ensure that they could not easily draw on rule-based knowledge about how to request in English.</p> <hd id="AN0149596508-14">Judgement tests</hd> <p>The use of GJTs has a long history in both linguistics and SLA research. In much of the research, they have been used to measure learners' knowledge of grammatical structures without any consideration of what type of knowledge is involved. In Ellis ([<reflink idref="bib12" id="ref91">12</reflink>]) and follow-up studies, however, attempts were made to design tests with different features (e.g. timed vs. untimed; aural vs. written) in order to distinguish measurements of implicit and explicit knowledge.</p> <p>GJTs can come in various forms (see Ellis [<reflink idref="bib10" id="ref92">10</reflink>]). For example, test-takers can be simply asked to judge whether the sentences are grammatical or ungrammatical, to indicate the part that is ungrammatical, and/or to correct the errors in the ungrammatical sentences. Thus a judgement test can also double up as an error correction test. The type of test that has been used in the implicit/explicit knowledge studies has typically only asked for a grammaticality judgement. Asking for test-takers to indicate and correct errors is impractical in timed and aural tests.</p> <p>One of the issues addressed is whether responses to the grammatical and ungrammatical sentences draw on different knowledge sources. Ellis ([<reflink idref="bib12" id="ref93">12</reflink>]), for example, found that scores derived from the ungrammatical sentences in the untimed GJT loaded much more strongly on the explicit factor than scores for the grammatical sentences. Total GJT scores (i.e. for both grammatical and ungrammatical sentences) loaded on both the explicit and implicit factors although more strongly on the explicit factor. Gutiérrez ([<reflink idref="bib20" id="ref94">20</reflink>]) found that there were statistically significant differences between the learners' responses to the grammatical and ungrammatical sentences in both the timed and untimed tests in her study and proposed learners resort to their implicit knowledge when judging grammatical sentences and to their explicit knowledge when judging ungrammatical ones. However, Kim and Nam ([<reflink idref="bib22" id="ref95">22</reflink>]) found that it was the ungrammatical sentences in an aural GJT that loaded on the implicit factor. Clearly, results for grammatical versus ungrammatical sentences have not been consistent across studies although the weight of the evidence supports Ellis' ([<reflink idref="bib12" id="ref96">12</reflink>]) finding, namely that ungrammatical sentences are the more likely to elicit explicit knowledge. Vafaee, Suzuki and Kachisnke ([<reflink idref="bib39" id="ref97">39</reflink>]), for example, reported that the confirmatory factor model with the best fit in their study identified the ungrammatical sentences in both a timed and untimed GJT as loading together with scores on the MKT on a factor they labelled explicit knowledge.</p> <p>There is also evidence that the modality of a GJT affects the kind of knowledge tapped by a test. This makes good sense as responding to aural sentences requires real-time language processing especially if the items in the test are time-pressured. In Kim and Nam's study, for example, the reason why the ungrammatical sentences loaded on the implicit factor was probably because the sentences were presented aurally (i.e. the learners did not have time to access their explicit knowledge).</p> <p>Drawing together the results from various studies, it is possible to identify the features of GJTs that influence the type of knowledge learners are likely to draw on. I have summarised these in Table 4. This table suggests that it is the grammatical sentences in an aural timed GJT that are most likely to tap into learners' implicit knowledge. This is an idealisation, however, as the various design features interact in complex ways that are not entirely predictable. Also, given that any kind of GJT requires learners to focus on form, it could be argued that GJTs can only distinguish automatised and non-automatised explicit knowledge at best. Kim and Nam's factor analysis resulted in a three factor solution that they labelled 'implicit strongest' (loadings just for the EIT), 'implicit weakest' (loadings for written timed GJT and aural TGJT) and 'explicit (loading for the MKT) but an alternative description of these three factors, however, could be 'implicit', 'automatised explicit' and 'non-automatised explicit' with the two GJTs serving as a measure of automatised explicit knowledge.</p> <p>Table 4. Design features of GJTs influencing the type of knowledge tapped.</p> <p> <ephtml> &lt;table&gt;&lt;thead valign="bottom"&gt;&lt;tr&gt;&lt;td&gt;Design features&lt;/td&gt;&lt;td&gt;Implicit knowledge&lt;/td&gt;&lt;td&gt;Explicit knowledge&lt;/td&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Grammaticality&lt;/td&gt;&lt;td&gt;Grammatical sentences&lt;/td&gt;&lt;td&gt;Ungrammatical sentences&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Timing&lt;/td&gt;&lt;td&gt;Timed&lt;/td&gt;&lt;td&gt;Untimed&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Modality&lt;/td&gt;&lt;td&gt;Aural&lt;/td&gt;&lt;td&gt;Written&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>GJT scores by themselves cannot convincingly tell us what kind of knowledge learners draw on. This is why researchers investigating implicit learning have also obtained subjective ratings (i.e. confidence ratings and source attributions) along with retrospective reports from learners in order to more clearly establish whether their judgements were accompanied by conscious awareness and searching for a rule. Arguably, then, if GJTs are to be used as measures of implicit knowledge, they must incorporate methods for investigating the nature of the judgements that learners make.</p> <p>Judgement tests have not figured in interlanguage pragmatics research to the best of my knowledge. If they are to be used, then learners would need to judge the appropriateness of utterances in relation to their situational contexts. That is, it would be necessary to describe the situational context for each item in the test. Also, whereas grammaticality is an either-or phenomenon appropriateness is best viewed as scalar so learners might be asked to judge the appropriateness of utterances on a scale from 'very appropriate' to 'entirely inappropriate'. Aural stimuli along with timed responses and subjective ratings offer the greatest likelihood of measuring implicit (or automatised explicit) knowledge.</p> <hd id="AN0149596508-15">Metalinguistic knowledge test</hd> <p>The MKT used in Ellis et al. ([<reflink idref="bib14" id="ref98">14</reflink>]) consisted of three parts. In the first part, learners were asked to select the best explanation for grammatical errors in sentences from the four choices provided for each sentence. The second part tested receptive knowledge of metalinguistic terms and the third part productive knowledge.</p> <p>The MKT is most clearly a test of explicit knowledge. In all the factor analysis studies that included such a test, scores loaded on the factor labelled explicit. In Ellis ([<reflink idref="bib12" id="ref99">12</reflink>]), ungrammatical sentences from the untimed GJT loaded along with scores from the MKT. In Vafaee, Suzuki and Kachisnke ([<reflink idref="bib39" id="ref100">39</reflink>]), scores from the ungrammatical sentences in both their timed and untimed GJT loaded with the MKT scores. In Kim and Nam's ([<reflink idref="bib22" id="ref101">22</reflink>]) study, the MKT loaded on its own factor. Elder ([<reflink idref="bib9" id="ref102">9</reflink>]) reported a thorough validation study of the MKT used in Ellis ([<reflink idref="bib12" id="ref103">12</reflink>]) as a test of explicit knowledge.</p> <p>There would seem little value in testing L2 learners' knowledge of meta-pragmatic terms as these are very technical and unlikely to be known by most learners. Thus parts 2 and 3 of Ellis et al.'s test would have no counterpart in a meta-pragmatic test. But the multiple choice format of part 1 could be adapted by asking learners to select the best explanation for why a particular utterance is inappropriate from the explanations provided. Alternatively, learners could be asked to provide their own explanations although this would confound learners' meta-pragmatic understandings with their ability to verbalise them.</p> <hd id="AN0149596508-16">Psycholinguistic measures</hd> <p>As previously discussed, Ellis ([<reflink idref="bib12" id="ref104">12</reflink>]) characterised tests of implicit knowledge as tests that (<reflink idref="bib1" id="ref105">1</reflink>) involve no conscious awareness of linguistic form, (<reflink idref="bib2" id="ref106">2</reflink>) are time-pressured to induce online processing, (<reflink idref="bib3" id="ref107">3</reflink>) focus on meaning and (<reflink idref="bib4" id="ref108">4</reflink>) do not involve the application of meta-language. We have seen that a number of factor-analytic studies reported that the EIT most convincingly satisfies these criteria. However, Suzuki and DeKeyser ([<reflink idref="bib36" id="ref109">36</reflink>]), Vafaee, Suzuki and Kachisnke ([<reflink idref="bib39" id="ref110">39</reflink>]) and Suzuki ([<reflink idref="bib35" id="ref111">35</reflink>]) have challenged this claim on the grounds that time pressure cannot limit access to explicit knowledge sufficiently to ensure that implicit knowledge is drawn on. These studies included the measures that Ellis claimed to be best measures of implicit knowledge (i.e. the EIT and timed GJT) along with measures drawn from the psycholinguistic literature. They found that that these latter measures loaded on a separate factor from the other measures and went on to argue that the psycholinguistic measures served as measures of implicit knowledge while the EIT and the timed GJT were best seen as measures of automatised explicit knowledge.</p> <p>The psycholinguistic measures were based on responses derived from three different tests. I have summarised the descriptions of these tests in Suzuki ([<reflink idref="bib35" id="ref112">35</reflink>]):</p> <p></p> <ulist> <item> The visual word paradigm: This involved tracking learners' eye movements. Learners were presented with a scene involving four pictures. They then listened to sentences and the eye movements they directed at specific pictures as they listened were recorded. The pictures included a picture representing the grammatical target (e.g. the agent of the sentence), a competitor target and a distractor. Learning was considered to be evident if the learners displayed sensitivity to the grammatical target by looking at the relevant picture more frequently than at the pictures representing the competitor target or the distractor picture.</item> <p></p> <item> Word monitoring task: In this test, a word appears on a screen. Learners are told to listen[<reflink idref="bib3" id="ref113">3</reflink>] to a sentence and as soon as they hear the target word to press a button. The purpose of this is to involve the learners in dual processing; that is, because they have to consciously listen for the word they are not able to consciously attend to the grammaticality of the sentences. The sentences were a mixture of grammatical and ungrammatical but learners were not told this. Each sentence was followed by a comprehension question to ensure that the learners' were focused on meaning. Response times for the region of interest in each sentence (i.e. the target word and the immediately following word) were recorded. A measure of grammatical sensitivity was calculated by subtracting the time taken to respond in the region of interest in the ungrammatical sentences from the time taken in the grammatical sentences. The assumption here is that sensitivity to grammatical violations while learners are distracted from attending to form reflects implicit knowledge.</item> <p></p> <item> Self-paced reading task: Learners are asked to read a sentence word by word as quickly as possible. The learners press a computer button to bring up each word in the sentence. As they do so the preceding word on their screen disappears. When learners have finished reading the whole sentence they answer a comprehension question. As in the word monitoring task, some of the sentences are grammatical and some ungrammatical and response times are recorded for the region of interest in each sentence. Grammatical sensitivity scores were calculated in the same way as for the word monitoring task.</item> </ulist> <p>If these tests were measuring the same construct (implicit knowledge) one might expect that they would be strongly inter-correlated. Suzuki did find a weak correlation between scores on the word monitoring task and the self-paced reading task but the correlations involving the visual-world task were all very weak and non-significant. In general scores on all these tests were much lower (but with very high standard deviations) than scores on the tests considered more likely to measure explicit knowledge (i.e. a timed and an untimed GJT and a controlled production test). It is therefore not so surprising to find that the psycholinguistic tests and the explicit tests loaded on separate factors. Also, while these psycholinguistic tests may be well-suited to experimental studies, they do not have high face validity as language tests as they involve very artificial procedures (e.g. reading a sentence word by word). Thus doubts must exist about their value if the aim is to design tests of implicit and explicit knowledge that can serve as stand-alone tests.</p> <p>It is also difficult to see how these psycholinguistic tests could be adapted to measure pragmatic knowledge as they all rely on how learners process a specific word in the area of interest in a sentence. Pragmatic meanings are not usually conveyed by a single word but by multiple linguistic devices. A complex request such as "Excuse me, would you mind closing the window", for example, does not contain a clearly identifiable region of interest. Thus what works for grammatical sensitivity may not work for pragmatic sensitivity. A psycholinguistic test of pragmatic knowledge is only feasible if it is possible to identify individual words that determine the social (in)appropriateness of an utterance. Also, as noted for many of the other tests, assessing whether learners are pragmatically sensitive will require providing information about the situational context of each utterance.</p> <hd id="AN0149596508-17">Some final comments</hd> <p>The tests designed to measure implicit knowledge almost invariably involved specific elements in isolated sentences.[<reflink idref="bib4" id="ref114">4</reflink>] This is not a problem when it comes to testing grammar but it is a problem for pragmatics testing as pragmatic features are multiple within an utterance and also cross utterance boundaries, i.e. they are discoursal. A key question, then, is whether it is possible to design tests that assess pragmatic ability in discourse and, if it is, how this can be done. The research on grammar testing offers little help here.</p> <p>If, however, the aim is to test implicit and explicit knowledge of individual speech acts, which after all has been a major focus in interlanguage pragmatics, then the grammar tests do provide ideas for how this might be done. The characteristics of a test most likely to elicit implicit pragmatic knowledge are as follows:</p> <p></p> <ulist> <item> It will involve dual processing (i.e. learners will be asked to accomplish a secondary task that distracts their attention from linguistic form).</item> <p></p> <item> It will assess receptive or productive pragmatic knowledge separately or both together (as in an EIT).</item> <p></p> <item> It will assess pragmatic (not semantic) meaning (i.e. the social appropriateness of utterances).</item> <p></p> <item> It will involve real-time language processing which can best be achieved by the use of aural (as opposed to written) stimuli and by imposing time limits on the learner's responses.</item> <p></p> <item> The time taken for learners to respond to individual test items will need to be recorded.</item> <p></p> <item> It will be supported by the collection of subjective ratings and self-report data from the learners to show to what extent they manifested conscious awareness of the pragmatic targets of the test.</item> </ulist> <p>Any test involving the comprehension or production of language will potentially elicit the use of implicit and explicit knowledge and the extent to which it leads to the use of one or the other type or both types will depend on how the language is represented in the minds of individual learners. At best, then, all a test can do is bias learners to the deployment of one type of language. Learners who lack implicit knowledge will have to use their explicit knowledge irrespective of the kind of test. They will just do badly in those tests designed to assess their implicit knowledge.</p> <p>Finally, it may not ultimately be possible to distinguish true implicit from automatised explicit knowledge. Some progress in achieving this has been made for grammar by using psycholinguistic tests such as the word monitoring task. But such tests may not prove adaptable to the testing of pragmatic aspects of language. It can also be argued that if the purpose is simply to develop a battery of tests that distinguish different types of pragmatic ability, simply showing that particular groups of learners differ in terms of the extent to which they can access their pragmatic knowledge automatically may suffice. From a theoretical perspective, it is clearly desirable to distinguish implicit and explicit pragmatic knowledge. From a practical point of view, it may prove impossible to do so and, in any case, functionally it is not necessary.</p> <hd id="AN0149596508-18">Disclosure statement</hd> <p>No potential conflict of interest was reported by the authors.</p> <hd id="AN0149596508-19">Appendix</hd> <p></p> <hd id="AN0149596508-20">Example of question and prompts from Qin and van Compernolle (forthcoming)</hd> <p>Miles plans to travel to Beijing. He knows his high school classmate, Lucy, a graduate student there. Mike calls Lucy and asks if he can stay in her dorm for a few nights.</p> <p>She replies: <emph>There are many people living in my dorm and it is already fully occupied</emph> (in Chinese). What does Lucy mean?</p> <p></p> <ulist> <item> Mike cannot stay in Lucy's dorm</item> <p></p> <item> Lucy's dorm is already fully occupied.</item> <p></p> <item> Lucy invites Mile to visit Beijing.</item> <p></p> <item> Lucy's dorm is very small.</item> <p></p> <item> Lucy plans to have Mike stay in Beijing for a long time.</item> </ulist> <p>Prompt 1: That is not the right answer. Listen to the clip and try again.</p> <p>Prompt 2: That is not right either. Why does Lucy talk about her dorm?</p> <p>Prompt 3: That is not right either. Is Lucy going to allow Mike to stay with her?</p> <ref id="AN0149596508-21"> <title> Notes </title> <blist> <bibl id="bib1" idref="ref6" type="bt">1</bibl> <bibtext> Neither Loewen and Ellis ([24]) nor the follow-up factor-analytic studies tested a model involving a single factor fit. Vafaee, Suzuki and Kachisnke ([39]) argued that is it essential to do so.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref7" type="bt">2</bibl> <bibtext> The EIT in Ellis ([12]) was administered face-to-face with individual learners. The EITs in both Spada, Shiu and Tomita ([34]) and Kim and Nam ([22]) were administered by computer, allowing for the time taken to reproduce a stimulus sentence to be strictly controlled. Interestingly, Suzuki and DeKeyser ([36]) who found that EIT scores correlated with scores from a MKT allowed 8 seconds for learners' to repeat a sentence, far longer than in any of the other EIT studies.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref8" type="bt">3</bibl> <bibtext> Another version of the word monitoring test (Vafaee, Suzuki and Kachinske [39]) had learners read rather than listen to sentences.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref22" type="bt">4</bibl> <bibtext> The exception is the use of free production tasks – e.g. the oral production task in Ellis ([12]).</bibtext> </blist> </ref> <ref id="AN0149596508-22"> <title> References </title> <blist> <bibtext> Bowden, H. 2016. Assessing second-language oral proficiency for research: the Spanish elicitation test. Studies in Second Language Acquisition 38 : 647 – 675. doi: 10.1017/S0272263115000443</bibtext> </blist> <blist> <bibtext> Bowles, M. 2011. Measuring implicit and explicit linguistic knowledge: what can heritage language learners contribute? Studies in Second Language Acquisition 33 : 247 – 271. doi: 10.1017/S0272263110000756</bibtext> </blist> <blist> <bibtext> van Compernolle, R. and H. Zhang. 2014. Dynamic assessment of elicited imitation: a case analysis of an advanced L2 English speaker. Language Testing 31 : 395 – 412. doi: 10.1177/0265532213520303</bibtext> </blist> <blist> <bibtext> de Jong, N. 2005a. Learning second language grammar by listening. Unpublished PhD thesis, Graduate School of Linguistics, the Netherlands.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref89" type="bt">5</bibl> <bibtext> de Jong, N. 2005b. Can second language grammar be learned through listening? An experimental study. Studies in Second Language Acquisition 27 : 205 – 234. doi: 10.1017/S0272263105050114</bibtext> </blist> <blist> <bibl id="bib6" idref="ref19" type="bt">6</bibl> <bibtext> DeKeyser, R. 2003. Implicit and explicit learning. In Handbook of Second Language Acquisition, ed. C. Doughty and M. Long, 313 – 348. Malden, MA : Blackwell.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref9" type="bt">7</bibl> <bibtext> Dienes, Z. and J. Perner. 1999. A theory of implicit and explicit knowledge. Behavioral and Brain Sciences 22 : 735 – 808. doi: 10.1017/S0140525X99002186</bibtext> </blist> <blist> <bibl id="bib8" idref="ref59" type="bt">8</bibl> <bibtext> Dienes, Z., G.T.M. Altmann, L. Kwan and A. Goode. 1995. Unconscious knowledge of artificial grammars is applied strategically. Journal of Experimental Psychology: Learning, Memory, and Cognition 21, no. 5 : 1322 – 1338.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref102" type="bt">9</bibl> <bibtext> Elder, C. 2009. Validating a test of metalinguistic knowledge. In Implicit and Explicit Knowledge in Second Language Learning, Testing and Teaching, ed. R. Ellis, S. Loewen, C. Elder, R. Erlam, J. Philp and H. Reinders, 113 – 138. Bristol : Multilingual Matters.</bibtext> </blist> <blist> <bibtext> Ellis, R. 1991. Grammatically judgments and second language acquisition. Studies in Second Language Acquisition 13, no. 2 : 161 – 186. doi: 10.1017/S0272263100009931</bibtext> </blist> <blist> <bibtext> Ellis, R. 2004. The definition and measurement of explicit knowledge. Language Learning 54: 227–275.</bibtext> </blist> <blist> <bibtext> Ellis, R. 2005. Measuring implicit and explicit knowledge of a second language: a psychometric study. Studies in Second Language Acquisition 27, no. 2 : 141 – 172.</bibtext> </blist> <blist> <bibtext> Ellis, R. 2006. Modelling learning difficulty and second language proficiency: the differential contributions of implicit and explicit knowledge. Applied Linguistics 27 : 431 – 463. doi: 10.1093/applin/aml022</bibtext> </blist> <blist> <bibtext> Ellis, R., S. Loewen, C. Elder, R. Erlam, J. Philp and H. Reinders. 2009. Implicit and Explicit Knowledge in Second Language Learning, Testing and Teaching. Bristol : Multilingual Matters.</bibtext> </blist> <blist> <bibtext> Enochs, K. and S. Yoshitake-Strain. 1999. Evaluating six measures of EFL learners' pragmatic competence. JALT Journal 21, no. 1 : 29 – 50.</bibtext> </blist> <blist> <bibtext> Erlam, R. 2006. Elicited imitation as a measure of L2 implicit knowledge: an empirical validation study. Applied Linguistics 27, no. 3 : 464 – 491. doi: 10.1093/applin/aml001</bibtext> </blist> <blist> <bibtext> Gaillard, S. and A. Tremblay. 2016. Linguistic proficiency assessment in second language acquisition research: the elicited imitation task. Language Learning 66 : 419 – 447. doi: 10.1111/lang.12157</bibtext> </blist> <blist> <bibtext> Godfroid, A. 2016. The effects of implicit instruction on implicit and explicit knowledge development. Studies in Second Language Acquisition 38: 177–215.</bibtext> </blist> <blist> <bibtext> Golato, A. 2003. Studying compliment responses: a comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics 24 : 90 – 121. doi: 10.1093/applin/24.1.90</bibtext> </blist> <blist> <bibtext> Gutiérrez, X. 2013. The construct validity of grammaticality judgment tests as measures of implicit and explicit knowledge. Studies in Second Language Acquisition 35 : 423 – 449. doi: 10.1017/S0272263113000041</bibtext> </blist> <blist> <bibtext> Kerz, E., D. Wiechmann and F.B. Riedel. 2017. Implicit learning in the crowd: investigating the role of awareness in the acquisition of L2 knowledge. Studies in Second Language Acquisition 39 : 711 – 734. doi: 10.1017/S027226311700002X</bibtext> </blist> <blist> <bibtext> Kim, J-E. and H. Nam. 2017. Measures of implicit knowledge revisited: processing modes, time pressure and modality. Studies in Second Language Acquisition 39 : 431 – 457. doi: 10.1017/S0272263115000510</bibtext> </blist> <blist> <bibtext> Lantolf, J.P. and M.E. Poehner. 2011. Dynamic assessment in the classroom: Vygotskian praxis for second language development. Language Teaching Research 15 : 11 – 33. doi: 10.1177/1362168810383328</bibtext> </blist> <blist> <bibtext> Loewen, S. and R. Ellis. 2007. Confirming the operational definitions of explicit and implicit knowledge in Ellis (2005). Studies in Second Language Acquisition 29 : 119 – 126.</bibtext> </blist> <blist> <bibtext> Loschky, L. and R. Bley-Vroman. 1993. Grammar and task-based methodology. In Tasks and Language Learning: Integrating Theory and Practice, ed. G. Crookes and S. Gass, 123 – 167. Clevedon : Multilingual Matters.</bibtext> </blist> <blist> <bibtext> Macrory, G. and V. Stone. 2000. Pupil progress in the acquisition of the perfect tense in French: the relationship between knowledge and use. Language Teaching Research 4 : 55 – 82. doi: 10.1177/136216880000400104</bibtext> </blist> <blist> <bibtext> Ortega, L. 2000. Understanding syntactic complexity: The measurement of change in the syntax of instructed L2 Spanish learners. Unpublished Ph.D., University of Hawai'i at Manoa.</bibtext> </blist> <blist> <bibtext> Ortega, L., N. Iwashita, S. Rabie and J.M. Norris. 1999. A Multilanguage Comparison of Measures of Syntactic Complexity. Hawai'i : University of Hawai'i, National Language Resource Center.</bibtext> </blist> <blist> <bibtext> Paradis, M. 2004. A Neurolinguistic Theory of Bilingualism. Amsterdam : John Benjamins.</bibtext> </blist> <blist> <bibtext> Qin and van Compernolle (forthcoming). Computerized dynamic assessment of implicature comprehension in L2 Chinese. Language Learning and Technology.</bibtext> </blist> <blist> <bibtext> Rebuschat, P. 2013. Measuring implicit and explicit knowledge in second language research. Language Learning 63 : 595 – 626. doi: 10.1111/lang.12010</bibtext> </blist> <blist> <bibtext> Rebuschat, P., P. Hamrick, K. Riestenberg, R. Sachs and N. Ziegler. 2015. Triangulating measures of awareness: a contribution to the debate of learning without awareness. Studies in Second Language Acquisition 37 : 299 – 334. doi: 10.1017/S0272263115000145</bibtext> </blist> <blist> <bibtext> Shintani, N. and R. Ellis. 2013. The comparative effect of direct written corrective feedback and metalinguistic explanation on learners' explicit and implicit knowledge of the English indefinite article. Journal of Second Language Writing 22 : 286 – 306. doi: 10.1016/j.jslw.2013.03.011</bibtext> </blist> <blist> <bibtext> Spada, N., J. Shiu and Y. Tomita. 2015. Validating an elicited imitation task as a measure of implicit knowledge: comparisons with other validation studies. Language Learning 65, no. 3 : 723 – 751. doi: 10.1111/lang.12129</bibtext> </blist> <blist> <bibtext> Suzuki, Y. 2017. Validity of new measures of implicit knowledge: distinguishing implicit knowledge from automatized explicit knowledge. Applied Psycholinguistics 38 : 1229 – 1261. doi: 10.1017/S014271641700011X</bibtext> </blist> <blist> <bibtext> Suzuki, Y. and R.M. DeKeyser. 2015. Comparing elicited imitation and word monitoring as measures of implicit knowledge. Language Learning 65, no. 4 : 860 – 895. doi: 10.1111/lang.12138</bibtext> </blist> <blist> <bibtext> Taguchi, N. 2012. Context, Individual Differences, and Pragmatic Competence. New York : Multilingual Matters.</bibtext> </blist> <blist> <bibtext> Ullman, M. 2004. Contributions of memory circuits to language: the declarative/procedural model. Cognition 92 : 231 – 270. doi: 10.1016/j.cognition.2003.10.008</bibtext> </blist> <blist> <bibtext> Vafaee, P., Y. Suzuki and I. Kachisnke. 2017. Validating grammaticality judgment tests: evidence from two psycholinguistic measures. Studies in Second Language Acquisition 39 : 59 – 95. doi: 10.1017/S0272263115000455</bibtext> </blist> <blist> <bibtext> Williams, J. 2005. Learning without awareness. Studies in Second Language Acquisition 27 : 269 – 304. doi: 10.1017/S0272263105050138</bibtext> </blist> <blist> <bibtext> Zhang, R. 2015. Measuring university-level L2 learners' implicit and explicit linguistic knowledge. Studies in Second Language Acquisition 37 : 457 – 486. doi: 10.1017/S0272263114000370</bibtext> </blist> </ref> <aug> <p>By Rod Ellis and Carsten Roever</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib40" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib18" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib12" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib13" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib11" firstref="ref10"></nolink> <nolink nlid="nl6" bibid="bib29" firstref="ref11"></nolink> <nolink nlid="nl7" bibid="bib38" firstref="ref12"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref16"></nolink> <nolink nlid="nl9" bibid="bib34" firstref="ref17"></nolink> <nolink nlid="nl10" bibid="bib41" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib36" firstref="ref20"></nolink> <nolink nlid="nl12" bibid="bib37" firstref="ref21"></nolink> <nolink nlid="nl13" bibid="bib32" firstref="ref24"></nolink> <nolink nlid="nl14" bibid="bib21" firstref="ref26"></nolink> <nolink nlid="nl15" bibid="bib22" firstref="ref29"></nolink> <nolink nlid="nl16" bibid="bib35" firstref="ref30"></nolink> <nolink nlid="nl17" bibid="bib31" firstref="ref32"></nolink> <nolink nlid="nl18" bibid="bib24" firstref="ref38"></nolink> <nolink nlid="nl19" bibid="bib14" firstref="ref40"></nolink> <nolink nlid="nl20" bibid="bib25" firstref="ref52"></nolink> <nolink nlid="nl21" bibid="bib19" firstref="ref61"></nolink> <nolink nlid="nl22" bibid="bib15" firstref="ref62"></nolink> <nolink nlid="nl23" bibid="bib17" firstref="ref63"></nolink> <nolink nlid="nl24" bibid="bib28" firstref="ref64"></nolink> <nolink nlid="nl25" bibid="bib27" firstref="ref67"></nolink> <nolink nlid="nl26" bibid="bib16" firstref="ref74"></nolink> <nolink nlid="nl27" bibid="bib23" firstref="ref83"></nolink> <nolink nlid="nl28" bibid="bib33" firstref="ref87"></nolink> <nolink nlid="nl29" bibid="bib30" firstref="ref90"></nolink> <nolink nlid="nl30" bibid="bib10" firstref="ref92"></nolink> <nolink nlid="nl31" bibid="bib39" firstref="ref97"></nolink> |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1292126 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: The Measurement of Implicit and Explicit Knowledge – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Ellis%2C+Rod%22">Ellis, Rod</searchLink><br /><searchLink fieldCode="AR" term="%22Roever%2C+Carsten%22">Roever, Carsten</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Language+Learning+Journal%22"><i>Language Learning Journal</i></searchLink>. 2021 49(2):160-175. – Name: Avail Label: Availability Group: Avail Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 16 – Name: DatePubCY Label: Publication Date Group: Date Data: 2021 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Information Analyses – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Grammar%22">Grammar</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Metalinguistics%22">Metalinguistics</searchLink><br /><searchLink fieldCode="DE" term="%22Psycholinguistics%22">Psycholinguistics</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Tests%22">Language Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Oral+Language%22">Oral Language</searchLink><br /><searchLink fieldCode="DE" term="%22Knowledge+Level%22">Knowledge Level</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Taxonomy%22">Taxonomy</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+Making%22">Decision Making</searchLink><br /><searchLink fieldCode="DE" term="%22Criticism%22">Criticism</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Content%22">Test Content</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Pragmatics%22">Pragmatics</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Processing%22">Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Factor+Analysis%22">Factor Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Error+Correction%22">Error Correction</searchLink><br /><searchLink fieldCode="DE" term="%22Imitation%22">Imitation</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Format%22">Test Format</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1080/09571736.2018.1504229 – Name: ISSN Label: ISSN Group: ISSN Data: 0957-1736 – Name: Abstract Label: Abstract Group: Ab Data: This article presents a review of research that has investigated ways of measuring implicit and explicit knowledge of a second language (L2), focusing on grammar. It begins by defining implicit and explicit knowledge in terms of a distinguishing set of criteria. Two ways of investigating implicit knowledge are discussed -- through experimental studies of implicit learning and by means of factor-analytic studies. This provides the basis for a taxonomy of tests designed to provide separate measures of the two types of knowledge. Proposals for oral production tests, comprehension tests, judgements tests, tests of metalinguistic knowledge and tests derived from psycholinguistic research are examined and critiqued. The article concludes by suggesting there is a need for tests of L2 pragmatic knowledge to complement those available for grammar and offers a number of suggestions for the design of such tests. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2021 – Name: AN Label: Accession Number Group: ID Data: EJ1292126 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1292126 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1080/09571736.2018.1504229 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 16 StartPage: 160 Subjects: – SubjectFull: Grammar Type: general – SubjectFull: Second Language Learning Type: general – SubjectFull: Metalinguistics Type: general – SubjectFull: Psycholinguistics Type: general – SubjectFull: Language Tests Type: general – SubjectFull: Oral Language Type: general – SubjectFull: Knowledge Level Type: general – SubjectFull: Comparative Analysis Type: general – SubjectFull: Taxonomy Type: general – SubjectFull: Decision Making Type: general – SubjectFull: Criticism Type: general – SubjectFull: Test Content Type: general – SubjectFull: Test Construction Type: general – SubjectFull: Pragmatics Type: general – SubjectFull: Language Processing Type: general – SubjectFull: Factor Analysis Type: general – SubjectFull: Error Correction Type: general – SubjectFull: Imitation Type: general – SubjectFull: Test Format Type: general Titles: – TitleFull: The Measurement of Implicit and Explicit Knowledge Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Ellis, Rod – PersonEntity: Name: NameFull: Roever, Carsten IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2021 Identifiers: – Type: issn-print Value: 0957-1736 Numbering: – Type: volume Value: 49 – Type: issue Value: 2 Titles: – TitleFull: Language Learning Journal Type: main |
| ResultId | 1 |
