View in EDS HTML Full Text PDF Full Text

Parametric Bootstrap Mantel-Haenszel Statistic for Aggregated Testlet Effects

Saved in:
Bibliographic Details
Title: Parametric Bootstrap Mantel-Haenszel Statistic for Aggregated Testlet Effects
Language: English
Authors: Youn Seon Lim (ORCID 0000-0003-0225-1527)
Source: Journal of Educational Measurement. 2025 62(4):503-530.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 28
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Secondary Education
Descriptors: Sampling, Statistical Inference, Tests, Statistical Analysis, Test Items, Achievement Tests, International Assessment, Foreign Countries, Secondary School Students, Cognitive Measurement
Assessment and Survey Identifiers: Program for International Student Assessment
DOI: 10.1111/jedm.12440
ISSN: 0022-0655
1745-3984
Abstract: While testlets have proven useful for assessing complex skills, the stem shared by multiple items often induces correlations between responses, leading to violations of local independence (LI), which can result in biased parameter and ability estimates. Diagnostic procedures for detecting testlet effects typically involve model comparisons testing for the inclusion of extra testlet parameters or, at the item level, testing for pairwise LI. Rosenbaum's adaptation of the Mantel-Haenszel (MH) X[superscript 2]-statistic belongs to the latter category. The MH X[superscript 2]-statistic has also been used in cognitive diagnosis for detecting violations of LI and for the identification of testlet effects. However, this approach is not without limitations, as it lacks a rationale for integrating multiple pairwise MH X[superscript 2]-statistics and any notion of the sampling distribution of such an integrated statistic. In this article, a procedure for integrating multiple pairwise MH X[superscript 2]-statistics to evaluate testlet effects in cognitive diagnosis is proposed. The unknown sampling distribution issue is addressed by implementing a parametric bootstrap resampling scheme. Results from simulation studies demonstrate the performance of the proposed parametric bootstrap testlet MH X[superscript 2]-statistic, and its application to the 2015 PISA Collaborative Problem Solving (CPS) data set illustrates the method's practical merits.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1491369
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHPy9CCMnNBXTdVLfi4Poc0AAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDIIF3iNu4h7y8GX5uAIBEICBmwKx45RQCm51DNQaK4RTciXSR4U5YLovGVmhmQ2pe5D31GomnFJtUynGx1kh35e_WE1YvbwMjf8KXeYTO8zIOPp0ugMVeGa4qikwJ9BXG_SW00DPbfoyz7EK82TO2oCVDG6S0lOc0sVL3Rh5j6AVL7K8QxVShXchJi0PDc4MNktwSJ5nae3kD3b9XSVeiY3JTHodzPkFDzTMFyXX
Text:
  Availability: 1
  Value: <anid>AN0190280385;mea01dec.25;2025Dec18.05:08;v2.2.500</anid> <title id="AN0190280385-1">Parametric Bootstrap Mantel‐Haenszel Statistic for Aggregated Testlet Effects </title> <sbt id="AN0190280385-2">Introduction</sbt> <p>While testlets have proven useful for assessing complex skills, the stem shared by multiple items often induces correlations between responses, leading to violations of local independence (LI), which can result in biased parameter and ability estimates. Diagnostic procedures for detecting testlet effects typically involve model comparisons testing for the inclusion of extra testlet parameters or, at the item level, testing for pairwise LI. Rosenbaum's adaptation of the Mantel‐Haenszel (MH) χ2$\chi ^2$‐statistic belongs to the latter category. The MH χ2$\chi ^2$‐statistic has also been used in cognitive diagnosis for detecting violations of LI and for the identification of testlet effects. However, this approach is not without limitations, as it lacks a rationale for integrating multiple pairwise MH χ2$\chi ^2$‐statistics and any notion of the sampling distribution of such an integrated statistic. In this article, a procedure for integrating multiple pairwise MH χ2$\chi ^2$‐statistics to evaluate testlet effects in cognitive diagnosis is proposed. The unknown sampling distribution issue is addressed by implementing a parametric bootstrap resampling scheme. Results from simulation studies demonstrate the performance of the proposed parametric bootstrap testlet MH χ2$\chi ^2$‐statistic, and its application to the 2015 PISA Collaborative Problem Solving (CPS) data set illustrates the method's practical merits.</p> <p>In educational and psychological measurement, the term "testlet" is used to describe a specific format of test item, which consists of a narrative section that provides context and key information regarding a particular task or problem, and a collection of questions or items that require the correct response to be based on the processing of the information presented in the narrative (e.g., Wainer et al., [<reflink idref="bib67" id="ref1">67</reflink>]; Wainer & Kiely, [<reflink idref="bib65" id="ref2">65</reflink>]). Testlets have become a popular choice for STEM assessments because this item format is well‐suited for measuring contextual knowledge and complex skills without burdening examinees with the need to read through multiple problem descriptions.</p> <p>Local independence of item responses, given the measured latent trait, is a critical assumption in latent variable modeling. The grouping of items according to a common theme, however, often introduces correlations among item responses that cannot be fully accounted for by the measured trait, leading to a violation of the local independence assumption (Bradlow et al., [<reflink idref="bib6" id="ref3">6</reflink>]), a phenomenon sometimes referred to as the "testlet effect." Ignoring testlet effects in item response theory (IRT) models for continuous traits has been shown to result in biased model parameter estimates and their standard errors (e.g., Bradlow et al., [<reflink idref="bib6" id="ref4">6</reflink>]; Sireci et al., [<reflink idref="bib59" id="ref5">59</reflink>]; Wainer et al., [<reflink idref="bib68" id="ref6">68</reflink>]).</p> <p>The testlet format has also been adopted for cognitive diagnosis (CD), a restricted latent class paradigm in educational and psychological measurement that combines the goals of formative assessment with strict psychometric standards (e.g., von Davier & Lee, [<reflink idref="bib64" id="ref7">64</reflink>]). Like in IRT, local independence of the item responses is a fundamental assumption in CD to ensure valid parameter estimation (e.g., Rupp et al., [<reflink idref="bib55" id="ref8">55</reflink>]). The detrimental impact of violations of local independence on the accuracy of item parameter estimates, their standard errors, and the classification of examinees has been demonstrated in several studies (Chen et al., [<reflink idref="bib10" id="ref9">10</reflink>]; Hansen, [<reflink idref="bib26" id="ref10">26</reflink>]; Hansen et al., [<reflink idref="bib27" id="ref11">27</reflink>]; Lim & Drasgow, 2019).</p> <p>Extant diagnostic procedures for detecting testlet effects either involve comparing the fit of models with and without extra testlet parameters or, at the level of item pairs, testing for potential violations of the local independence assumption. Examples are the studies of Hansen ([<reflink idref="bib26" id="ref12">26</reflink>]), Hansen et al. ([<reflink idref="bib27" id="ref13">27</reflink>]), and Lim and Drasgow (2019). The latter proposed the use of Rosenbaum ([<reflink idref="bib53" id="ref14">53</reflink>], [<reflink idref="bib54" id="ref15">54</reflink>]) Mantel‐Haenszel (MH) <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> statistic (Mantel & Haenszel, [<reflink idref="bib48" id="ref16">48</reflink>]) for the discovery of pairwise item dependencies in CD. Yet, the application of Rosenbaum's MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic in CD is not without limitations. Different from Rosenbaum's original studies, in which observable total test scores were employed as the stratification variable, Lim and Drasgow (2019) proposed to use examinees' latent proficiency class membership as the grouping variable. But the true class membership of examinees is unknown and must be estimated from the item responses. Consequently, the sampling distribution of the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic is no longer an exact chi‐squared distribution, but rather an approximation thereof. A further limitation is that item pairs can only be inspected one at a time. How might one integrate the multiple MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistics obtained for a collection of items that form a testlet and whose responses are therefore very likely correlated in violation of the local independence assumption? It is evident that the obtained pairwise MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistics must be integrated somehow to support an inference about the presence or absence of a testlet effect that is due to an entire cluster of items forming a testlet. However, currently, no rationale exists for how to integrate multiple MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0011" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistics that are very likely not independent, and more crucially, the sampling distribution of such an integrated MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic is unknown.</p> <p>In this article, a procedure for integrating multiple pairwise MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistics to evaluate testlet effects in cognitive diagnosis is proposed. The unknown sampling distribution issue is addressed by implementing a parametric bootstrap resampling scheme. The proposed method can be applied to any CD assessment data that are suspected of being contaminated by testlet effects—without first having to establish the presence of testlet effects by fitting the data with complex parametric testlet CDMs. The issue of the unknown sampling distribution is addressed through the implementation of a parametric bootstrap resampling scheme.</p> <p>The next section provides a brief overview of CD and some of its key concepts, of local independence, testlets, and the modeling of testlet effects. The third section presents a summary of the existing approaches to the detection of testlet effects. In Section 4, the parametric bootstrap testlet MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic is introduced. Simulation studies are reported for investigating the type I error control and the power of the proposed parametric bootstrap MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic under various experimental conditions. Data with and without testlet effects were generated using the Deterministic Input, Noisy "AND" Gate (DINA) model (Haertel, [<reflink idref="bib28" id="ref17">28</reflink>]; Junker & Sijtsma, [<reflink idref="bib35" id="ref18">35</reflink>]; Macready & Dayton, [<reflink idref="bib47" id="ref19">47</reflink>]) and the saturated loglinear cognitive diagnosis model (LCDM; Henson et al., [<reflink idref="bib31" id="ref20">31</reflink>]). The application of the proposed parametric bootstrap testlet MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic to the 2015 Program for International Student Assessment (PISA) collaborative problem solving (CPS) assessment data illustrates the usage in practice. The paper concludes with a discussion of the findings and implications for future research.</p> <hd id="AN0190280385-3">Reviews: Key Technical Concepts</hd> <p></p> <hd id="AN0190280385-4">Cognitive Diagnosis</hd> <p>CD is the process of identifying the specific cognitive abilities that an individual possesses or lacks. CD models (CDMs) describe cognitive ability as a composite of multiple latent skills, or attributes, which an examinee may have either mastered or failed to master. The mastery of different combinations of attributes allows for the identification of distinct proficiency classes, which are represented as K‐dimensional binary vectors, denoted as <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi mathvariant="bold-italic">α</mi></mrow><mo>=</mo><msup><mrow><mo>(</mo><msub><mi>α</mi><mn>1</mn></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mi>α</mi><mi>k</mi></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mi>α</mi><mi>K</mi></msub><mo>)</mo></mrow><mo>′</mo></msup></mrow><annotation encoding="application/x-tex">$\bm {\alpha } = (\alpha _1,\ldots, \alpha _k, \ldots, \alpha _K)^{\prime }$</annotation></semantics></math> </ephtml> , where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0018" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>α</mi><mi>k</mi></msub><mo>=</mo><mn>0</mn><mo>,</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">$\alpha _k = 0, 1$</annotation></semantics></math> </ephtml> indicates failure or mastery, respectively. The items of a CD test are defined by their q‐vectors, which document the individual attribute profiles necessary for answering an item correctly. The q‐vectors constitute the rows of the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0019" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math> </ephtml> ‐items‐by‐ <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0020" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math> </ephtml> ‐attributes Q‐matrix of a CD assessment, with entries <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0021" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>q</mi><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>=</mo><mn>0</mn><mo>,</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">$q_{jk} = 0, 1$</annotation></semantics></math> </ephtml> indicating whether the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0022" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>k</mi><mrow><mi>t</mi><mi>h</mi></mrow></msup><annotation encoding="application/x-tex">$k^{th}$</annotation></semantics></math> </ephtml> attribute is required by the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0023" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> th item (Tatsuoka, [<reflink idref="bib62" id="ref21">62</reflink>]). The Q‐matrix of an assessment must be complete; otherwise, the identifiability of the CDM fitted to the data cannot be guaranteed. The objective of CD is to estimate an examinee's proficiency class based on their item responses.</p> <hd id="AN0190280385-5">Local Independence</hd> <p>"Local independence means that within any group of examinees all characterized by the same values <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0024" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>θ</mi><mn>1</mn></msub><mo>,</mo><msub><mi>θ</mi><mn>2</mn></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mi>θ</mi><mi>k</mi></msub></mrow><annotation encoding="application/x-tex">$\theta _1, \theta _2, \ldots, \theta _k$</annotation></semantics></math> </ephtml> , the (conditional) distributions of the item scores are all independent of each other. [ <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0025" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtext>...</mtext><annotation encoding="application/x-tex">$\ldots$</annotation></semantics></math> </ephtml> ] The assumption of local independence is equivalent to the assumption that the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0026" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>θ</mi><mn>1</mn></msub><mo>,</mo><msub><mi>θ</mi><mn>2</mn></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mi>θ</mi><mi>k</mi></msub></mrow><annotation encoding="application/x-tex">$\theta _1, \theta _2, \ldots, \theta _k$</annotation></semantics></math> </ephtml> under consideration span the complete latent space" (Lord & Novick, [<reflink idref="bib44" id="ref22">44</reflink>], p. 361).</p> <p>For IRT models, the literature reporting negative consequences of violations of local independence is vast; a few examples may suffice here: (a) on test reliability and validity (Bradlow et al., [<reflink idref="bib6" id="ref23">6</reflink>]; Marais & Andrich, [<reflink idref="bib49" id="ref24">49</reflink>]; Sireci et al., [<reflink idref="bib59" id="ref25">59</reflink>]; Wainer & Wang, [<reflink idref="bib66" id="ref26">66</reflink>]; Yen, [<reflink idref="bib73" id="ref27">73</reflink>]), (b) on item parameter estimation (Ackerman, [<reflink idref="bib1" id="ref28">1</reflink>]; Bradlow et al., [<reflink idref="bib6" id="ref29">6</reflink>]; Wainer & Wang, [<reflink idref="bib66" id="ref30">66</reflink>]), (c) on item fit (Marais & Andrich, [<reflink idref="bib49" id="ref31">49</reflink>]), and (d) on test equating and scaling (Bishop & Omar, [<reflink idref="bib4" id="ref32">4</reflink>]; Lee et al., [<reflink idref="bib38" id="ref33">38</reflink>]; Li et al., [<reflink idref="bib39" id="ref34">39</reflink>]).</p> <p>In the context of CD, local independence concerns whether the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0027" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math> </ephtml> ‐dimensional (latent) attribute space of an assessment has been adequately determined so that the proficiency classes of examinees are exhaustively identified. If this condition is fulfilled, then the item responses are independent of each other when conditioning on an examinee's attribute profile <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0028" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_i$</annotation></semantics></math> </ephtml> , where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0029" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>=</mo><mn>1</mn><mo>,</mo><mtext>...</mtext><mo>,</mo><mi>N</mi></mrow><annotation encoding="application/x-tex">$i=1, \ldots, N$</annotation></semantics></math> </ephtml> is the examinee index (Rupp et al., [<reflink idref="bib55" id="ref35">55</reflink>]). Put another way, local independence is equivalent to the assumption that the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0030" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math> </ephtml> latent variables, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0031" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>α</mi><mn>1</mn></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mi>α</mi><mi>K</mi></msub></mrow><annotation encoding="application/x-tex">$\alpha _1, \ldots, \alpha _K$</annotation></semantics></math> </ephtml> , span the entire latent ability space—casually speaking, no skill has been overlooked. In fact, violations of local independence typically point at the likely misspecification of the collection of attributes assumed to underly a test. A formal equivalent to local independence is that all conditional item response covariances equal zero (e.g., Rosenbaum, [<reflink idref="bib53" id="ref36">53</reflink>], [<reflink idref="bib54" id="ref37">54</reflink>]; Stout, [<reflink idref="bib60" id="ref38">60</reflink>]): <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0032" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>Cov</mtext><mfenced separators="" open="(" close=")"><msub><mi>Y</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>,</mo><msub><mi>Y</mi><mrow><mi>i</mi><msup><mi>j</mi><mo>′</mo></msup></mrow></msub><mo>∣</mo><mrow><msub><mi mathvariant="bold">α</mi><mi mathvariant="bold">i</mi></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><mn>0</mn><mspace width="1em" /><mo>∀</mo><mspace width="0.33em" /><mi>j</mi><mo>,</mo><msup><mi>j</mi><mo>′</mo></msup><mo>∈</mo><mrow><mo>{</mo><mn>1</mn><mo>,</mo><mtext>...</mtext><mo>,</mo><mi>J</mi><mo>}</mo></mrow></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \mbox{Cov} {\left(Y_{ij},Y_{ij^{\prime }} \mid \bm {\alpha _i} \right)} = 0\quad \forall\ j, j^{\prime } \in \lbrace 1, \ldots, J \rbrace \end{equation*}$$</annotation></semantics></math> </ephtml> when conditioning on an examinee's attribute profile <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0033" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_i$</annotation></semantics></math> </ephtml> . A stronger version of local independence is given by the factorization of the joint item response probabilities 1 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0034" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mrow><mo>(</mo><msub><mi mathvariant="bold-italic">Y</mi><mi>i</mi></msub><mo>∣</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><munderover><mo>∏</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>J</mi></munderover><mi>P</mi><mfenced separators="" open="(" close=")"><msub><mi>Y</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>∣</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} P(\bm {Y}_i \mid \bm {\alpha }_i) = \prod _{j = 1}^J P{\left(Y_{ij} \mid \bm {\alpha }_i \right)} \end{equation}$$</annotation></semantics></math> </ephtml> (e.g., Rupp et al., [<reflink idref="bib55" id="ref39">55</reflink>]). Within the context of CD, the most significant negative consequence of violations of local independence is the increased risk of examinee misclassification (e.g., Hansen, [<reflink idref="bib26" id="ref40">26</reflink>]; Hansen et al., [<reflink idref="bib27" id="ref41">27</reflink>]; Lim & Drasgow, 2019; Lim, [<reflink idref="bib41" id="ref42">41</reflink>]).</p> <hd id="AN0190280385-6">Testlets</hd> <p>As mentioned earlier, a testlet, or "item bundle" (Rosenbaum, [<reflink idref="bib54" id="ref43">54</reflink>]), is a cluster of items that share a common stimulus, called the "stem," that provides the context and the key information required for correctly responding to the set of items designed to assess an examinee's ability to process the input from the stem (Wainer & Kiely, [<reflink idref="bib65" id="ref44">65</reflink>]). Typical examples of items in testlet format include text passages in language comprehension tests coupled with multiple questions regarding the content of the text. Another common type of testlet is the fill‐in‐the‐blanks item, in which the items are embedded in the text as empty spaces to be filled in (e.g., Klein‐Braley, [<reflink idref="bib37" id="ref45">37</reflink>]). In reiterating, the testlet format has many appealing features like offering an effective and economic mode of conducting assessments that target contextual knowledge and complex skills but without imposing the burden of long‐winded test instructions on examinees (e.g., DeMars, [<reflink idref="bib18" id="ref46">18</reflink>]; Wainer et al., [<reflink idref="bib67" id="ref47">67</reflink>]; Wainer & Kiely, [<reflink idref="bib65" id="ref48">65</reflink>]). A significant issue of testlets, however, is the likely local dependence among responses arising from the shared item stem—in plain English: examinee answers to one question may "carry over" in affecting the responses to subsequent items.</p> <hd id="AN0190280385-7">Modeling Testlet Effects</hd> <p>A common method employed in IRT to remedy local dependence due to testlet effects consists in augmenting the model‐specific item response function (IRF) with a testlet‐specific random effect term. The testlet IRT model (Bradlow et al., [<reflink idref="bib6" id="ref49">6</reflink>]; Wainer et al., [<reflink idref="bib68" id="ref50">68</reflink>]), for example, has for each testlet an extra normal random effect with testlet‐specific variance included in its IRF to account for the nuisance variability originating from the correlations between the items within that testlet. The latent ability trait is assumed to be independent of the testlet specific effects. The bi‐factor model (DeMars, [<reflink idref="bib17" id="ref51">17</reflink>]; Gibbons & Hedeker, [<reflink idref="bib23" id="ref52">23</reflink>]) is based on similar idea: a general factor represents the overall latent ability trait measured by all items and testlet‐specific factors account for the variance shared among items within each testlet that is not absorbed by the general factor. Thus, item responses are perceived as influenced by the general trait and the testlet‐specific trait; both of which are typically assumed to be uncorrelated. Two‐tier models (Cai, [<reflink idref="bib7" id="ref53">7</reflink>]) extend the bi‐factor approach by including a hierarchical structure where item responses depend on both a general trait and multiple testlet‐specific traits. This hierarchical structure allows for the modeling of complex dependencies within testlets. All three approaches assume that (<reflink idref="bib1" id="ref54">1</reflink>) general and testlet‐specific traits are normally distributed; (<reflink idref="bib2" id="ref55">2</reflink>) both the general and testlet‐specific traits capture the local dependence within testlets; (<reflink idref="bib3" id="ref56">3</reflink>) each testlet‐specific dimension is mutually exclusive; and (<reflink idref="bib4" id="ref57">4</reflink>) general and testlet‐specific traits are independent (e.g., Xu et al., [<reflink idref="bib72" id="ref58">72</reflink>]).</p> <p>In CD, testlet‐effect models have been proposed that, similarly, incorporate a testlet‐specific random effect to account for within‐testlet dependencies (Hansen, [<reflink idref="bib26" id="ref59">26</reflink>]; Hansen et al., [<reflink idref="bib27" id="ref60">27</reflink>]; Ma et al., [<reflink idref="bib46" id="ref61">46</reflink>]; Sha, [<reflink idref="bib58" id="ref62">58</reflink>]; Xu et al., [<reflink idref="bib72" id="ref63">72</reflink>]; Zhan et al., [<reflink idref="bib75" id="ref64">75</reflink>], [<reflink idref="bib76" id="ref65">76</reflink>]). Like in IRT models, these CDMs typically assume that general and testlet‐specific traits are independent. According to Xu et al. ([<reflink idref="bib72" id="ref66">72</reflink>]), however, a major difference between IRT and CDMs, despite certain conceptual links, consists in the mixing of discrete latent attribute profiles with continuous testlet effects into an IRF. Hansen et al. ([<reflink idref="bib27" id="ref67">27</reflink>]) devised a testlet CDM based on the saturated LCDM (Henson et al., [<reflink idref="bib31" id="ref68">31</reflink>]), as its link function conveniently accommodates the linear predictor formed by the sum of attribute main effects and their higher‐order interactions plus the random testlet effect, the term added to the IRF to account for the dependency among items within a testlet (see equation 9 in Hansen et al., [<reflink idref="bib27" id="ref69">27</reflink>]; p. 228—repeated here for convenience)2 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0035" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo form="prefix">logit</mo><mfenced separators="" open="(" close=")"><mi>P</mi><mo>(</mo><msub><mi>Y</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>∣</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>,</mo><msub><mi>ξ</mi><mi>t</mi></msub><mo>)</mo></mfenced><mo linebreak="goodbreak">=</mo><msub><mi>λ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msubsup><mi mathvariant="bold-italic">λ</mi><mi>j</mi><mo>′</mo></msubsup><mi mathvariant="bold-italic">h</mi><mrow><mo>(</mo><msub><mi mathvariant="bold">q</mi><mi>j</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>)</mo></mrow><mo linebreak="goodbreak">+</mo><msub><mi>β</mi><mi>t</mi></msub><msub><mi>ξ</mi><mi>t</mi></msub><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \operatorname{logit}{\left(P (Y_{ij} = 1 \mid \bm {\alpha }_i, \xi _t) \right)} = \lambda _{j0} + \bm {\lambda }^\prime _j \bm {h} (\mathbf {q}_j,\bm {\alpha }_i) + \beta _{t} \xi _t, \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0036" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>Y</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><annotation encoding="application/x-tex">$Y_{ij}$</annotation></semantics></math> </ephtml> denotes the response of examinee <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0037" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> to item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0038" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> in testlet <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0039" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0040" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>λ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><annotation encoding="application/x-tex">$\lambda _{j0}$</annotation></semantics></math> </ephtml> denotes the intercept parameter (equivalent to the guessing parameter <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0041" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>g</mi><mi>j</mi></msub><annotation encoding="application/x-tex">$g_j$</annotation></semantics></math> </ephtml> ), <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0042" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi mathvariant="bold-italic">λ</mi><mi>j</mi><mo>′</mo></msubsup><mi mathvariant="bold-italic">h</mi><mrow><mo>(</mo><msub><mi mathvariant="bold">q</mi><mi>j</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\bm {\lambda }^\prime _j \bm {h}(\mathbf {q}_j,\bm {\alpha }_i)$</annotation></semantics></math> </ephtml> is the linear predictor, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0043" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>β</mi><mi>t</mi></msub><annotation encoding="application/x-tex">$\beta _t$</annotation></semantics></math> </ephtml> is the slope of the testlet‐specific parameter <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0044" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>ξ</mi><mi>t</mi></msub><annotation encoding="application/x-tex">$\xi _t$</annotation></semantics></math> </ephtml> that is shared by all items in testlet <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0045" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math> </ephtml> . One should notice that Hansen et al. ([<reflink idref="bib27" id="ref70">27</reflink>]) parameterized the testlet effect as an item‐related phenomenon. Constraining the linear predictor so that certain <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0046" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>λ</mi><annotation encoding="application/x-tex">$\lambda$</annotation></semantics></math> </ephtml> ‐coefficients are set to zero allows for the reexpression of specific CDMs as submodels of the model presented in Equation 2; for example, the DINA model with a testlet‐effect added and reparameterized as a (reduced) LCDM (see also Sha, [<reflink idref="bib58" id="ref71">58</reflink>])3 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0047" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>logit</mi><mrow><mo>(</mo><mi>P</mi><mrow><mo>(</mo><msub><mi>Y</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>|</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>,</mo><msub><mi>ξ</mi><mi>t</mi></msub><mo>)</mo></mrow><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><msub><mi>λ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>λ</mi><mrow><mi>j</mi><mo>(</mo><mn>12</mn><mo>,</mo><mrow><mtext>...</mtext><mo>,</mo></mrow><mi>K</mi><mo>)</mo></mrow></msub><munderover><mo>∏</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msubsup><mi>α</mi><mrow><mi>i</mi><mi>k</mi></mrow><msub><mi>q</mi><mrow><mi>j</mi><mi>k</mi></mrow></msub></msubsup><mo linebreak="goodbreak">+</mo><msub><mi>β</mi><mi>t</mi></msub><msub><mi>ξ</mi><mi>t</mi></msub><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} {\mathrm{logit}}(P({Y}_{ij}=1\vert {\bm{\alpha}}_{i},{\xi}_{t}))={\lambda}_{j0}+{\lambda}_{j(12,{{\ldots},}K)}\prod _{k=1}^{K}{\alpha}_{{ik}}^{{q}_{{jk}}}+{\beta}_{t}{\xi}_{t}, \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0048" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>λ</mi><mrow><mn>0</mn><mi>j</mi></mrow></msub><mo>=</mo><mi>ln</mi><mrow><mo>(</mo><mrow><msub><mi>g</mi><mi>j</mi></msub><mo>/</mo><mrow><mo>(</mo><mn>1</mn><mo>−</mo><msub><mi>g</mi><mi>j</mi></msub><mo>)</mo></mrow></mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\lambda _{0j} = \ln ({g_j /(1-g_j)})$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0049" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>λ</mi><mrow><mi>j</mi><mo>(</mo><mn>12</mn><mtext>...</mtext><mi>K</mi><mo>)</mo></mrow></msub><mo>=</mo><mo>−</mo><msub><mi>λ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo>+</mo><mi>ln</mi><mrow><mo>(</mo><mrow><mo>(</mo><mn>1</mn><mo>−</mo><msub><mi>s</mi><mi>j</mi></msub><mo>)</mo></mrow><mo>/</mo><msub><mi>s</mi><mi>j</mi></msub><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">${\lambda}_{j(12{{\ldots}}K)}=-{\lambda}_{j0}+\ln ((1-{s}_{j})/{s}_{j})$</annotation></semantics></math> </ephtml> are guessing and slipping parameters, respectively. Notice that this model also implements the testlet effect <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0050" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>ξ</mi><mi>t</mi></msub><annotation encoding="application/x-tex">$\xi _t$</annotation></semantics></math> </ephtml> as an item‐related parameter.</p> <p>Xu et al. ([<reflink idref="bib72" id="ref72">72</reflink>]) raised concerns about the ecological validity of the conceptualization and implementation of the testlet effects as item‐related phenomena in these CDMs. As Xu et al. ([<reflink idref="bib72" id="ref73">72</reflink>]) argue, in the case of CDMs, different from IRT models, the discrete nature of the latent attribute profiles induces potential dependency between them and the testlet effects. To account for this dependency, Xu et al. ([<reflink idref="bib72" id="ref74">72</reflink>]) propose to treat the testlet effect as a random interaction term involving the items in a testlet and examinees. The corresponding CDM is called the Interaction Testlet DINA (IT‐DINA) model (equation 8, Xu et al., [<reflink idref="bib72" id="ref75">72</reflink>]). Notice that the form of the IRF of the IT‐DINA presented here uses the logit link (instead of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0051" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi mathvariant="normal">Φ</mi><annotation encoding="application/x-tex">$\Phi$</annotation></semantics></math> </ephtml> that was employed in Xu et al., [<reflink idref="bib72" id="ref76">72</reflink>], as the link function)4 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0052" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mo form="prefix">logit</mo></mrow><mfenced separators="" open="(" close=")"><mi>P</mi><mfenced separators="" open="(" close=")"><msub><mi>Y</mi><mi>ij</mi></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>∣</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">ξ</mi><mi>ti</mi></msub></mfenced></mfenced><mo linebreak="badbreak">=</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>1</mn></mrow></msub><msub><mi>η</mi><mi>ij</mi></msub><mo linebreak="goodbreak">+</mo><msub><mi>β</mi><mi>jt</mi></msub><msub><mi>ξ</mi><mrow><mi>ti</mi><mn>1</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>γ</mi><mi>jt</mi></msub><mrow><mo>(</mo><mn>2</mn><msub><mi>η</mi><mi>ij</mi></msub><mo linebreak="goodbreak">−</mo><mn>1</mn><mo>)</mo></mrow><msub><mi>ξ</mi><mrow><mi>ti</mi><mn>2</mn></mrow></msub><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \operatorname{\mathrm{logit}}\left(P\left({Y}_{\textit{ij}}=1\mid {\bm{\alpha}}_{i},{\bm{\xi}}_{\textit{ti}}\right)\right)={\delta}_{j0}+{\delta}_{j1}{\eta}_{\textit{ij}}+{\beta}_{\textit{jt}}{\xi}_{\textit{ti}1}+{\gamma}_{\textit{jt}}(2{\eta}_{\textit{ij}}-1){\xi}_{\textit{ti}2}, \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0053" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>δ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo>=</mo><msub><mi>λ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub></mrow><annotation encoding="application/x-tex">${\delta}_{j0}={\lambda}_{j0}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0054" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>δ</mi><mrow><mi>j</mi><mn>1</mn></mrow></msub><mo>=</mo><msub><mi>λ</mi><mrow><mi>j</mi><mo>(</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mrow><mtext>...</mtext><mo>,</mo></mrow><mi>K</mi><mo>)</mo></mrow></msub></mrow><annotation encoding="application/x-tex">${\delta}_{j1}={\lambda}_{j(1,2,{{\ldots},}K)}$</annotation></semantics></math> </ephtml> as defined earlier in connection with Equation 3. The ideal response of examinee <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0055" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> to item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0056" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> is denoted as <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0057" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>η</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mi mathvariant="script">I</mi><mrow><mo>[</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>⪰</mo><msub><mi mathvariant="bold">q</mi><mi>j</mi></msub><mo>]</mo></mrow></mrow><annotation encoding="application/x-tex">$\eta _{ij} = \mathcal {I}[ \bm {\alpha }_i \succeq \mathbf {q}_j ]$</annotation></semantics></math> </ephtml> ( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0058" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="script">I</mi><mo>(</mo><mo>·</mo><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathcal {I}(\cdot)$</annotation></semantics></math> </ephtml> is the indicator function); alternatively: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0059" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>η</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><msubsup><mo>∏</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></msubsup><msubsup><mi>α</mi><mrow><mi>i</mi><mi>k</mi></mrow><msub><mi>q</mi><mrow><mi>j</mi><mi>k</mi></mrow></msub></msubsup></mrow><annotation encoding="application/x-tex">$\eta _{ij} = \prod _{k=1}^{K} \alpha _{ik}^{q_{jk}}$</annotation></semantics></math> </ephtml> ; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0060" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">ξ</mi><mrow><mi>t</mi><mi>i</mi></mrow></msub><mo>=</mo><msup><mrow><mo>(</mo><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>1</mn></mrow></msub><mo>,</mo><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>2</mn></mrow></msub><mo>)</mo></mrow><mo>′</mo></msup></mrow><annotation encoding="application/x-tex">$\bm {\xi }_{ti} = (\xi _{ti1}, \xi _{ti2})^{\prime }$</annotation></semantics></math> </ephtml> , where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0061" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>1</mn></mrow></msub><mo>,</mo><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>2</mn></mrow></msub><mover><mo>∼</mo><mtext>iid</mtext></mover><mi mathvariant="script">N</mi><mrow><mo>(</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$ \xi _{ti1}, \xi _{ti2} \stackrel{\mbox{iid}}{\sim } \mathcal {N} (0, 1)$</annotation></semantics></math> </ephtml> are two examinee‐specific traits on testlet <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0062" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math> </ephtml> . In particular, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0063" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>1</mn></mrow></msub><annotation encoding="application/x-tex">$\xi _{ti1}$</annotation></semantics></math> </ephtml> is the unique testlet effect, whose mean is unaffected by <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0064" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_i$</annotation></semantics></math> </ephtml> , whereas <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0065" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>2</mn></mrow></msub><annotation encoding="application/x-tex">$\xi _{ti2}$</annotation></semantics></math> </ephtml> is linked to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0066" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_i$</annotation></semantics></math> </ephtml> through <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0067" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>η</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><annotation encoding="application/x-tex">$\eta _{ij}$</annotation></semantics></math> </ephtml> , which is reflected in the parameterization <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0068" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mo>(</mo><mn>2</mn><msub><mi>η</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>−</mo><mn>1</mn><mo>)</mo></mrow><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>2</mn></mrow></msub></mrow><annotation encoding="application/x-tex">$(2 \eta _{ij} - 1) \xi _{ti2}$</annotation></semantics></math> </ephtml> .</p> <p>Zhan et al. ([<reflink idref="bib76" id="ref77">76</reflink>]) and Ma et al. ([<reflink idref="bib46" id="ref78">46</reflink>]) have also proposed testlet CDMs that feature testlet effects as item‐by‐examinee normal random interactions, which allow for the possibility that testlet effects have a differential impact on examinees such that some benefit from testlet effects and others do not, or are even disadvantaged by them.</p> <hd id="AN0190280385-8">Methods for Detecting Testlet Effects</hd> <p>Common approaches involve either model comparisons testing for the inclusion of extra testlet parameters or, at the item level, testing for pairwise local independence. The effects of testlets and their diagnostics have been the subject of extensive research within the context of IRT. In a comprehensive simulation study, DeMars ([<reflink idref="bib18" id="ref79">18</reflink>]) investigated the efficacy of various indices for detecting testlet effects in comparing a unidimensional with a bifactor IRT model, where the latter included a nuisance factor to capture the possible testlet effect. Conventional model fit statistics, including the log‐likelihood difference test (e.g., Bock & Gibbons, [<reflink idref="bib5" id="ref80">5</reflink>]), AIC (Akaike, 1974), BIC (Schwarz, 1976), and sample‐size adjusted BIC (SSA‐BIC; Sclove, 1987) were employed for the purpose of comparing the testlet and non‐testlet models. Moreover, Stout's (2005) DIMTEST was utilized. The various methods were then evaluated in terms of their respective type I and type II error rates. The BIC, SSA‐BIC, and DIMTEST were found to be reasonably sensitive to the presence of testlet effects.</p> <p>Other methods for assessing testlet effects include pairwise item fit indices. For example, the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0069" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic (Mantel & Haenszel, [<reflink idref="bib48" id="ref81">48</reflink>]), stratified along the levels of total test scores (Rosenbaum, [<reflink idref="bib53" id="ref82">53</reflink>]), and the Q3 measure, the correlation between the residuals for item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0070" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> and item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0071" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>j</mi><mo>′</mo></msup><annotation encoding="application/x-tex">$j^{\prime }$</annotation></semantics></math> </ephtml> (Yen, [<reflink idref="bib74" id="ref83">74</reflink>], [<reflink idref="bib73" id="ref84">73</reflink>]). Pairwise indices for testlet effects in three‐parameter testlet IRT models are reviewed and evaluated in Kim et al. ([<reflink idref="bib36" id="ref85">36</reflink>]).</p> <p>Although there are conceptual connections between CD and IRT models, the diagnostic indices evaluated in large‐scale studies for IRT models do not display comparable sensitivity within a CD context. Therefore, they cannot simply be transferred to CD. As Xu et al. ([<reflink idref="bib72" id="ref86">72</reflink>]) have observed, these complications are likely to arise from the combination of discrete attribute profiles with continuous testlet effects into an IRF. Indeed, only a limited number of studies have explored methods for detecting testlet effects in the context of CD. Hansen et al. ([<reflink idref="bib27" id="ref87">27</reflink>]) investigated the usefulness of an overall model fit statistic, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0072" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> (Maydeu‐Olivares & Joe, [<reflink idref="bib51" id="ref88">51</reflink>]), and an item pairwise Chi‐squared test statistic, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0073" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>χ</mi><mrow><mi>j</mi><msup><mi>j</mi><mo>′</mo></msup></mrow><mn>2</mn></msubsup><annotation encoding="application/x-tex">$\chi ^2_{jj^{\prime }}$</annotation></semantics></math> </ephtml> (Chen & Thissen, [<reflink idref="bib9" id="ref89">9</reflink>]), for detecting testlet effects. The <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0074" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic demonstrated excellent type I error rates and proved to be highly sensitive to the presence of testlet effects, whereas <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0075" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>χ</mi><mrow><mi>j</mi><msup><mi>j</mi><mo>′</mo></msup></mrow><mn>2</mn></msubsup><annotation encoding="application/x-tex">$\chi ^2_{jj^{\prime }}$</annotation></semantics></math> </ephtml> exhibited sensitivity only under certain conditions of model misspecification.</p> <p>In a recent study, Lim ([<reflink idref="bib41" id="ref90">41</reflink>]) examined the efficacy of various item pairwise indices in detecting testlet effects. The performance of the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0076" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic (Mantel & Haenszel, [<reflink idref="bib48" id="ref91">48</reflink>]; Rosenbaum, [<reflink idref="bib53" id="ref92">53</reflink>], [<reflink idref="bib54" id="ref93">54</reflink>]; see also Lim & Drasgow, [<reflink idref="bib42" id="ref94">42</reflink>], [<reflink idref="bib43" id="ref95">43</reflink>]), the Chi‐squared statistic <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0077" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>χ</mi><mrow><mi>j</mi><msup><mi>j</mi><mo>′</mo></msup></mrow><mn>2</mn></msubsup><annotation encoding="application/x-tex">$\chi ^2_{jj^{\prime }}$</annotation></semantics></math> </ephtml> (Chen & Thissen, [<reflink idref="bib9" id="ref96">9</reflink>]), and the absolute deviations of observed and predicted correlations <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0078" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>r</mi><mrow><mi>j</mi><msup><mi>j</mi><mo>′</mo></msup></mrow></msub><annotation encoding="application/x-tex">$r_{jj^{\prime }}$</annotation></semantics></math> </ephtml> (Chen et al., [<reflink idref="bib11" id="ref97">11</reflink>]) were evaluated using data conforming to three different CDMs: the DINA model (Haertel, [<reflink idref="bib28" id="ref98">28</reflink>]; Junker & Sijtsma, [<reflink idref="bib35" id="ref99">35</reflink>]; Macready & Dayton, [<reflink idref="bib47" id="ref100">47</reflink>]), the Additive Cognitive Diagnosis Model (A‐CDM; de la Torre, [<reflink idref="bib19" id="ref101">19</reflink>]), and the (saturated) LCDM (Henson et al., [<reflink idref="bib31" id="ref102">31</reflink>]). The IRF of each of the three models was augmented by a testlet‐specific parameter. To illustrate, consider the IRF of the DINA model with a testlet‐effect added and re‐parameterized as a (reduced) LCDM, as shown in Equation 3. All three indices demonstrated acceptable type I error rates and power.</p> <p>In concluding this section, it seems appropriate to offer a comment. The measures and statistics reviewed here for establishing testlet effects operate either at the level of the entire test, in comparison with a test containing no testlet(s), or at the level of individual item pairs. It should be noted that none of the aforementioned indices are suitable for examining the presence of an aggregated testlet effect as it may originate from taking into account all items in a cluster at once, which is a subtle but important distinction. To illustrate this point, consider the following example constructed from material presented in Xu et al. ([<reflink idref="bib72" id="ref103">72</reflink>]). Two distinct data sets were generated, each comprising 1,000 observations. One data set exhibited a testlet effect, while the other did not. The IT‐DINA model (equation 8, Xu et al., [<reflink idref="bib72" id="ref104">72</reflink>]) presented earlier in Equation 4 was used with the logit link and is repeated here for convenience <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0079" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mo form="prefix">logit</mo></mrow><mfenced separators="" open="(" close=")"><mi>P</mi><mfenced separators="" open="(" close=")"><msub><mi>Y</mi><mi>ij</mi></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>∣</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">ξ</mi><mi>ti</mi></msub></mfenced></mfenced><mo linebreak="badbreak">=</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>1</mn></mrow></msub><msub><mi>η</mi><mi>ij</mi></msub><mo linebreak="goodbreak">+</mo><msub><mi>β</mi><mi>jt</mi></msub><msub><mi>ξ</mi><mrow><mi>ti</mi><mn>1</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>γ</mi><mi>jt</mi></msub><mrow><mo>(</mo><mn>2</mn><msub><mi>η</mi><mi>ij</mi></msub><mo linebreak="goodbreak">−</mo><mn>1</mn><mo>)</mo></mrow><msub><mi>ξ</mi><mrow><mi>ti</mi><mn>2</mn></mrow></msub><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \operatorname{\mathrm{logit}}\left(P\left({Y}_{\textit{ij}}=1\mid {\bm{\alpha}}_{i},{\bm{\xi}}_{\textit{ti}}\right)\right)={\delta}_{j0}+{\delta}_{j1}{\eta}_{\textit{ij}}+{\beta}_{\textit{jt}}{\xi}_{\textit{ti}1}+{\gamma}_{\textit{jt}}(2{\eta}_{\textit{ij}}-1){\xi}_{\textit{ti}2}. \end{equation*}$$</annotation></semantics></math> </ephtml> The slipping and guessing parameters for all items were set to 0.1; the Q‐matrix from table 3 in Xu et al., 2024, which is shown in Table 1 and consists of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0080" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>28</mn></mrow><annotation encoding="application/x-tex">$J = 28$</annotation></semantics></math> </ephtml> items and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0081" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">$K=3$</annotation></semantics></math> </ephtml> attributes, was used. The examinee attribute patterns were created using the multivariate normal threshold model (Chiu et al., [<reflink idref="bib14" id="ref105">14</reflink>]). For each examinee, a continuous <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0082" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math> </ephtml> ‐dimensional vector, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0083" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">θ</mi><mi>i</mi></msub><mo>=</mo><msup><mrow><mo>(</mo><msub><mi>θ</mi><mrow><mi>i</mi><mn>1</mn></mrow></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mi>θ</mi><mrow><mi>i</mi><mi>K</mi></mrow></msub><mo>)</mo></mrow><mo>′</mo></msup></mrow><annotation encoding="application/x-tex">$\bm {\theta }_i = (\theta _{i1}, \ldots, \theta _{iK})^{\prime }$</annotation></semantics></math> </ephtml> , was drawn from a multivariate normal distribution, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0084" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi><mi>V</mi><mi>N</mi><mfenced separators="" open="(" close=")"><msub><mn mathvariant="bold">0</mn><mi>K</mi></msub><mo>,</mo><mrow><mi mathvariant="bold">Σ</mi></mrow></mfenced></mrow><annotation encoding="application/x-tex">$MVN \left(\mathbf {0}_K, \bm {\Sigma } \right)$</annotation></semantics></math> </ephtml> , with zero expectation vector and the variance‐covariance matrix <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0085" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">Σ</mi></mrow><annotation encoding="application/x-tex">$\bm {\Sigma }$</annotation></semantics></math> </ephtml> having unit variances and common covariances <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0086" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mrow><mi>k</mi><msup><mi>k</mi><mo>′</mo></msup></mrow></msub><mo>=</mo><mn>0.3</mn></mrow><annotation encoding="application/x-tex">$\sigma _{kk^{\prime }} = 0.3$</annotation></semantics></math> </ephtml> . The <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0087" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math> </ephtml> ‐dimensional continuous vector was dichotomized according to <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0088" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>α</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifθik≥Φ−1k(K+1)0otherwise</mfenced><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \alpha _{ik} = {\begin{cases} 1 & \mbox{if $\theta _{ik} \ge \Phi ^{-1} {\left(\frac{k}{(K+1)} \right)}$} \\ 0 & \mbox{otherwise} \end{cases}}. \end{equation*}$$</annotation></semantics></math> </ephtml> The 1,000 synthetic response vectors in the first data set were all "contaminated" <bold>with</bold> testlet effects. When generating the second data set <bold>without</bold> testlet effects, the terms <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0089" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>β</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>1</mn></mrow></msub><mo>+</mo><msub><mi>γ</mi><mrow><mi>j</mi><mi>t</mi></mrow></msub><mrow><mo>(</mo><mn>2</mn><msub><mi>η</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>−</mo><mn>1</mn><mo>)</mo></mrow><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>2</mn></mrow></msub></mrow><annotation encoding="application/x-tex">$\beta _{jt} \xi _{ti1} + \gamma _{jt} (2 \eta _{ij} - 1) \xi _{ti2}$</annotation></semantics></math> </ephtml> were excluded from the IRF.</p> <p>1 Table Q‐Matrix with J=28$J = 28$ Items Using K=3$K = 3$ Attributes (table 3, Xu et al., 2024)</p> <p> <ephtml> <table><thead><tr><th>Item</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0092" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>1</mn></msub><annotation encoding="application/x-tex">$\alpha _1$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0093" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$\alpha _2$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0094" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>3</mn></msub><annotation encoding="application/x-tex">$\alpha _3$</annotation></semantics></math></p></th><th>Testlet <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0095" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>1</td></tr><tr><td>2</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>3</td><td>0</td><td>1</td><td>0</td><td>1</td></tr><tr><td>4</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>5</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>6</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>7</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>8</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>9</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>10</td><td>0</td><td>0</td><td>1</td><td>2</td></tr><tr><td>11</td><td>0</td><td>1</td><td>0</td><td>2</td></tr><tr><td>12</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>13</td><td>0</td><td>0</td><td>1</td><td>2</td></tr><tr><td>14</td><td>0</td><td>0</td><td>1</td><td>3</td></tr><tr><td>15</td><td>1</td><td>0</td><td>0</td><td>3</td></tr><tr><td>16</td><td>1</td><td>0</td><td>0</td><td>3</td></tr><tr><td>17</td><td>0</td><td>0</td><td>1</td><td>3</td></tr><tr><td>18</td><td>0</td><td>1</td><td>0</td><td>3</td></tr><tr><td>19</td><td>1</td><td>0</td><td>0</td><td>3</td></tr><tr><td>20</td><td>1</td><td>0</td><td>0</td><td>4</td></tr><tr><td>21</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>22</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>23</td><td>0</td><td>1</td><td>0</td><td>4</td></tr><tr><td>24</td><td>0</td><td>1</td><td>0</td><td>4</td></tr><tr><td>25</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>26</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>27</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>28</td><td>0</td><td>1</td><td>0</td><td>4</td></tr></tbody></table> </ephtml> </p> <p>Both data sets were fitted with the DINA EM‐algorithm implemented in the R‐package GDINA (Ma & de la Torre, [<reflink idref="bib45" id="ref106">45</reflink>]) without estimating any testlet effect; that is, in both instances, the data were treated as if they were not contaminated with a testlet effect. The objective was to investigate the impact of failing to consider the potential influence of a testlet effect. A comparison of the relative and absolute fit indices revealed that the contaminated data set yielded a substantially lower fit value than the non‐contaminated data set (see Table 2).</p> <p>2 Table Q‐Matrices for J=28$J = 28$ Items Using K=3$K = 3$ Attributes</p> <p> <ephtml> <table><thead><tr><th>Indices</th><th>With Testlet</th><th>Without Testlet</th></tr></thead><tbody><tr><td>Relative Fit</td><td /><td /></tr><tr><td>−2LL</td><td>28199.05</td><td>22063.14</td></tr><tr><td>AIC</td><td>28325.05</td><td>22189.14</td></tr><tr><td>BIC</td><td>28634.24</td><td>22498.33</td></tr><tr><td>Absolute Fit</td><td /><td /></tr><tr><td>RMSEA2</td><td>0.0536</td><td>0.0000</td></tr><tr><td>90% CI</td><td>[0.0506, 0.0567]</td><td>[0, 0.0058]</td></tr><tr><td>No. of</td><td /><td /></tr><tr><td>Parameters</td><td>63</td><td>63</td></tr><tr><td>PAR Score</td><td>0.950</td><td>0.999</td></tr></tbody></table> </ephtml> </p> <p>To further establish the presence of a testlet effect, the pattern‐wise agreement rate (PAR) was computed, which is defined as the proportion of examinees who were correctly classified <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0098" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>PAR</mi><mo linebreak="badbreak">=</mo><mfrac><mrow><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></msubsup><mi mathvariant="script">I</mi><mrow><mo>[</mo><msub><mover accent="true"><mi mathvariant="bold-italic">α</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>=</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>]</mo></mrow></mrow><mi>N</mi></mfrac></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \textit{PAR} = \frac{\sum ^N_{i=1} \mathcal {I}[\widehat{\bm {\alpha }}_i = \bm {\alpha }_i]}{N} \end{equation*}$$</annotation></semantics></math> </ephtml> (recall that for simulated data, the true proficiency class membership of examinees is known). It was anticipated that the classification of examinees would be affected by the failure of accounting for the presence of a testlet effect. Indeed, the PAR score for accurate examinee classification decreased by approximately <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0099" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>5</mn><mo>%</mo></mrow><annotation encoding="application/x-tex">$5\%$</annotation></semantics></math> </ephtml> when the data were contaminated with the testlet effect, as previously described: What can be said about diagnosing the testlet effect? The model fit indices and the PAR scores suggest poorer fit for the testlet‐contaminated data; however, further diagnostic insights cannot be gained from these measures.</p> <p>As an alternative diagnostic of the presence of a testlet effect, the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0100" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic was computed for all item pairs using the estimated proficiency class membership as stratification variable (the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0101" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic is discussed in greater detail in the next section). For the contaminated data, for instance, in approximately one‐third of the 36 item pairs from the fourth testlet, which is the largest testlet in the assessment, the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0102" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic produced a non‐significant result ( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0103" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>0.05</mn></mrow><annotation encoding="application/x-tex">$\alpha = 0.05$</annotation></semantics></math> </ephtml> ); thus, indicating that the item pair was not identified as belonging to that testlet. In case of the non‐contaminated data, only one item pair was (incorrectly) flagged by the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0104" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic (which is roughly in line with the findings of type‐I‐error studies conducted by Lim, 2020). In conclusion, a diagnostic device that does not require fitting the item responses with a complex testlet CDM using computationally expensive EM or MCMC algorithms, but that can reliably identify the aggregated testlet effect originating from an item cluster in a CD assessment would be highly desirable.</p> <hd id="AN0190280385-9">The Parametric Bootstrap Mantel‐Haenszel Statistic for Aggregated Testlet Effects</hd> <p></p> <hd id="AN0190280385-10">The "Classic" Mantel‐Haenszel Chi‐Squared Statistic</hd> <p>The MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0105" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic, also known as Cochran‐Mantel‐Haenszel (CMH) chi‐squared statistic, is used to assess the conditional independence of two binary random variables (Cochran, [<reflink idref="bib16" id="ref107">16</reflink>]; Mantel & Haenszel, [<reflink idref="bib48" id="ref108">48</reflink>]). A common scenario is that the frequencies observed on the binary random variables, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0106" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><mi>j</mi></msub><annotation encoding="application/x-tex">$X_j$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0107" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><msup><mi>j</mi><mo>′</mo></msup></msub><annotation encoding="application/x-tex">$X_{j^{\prime }}$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0108" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>≤</mo><mi>j</mi><mo><</mo><msup><mi>j</mi><mo>′</mo></msup><mo>≤</mo><mi>J</mi></mrow><annotation encoding="application/x-tex">$1 \le j < j^{\prime } \le J$</annotation></semantics></math> </ephtml> , have been aggregated into multiple <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0109" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>×</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">$2 \times 2$</annotation></semantics></math> </ephtml> cross tabulations conditioned on and arranged along the levels of a third categorical variable, called the stratification variable <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0110" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>C</mi><annotation encoding="application/x-tex">$C$</annotation></semantics></math> </ephtml> , with levels <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0111" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mn>1</mn><mo>,</mo><mtext>...</mtext><mo>,</mo><mi>C</mi></mrow><annotation encoding="application/x-tex">$c = 1, \ldots, C$</annotation></semantics></math> </ephtml> , referring to different locations, time points, individuals, or groups of subjects, and so on. Let <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0112" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>f</mi><mrow><mn>11</mn><mi>c</mi></mrow></msub><annotation encoding="application/x-tex">$f_{11c}$</annotation></semantics></math> </ephtml> be the frequency of examinees at level <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0113" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>c</mi><annotation encoding="application/x-tex">$c$</annotation></semantics></math> </ephtml> of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0114" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>C</mi><annotation encoding="application/x-tex">$C$</annotation></semantics></math> </ephtml> , who endorsed both items (coded as 1, as opposed to 0, which indicates non‐endorsement). The marginal frequencies are the row totals <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0115" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>f</mi><mrow><mn>1</mn><mo>+</mo><mi>c</mi></mrow></msub><annotation encoding="application/x-tex">$f_{1+c}$</annotation></semantics></math> </ephtml> and the column totals <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0116" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>f</mi><mrow><mo>+</mo><mn>1</mn><mi>c</mi></mrow></msub><annotation encoding="application/x-tex">$f_{+1c}$</annotation></semantics></math> </ephtml> ; the total sample size in the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0117" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>c</mi><annotation encoding="application/x-tex">$c$</annotation></semantics></math> </ephtml> th stratum is denoted as <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0118" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>f</mi><mrow><mo>+</mo><mo>+</mo><mi>c</mi></mrow></msub><annotation encoding="application/x-tex">$f_{++c}$</annotation></semantics></math> </ephtml> . All strata with a sample size <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0119" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>f</mi><mrow><mo>+</mo><mo>+</mo><mi>c</mi></mrow></msub><mo>≥</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">$f_{++c} \ge 1$</annotation></semantics></math> </ephtml> are included in the analysis. In the event that any cell count in a table is zero, the Haldane correction can be applied to each cell in the table to obtain a more accurate significance level for the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0120" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic (e.g., Li et al., 1979).</p> <p>The MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0121" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic for the item pair <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0122" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><mi>j</mi></msub><annotation encoding="application/x-tex">$X_j$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0123" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><msup><mi>j</mi><mo>′</mo></msup></msub><annotation encoding="application/x-tex">$X_{j^{\prime }}$</annotation></semantics></math> </ephtml> is then defined as 5 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0124" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>MH</mtext><mspace width="0.33em" /><mrow><msup><mi>χ</mi><mn>2</mn></msup><mfenced separators="" open="(" close=")"><msub><mi>X</mi><mi>j</mi></msub><mo>,</mo><msub><mi>X</mi><msup><mi>j</mi><mo>′</mo></msup></msub></mfenced></mrow><mo linebreak="badbreak">=</mo><mfrac><msup><mfenced separators="" open="(" close=")"><msub><mo>∑</mo><mi>c</mi></msub><msub><mi>f</mi><mrow><mn>11</mn><mi>c</mi></mrow></msub><mo>−</mo><msub><mo>∑</mo><mi>c</mi></msub><mi>E</mi><mrow><mo>(</mo><msub><mi>f</mi><mrow><mn>11</mn><mi>c</mi></mrow></msub><mo>)</mo></mrow></mfenced><mn>2</mn></msup><mrow><msub><mo>∑</mo><mi>c</mi></msub><mtext>Var</mtext><mrow><mo>(</mo><msub><mi>f</mi><mrow><mn>11</mn><mi>c</mi></mrow></msub><mo>)</mo></mrow></mrow></mfrac></mrow><annotation encoding="application/x-tex">$$\begin{equation} \mbox{MH $\chi ^2 {\left(X_j, X_{j^{\prime }} \right)}$} = \frac{{\left(\sum _{c} f_{11c} - \sum _{c} E(f_{11c}) \right)}^2}{\sum _{c} \text{Var}(f_{11c})} \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0125" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi><mrow><mo>(</mo><msub><mi>f</mi><mrow><mn>11</mn><mi>c</mi></mrow></msub><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><mfrac><mrow><msub><mi>f</mi><mrow><mn>1</mn><mo>+</mo><mi>c</mi></mrow></msub><msub><mi>f</mi><mrow><mo>+</mo><mn>1</mn><mi>c</mi></mrow></msub></mrow><msub><mi>f</mi><mrow><mo>+</mo><mo>+</mo><mi>c</mi></mrow></msub></mfrac></mrow><annotation encoding="application/x-tex">$$\begin{equation*} E(f_{11c}) = \frac{f_{1+c} f_{+1c} }{f_{++c} } \end{equation*}$$</annotation></semantics></math> </ephtml> and <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0126" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>Var</mtext><mrow><mo>(</mo><msub><mi>f</mi><mrow><mn>11</mn><mi>c</mi></mrow></msub><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><mfrac><mrow><msub><mi>f</mi><mrow><mn>0</mn><mo>+</mo><mi>c</mi></mrow></msub><msub><mi>f</mi><mrow><mn>1</mn><mo>+</mo><mi>c</mi></mrow></msub><msub><mi>f</mi><mrow><mo>+</mo><mn>0</mn><mi>c</mi></mrow></msub><msub><mi>f</mi><mrow><mo>+</mo><mn>1</mn><mi>c</mi></mrow></msub></mrow><mrow><msubsup><mi>f</mi><mrow><mo>+</mo><mo>+</mo><mi>c</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><msub><mi>f</mi><mrow><mo>+</mo><mo>+</mo><mi>c</mi></mrow></msub><mo>−</mo><mn>1</mn><mo>)</mo></mrow></mrow></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \mbox{Var} (f_{11c}) = \frac{ f_{0+c} f_{1+c} f_{+0c} f_{+1c} }{ f_{++c}^2 (f_{++c} - 1) }. \end{equation*}$$</annotation></semantics></math> </ephtml> Under the null hypothesis of conditional independence, the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0127" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic is distributed as a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0128" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> random variable with one degree of freedom.</p> <p>In a measurement context, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0129" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><mi>j</mi></msub><annotation encoding="application/x-tex">$X_j$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0130" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><msup><mi>j</mi><mo>′</mo></msup></msub><annotation encoding="application/x-tex">$X_{j^{\prime }}$</annotation></semantics></math> </ephtml> typically denote two binary items and the stratification variable <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0131" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>C</mi><annotation encoding="application/x-tex">$C$</annotation></semantics></math> </ephtml> refers to test sum‐scores or any other categorical covariate of interest. For IRT models, the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0132" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic has been used to assess local independence (Rosenbaum, [<reflink idref="bib53" id="ref109">53</reflink>], [<reflink idref="bib54" id="ref110">54</reflink>]) as well as to detect differential item functioning (DIF) across two groups of examinees: the focal and the reference group (e.g., Holland & Thayer, [<reflink idref="bib33" id="ref111">33</reflink>]). For the purpose of detecting DIF in IRT, examinees are typically stratified based on their observed total test scores.</p> <p>The MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0133" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic has also been used in CD to detect potential local dependence among item pairs and testlet effects (Lim, [<reflink idref="bib41" id="ref112">41</reflink>]; Lim & Drasgow, [<reflink idref="bib42" id="ref113">42</reflink>], [<reflink idref="bib43" id="ref114">43</reflink>]). The application of the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0134" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic in CD necessitates a subtle yet technically crucial modification: examinees are stratified according to their estimated latent attribute vector, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0135" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi mathvariant="bold-italic">α</mi><mo>̂</mo></mover><mrow><mi>i</mi><mo>∈</mo><mi>c</mi></mrow></msub><mo>=</mo><msub><mover accent="true"><mi mathvariant="bold-italic">α</mi><mo>̂</mo></mover><mi>c</mi></msub></mrow><annotation encoding="application/x-tex">$\widehat{\bm {\alpha }}_{i \in c} = \widehat{\bm {\alpha }}_{c}$</annotation></semantics></math> </ephtml> , with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0136" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mn>1</mn><mo>,</mo><mtext>...</mtext><mo>,</mo><mi>C</mi><mo>=</mo><msup><mn>2</mn><mi>K</mi></msup></mrow><annotation encoding="application/x-tex">$c = 1, \ldots, C = 2^K$</annotation></semantics></math> </ephtml> the proficiency class index. (Recall that each proficiency class is uniquely identified based on its attribute vector <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0137" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>c</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_c$</annotation></semantics></math> </ephtml> , which documents which particular attributes individuals in class <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0138" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>c</mi><annotation encoding="application/x-tex">$c$</annotation></semantics></math> </ephtml> have mastered and which they have not. As each examinee is a member of one and only one proficiency class, the more compact notations <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0139" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>c</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_c$</annotation></semantics></math> </ephtml> or <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0140" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_i$</annotation></semantics></math> </ephtml> instead of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0141" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mrow><mi>i</mi><mo>∈</mo><mi>c</mi></mrow></msub><annotation encoding="application/x-tex">$\bm {\alpha }_{i \in c}$</annotation></semantics></math> </ephtml> are used if context permits.)</p> <p>Under the null hypothesis, the test statistic would have a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0142" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> ‐distribution with 1 degree of freedom provided examinees' true latent attribute vectors <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0143" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold-italic">α</mi></mrow><annotation encoding="application/x-tex">$\bm {\alpha }$</annotation></semantics></math> </ephtml> were known and used as levels <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0144" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>c</mi><annotation encoding="application/x-tex">$c$</annotation></semantics></math> </ephtml> of the stratification variable <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0145" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>C</mi><annotation encoding="application/x-tex">$C$</annotation></semantics></math> </ephtml> . However, the true <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0146" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold-italic">α</mi></mrow><annotation encoding="application/x-tex">$\bm {\alpha }$</annotation></semantics></math> </ephtml> vectors of examinees are never known. Therefore, the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0147" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic has only an asymptotic <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0148" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> ‐distribution with 1 degree of freedom as the number of items <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0149" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math> </ephtml> and the sample sizes in each <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0150" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>×</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">$2 \times 2$</annotation></semantics></math> </ephtml> table increase (Lim & Drasgow, [<reflink idref="bib42" id="ref115">42</reflink>], [<reflink idref="bib43" id="ref116">43</reflink>]).</p> <hd id="AN0190280385-11">The Mantel‐Haenszel Chi‐Squared Statistic for Aggregated Testlet Effects</hd> <p>Recall that the sum of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0151" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>W</mi><annotation encoding="application/x-tex">$W$</annotation></semantics></math> </ephtml> independent <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0152" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> random variables, each with one degree of freedom, is itself distributed as a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0153" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> random variable with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0154" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>W</mi><annotation encoding="application/x-tex">$W$</annotation></semantics></math> </ephtml> degrees of freedom. Hence, one could consider extending Equation 5 in order to incorporate the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0155" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic obtained for the item pairs <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0156" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>(</mo><msub><mi>X</mi><mi>j</mi></msub><mo>,</mo><msub><mi>X</mi><msup><mi>j</mi><mo>′</mo></msup></msub><mo>)</mo></mrow><annotation encoding="application/x-tex">$(X_j, X_{j^{\prime }})$</annotation></semantics></math> </ephtml> , with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0157" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi><mo>≠</mo><msup><mi>j</mi><mo>′</mo></msup></mrow><annotation encoding="application/x-tex">$j \ne j^{\prime }$</annotation></semantics></math> </ephtml> , across all <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0158" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mfenced separators="" open="(" close=")"><mfrac linethickness="0pt"><msup><mi>J</mi><mrow><mo>(</mo><mi>T</mi><msub><mi>L</mi><mi>t</mi></msub><mo>)</mo></mrow></msup><mn>2</mn></mfrac></mfenced><annotation encoding="application/x-tex">$\binom{J^{(TL_t)}}{2}$</annotation></semantics></math> </ephtml> item pairs formed from the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0159" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>J</mi><mrow><mo>(</mo><mi>T</mi><msub><mi>L</mi><mi>t</mi></msub><mo>)</mo></mrow></msup><annotation encoding="application/x-tex">$J^{(TL_t)}$</annotation></semantics></math> </ephtml> items in testlet <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0160" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math> </ephtml> , with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0161" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><msub><mi>L</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">$TL_t$</annotation></semantics></math> </ephtml> denoting testlet <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0162" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0163" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><mo>=</mo><mn>1</mn><mo>,</mo><mtext>...</mtext><mo>,</mo><mi>T</mi></mrow><annotation encoding="application/x-tex">$t = 1, \ldots, T$</annotation></semantics></math> </ephtml> the index of the testlets in a given assessment. Concretely, such an aggregated MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0164" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic could look like 6 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0165" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>AGG MHχ2(t)=∑Xj,Xj′∈TLt;j<j′MHχ2Xj,Xj′=∑Xj,Xj′∈TLt;j<j′∑cf11c−∑cE(f11c)2∑cVar(f11c).<annotation encoding="application/x-tex">$$\begin{eqnarray} \mbox{AGG MH $\chi ^2 (t)$} & = & \sum _{X_j, X_{j^{\prime }} \in TL_t; j < j^{\prime }} \mbox{MH $\chi ^2 {\left(X_j, X_{j^{\prime }} \right)}$} \nonumber \\ & = & \sum _{X_j, X_{j^{\prime }} \in TL_t; j < j^{\prime }} \frac{ {\left(\sum _{c} f_{11c} - \sum _{c} E(f_{11c}) \right)}^2 }{ \sum _{c} \mbox{Var}(f_{11c}) }. \end{eqnarray}$$</annotation></semantics></math> </ephtml> However, this seemingly straightforward and appealing idea encounters several theoretical obstacles. First, recall that in case of the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0166" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic in Equation 5, the levels <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0167" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>c</mi><annotation encoding="application/x-tex">$c$</annotation></semantics></math> </ephtml> of the stratification variable <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0168" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>C</mi><annotation encoding="application/x-tex">$C$</annotation></semantics></math> </ephtml> , the true proficiency classes <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0169" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">α</mi><mi>c</mi></msub><annotation encoding="application/x-tex">$\bm {\alpha }_c$</annotation></semantics></math> </ephtml> , are unknown and must be estimated from the data. However, using the sample estimates <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0170" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi mathvariant="bold-italic">α</mi><mo>̂</mo></mover><mn>1</mn></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mover accent="true"><mi mathvariant="bold-italic">α</mi><mo>̂</mo></mover><mi>C</mi></msub></mrow><annotation encoding="application/x-tex">$\widehat{\bm {\alpha }}_1, \ldots, \widehat{\bm {\alpha }}_C$</annotation></semantics></math> </ephtml> causes the sampling distribution of the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0171" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic to be no longer exact, but only approximate <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0172" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> random variables. Second, the pairs of items formed by those in a testlet are not independent, as they overlap. Hence, the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0173" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistics are not independent but correlated approximate <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0174" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> random variables. As a further complication, correlated <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0175" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> random variables do not follow a central but a mixture chi‐squared distribution whose degrees of freedom can only be approximated, which, however, requires knowledge of the correlations between the approximate <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0176" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>X</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$X^2$</annotation></semantics></math> </ephtml> random variables. But these correlations are typically unknown and cannot be estimated because replications are not available. As an alternative, Ferrari ([<reflink idref="bib21" id="ref117">21</reflink>]) discussed the approximation of the distribution of a sum of correlated chi‐squared random variables through a gamma distribution (recall that the chi‐squared distribution is a special gamma with shape parameter <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0177" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mfrac><mi>ν</mi><mn>2</mn></mfrac></mrow><annotation encoding="application/x-tex">$\alpha = \frac{\nu }{2}$</annotation></semantics></math> </ephtml> and scale parameter <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0178" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">$\theta = 2$</annotation></semantics></math> </ephtml> ; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0179" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>ν</mi><annotation encoding="application/x-tex">$\nu$</annotation></semantics></math> </ephtml> denotes the degrees of freedom of the chi‐squared distribution). However, specification of the shape and scale parameters of the approximating gamma distribution also requires knowledge of the correlations between the MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0180" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistics.</p> <p>As a solution to these impediments, a parametric bootstrap procedure (Carlin & Gelfand, [<reflink idref="bib8" id="ref118">8</reflink>]; Efron, [<reflink idref="bib20" id="ref119">20</reflink>]; Geyer, [<reflink idref="bib22" id="ref120">22</reflink>]; Hinkley, [<reflink idref="bib32" id="ref121">32</reflink>]) is proposed here for identifying the sampling distribution of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0181" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic. The following computational steps are required:</p> <p></p> <ulist> <item> 1. Fit the data collected with a CD assessment presumably containing one or more testlets with a CDM that <emph>does not</emph> include any testlet parameter(s); in other words, assume no testlet effect is present and the assumption of local independence holds (i.e., that the null hypothesis is true).</item> <p></p> <item> 2. The estimates of the item parameter vectors obtained in Step 1, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0182" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi mathvariant="bold-italic">β</mi><mo>̂</mo></mover><mn>1</mn></msub><mo>,</mo><mtext>...</mtext><mo>,</mo><msub><mover accent="true"><mi mathvariant="bold-italic">β</mi><mo>̂</mo></mover><mi>J</mi></msub></mrow><annotation encoding="application/x-tex">$\widehat{\bm {\beta }}_1, \ldots, \widehat{\bm {\beta }}_J$</annotation></semantics></math> </ephtml> , are then used to simulate a large number of datasets, say, 1,000 or more, which constitute the bootstrap sample.</item> <p></p> <item> 3. For each data set in the bootstrap sample, the statistic AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0183" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> is calculated for each testlet <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0184" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math> </ephtml> in the assessment, which consists of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0185" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>J</mi><mrow><mo>(</mo><mi>T</mi><msub><mi>L</mi><mi>t</mi></msub><mo>)</mo></mrow></msup><annotation encoding="application/x-tex">$J^{(TL_t)}$</annotation></semantics></math> </ephtml> items. For example, for a bootstrap sample comprising 1,000 data sets, each containing <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0186" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><mo>=</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">$t=4$</annotation></semantics></math> </ephtml> testlets, the computational results form a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0187" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>(</mo><mn>1</mn><mo>,</mo><mn>000</mn><mo>×</mo><mn>4</mn><mo>)</mo></mrow><annotation encoding="application/x-tex">$(1,000 \times 4)$</annotation></semantics></math> </ephtml> matrix, with the individual cell entries denoting the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0188" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic of a particular testlet.</item> <p></p> <item> 4. The columns of estimates obtained in Step 3 are then sorted in ascending order to obtain the bootstrap sampling distribution for the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0189" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistics of each testlet, which is supposed to approximate its true sampling distribution under the null hypothesis of conditional independence, that is, in the absence of testlet effects.</item> <p></p> <item> 5. The critical value of the bootstrap sampling distribution of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0190" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> is then estimated as its 95th percentile. Additionally, the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0191" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>p</mi><annotation encoding="application/x-tex">$p$</annotation></semantics></math> </ephtml> ‐value associated with AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0192" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> computed for the original data set is obtained.</item> </ulist> <p>The rationale behind this bootstrap resampling procedure follows a common approach to finding <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0193" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>p</mi><annotation encoding="application/x-tex">$p$</annotation></semantics></math> </ephtml> ‐values for unknown null distributions and has a large sample justification (see, e.g., Beran, [<reflink idref="bib3" id="ref122">3</reflink>]). As the number of items <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0194" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math> </ephtml> and the sample size increases, the parameter estimates of the CDM fitted in Step 1 converge to their true values. The bootstrap data sets are independently and identically distributed under the null model, as they are generated based on the model parameter estimates of the original data set that is assumed not to include any testlet effect.</p> <hd id="AN0190280385-12">Simulation Studies</hd> <p>In order to assess the sensitivity of the proposed parametric bootstrap Mantel‐Haenszel statistic for testlets, AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0195" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> , simulation studies were conducted using synthetic data. The performance was evaluated within two distinct paradigms: the type I error control of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0196" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> and its power. Two studies were conducted for each paradigm, with two different CDMs, four Q‐matrices and a broad range of experimental factors.</p> <p>Earlier, it was mentioned that extant measures and statistics for establishing testlet effects either employ a model‐comparison paradigm contrasting the fit of models with and without testlet effects, or operate at the level of individual item pairs. The conclusion was that none of these approaches was suitable for examining the presence of an aggregated testlet effect as it may originate from taking into account all items in a cluster at once. In a remarkable study, Hansen et al. ([<reflink idref="bib27" id="ref123">27</reflink>]) discussed criteria for detecting testlet effects in CD modeling. They emphasized that full‐information test statistics such as likelihood ratio <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0197" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>G</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$G^2$</annotation></semantics></math> </ephtml> are inadequate for the multinomial setting of CD, where the sparseness of high‐dimensional response‐pattern cross‐classifications typically causes the break‐down of the asymptotic chi‐squared approximations for the full‐information test statistics. Hence, Hansen et al. ([<reflink idref="bib27" id="ref124">27</reflink>]) proposed to use instead limited‐information overall fit statistics such as Maydeu‐Olivares and Joe ([<reflink idref="bib51" id="ref125">51</reflink>]) <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0198" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> . In several simulation studies investigating the usefulness of the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0199" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic for detecting testlet effects within the context of CD modeling, Hansen et al. ([<reflink idref="bib27" id="ref126">27</reflink>]) observed that (i)  <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0200" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> displayed excellent type I error control (i.e., when the model was correctly specified as not including a testlet effect), and (ii) proved to be extremely sensitive to model misspecifications (i.e., the ability to detect a model that ignores the testlet structure of the data generating model—that is, statistical power). Hence, in following Hansen et al. ([<reflink idref="bib27" id="ref127">27</reflink>]) the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0201" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic was used to provide a comparative standard for evaluating the type I error control and the power of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0202" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic.</p> <p>In the current study, for each experimental cell, across the 500 replicated data sets, the proportion of rejections of the null hypothesis of good fit based on the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0203" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic was computed. This proportion served as an external criterion for assessing the type I error control and the power of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0204" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic.</p> <hd id="AN0190280385-13">Type I Error Studies Evaluating AGG MH χ2(t)$\chi ^2 (t)$</hd> <p> <emph>Design: Study I—Using the DINA Model</emph>. The "regular" DINA model (in the parameterization of Equations 3 and 4, respectively, repeated here for convenience) <emph>without</emph> testlet parameter(s), <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0206" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mo form="prefix">logit</mo></mrow><mfenced separators="" open="(" close=")"><mi>P</mi><mo>(</mo><msub><mi>Y</mi><mi>ij</mi></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>∣</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>)</mo></mfenced><mo linebreak="badbreak">=</mo><msub><mi>λ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>λ</mi><mrow><mi>j</mi><mo>(</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mtext>...</mtext><mi mathvariant="normal">,</mi><mi>K</mi><mo>)</mo></mrow></msub><munderover><mo>∏</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msubsup><mi>α</mi><mi>ik</mi><msub><mi>q</mi><mi>jk</mi></msub></msubsup><mo linebreak="goodbreak">=</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>1</mn></mrow></msub><msub><mi>η</mi><mi>ij</mi></msub><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \operatorname{\mathrm{logit}}\left(P({Y}_{\textit{ij}}=1\mid {\bm{\alpha}}_{i})\right)={\lambda}_{j0}+{\lambda}_{j(1,2,\text{\ensuremath{\ldots},}K)}\prod _{k=1}^{K}{\alpha}_{\textit{ik}}^{{q}_{\textit{jk}}}={\delta}_{j0}+{\delta}_{j1}{\eta}_{\textit{ij}}, \end{equation*}$$</annotation></semantics></math> </ephtml> was employed in conjunction with the Q‐matrix of table 3 in Xu et al. ([<reflink idref="bib72" id="ref128">72</reflink>]), which was previously presented in Table 1, to generate the data. The number of attributes was <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0207" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">$K = 3$</annotation></semantics></math> </ephtml> , and the test length was <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0208" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>28</mn></mrow><annotation encoding="application/x-tex">$J = 28$</annotation></semantics></math> </ephtml> . Slipping and guessing were set to 0.1 across all items; examinee attribute patterns were generated using the multivariate normal threshold model (Chiu et al., [<reflink idref="bib14" id="ref129">14</reflink>]) as described previously (i.e., zero expectation vector and the variance‐covariance matrix having unit variances and common covariances <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0209" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mrow><mi>k</mi><msup><mi>k</mi><mo>′</mo></msup></mrow></msub><mo>=</mo><mn>0.3</mn></mrow><annotation encoding="application/x-tex">$\sigma _{kk^{\prime }} = 0.3$</annotation></semantics></math> </ephtml> ). The data were generated with the simGDINA function in the R package GDINA (Ma & de la Torre, [<reflink idref="bib45" id="ref130">45</reflink>]). Three different sample sizes were used: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0210" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>=</mo><mn>500</mn><mo>,</mo><mn>1</mn><mo>,</mo><mn>000</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>000</mn></mrow><annotation encoding="application/x-tex">$N = 500, 1,000, 2,000$</annotation></semantics></math> </ephtml> . For each cell, 500 replicated data sets were generated.</p> <p>As the items in the Q‐matrix shown in Table 1 all have single‐attribute q‐vectors and might, thus, be considered as "too easy," a second, more complex‐structured Q‐matrix from Xu et al. ([<reflink idref="bib72" id="ref131">72</reflink>]; table 10) was used in addition that contained <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0211" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>28</mn></mrow><annotation encoding="application/x-tex">$J = 28$</annotation></semantics></math> </ephtml> items with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0212" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>5</mn></mrow><annotation encoding="application/x-tex">$K = 5$</annotation></semantics></math> </ephtml> attributes. The maximum number of attributes used per item in this Q‐matrix was three (see Table 3).</p> <p>3 Table Q‐Matrix with J=28$J = 28$ Items Using K=5$K = 5$ Attributes (table 10 , Xu et al., 2024)</p> <p> <ephtml> <table><thead><tr><th>Item</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0215" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>1</mn></msub><annotation encoding="application/x-tex">$\alpha _1$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0216" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$\alpha _2$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0217" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>3</mn></msub><annotation encoding="application/x-tex">$\alpha _3$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0218" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>4</mn></msub><annotation encoding="application/x-tex">$\alpha _4$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0219" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>5</mn></msub><annotation encoding="application/x-tex">$\alpha _5$</annotation></semantics></math></p></th><th>Testlet <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0220" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>t</mi><annotation encoding="application/x-tex">$t$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>2</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>3</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>4</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td></tr><tr><td>5</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td></tr><tr><td>6</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>7</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>2</td></tr><tr><td>8</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>2</td></tr><tr><td>9</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>10</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>2</td></tr><tr><td>11</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>2</td></tr><tr><td>12</td><td>0</td><td>0</td><td>1</td><td>1</td><td>0</td><td>2</td></tr><tr><td>13</td><td>1</td><td>0</td><td>0</td><td>1</td><td>0</td><td>2</td></tr><tr><td>14</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>3</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>3</td></tr><tr><td>16</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>3</td></tr><tr><td>17</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>3</td></tr><tr><td>18</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>3</td></tr><tr><td>19</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>3</td></tr><tr><td>20</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>4</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>4</td></tr><tr><td>22</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>4</td></tr><tr><td>23</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>4</td></tr><tr><td>24</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>25</td><td>1</td><td>0</td><td>1</td><td>1</td><td>0</td><td>4</td></tr><tr><td>26</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td><td>4</td></tr><tr><td>27</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>28</td><td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>4</td></tr></tbody></table> </ephtml> </p> <p> <emph>Results: Study I—Using the DINA Model</emph>. The proportions of incorrect rejections (i.e., type I errors) are reported in Table 4 for the two different Q‐matrices with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0221" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">$K=3$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0222" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>5</mn></mrow><annotation encoding="application/x-tex">$K=5$</annotation></semantics></math> </ephtml> attributes each with four testlets (including also their average) and the three different sample sizes employed. The observed type I error rates are in many instances equal to the nominal level of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0223" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>0.05</mn></mrow><annotation encoding="application/x-tex">$\alpha = 0.05$</annotation></semantics></math> </ephtml> ; there are a few deviations where the type I error rates drop to 0.4 or increase to 0.6. Remarkably, the values obtained for the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0224" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic confirm these findings.</p> <p>4 Table DINA: Results of the Type I Error Study Evaluating AGG MH χ2(t)$\chi ^2 (t)$; Values of the M2$M_2$ Statistic Included as Reference Measures</p> <p> <ephtml> <table><thead><tr valign="bottom"><th /><th>Testlet 1</th><th>Testlet 2</th><th>Testlet 3</th><th>Testlet 4</th><th /><th /></tr><tr><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0227" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>N</mi><annotation encoding="application/x-tex">$N$</annotation></semantics></math></p></th><th>(Items 1‐6)</th><th>(Items 7‐13)</th><th>(Items 14‐19)</th><th>(Items 20‐28)</th><th>Average</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0228" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>Q‐Matrix: <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0229" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>J</mi><mo>=</mo>28<annotation encoding="application/x-tex">$J = 28$</annotation></semantics></math></p>, <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0230" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><mo>=</mo>3<annotation encoding="application/x-tex">$K = 3$</annotation></semantics></math></p></td><td /></tr><tr><td>500</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.07</td></tr><tr><td>1,000</td><td>0.05</td><td>0.04</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.05</td></tr><tr><td>2,000</td><td>0.04</td><td>0.04</td><td>0.05</td><td>0.05</td><td>0.04</td><td>0.04</td></tr><tr><td>Q‐Matrix: <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0231" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>J</mi><mo>=</mo>28<annotation encoding="application/x-tex">$J = 28$</annotation></semantics></math></p>, <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0232" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><mo>=</mo>5<annotation encoding="application/x-tex">$K = 5$</annotation></semantics></math></p></td><td /></tr><tr><td>500</td><td>0.04</td><td>0.06</td><td>0.05</td><td>0.06</td><td>0.05</td><td>0.06</td></tr><tr><td>1,000</td><td>0.05</td><td>0.06</td><td>0.04</td><td>0.05</td><td>0.05</td><td>0.06</td></tr><tr><td>2,000</td><td>0.06</td><td>0.04</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.06</td></tr></tbody></table> </ephtml> </p> <p> <emph>Design: Study II—Using the Saturated LCDM</emph>. The saturated LCDM <emph>without</emph> any testlet effects was used having IRF <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0233" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>logitPYij=1∣αi=δj0+∑k=1Kδjkαikqjk+∑k′=k+1K∑k=1K−1δjkk′αikαik′qjkqjk′+⋯+δj1,2,⋯,K∏k=1Kαikqjk.<annotation encoding="application/x-tex">$$\begin{eqnarray*} {\operatorname{\mathrm{logit}}\left(P\left({Y}_{\textit{ij}}=1\mid {\bm{\alpha}}_{i}\right)\right)}& =& {{\delta}_{j0}+\sum _{k=1}^{K}{\delta}_{\textit{jk}}{\alpha}_{\textit{ik}}{q}_{\textit{jk}}+\sum _{{k}^{\prime}=k+1}^{K}\sum _{k=1}^{K-1}{\delta}_{\textit{jkk}\prime}{\alpha}_{\textit{ik}}{\alpha}_{i{k}^{\prime}}{q}_{\textit{jk}}{q}_{j{k}^{\prime}}+\cdots +}\\ & & {{\delta}_{j\left(1,2,\text{\ensuremath{\cdots},}K\right)}\prod _{k=1}^{K}{\alpha}_{\textit{ik}}{q}_{\textit{jk}}.} \end{eqnarray*}$$</annotation></semantics></math> </ephtml> Q‐matrices were randomly generated for each replicated data set using the genQ routine in the R package cdmTools (Nájera et al., [<reflink idref="bib52" id="ref132">52</reflink>]). Each Q‐matrix contained <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0234" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math> </ephtml> items and had either <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0235" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">$K = 3$</annotation></semantics></math> </ephtml> or <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0236" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>5</mn></mrow><annotation encoding="application/x-tex">$K = 5$</annotation></semantics></math> </ephtml> attributes. To create Q‐matrices containing <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0237" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math> </ephtml>  = 48 items, the Q‐matrices with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0238" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math> </ephtml> items were duplicated and stacked. The genQ routine imposes on each randomly generated Q‐matrix the strict identifiability conditions as outlined in Chen et al. ([<reflink idref="bib12" id="ref133">12</reflink>]) and Xu and Shang ([<reflink idref="bib71" id="ref134">71</reflink>]). As examples, two Q‐matrices are presented in Table 5.</p> <p>5 Table Q‐Matrices for J=24$J = 24$ Items Using K=3$K=3$ and K=5$K=5$ Attributes</p> <p> <ephtml> <table><thead><tr><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0242" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0243" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 5</th></tr><tr><th>Item</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0244" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>1</mn></msub><annotation encoding="application/x-tex">$\alpha _1$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0245" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$\alpha _2$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0246" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>3</mn></msub><annotation encoding="application/x-tex">$\alpha _3$</annotation></semantics></math></p></th><th>Item</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0247" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>1</mn></msub><annotation encoding="application/x-tex">$\alpha _1$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0248" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$\alpha _2$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0249" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>3</mn></msub><annotation encoding="application/x-tex">$\alpha _3$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0250" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>4</mn></msub><annotation encoding="application/x-tex">$\alpha _4$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0251" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>5</mn></msub><annotation encoding="application/x-tex">$\alpha _5$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>1</td><td>0</td><td>0</td><td>1</td><td>1</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>2</td><td>0</td><td>0</td><td>1</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>3</td><td>1</td><td>1</td><td>0</td><td>3</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>4</td><td>0</td><td>1</td><td>0</td><td>4</td><td>1</td><td>0</td><td>0</td><td>1</td><td>1</td></tr><tr><td>5</td><td>1</td><td>0</td><td>0</td><td>5</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>6</td><td>1</td><td>1</td><td>1</td><td>6</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>7</td><td>1</td><td>1</td><td>0</td><td>7</td><td>1</td><td>1</td><td>1</td><td>0</td><td>1</td></tr><tr><td>8</td><td>1</td><td>0</td><td>0</td><td>8</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>9</td><td>1</td><td>0</td><td>0</td><td>9</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>10</td><td>0</td><td>0</td><td>1</td><td>10</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>11</td><td>0</td><td>1</td><td>1</td><td>11</td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td></tr><tr><td>12</td><td>1</td><td>1</td><td>1</td><td>12</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>13</td><td>1</td><td>0</td><td>0</td><td>13</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>14</td><td>0</td><td>1</td><td>1</td><td>14</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td></tr><tr><td>15</td><td>1</td><td>1</td><td>1</td><td>15</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>16</td><td>1</td><td>0</td><td>0</td><td>16</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>17</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>18</td><td>0</td><td>0</td><td>1</td><td>18</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td></tr><tr><td>19</td><td>0</td><td>1</td><td>0</td><td>19</td><td>1</td><td>0</td><td>0</td><td>1</td><td>1</td></tr><tr><td>20</td><td>0</td><td>0</td><td>1</td><td>20</td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td></tr><tr><td>21</td><td>1</td><td>0</td><td>0</td><td>21</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>22</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td></tr><tr><td>23</td><td>1</td><td>1</td><td>1</td><td>23</td><td>1</td><td>1</td><td>0</td><td>1</td><td>0</td></tr><tr><td>24</td><td>0</td><td>1</td><td>0</td><td>24</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr></tbody></table> </ephtml> </p> <p>In this study, only two distinct sample sizes were used: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0252" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>=</mo><mn>500</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>000</mn></mrow><annotation encoding="application/x-tex">$N = 500, 2,000$</annotation></semantics></math> </ephtml> . In each setting, the number of items per testlet was calculated by dividing the total number of items in the test by the number of testlets: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0253" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>J</mi><mrow><mo>(</mo><mi>T</mi><msub><mi>L</mi><mi>j</mi></msub><mo>)</mo></mrow></msup><mo>=</mo><mi>J</mi><mo>/</mo><mi>T</mi></mrow><annotation encoding="application/x-tex">$J^{(TL_j)} = J/T$</annotation></semantics></math> </ephtml> ; hence, the number of testlets was <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0254" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>4</mn><mo>,</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">$T = 4, 6$</annotation></semantics></math> </ephtml> , all containing the same number of items. Examinee attribute patterns were generated using the previously described multivariate normal threshold model (Chiu et al., [<reflink idref="bib14" id="ref135">14</reflink>]): zero expectation vector and the variance‐covariance matrix having unit variances and common covariances <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0255" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mrow><mi>k</mi><msup><mi>k</mi><mo>′</mo></msup></mrow></msub><mo>=</mo><mn>0.3</mn></mrow><annotation encoding="application/x-tex">$\sigma _{kk^{\prime }} = 0.3$</annotation></semantics></math> </ephtml> in one condition, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0256" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mrow><mi>k</mi><msup><mi>k</mi><mo>′</mo></msup></mrow></msub><mo>=</mo><mn>0.6</mn></mrow><annotation encoding="application/x-tex">$\sigma _{kk^{\prime }} = 0.6$</annotation></semantics></math> </ephtml> in the second. The item parameters <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0257" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">δ</mi><mi>j</mi></msub><annotation encoding="application/x-tex">$\bm {\delta }_j$</annotation></semantics></math> </ephtml> were randomly generated with the constraint <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0258" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.20</mn><mo>≤</mo><mi>P</mi><mo>(</mo><msub><mi>Y</mi><mi>j</mi></msub><mo>=</mo><mn>1</mn><mo>∣</mo><mrow><mi mathvariant="bold-italic">α</mi></mrow><mo>)</mo><mo>≤</mo><mn>0.80</mn></mrow><annotation encoding="application/x-tex">$0.20 \le P(Y_j = 1 \mid \bm {\alpha }) \le 0.80$</annotation></semantics></math> </ephtml> across all items. The data were generated with the simGDINA function in the R package GDINA (Ma & de la Torre, [<reflink idref="bib45" id="ref136">45</reflink>]). In summary, completely crossing all experimental factors resulted in <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0259" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>×</mo><mn>2</mn><mo>×</mo><mn>2</mn><mo>×</mo><mn>2</mn><mo>=</mo><mn>16</mn></mrow><annotation encoding="application/x-tex">$2 \times 2 \times 2 \times 2 = 16$</annotation></semantics></math> </ephtml> cells. For each cell, 500 replicated data sets were generated.</p> <p> <emph>Results: Study II—Using the Saturated LCDM</emph>. The proportions of incorrect rejections (i.e., type I errors) are reported in Table 6. For the sake of brevity, only the average proportions of incorrect rejections are reported, as each testlet contained the same number of items. As in the DINA study, the type I error rates exhibited fluctuations around the nominal level of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0260" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>0.05</mn></mrow><annotation encoding="application/x-tex">$\alpha = 0.05$</annotation></semantics></math> </ephtml> . The values obtained for the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0261" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic confirm these results.</p> <p>6 Table Saturated LCDM: Results of the Type I Error Study Evaluating AGG MH χ2(t)$\chi ^2 (t)$; Values of the M2$M_2$ Statistic Are Included as Reference Measures</p> <p> <ephtml> <table><thead><tr><th /><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0264" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0265" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0266" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0267" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.6</th></tr><tr><th /><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0268" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math></p> = 24</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0269" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math></p> = 48</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0270" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math></p> = 24</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0271" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math></p> = 48</th></tr><tr><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0272" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>N</mi><annotation encoding="application/x-tex">$N$</annotation></semantics></math></p></th><th>AGG</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0273" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math></p></th><th>AGG</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0274" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math></p></th><th>AGG</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0275" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math></p></th><th>AGG</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0276" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msup><mi>M</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$M^2$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td align="center">Testlet Size = 4, <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0277" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 3</td></tr><tr><td>500</td><td>0.04</td><td>0.07</td><td>0.05</td><td>0.04</td><td>0.05</td><td>0.06</td><td>0.05</td><td>0.05</td></tr><tr><td>2,000</td><td>0.04</td><td>0.04</td><td>0.05</td><td>0.06</td><td>0.04</td><td>0.07</td><td>0.05</td><td>0.06</td></tr><tr><td align="center">Testlet Size = 4, <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0278" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 5</td></tr><tr><td>500</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.07</td><td>0.06</td><td>0.07</td><td>0.06</td><td>0.05</td></tr><tr><td>2,000</td><td>0.05</td><td>0.04</td><td>0.04</td><td>0.05</td><td>0.05</td><td>0.06</td><td>0.05</td><td>0.06</td></tr><tr><td /><td align="center">Testlet Size = 6, <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0279" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 3</td></tr><tr><td>500</td><td>0.05</td><td>0.05</td><td>0.06</td><td>0.06</td><td>0.05</td><td>0.06</td><td>0.05</td><td>0.07</td></tr><tr><td>2,000</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.07</td><td>0.04</td><td>0.05</td><td>0.05</td><td>0.06</td></tr><tr><td /><td align="center">Testlet Size = 6, <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0280" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 5</td></tr><tr><td>500</td><td>0.06</td><td>0.06</td><td>0.05</td><td>0.08</td><td>0.05</td><td>0.05</td><td>0.06</td><td>0.06</td></tr><tr><td>2,000</td><td>0.05</td><td>0.05</td><td>0.04</td><td>0.07</td><td>0.05</td><td>0.06</td><td>0.04</td><td>0.04</td></tr></tbody></table> </ephtml> </p> <hd id="AN0190280385-14">Power Studies Evaluating AGG MH χ2(t)$\chi ^2 (t)$</hd> <p> <emph>Design: Study I—Using the IT‐DINA Model</emph>. The IT‐DINA model (equation 8, Xu et al., [<reflink idref="bib72" id="ref137">72</reflink>]) <emph>with</emph> testlet parameter(s) was employed. The IRF, which has been previously presented in Equation 4, is reiterated here for the sake of convenience <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0282" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mo form="prefix">logit</mo></mrow><mfenced separators="" open="(" close=")"><mi>P</mi><mfenced separators="" open="(" close=")"><msub><mi>Y</mi><mi>ij</mi></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>∣</mo><msub><mi mathvariant="bold-italic">α</mi><mi>i</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">ξ</mi><mi>ti</mi></msub></mfenced></mfenced><mo linebreak="badbreak">=</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>0</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>δ</mi><mrow><mi>j</mi><mn>1</mn></mrow></msub><msub><mi>η</mi><mi>ij</mi></msub><mo linebreak="goodbreak">+</mo><msub><mi>β</mi><mi>jt</mi></msub><msub><mi>ξ</mi><mrow><mi>ti</mi><mn>1</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>γ</mi><mi>jt</mi></msub><mrow><mo>(</mo><mn>2</mn><msub><mi>η</mi><mi>ij</mi></msub><mo linebreak="goodbreak">−</mo><mn>1</mn><mo>)</mo></mrow><msub><mi>ξ</mi><mrow><mi>ti</mi><mn>2</mn></mrow></msub><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \operatorname{\mathrm{logit}}\left(P\left({Y}_{\textit{ij}}=1\mid {\bm{\alpha}}_{i},{\bm{\xi}}_{\textit{ti}}\right)\right)={\delta}_{j0}+{\delta}_{j1}{\eta}_{\textit{ij}}+{\beta}_{\textit{jt}}{\xi}_{\textit{ti}1}+{\gamma}_{\textit{jt}}(2{\eta}_{\textit{ij}}-1){\xi}_{\textit{ti}2}. \end{equation*}$$</annotation></semantics></math> </ephtml> The settings for generating the data were identical to those used for the type I error study (i.e., the Q‐matrix from Xu et al., 2024, which was shown previously in Table 1, comprising <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0283" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">$K = 3$</annotation></semantics></math> </ephtml> attributes and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0284" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>28</mn></mrow><annotation encoding="application/x-tex">$J = 28$</annotation></semantics></math> </ephtml> items, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0285" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">$T = 4$</annotation></semantics></math> </ephtml> testlets comprising 6, 7, 6, and 9 items, respectively). However, now, the data were also contaminated with testlet effects of varying sizes. In contrast with the approach taken by Xu et al., [<reflink idref="bib72" id="ref138">72</reflink>], the two testlet effects, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0286" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>1</mn></mrow></msub><mo>,</mo><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>2</mn></mrow></msub><mover><mo>∼</mo><mtext>iid</mtext></mover><mi mathvariant="script">N</mi><mrow><mo>(</mo><mn>0</mn><mo>,</mo><msup><mi>σ</mi><mn>2</mn></msup><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$ \xi _{ti1}, \xi _{ti2} \stackrel{\mbox{iid}}{\sim } \mathcal {N} (0, \sigma ^2)$</annotation></semantics></math> </ephtml> , exhibited varying variances <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0287" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>σ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\sigma ^2$</annotation></semantics></math> </ephtml> corresponding to the six levels of magnitude of the testlet effect: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0288" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>=</mo><mn>0.25</mn><mo>,</mo><mn>0.50</mn><mo>,</mo><mn>0.75</mn><mo>,</mo><mn>1.0</mn><mo>,</mo><mn>1.25</mn><mo>,</mo><mn>1.50</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 = 0.25, 0.50, 0.75, 1.0, 1.25, 1.50$</annotation></semantics></math> </ephtml> .</p> <p>A brief comment regarding the choice of these specific levels as they were derived from reports in the literature seems warranted. Most of the studies that inspired the usage of testlets in CD were conducted within the item response theory (IRT) paradigm, where the magnitude of a testlet effect is perceived as driven by the variance of the underlying random effect (Wainer & Wang, [<reflink idref="bib66" id="ref139">66</reflink>]). Said differently, the testlet effect variance indicates the degree of local dependence among the items within a given testlet. Jiao et al. ([<reflink idref="bib34" id="ref140">34</reflink>]) used in their simulations a variance of 0.25 and 1 to represent small and large testlet effects. Wainer and Wang ([<reflink idref="bib66" id="ref141">66</reflink>]) report a testlet effect with an estimated variance of 0.19 that does not affect parameter estimation. Wang et al. ([<reflink idref="bib70" id="ref142">70</reflink>]) used values of 0.0 (no testlet effect), 0.5, and 1.0. Wang and Wilson ([<reflink idref="bib69" id="ref143">69</reflink>]) used variances of 0.25, 0.5, 0.75, and 1.00 in their simulations to represent small to large testlet effects.</p> <p>The few studies investigating testlet modeling in CD mostly mirror what has been done in IRT. Sha ([<reflink idref="bib58" id="ref144">58</reflink>]) ran a set of large‐scale simulations using a testlet DINA model, with testlet effects having variances of 0 (= no testlet effect), 0.25, 1, 4, and 9. Hansen et al. ([<reflink idref="bib27" id="ref145">27</reflink>]) used in their simulations testlet effects with variances of 0 (= no testlet effect), 1, and 4. In a small‐scale simulation, Zhan et al. ([<reflink idref="bib76" id="ref146">76</reflink>]) used a testlet effect with a variance of 0.5, which they described as "moderate." Ma et al. ([<reflink idref="bib46" id="ref147">46</reflink>]) used testlet effects of 0.5 and 2.25 qualified as moderate and large, respectively. Guo et al. ([<reflink idref="bib25" id="ref148">25</reflink>]) used for their simulations testlet effects having variances of 0.25, 0.5, and 1. Xu et al. ([<reflink idref="bib72" id="ref149">72</reflink>]) used testlet effect sizes of 0, 1, and 1.5.</p> <p>Thus, a general consensus seems to have emerged that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0289" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>≤</mo><mn>0.25</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 \le 0.25$</annotation></semantics></math> </ephtml> corresponds to a negligible testlet effect (e.g., Glas et al., [<reflink idref="bib24" id="ref150">24</reflink>]), whereas values beyond 0.50 appear to be seen as substantial testlet effect (e.g., Wainer et al., [<reflink idref="bib67" id="ref151">67</reflink>]; Wang et al., [<reflink idref="bib70" id="ref152">70</reflink>]). Thus, a testlet effect of 0.25 was chosen as the smallest effect size in these simulations. (Small‐scale experimentation with different low‐variance effect sizes—not reported here—also suggested that testlet effects below 0.25 do not impact the estimation process.)</p> <p>The following sample sizes were employed: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0290" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>=</mo><mn>500</mn><mo>,</mo><mn>1</mn><mo>,</mo><mn>000</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>000</mn></mrow><annotation encoding="application/x-tex">$N = 500, 1,000, 2,000$</annotation></semantics></math> </ephtml> . Completely crossing all factors resulted in a total of 18 experimental conditions, with 500 replicated data sets produced for each condition.</p> <p> <emph>Results: Study I—Using the IT‐DINA Model</emph>. The proportions of correct rejections (i.e., the power) are reported in Table 7 for the "simple" Q‐matrix including only single‐attribute items, with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0291" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">$K=3$</annotation></semantics></math> </ephtml> attributes, for the four testlets (including also their average) and the three different sample sizes used. Table 8 shows the proportions of correct rejections (i.e., the power) for the "complex" Q‐matrix (i.e., <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0292" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>5</mn></mrow><annotation encoding="application/x-tex">$K=5$</annotation></semantics></math> </ephtml> attributes; items using up to three attributes). Both tables also present the values of the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0293" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic. The power of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0294" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> in the experimental condition with a testlet effect of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0295" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>=</mo><mn>0.25</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 = 0.25$</annotation></semantics></math> </ephtml> lends support to the notion that testlet effects of size <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0296" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>≤</mo><mn>0.25</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 \le 0.25$</annotation></semantics></math> </ephtml> can be safely ignored. However, when the testlet effect size is increased to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0297" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>≤</mo><mn>0.50</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 \le 0.50$</annotation></semantics></math> </ephtml> , then the power of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0298" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> attains satisfactory levels (with the possible exception for Testlet 3 when <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0299" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>=</mo><mn>500</mn></mrow><annotation encoding="application/x-tex">$N = 500$</annotation></semantics></math> </ephtml> ). For testlet effects beyond <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0300" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>=</mo><mn>0.50</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 = 0.50$</annotation></semantics></math> </ephtml> , the power of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0301" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> is at its maximum. These findings are also reflected in the values observed for the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0302" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml>  statistic.</p> <p>7 Table IT‐DINA with Q‐Matrix (J=28,K=3$J=28, K=3$): Results of the Power Study Evaluating AGG MH χ2(t)$\chi ^2 (t)$; Values of the M2$M_2$ Statistic Are Included as Reference Measures</p> <p> <ephtml> <table><thead><tr><th>Testlet</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0306" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>N</mi><annotation encoding="application/x-tex">$N$</annotation></semantics></math></p></th><th>Testlet 1</th><th>Testlet 2</th><th>Testlet 3</th><th>Testlet 4</th><th>Average</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0307" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>0.25</td><td>500</td><td>0.13</td><td>0.21</td><td>0.10</td><td>0.16</td><td>0.15</td><td>0.07</td></tr><tr><td /><td>1,000</td><td>0.27</td><td>0.23</td><td>0.17</td><td>0.28</td><td>0.24</td><td>0.09</td></tr><tr><td /><td>2,000</td><td>0.29</td><td>0.21</td><td>0.16</td><td>0.19</td><td>0.35</td><td>0.10</td></tr><tr><td>0.50</td><td>500</td><td>0.87</td><td>0.82</td><td>0.67</td><td>0.95</td><td>0.83</td><td>0.46</td></tr><tr><td /><td>1,000</td><td>0.97</td><td>0.96</td><td>0.90</td><td>1.00</td><td>0.96</td><td>0.85</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>0.75</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>1.00</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>1.25</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>1.50</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr></tbody></table> </ephtml> </p> <p>8 Table IT‐DINA with Q‐Matrix (J=28,K=5$J=28, K=5$): Results of the Power Study Evaluating AGG MH χ2(t)$\chi ^2 (t)$; Values of the M2$M_2$ Statistic Are Included as Reference Measures</p> <p> <ephtml> <table><thead><tr><th>Testlet</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0311" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>N</mi><annotation encoding="application/x-tex">$N$</annotation></semantics></math></p></th><th>Testlet 1</th><th>Testlet 2</th><th>Testlet 3</th><th>Testlet 4</th><th>Average</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0312" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>0.25</td><td>500</td><td>0.09</td><td>0.17</td><td>0.12</td><td>0.19</td><td>0.14</td><td>0.08</td></tr><tr><td /><td>1,000</td><td>0.09</td><td>0.17</td><td>0.14</td><td>0.29</td><td>0.18</td><td>0.08</td></tr><tr><td /><td>2,000</td><td>0.16</td><td>0.21</td><td>0.12</td><td>0.36</td><td>0.21</td><td>0.10</td></tr><tr><td>0.50</td><td>500</td><td>0.52</td><td>0.70</td><td>0.49</td><td>0.85</td><td>0.64</td><td>0.36</td></tr><tr><td /><td>1,000</td><td>0.68</td><td>0.93</td><td>0.73</td><td>0.99</td><td>0.83</td><td>0.73</td></tr><tr><td /><td>2,000</td><td>0.95</td><td>1.00</td><td>0.96</td><td>1.00</td><td>1.00</td><td>0.98</td></tr><tr><td>0.75</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>1.00</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>1.25</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>1.50</td><td>500</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>2,000</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr></tbody></table> </ephtml> </p> <p> <emph>Design: Study II—Using the Saturated LCDM</emph>. The design of this study was based on that of the type I error study. Recall the key features included randomly generated Q‐matrices with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0313" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math> </ephtml> items and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0314" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">$K = 3$</annotation></semantics></math> </ephtml> or <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0315" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>5</mn></mrow><annotation encoding="application/x-tex">$K = 5$</annotation></semantics></math> </ephtml> attributes, respectively (see the examples in previous Table 5). These were duplicated and stacked for the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0316" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math> </ephtml> item condition. The sample size was <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0317" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>=</mo><mn>500</mn></mrow><annotation encoding="application/x-tex">$N = 500$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0318" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>=</mo><mn>2</mn><mo>,</mo><mn>000</mn></mrow><annotation encoding="application/x-tex">$N=2,000$</annotation></semantics></math> </ephtml> . The number of testlets was <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0319" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>4</mn><mo>,</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">$T = 4, 6$</annotation></semantics></math> </ephtml> , all containing the same number of items. The examinee attribute patterns were generated using the multivariate normal threshold model, where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0320" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mrow><mi>k</mi><msup><mi>k</mi><mo>′</mo></msup></mrow></msub><mo>=</mo><mn>0.3</mn></mrow><annotation encoding="application/x-tex">$\sigma _{kk^{\prime }} = 0.3$</annotation></semantics></math> </ephtml> in one condition, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0321" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mrow><mi>k</mi><msup><mi>k</mi><mo>′</mo></msup></mrow></msub><mo>=</mo><mn>0.6</mn></mrow><annotation encoding="application/x-tex">$\sigma _{kk^{\prime }} = 0.6$</annotation></semantics></math> </ephtml> in the second. The item parameters <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0322" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">δ</mi><mi>j</mi></msub><annotation encoding="application/x-tex">$\bm {\delta }_j$</annotation></semantics></math> </ephtml> were randomly generated with the constraint <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0323" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.20</mn><mo>≤</mo><mi>P</mi><mo>(</mo><msub><mi>Y</mi><mi>j</mi></msub><mo>=</mo><mn>1</mn><mo>∣</mo><mrow><mi mathvariant="bold-italic">α</mi></mrow><mo>)</mo><mo>≤</mo><mn>0.80</mn></mrow><annotation encoding="application/x-tex">$0.20 \le P (Y_j = 1 \mid \bm {\alpha }) \le 0.80$</annotation></semantics></math> </ephtml> across all items.</p> <p>However, in contrast to the type I error study, now the the saturated LCDM was augmented <emph>with</emph> a normal random item‐by‐examinee interaction effect for testlet <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0324" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><msub><mi>L</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">$TL_t$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0325" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ξ</mi><mrow><mi>t</mi><mi>i</mi><mn>3</mn></mrow></msub><mo>∼</mo><mi mathvariant="script">N</mi><mrow><mo>(</mo><mn>0</mn><mo>,</mo><msup><mi>σ</mi><mn>2</mn></msup><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\xi _{ti3} {\sim } \mathcal {N} (0, \sigma ^2)$</annotation></semantics></math> </ephtml> , where the size of the variance <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0326" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>=</mo><mn>0.25</mn><mo>,</mo><mn>0.50</mn><mo>,</mo><mn>0.75</mn><mo>,</mo><mn>1.0</mn><mo>,</mo><mn>1.25</mn><mo>,</mo><mn>1.50</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 = 0.25, 0.50, 0.75, 1.0, 1.25, 1.50$</annotation></semantics></math> </ephtml> corresponds to the magnitude of the testlet effect <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0327" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>logitPYij=1∣αi=δj0+∑k=1Kδjkαikqjk+∑k′=k+1K∑k=1K−1δjkk′αikαik′qjkqjk′+⋯+δj1,2,⋯,K∏k=1Kαikqjk+βjtξti3.<annotation encoding="application/x-tex">$$\begin{equation*} \def\eqcellsep{&}\begin{array}{ccc} {\operatorname{\mathrm{logit}}\left(P\left({Y}_{\textit{ij}}=1\mid {\bm{\alpha}}_{i}\right)\right)}& =& {{\delta}_{j0}+\sum _{k=1}^{K}{\delta}_{\textit{jk}}{\alpha}_{\textit{ik}}{q}_{\textit{jk}}+\sum _{{k}^{\prime}=k+1}^{K}\sum _{k=1}^{K-1}{\delta}_{\textit{jkk}\prime}{\alpha}_{\textit{ik}}{\alpha}_{i{k}^{\prime}}{q}_{\textit{jk}}{q}_{j{k}^{\prime}}+\cdots +}\\ & & {{\delta}_{j\left(1,2,\text{\ensuremath{\cdots},}K\right)}\prod _{k=1}^{K}{\alpha}_{\textit{ik}}{q}_{\textit{jk}}+{\beta}_{\textit{jt}}{\xi}_{\textit{ti}3}.}\end{array} \end{equation*}$$</annotation></semantics></math> </ephtml> Notice that this differs from the IT‐DINA model, as the saturated testlet LCDM was not equipped with a second testlet component involving examinees' ideal responses, as no unequivocally accepted convention currently exists for translating the probabilistic ideal responses of the saturated LCDM into binary ideal responses. Chiu et al. (2018; see also, Chiu & Köhn, [<reflink idref="bib13" id="ref153">13</reflink>]) proposed a method for estimating an examinee's ideal response for general CDMs using a weighted sum of the disjunctive and conjunctive ideal responses (associated with the DINO and DINA model, respectively). However, this weighted‐sum estimate itself requires an initial classification of examinees based on the DINA model, which may already introduce an early bias into the ideal response estimate. Therefore, it was determined that the IRF of the saturated testlet LCDM would be utilized without a second testlet term that would have explicitly included examinees' ideal responses. Ultimately, the complete crossing of all experimental factors resulted in <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0328" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>×</mo><mn>2</mn><mo>×</mo><mn>2</mn><mo>×</mo><mn>2</mn><mo>×</mo><mn>6</mn><mo>=</mo><mn>96</mn></mrow><annotation encoding="application/x-tex">$2 \times 2 \times 2 \times 2 \times 6 = 96$</annotation></semantics></math> </ephtml> cells, with 500 replicated data sets produced for each of them.</p> <p> <emph>Results: Study II—Using the Saturated LCDM</emph>. The proportions of correct rejections (i.e., the power) are reported in Tables 9 and 10. The former presents only the mean proportions of incorrect rejections for the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0329" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> statistic (like in the type I error study), as each testlet contained the same number of items. The power of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0330" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> when the testlet effect was <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0331" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>=</mo><mn>0.25</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 = 0.25$</annotation></semantics></math> </ephtml> supports the notion that testlet effects of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0332" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>≤</mo><mn>0.25</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 \le 0.25$</annotation></semantics></math> </ephtml> can be safely ignored. However, when the testlet effect size is increased to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0333" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>=</mo><mn>0.50</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 = 0.50$</annotation></semantics></math> </ephtml> , then the power of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0334" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> attains satisfactory levels. For testlet effects beyond <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0335" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>σ</mi><mn>2</mn></msup><mo>=</mo><mn>0.50</mn></mrow><annotation encoding="application/x-tex">$\sigma ^2 = 0.50$</annotation></semantics></math> </ephtml> , the power of AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0336" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> attains its maximum. Table 10 reports the values of the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0337" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic corresponding to the mean proportions of incorrect rejections for the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0338" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> statistic as shown in Table 9. The observations on the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0339" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic support the findings on the power of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0340" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> statistic reported in Table 9.</p> <p>9 Table Saturated LCDM: Results of the Power Study Evaluating the AGG MH χ2(t)$\chi ^2 (t)$ Statistic</p> <p> <ephtml> <table><thead><tr><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0342" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0343" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 5</th></tr><tr><th /><th /><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0344" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0345" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0346" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0347" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.6</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0348" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0349" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0350" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0351" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = .6</th></tr><tr><th /><th>Testlet</th><th /><th /><th /><th /><th /><th /><th /><th /></tr><tr><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0352" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>N</mi><annotation encoding="application/x-tex">$N$</annotation></semantics></math></p></th><th>Effect Size</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0353" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0354" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0355" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0356" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0357" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0358" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0359" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0360" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td /><td /><td>Testlet Size = 4</td></tr><tr><td>500</td><td>0.25</td><td>0.12</td><td>0.27</td><td>0.12</td><td>0.29</td><td>0.12</td><td>0.21</td><td>0.12</td><td>0.57</td></tr><tr><td /><td>0.50</td><td>0.68</td><td>1.00</td><td>0.68</td><td>1.00</td><td>0.57</td><td>0.99</td><td>0.53</td><td>1.00</td></tr><tr><td /><td>0.75</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>0.97</td><td>1.00</td><td>0.98</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>2,000</td><td>0.25</td><td>0.13</td><td>0.61</td><td>0.21</td><td>0.62</td><td>0.18</td><td>0.59</td><td>0.18</td><td>0.72</td></tr><tr><td /><td>0.50</td><td>0.99</td><td>1.00</td><td>0.99</td><td>1.00</td><td>0.93</td><td>1.00</td><td>0.93</td><td>1.00</td></tr><tr><td /><td>0.75</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td /><td>Testlet Size = 6</td></tr><tr><td>500</td><td>0.25</td><td>0.39</td><td>0.19</td><td>0.39</td><td>0.22</td><td>0.17</td><td>0.09</td><td>0.13</td><td>0.24</td></tr><tr><td /><td>0.50</td><td>0.69</td><td>0.93</td><td>0.69</td><td>1.00</td><td>0.41</td><td>1.00</td><td>0.42</td><td>0.92</td></tr><tr><td /><td>0.75</td><td>1.00</td><td>1.00</td><td>0.98</td><td>1.00</td><td>0.86</td><td>1.00</td><td>0.90</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>2,000</td><td>0.25</td><td>0.49</td><td>0.38</td><td>0.48</td><td>0.41</td><td>0.13</td><td>0.33</td><td>0.12</td><td>0.46</td></tr><tr><td /><td>0.50</td><td>0.94</td><td>1.00</td><td>0.95</td><td>1.00</td><td>1.00</td><td>0.79</td><td>0.79</td><td>1.00</td></tr><tr><td /><td>0.75</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr></tbody></table> </ephtml> </p> <p>10 Table Saturated LCDM: Results of the Power Study Evaluating the M2$M_2$ Statistic</p> <p> <ephtml> <table><thead><tr><th /><th /><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0362" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0363" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math></p> = 5</th></tr><tr><th /><th /><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0364" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0365" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0366" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0367" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.6</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0368" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0369" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.3</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0370" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>α</mi><annotation encoding="application/x-tex">${\alpha }$</annotation></semantics></math></p> with <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0371" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>ρ</mi><annotation encoding="application/x-tex">${\rho }$</annotation></semantics></math></p> = 0.6</th></tr><tr><th /><th>Testlet</th><th /><th /><th /><th /><th /><th /><th /><th /></tr><tr><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0372" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>N</mi><annotation encoding="application/x-tex">$N$</annotation></semantics></math></p></th><th>Effect Size</th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0373" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0374" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0375" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0376" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0377" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0378" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0379" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>24</mn></mrow><annotation encoding="application/x-tex">$J = 24$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0380" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>J</mi><mo>=</mo><mn>48</mn></mrow><annotation encoding="application/x-tex">$J = 48$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td /><td /><td>Testlet Size = 4</td></tr><tr><td>500</td><td>0.25</td><td>0.08</td><td>0.09</td><td>0.05</td><td>0.10</td><td>0.05</td><td>0.08</td><td>0.04</td><td>0.08</td></tr><tr><td /><td>0.50</td><td>0.46</td><td>0.96</td><td>0.44</td><td>0.94</td><td>0.26</td><td>0.86</td><td>0.24</td><td>0.86</td></tr><tr><td /><td>0.75</td><td>1.00</td><td>1.00</td><td>0.98</td><td>1.00</td><td>0.98</td><td>1.00</td><td>0.97</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>2,000</td><td>0.25</td><td>0.09</td><td>0.25</td><td>0.11</td><td>0.23</td><td>0.07</td><td>0.19</td><td>0.07</td><td>0.19</td></tr><tr><td /><td>0.50</td><td>0.99</td><td>1.00</td><td>0.99</td><td>1.00</td><td>0.94</td><td>1.00</td><td>0.94</td><td>1.00</td></tr><tr><td /><td>0.75</td><td>0.97</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td /><td>Testlet Size = 6</td></tr><tr><td>500</td><td>0.25</td><td>0.06</td><td>0.08</td><td>0.07</td><td>0.08</td><td>0.07</td><td>0.07</td><td>0.05</td><td>0.07</td></tr><tr><td /><td>0.50</td><td>0.26</td><td>0.79</td><td>0.25</td><td>0.80</td><td>0.20</td><td>0.69</td><td>0.22</td><td>1.66</td></tr><tr><td /><td>0.75</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>0.93</td><td>0.89</td><td>0.90</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>2,000</td><td>0.25</td><td>0.09</td><td>0.17</td><td>0.08</td><td>0.19</td><td>0.07</td><td>0.12</td><td>0.08</td><td>0.15</td></tr><tr><td /><td>0.50</td><td>0.91</td><td>1.00</td><td>0.94</td><td>1.00</td><td>0.81</td><td>1.00</td><td>0.82</td><td>1.00</td></tr><tr><td /><td>0.75</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.25</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td /><td>1.50</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td></tr></tbody></table> </ephtml> </p> <hd id="AN0190280385-15">Application to a Real Data Set</hd> <p></p> <hd id="AN0190280385-16">Design</hd> <p>The results of the simulation studies demonstrated the capacity of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0381" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> statistic to discern the existence of a testlet effect stemming from a cluster of items. In this third study, the efficacy of the proposed AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0382" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> statistic is investigated in practice using a "real‐world" data set collected in 2015 with the computer‐based PISA collaborative problem‐solving (CPS) test. The design of this study followed closely the one used for illustrative purposes in Xu et al. ([<reflink idref="bib72" id="ref154">72</reflink>]). From the main survey Cluster 1, four of the five testlets were selected. In accordance with Yavuz and Atar (2020), Xu et al. ([<reflink idref="bib72" id="ref155">72</reflink>]) presumed the PISA‐CPS assessment was to target three key competencies: shared understanding, appropriate action, and team organization. These were assessed using 28 items of high diagnosticity across four testlets (the Q‐matrix is presented below in Table 11). Xu et al. (2024) selected the data of 8,880 examinees from English‐speaking countries only. Polytomous responses were recoded into dichotomous scores and missing data were excluded.</p> <p>11 Table Q‐Matrix of the 2015 PISA‐CPS Assessment (table 3, Xu et al., 2024)</p> <p> <ephtml> <table><thead><tr><th>Item</th><th>CPS Code</th><th>Attributes</th><th>Testlet</th></tr><tr><th /><th /><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0383" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>1</mn></msub><annotation encoding="application/x-tex">$\alpha _1$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0384" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$\alpha _2$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0385" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>α</mi><mn>3</mn></msub><annotation encoding="application/x-tex">$\alpha _3$</annotation></semantics></math></p></th><th /></tr></thead><tbody><tr><td>1</td><td>CC104101</td><td>0</td><td>1</td><td>0</td><td>1</td></tr><tr><td>2</td><td>CC104102</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>3</td><td>CC104103</td><td>0</td><td>1</td><td>0</td><td>1</td></tr><tr><td>4</td><td>CC104105</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>5</td><td>CC104106</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>6</td><td>CC104107</td><td>1</td><td>0</td><td>0</td><td>1</td></tr><tr><td>7</td><td>CC106101</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>8</td><td>CC106102</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>9</td><td>CC106103</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>10</td><td>CC106104</td><td>0</td><td>0</td><td>1</td><td>2</td></tr><tr><td>11</td><td>CC106105</td><td>0</td><td>1</td><td>0</td><td>2</td></tr><tr><td>12</td><td>CC106106</td><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>13</td><td>CC106107</td><td>0</td><td>0</td><td>1</td><td>2</td></tr><tr><td>14</td><td>CC104201</td><td>0</td><td>0</td><td>1</td><td>3</td></tr><tr><td>15</td><td>CC104202</td><td>1</td><td>0</td><td>0</td><td>3</td></tr><tr><td>16</td><td>CC104203</td><td>1</td><td>0</td><td>0</td><td>3</td></tr><tr><td>17</td><td>CC104204</td><td>0</td><td>0</td><td>1</td><td>3</td></tr><tr><td>18</td><td>CC104205</td><td>0</td><td>1</td><td>0</td><td>3</td></tr><tr><td>19</td><td>CC104206</td><td>1</td><td>0</td><td>0</td><td>3</td></tr><tr><td>20</td><td>CC106201</td><td>1</td><td>0</td><td>0</td><td>4</td></tr><tr><td>21</td><td>CC106202</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>22</td><td>CC106203</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>23</td><td>CC106204</td><td>0</td><td>1</td><td>0</td><td>4</td></tr><tr><td>24</td><td>CC106205</td><td>0</td><td>1</td><td>0</td><td>4</td></tr><tr><td>25</td><td>CC106206</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>26</td><td>CC106207</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>27</td><td>CC106208</td><td>0</td><td>0</td><td>1</td><td>4</td></tr><tr><td>28</td><td>CC106209</td><td>0</td><td>1</td><td>0</td><td>4</td></tr></tbody></table> </ephtml> </p> <hd id="AN0190280385-17">Results</hd> <p>The data were analyzed using the DINA model presented in Equation 3. Based on the model parameter estimates, 1,000 parametric bootstrap samples were generated. From the resulting sampling distribution, the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0386" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>95</mn><mo>%</mo></mrow><annotation encoding="application/x-tex">$95\%$</annotation></semantics></math> </ephtml> ‐critical values for the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0387" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic were obtained for the four testlets, namely 23.92, 33.84, 26.86, and 51.46, respectively. In contrast, the observed values of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0388" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic for the four testlets were 240.30, 468.45, 85.17, and 359.05, respectively. Thus, the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0389" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic was capable of detecting all four testlets in the PISA‐CPS assessment. As an aside, notice that the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0390" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic for this data set was 3275.502, with 343 degrees of freedom and a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0391" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>p</mi><annotation encoding="application/x-tex">$p$</annotation></semantics></math> </ephtml> value of zero (indicating poor fit for the model ignoring the testlet structure).</p> <hd id="AN0190280385-18">Discussion</hd> <p>Testlets offer a number of benefits in the context of educational and psychological measurement. For instance, they have become a popular choice, for instance, in STEM assessments because this item format is well‐suited for measuring contextual knowledge and complex skills without placing an undue burden on examinees, who are not required to read through multiple problem descriptions. However, the grouping of items according to a common theme typically introduces correlations among item responses, thereby violating the local independence assumption. This phenomenon, known as the "testlet effect," can introduce significant bias if ignored. Extant diagnostic procedures and measures employed to detect the presence of testlet effects operate either at the level of the entire test, comparing the fit of models with and without additional testlet parameters, or at the level of individual item pairs testing for potential violations of local independence. None of these approaches is suitable for examining the presence of an aggregated testlet effect as it may originate from taking into account all items in a cluster at once, which is a subtle but important distinction. In this article, a novel variant of the MH statistic is proposed, the aggregated MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0392" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> ‐statistic, AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0393" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> , that requires the application of a parametric bootstrap re‐sampling procedure for statistical inference. The procedure is conceptually straightforward and suitable for any CD assessment data that are suspected of being contaminated by testlet effects. It is readily implemented, as it does not require fitting the data with complex and computationally expensive algorithms to establish the potential presence of testlet effects. The issue of the unknown sampling distribution is addressed through the implementation of a parametric bootstrap resampling scheme.</p> <p>The results of simulation studies employing the novel IT‐DINA model proposed by Xu et al. ([<reflink idref="bib72" id="ref156">72</reflink>]), and the saturated testlet LCDM demonstrated that the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0394" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic exhibits excellent control of the type I error rate under varying experimental condition, including sample size, test length, number of attributes, correlation among attributes, and the number of testlets. The power for detecting a testlet effect based on AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0395" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> is also excellent, provided that the testlet effect exceeds a level of 0.25. (Recall that a majority of researchers concur that a testlet effect of less than 0.25 can be disregarded.) The application of the new statistic to a real‐world data set conforming to the DINA model augmented by a testlet effect demonstrated the efficacy of the new approach in detecting testlet effects.</p> <p>In contrast to existing procedures for testlet diagnosis, which are limited to examining item pairs or entire data sets, but are incapable of identifying a testlet effect associated with a specific subset of items, the bootstrap re‐sampling AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0396" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic provides a means of identifying testlet effects within a data set that are linked to a particular item bundle. Moreover, usage of this diagnostic tool permits researchers and practitioners to refit a data set that has been contaminated by a testlet effect with a CD model that includes explicit parameterization to account for the nuisance variability introduced by testlet(s).</p> <p>In conclusion, three aspects of the current study should be addressed. First, the objective was to evaluate the efficacy and sensitivity of the bootstrap re‐sampling AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0397" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic when item responses are dichotomous. An obvious extension would be to investigate the performance of the proposed statistic when items are polytomous. The transition from dichotomous to polytomous responses may present a significant challenge when tests incorporate testlets. Currently, it remains uncertain whether testlet effects give rise to comparable issues when items are polytomous as they do when items are dichotomous. Closely related is the question whether the findings of this study also apply to testlet variants underlain by different CDMs like the Reduced RUM, (Hartz, [<reflink idref="bib29" id="ref157">29</reflink>]; Hartz & Roussos, [<reflink idref="bib30" id="ref158">30</reflink>]), the Linear Logistic Model (LLM), (Maris, [<reflink idref="bib50" id="ref159">50</reflink>]) or the A‐CDM (de la Torre, [<reflink idref="bib19" id="ref160">19</reflink>])? What impact may complex Q‐matrix compositions have, for example, when attributes are hierarchically organized?</p> <p>Second, in reiterating, to the best of our knowledge, currently no alternative method exists that would allow for the direct evaluation of the performance of the bootstrap re‐sampling AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0398" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic. As an ad hoc bridge of this methodological gap, in the current study, like in Cai et al. (2016), the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0399" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic (Maydeu‐Olivares & Joe, [<reflink idref="bib51" id="ref161">51</reflink>]) was used as an external criterion to evaluate the type I error control and the power of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0400" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic. Future research will show whether more direct alternatives can be developed. Within this context, recall that traditional model comparison approaches as they are used in IRT typically encounter difficulties when applied to CD. Pairwise item statistics may not consistently identify all items involved in a testlet. Hence, for the time being, researchers who wish to evaluate the aggregated testlet effect of a bundle of correlated items may find the approach offered by the bootstrap resampling AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0401" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic helpful.</p> <p>Third, and directly related, what guidance can be provided to researchers and practitioners in the field who wish to use the proposed procedure? Should the elaborate bootstrap sampling procedure raise concerns regarding CPU times? Are there certain factors that significantly affect the performance of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0402" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic?</p> <p>Based on the experience from the simulation studies, the answer to the first question tends to be "no." Depending on the complexity of the various experimental conditions as determined by the number of items, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0403" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math> </ephtml> (i.e., test length), the number of attributes, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0404" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>K</mi><annotation encoding="application/x-tex">$K$</annotation></semantics></math> </ephtml> , the level of error contamination of the data, the size of the correlation <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0405" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>ρ</mi><annotation encoding="application/x-tex">$\rho$</annotation></semantics></math> </ephtml> among the latent attribute dimensions, and the number of testlets, the generation of 1,000 bootstrap samples and the extraction of the corresponding bootstrap sampling distribution of the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0406" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic consumed CPU times ranging from 4 to 57 minutes. Notice that these computational steps were implemented in R on a 64‐bit machine with 16 GB working memory and an Apple M1 Pro processor. From a practical point of view, these CPU times seem within tolerable bounds—especially, as these computational operations need to be performed only once for a given data set. One should also keep in mind that CPU times generally depend on the computational facilities available and the programming language used for implementing the algorithm.</p> <p>With regard to the second question, the simulation studies suggest that among all the experimental factors controlled for (e.g., sample size, test length, item‐to‐testlet ratio, type of CDM, etc.) the strength of the testlet effect—that is, its variance—had the most significant impact across all the different experimental conditions. As a rough guideline, testlet effects with a variance in the range of 0.25‐0.50 are typically difficult to detect—not just with the AGG MH <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0407" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\chi ^2 (t)$</annotation></semantics></math> </ephtml> ‐statistic but also with the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12440:jedm12440-math-0408" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>M</mi><mn>2</mn></msub><annotation encoding="application/x-tex">$M_2$</annotation></semantics></math> </ephtml> statistic. In conclusion, the field of modeling testlets in CD and developing sensitive diagnostics is still relatively new and has numerous open questions that invite further research.</p> <ref id="AN0190280385-19"> <title> References </title> <blist> <bibl id="bib1" idref="ref28" type="bt">1</bibl> <bibtext> Ackerman, T. A. (1987). The robustness of LOGIST and BILOG IRT estimation programs to violations of local independence. Paper presented at the Annual Meeting of the American Educational Research Association, Washington, DC.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref55" type="bt">2</bibl> <bibtext> Akaike, H. (1987). Factor analysis and AIC. Psychometrika, 52, 317 – 332.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref56" type="bt">3</bibl> <bibtext> Beran, R. (1997). Diagnosing bootstrap success. Annals of the Institute of Statistical Mathematics, 49, 1 – 24.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref32" type="bt">4</bibl> <bibtext> B ishop, N. S., & Omar, M. H. (2002). Comparing vertical scales derived from dichotomous and polytomous IRT models for a test composed of testlets. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref80" type="bt">5</bibl> <bibtext> Bock, R. D., & Gibbons, R. (2010). Factor analysis of categorical item responses. In M. L. Nering & R. Ostini (Eds.), Handbook of polytomous item response theory models (pp. 155 – 184). New York, NY : Routledge.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref3" type="bt">6</bibl> <bibtext> Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153 – 168.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref53" type="bt">7</bibl> <bibtext> Cai, L. (2010). A two‐tier full‐information item factor analysis model with applications. Psychometrika, 75 (4), 581–612.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref118" type="bt">8</bibl> <bibtext> Carlin, B. P., & Gelfand, A. E. (1991) A sample reuse method for accurate parametric empirical Bayes confidence intervals. Journal of the Royal Statistical Society Series B, 53, 189 – 200.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref89" type="bt">9</bibl> <bibtext> Chen, W., & Thissen, D. (1997). Local dependence indices for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265 – 289.</bibtext> </blist> <blist> <bibtext> Chen, Y., Liu, Y., & Xu, S. (2018). Mutual information reliability for latent class analysis. Applied Psychological Measurement, 42 (6), 460–477.</bibtext> </blist> <blist> <bibtext> Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123 – 140.</bibtext> </blist> <blist> <bibtext> Chen, Y., Liu, J., Xu, G., & Ying, Z. (2015). Statistical analysis of Q‐matrix based diagnostic classification models. Journal of the American Statistical Association, 110, 850 – 866.</bibtext> </blist> <blist> <bibtext> Chiu, C. Y., & Köhn, H. F. (2019). Nonparametric methods in cognitively diagnostic assessment. In M. von Davier, & Y. S. Lee (Eds.) Handbook of diagnostic classification models: Models and model extensions, applications, software packages (pp. 107 – 132). Cham, CH : Springer Nature.</bibtext> </blist> <blist> <bibtext> Chiu, C.‐Y., Douglas, J. A., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633 – 665.</bibtext> </blist> <blist> <bibtext> Chiu, C.‐Y., Sun, Y., & Bian, Y. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83, 355 – 375.</bibtext> </blist> <blist> <bibtext> Cochran, W. G. (1954). Some methods for strengthening the common χ 2 $\chi ^2$ tests. Biometrics, 10 (4), 417–451.</bibtext> </blist> <blist> <bibtext> DeMars, C. E. (2006). Application of the bi‐factor multidimensional item response theory model to testlet‐based tests. Journal of Educational Measurement, 43 (2), 145–168.</bibtext> </blist> <blist> <bibtext> DeMars, C. E. (2012). Confirming testlet effects. Applied Psychological Measurement, 36, 104 – 121.</bibtext> </blist> <blist> <bibtext> de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179 – 199.</bibtext> </blist> <blist> <bibtext> Efron, B. (1979). Bootstrap methods: another look at the jackknife. Annals of Statistics, 7, 1 – 26.</bibtext> </blist> <blist> <bibtext> Ferrari, A. (2019). A note on sum and difference of correlated chi‐squared variables. arXiv preprint arXiv:1906.09982.</bibtext> </blist> <blist> <bibtext> Geyer, C. J. (1991) Constrained maximum likelihood exemplified by isotonic convex logistic regression. Journal of the American Statistical Association, 86, 717 – 724.</bibtext> </blist> <blist> <bibtext> Gibbons, R. D., & Hedeker, D. R. (1992). Full‐information item bi‐factor analysis. Psychometrika, 57 (3), 423–436.</bibtext> </blist> <blist> <bibtext> Glas, C. A. W., Wainer, H., & Bradlow, E. T. (2000). MML and EAP estimation in testlet‐based adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 271 – 288). Boston, MA : Kluwer‐Nijhoff.</bibtext> </blist> <blist> <bibtext> Guo, L., Zhou, W., & Li, X. (2024). Cognitive diagnosis testlet model for multiple‐choice items. Journal of Educational and Behavioral Statistics, 49 (1), 32–60.</bibtext> </blist> <blist> <bibtext> Hansen, M. (2013). Hierarchical item response models for cognitive diagnosis. Doctoral Dissertation, University of California, Los Angeles.</bibtext> </blist> <blist> <bibtext> Hansen, M., Cai, L., Monroe, S., & Li, Z. (2016). Limited information goodness‐of‐fit testing of diagnostic classification item response models. The British Journal of Mathematical and Statistical Psychology, 69 (3), 225–252.</bibtext> </blist> <blist> <bibtext> Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26 (4), 301–321.</bibtext> </blist> <blist> <bibtext> Hartz, S. M. (2002). A Bayesian framework for the Unified Model for assessing cognitive abilities: Blending theory with practicality. Doctoral dissertation. Available from ProQuest Dissertations and Theses database (UMI No. 3044108).</bibtext> </blist> <blist> <bibtext> Hartz, S. M., & Roussos, L. A. (2008). The fusion model for skill diagnosis: Blending theory with practicality (Research report No. RR‐08‐71). Princeton, NJ : Educational Testing Service.</bibtext> </blist> <blist> <bibtext> Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log‐linear models with latent variables. Psychometrika, 74 (2), 191–210.</bibtext> </blist> <blist> <bibtext> Hinkley, D. V. (1988). Bootstrap methods. Journal of the Royal Statistical Society Series B, 50, 321 – 337.</bibtext> </blist> <blist> <bibtext> Holland, W. P., & Thayer, D. T. (1988). Differential item performance and the Mantel Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129 – 145). Hillsdale, NJ : Lawrence Earlbaum Associates.</bibtext> </blist> <blist> <bibtext> Jiao, H., Kamata, A., Wang, S., & Jin, Y. (2012). A multilevel testlet model for dual local dependence. Journal of Educational Measurement, 49, 82 – 100.</bibtext> </blist> <blist> <bibtext> Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258 – 272.</bibtext> </blist> <blist> <bibtext> Kim, D., De Ayala, R. J., Ferdous, A. A., & Nering, M. L. (2011). The comparative performance of conditional independence indices. Applied Psychological Measurement, 35, 447 – 471.</bibtext> </blist> <blist> <bibtext> Klein‐Braley, C. (1997). C‐Tests in the context of reduced redundancy testing: An appraisal. Language Testing, 14 (1), 47–84.</bibtext> </blist> <blist> <bibtext> Lee, G., Kolen, M. J., Frisbie, D. A., & Ankenmann, R. D. (2002). Comparison of dichotomous and polytomous item response models in equating scores from tests composed of testlets. Applied Psychological Measurement, 25, 357 – 372.</bibtext> </blist> <blist> <bibtext> Li, Y., Bolt, D. M., & Fu, J. (2005). A test characteristic curve linking method for the testlet model. Applied Psychological Measurement, 29, 340 – 356.</bibtext> </blist> <blist> <bibtext> Li, Y., Bolt, D. M., & Fu, J. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30 (1), 3–21.</bibtext> </blist> <blist> <bibtext> Lim, Y. S. (2022). Detecting testlet effects in cognitive diagnosis models. In: Wiberg, M., Molenaar, D., González, J., Kim, J. S., & Hwang, H. (Eds.) Quantitative Psychology. IMPS 2021. Springer Proceedings in Mathematics & Statistics, vol 393. Cham, CH : Springer.</bibtext> </blist> <blist> <bibtext> Lim, Y. S., & Drasgow, F. (2019a). Conditional independence and dimensionality of nonparametric cognitive diagnostic models: A test for model fit. Journal of Classification, 36, 295 – 305.</bibtext> </blist> <blist> <bibtext> Lim, Y. S., & Drasgow, F. (2019b). Assessing the dimensionality of the latent attribute space in cognitive diagnosis through testing for conditional independence. In Wiberg, M., Culpepper, S., Janssen, R., González, J., & Molenaar, D. (Eds.) Quantitative psychology research (pp. 183 – 194). New York : Springer.</bibtext> </blist> <blist> <bibtext> Lord, F. M. & Novick, M. R. (1968). Statistical theories of mental test scores. Addison‐Wesley : Menlo Park, CA.</bibtext> </blist> <blist> <bibtext> Ma, W., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93, 1 – 26.</bibtext> </blist> <blist> <bibtext> Ma, W., Wang, C., & Xiao, J. (2023). A testlet diagnostic classification model with attribute hierarchies. Applied Psychological Measurement, 47 (3), 183–199.</bibtext> </blist> <blist> <bibtext> Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99 – 120.</bibtext> </blist> <blist> <bibtext> Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of National Cancer Institute, 22, 719 – 748.</bibtext> </blist> <blist> <bibtext> Marais, I. D., & Andrich, D. (2008). Effects of varying magnitude and patterns of local dependence in the unidimensional Rasch model. Journal of Applied Measurement, 9, 105 – 124.</bibtext> </blist> <blist> <bibtext> Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187 – 212.</bibtext> </blist> <blist> <bibtext> Maydeu‐Olivares, A., & Joe, H. (2006). Limited information goodness‐of‐fit testing in multidimensional contingency tables. Psychometrika, 71, 713 – 732.</bibtext> </blist> <blist> <bibtext> Nájera, P., Sorrel, M. A., & Abad, F. J. (2024). cdmTools : Useful tools for cognitive diagnosis modeling. R package version 1.0.5. https://cran.r‐project.org/web/packages/cdmTools/</bibtext> </blist> <blist> <bibtext> Rosenbaum, P. R. (1984). Testing the conditional independence and monotonicity assumption of item response theory. Psychometrika, 49, 425 – 436.</bibtext> </blist> <blist> <bibtext> Rosenbaum, P. R. (1988). Item bundles. Psychometrika, 53 (3), 349–359.</bibtext> </blist> <blist> <bibtext> Rupp, A., Templin, J., & Henson, R. (2010). Diagnostic assessment: Theory, methods, and applications. New York : Guilford.</bibtext> </blist> <blist> <bibtext> Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461 – 464.</bibtext> </blist> <blist> <bibtext> Sclove, S. L. (1987). Application of model‐selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333 – 343.</bibtext> </blist> <blist> <bibtext> Sha, S. (2016). Nonparametric diagnostic classification analysis for testlet‐based tests. Unpublished doctoral dissertation, The University of North Carolina at Greensboro.</bibtext> </blist> <blist> <bibtext> Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet‐based tests. Journal of Educational Measurement, 28, 237 – 247.</bibtext> </blist> <blist> <bibtext> Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589 – 617.</bibtext> </blist> <blist> <bibtext> Stout, W. (2005). Dimtest (Version 2.0) Computer software. Champaign, IL : William Stout Institute for Measurement.</bibtext> </blist> <blist> <bibtext> Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20 (4), 345–354.</bibtext> </blist> <blist> <bibtext> von Davier, M. (1997). Bootstrapping goodness‐of‐fit statistics for sparse categorical data: Results of a Monte Carlo study. Methods of Psychological Research, 2, 29 – 48.</bibtext> </blist> <blist> <bibtext> von Davier, M., & Lee, Y.‐S. (Eds.) (2019). Handbook of diagnostic classificaiton models. Models and model extensions, applications, software packages. Cham, CH : Springer.</bibtext> </blist> <blist> <bibtext> Wainer, H & Kiely, G. (1987). Item clusters and computerized adaptive testing: A case study for testlets. Journal of Educational Measurement, 24, 195 – 201.</bibtext> </blist> <blist> <bibtext> Wainer, H., & Wang, X. (2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37, 203 – 220.</bibtext> </blist> <blist> <bibtext> Wainer, H., Bradlow, E. T., & Du, Z. (2000). Testlet response theory: An analog for the 3PL model useful in testlet‐based adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 245 – 269). Dordrecht, Netherlands : Kluwer.</bibtext> </blist> <blist> <bibtext> Wainer, H., Bradlow, E. T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge, UK : Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Wang, W.‐C., & Wilson, M. (2005). The Rasch testlet model. Applied Psychological Measurement, 29, 126 – 149.</bibtext> </blist> <blist> <bibtext> Wang, X., Bradlow, E. T., & Wainer, H. (2002). A general Bayesian model for testlets: Theory and applications. Applied Psychological Measurement, 26, 190 – 128.</bibtext> </blist> <blist> <bibtext> Xu, G., & Shang, Z. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113, 1284 – 1295.</bibtext> </blist> <blist> <bibtext> Xu, X., Fang, G., Guo, J., Ying, Z., & Zhang, S. (2024). Diagnostic classification models for testlets: Methods and theory. Psychometrika, 89 (3), 851–876.</bibtext> </blist> <blist> <bibtext> Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187 – 213.</bibtext> </blist> <blist> <bibtext> Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three‐parameter logistic model. Applied Psychological Measurement, 8, 125 – 145.</bibtext> </blist> <blist> <bibtext> Zhan, P., Li, X., Wang, W.‐C., Bian, Y., & Wang, L. (2015). The multidimensional testlet‐effect cognitive diagnostic models. Acta Psychologica Sinica, 47 (5), 689–701.</bibtext> </blist> <blist> <bibtext> Zhan, P., Liao, M., & Bian, Y. (2018). Joint testlet cognitive diagnosis modeling for paired local item dependence in response times and response accuracy. Frontiers in Psychology, 9, 607.</bibtext> </blist> </ref> <aug> <p>By Youn Seon Lim</p> <p>Reported by Author</p> </aug> <nolink nlid="nl1" bibid="bib67" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib65" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib59" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib68" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib64" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib55" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib10" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib26" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib27" firstref="ref11"></nolink> <nolink nlid="nl10" bibid="bib53" firstref="ref14"></nolink> <nolink nlid="nl11" bibid="bib54" firstref="ref15"></nolink> <nolink nlid="nl12" bibid="bib48" firstref="ref16"></nolink> <nolink nlid="nl13" bibid="bib28" firstref="ref17"></nolink> <nolink nlid="nl14" bibid="bib35" firstref="ref18"></nolink> <nolink nlid="nl15" bibid="bib47" firstref="ref19"></nolink> <nolink nlid="nl16" bibid="bib31" firstref="ref20"></nolink> <nolink nlid="nl17" bibid="bib62" firstref="ref21"></nolink> <nolink nlid="nl18" bibid="bib44" firstref="ref22"></nolink> <nolink nlid="nl19" bibid="bib49" firstref="ref24"></nolink> <nolink nlid="nl20" bibid="bib66" firstref="ref26"></nolink> <nolink nlid="nl21" bibid="bib73" firstref="ref27"></nolink> <nolink nlid="nl22" bibid="bib38" firstref="ref33"></nolink> <nolink nlid="nl23" bibid="bib39" firstref="ref34"></nolink> <nolink nlid="nl24" bibid="bib60" firstref="ref38"></nolink> <nolink nlid="nl25" bibid="bib41" firstref="ref42"></nolink> <nolink nlid="nl26" bibid="bib37" firstref="ref45"></nolink> <nolink nlid="nl27" bibid="bib18" firstref="ref46"></nolink> <nolink nlid="nl28" bibid="bib17" firstref="ref51"></nolink> <nolink nlid="nl29" bibid="bib23" firstref="ref52"></nolink> <nolink nlid="nl30" bibid="bib72" firstref="ref58"></nolink> <nolink nlid="nl31" bibid="bib46" firstref="ref61"></nolink> <nolink nlid="nl32" bibid="bib58" firstref="ref62"></nolink> <nolink nlid="nl33" bibid="bib75" firstref="ref64"></nolink> <nolink nlid="nl34" bibid="bib76" firstref="ref65"></nolink> <nolink nlid="nl35" bibid="bib74" firstref="ref83"></nolink> <nolink nlid="nl36" bibid="bib36" firstref="ref85"></nolink> <nolink nlid="nl37" bibid="bib51" firstref="ref88"></nolink> <nolink nlid="nl38" bibid="bib42" firstref="ref94"></nolink> <nolink nlid="nl39" bibid="bib43" firstref="ref95"></nolink> <nolink nlid="nl40" bibid="bib11" firstref="ref97"></nolink> <nolink nlid="nl41" bibid="bib19" firstref="ref101"></nolink> <nolink nlid="nl42" bibid="bib14" firstref="ref105"></nolink> <nolink nlid="nl43" bibid="bib45" firstref="ref106"></nolink> <nolink nlid="nl44" bibid="bib16" firstref="ref107"></nolink> <nolink nlid="nl45" bibid="bib33" firstref="ref111"></nolink> <nolink nlid="nl46" bibid="bib21" firstref="ref117"></nolink> <nolink nlid="nl47" bibid="bib20" firstref="ref119"></nolink> <nolink nlid="nl48" bibid="bib22" firstref="ref120"></nolink> <nolink nlid="nl49" bibid="bib32" firstref="ref121"></nolink> <nolink nlid="nl50" bibid="bib52" firstref="ref132"></nolink> <nolink nlid="nl51" bibid="bib12" firstref="ref133"></nolink> <nolink nlid="nl52" bibid="bib71" firstref="ref134"></nolink> <nolink nlid="nl53" bibid="bib34" firstref="ref140"></nolink> <nolink nlid="nl54" bibid="bib70" firstref="ref142"></nolink> <nolink nlid="nl55" bibid="bib69" firstref="ref143"></nolink> <nolink nlid="nl56" bibid="bib25" firstref="ref148"></nolink> <nolink nlid="nl57" bibid="bib24" firstref="ref150"></nolink> <nolink nlid="nl58" bibid="bib13" firstref="ref153"></nolink> <nolink nlid="nl59" bibid="bib29" firstref="ref157"></nolink> <nolink nlid="nl60" bibid="bib30" firstref="ref158"></nolink> <nolink nlid="nl61" bibid="bib50" firstref="ref159"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1491369
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Parametric Bootstrap Mantel-Haenszel Statistic for Aggregated Testlet Effects
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Youn+Seon+Lim%22">Youn Seon Lim</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0225-1527">0000-0003-0225-1527</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2025 62(4):503-530.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 28
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Sampling%22">Sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Inference%22">Statistical Inference</searchLink><br /><searchLink fieldCode="DE" term="%22Tests%22">Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Cognitive+Measurement%22">Cognitive Measurement</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Program+for+International+Student+Assessment%22">Program for International Student Assessment</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.12440
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: While testlets have proven useful for assessing complex skills, the stem shared by multiple items often induces correlations between responses, leading to violations of local independence (LI), which can result in biased parameter and ability estimates. Diagnostic procedures for detecting testlet effects typically involve model comparisons testing for the inclusion of extra testlet parameters or, at the item level, testing for pairwise LI. Rosenbaum's adaptation of the Mantel-Haenszel (MH) X[superscript 2]-statistic belongs to the latter category. The MH X[superscript 2]-statistic has also been used in cognitive diagnosis for detecting violations of LI and for the identification of testlet effects. However, this approach is not without limitations, as it lacks a rationale for integrating multiple pairwise MH X[superscript 2]-statistics and any notion of the sampling distribution of such an integrated statistic. In this article, a procedure for integrating multiple pairwise MH X[superscript 2]-statistics to evaluate testlet effects in cognitive diagnosis is proposed. The unknown sampling distribution issue is addressed by implementing a parametric bootstrap resampling scheme. Results from simulation studies demonstrate the performance of the proposed parametric bootstrap testlet MH X[superscript 2]-statistic, and its application to the 2015 PISA Collaborative Problem Solving (CPS) data set illustrates the method's practical merits.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1491369
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1491369
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.12440
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 28
        StartPage: 503
    Subjects:
      – SubjectFull: Sampling
        Type: general
      – SubjectFull: Statistical Inference
        Type: general
      – SubjectFull: Tests
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Cognitive Measurement
        Type: general
      – SubjectFull: Program for International Student Assessment
        Type: general
    Titles:
      – TitleFull: Parametric Bootstrap Mantel-Haenszel Statistic for Aggregated Testlet Effects
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Youn Seon Lim
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 62
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1