View in EDS HTML Full Text PDF Full Text

Equivalent Diagnostic Classification Models

Saved in:
Bibliographic Details
Title: Equivalent Diagnostic Classification Models
Language: English
Authors: Maris, Gunter, Bechger, Timo
Source: Measurement: Interdisciplinary Research and Perspectives. Jan 2009 7(1):41-46.
Availability: Psychology Press. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 6
Publication Date: 2009
Document Type: Journal Articles
Opinion Papers
Descriptors: Factor Analysis, Classification, Psychometrics, Item Response Theory, Evaluation Problems, Models, Evidence, Measurement, Criterion Referenced Tests, Diagnostic Tests, Measurement Techniques, Educational Assessment, Educational Testing, Student Evaluation, Evaluation Methods, State of the Art Reviews
DOI: 10.1080/15366360802715478
ISSN: 1536-6367
Abstract: Rupp and Templin (2008) do a good job at describing the ever expanding landscape of Diagnostic Classification Models (DCM). In many ways, their review article clearly points to some of the questions that need to be answered before DCMs can become part of the psychometric practitioners toolkit. Apart from the issues mentioned in this article that are explicitly addressed in the article by Rupp and Templin (2008) there is one crucial issue that remains: equivalence of different DCMs, within or across different classes of DCMs. As DCMs claim to encode cognitive psychological theories, it is important that competing theories give rise to different models. That is, it should be possible to distinguish between competing theories on the basis of infinite observations. To the best of the authors' knowledge this issue has not yet been addressed. The situation is similar to that with the linear logistic test model (LLTM, Schleiblechner, 1972; Fischer, 1995) and the item response model with internal restrictions on item difficulty (MIRID, Butter, 1994; Butter, De Boeck, & Verhelst, 1998) that similarly try to encode substantive theories, and to the rotational invariance problem known from factor analysis. Since different, yet equivalent, models may lead to different diagnoses, and subsequently, different treatments, the problem is essential. This commentary addresses the equivalence problem. (Contains 1 footnote.)
Abstractor: ERIC
Number of References: 10
Entry Date: 2009
Accession Number: EJ831122
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEWyeJN2cHESoddPwqv38ylAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJPL9jl_NbFf_mgR6QIBEICBmn6beCgJEPifdvihQqav3N_E4VWJLeVh8VLCvzUnTGEmcFaSwR075I2_elEtjpFSEMEMk62nB432lEtNHiUOUdK0h0_PNOlfUgmNP5GwwkOmv7q-CKWalbXwe231jnHZleXEDvmUvXC2_bkb9VaLEt44OgIX43Ko8OUuCXUTEWEwFvQYapXhMEh-7X3v7WL6MybZZAVjk17THr0=
Text:
  Availability: 1
  Value: <anid>AN0036819613;p6i01jan.09;2019Mar14.13:55;v2.2.500</anid> <title id="AN0036819613-1">Equivalent Diagnostic Classification Models. </title> <p>[<reflink idref="bib9" id="ref1">9</reflink>]) do a good job at describing the ever expanding landscape of Diagnostic Classification Models (DCM). In many ways, their review article clearly points to some of the questions that need to be answered before DCMs can become part of the psychometric practitioners toolkit: To name a few of the outstanding, mainly statistical, problems:</p> <p></p> <ulist> <item> 1. The sort of cognitive theory needed for successful application of DCM is generally lacking, except maybe for rather trivial applications (where did all this interest in fraction-subtraction all of a sudden come from?).</item> <p></p> <item> 2. Identifiability of the parameters from the (distribution of the) observations remains problematic for most of the DCMs. As opposed to the much more clear cut situation with identifiability in, for instance, many of the monotone latent variable models (e.g., the Rasch model), the problem is much harder and much less trivial for DCMs. Rupp and Templin refer to the arbitrary, yet substantively significant, restrictions imposed by [<reflink idref="bib6" id="ref2">6</reflink>], pp. 193–195) on MCLCMs to solve identifiability problems. Even with these restrictions it is still unclear whether or not the parameters are identifiable from the observations. It can not be the case that substantive conclusions depend on arbitrary identifiability constraints.</item> <p></p> <item> 3. Estimating DCMs remains problematic and prohibits their large scale use in education (you can't have children sit for a test without the guarantee that they'll get a score, or in this case a bunch of them in return). The convergence problems encountered with many programs are likely to be related to the identifiability issues mentioned above.</item> <p></p> <item> 4. Evaluating fit of DCMs remains a crucial, yet largely uncovered topic.</item> </ulist> <p>Apart from the issues mentioned above that are explicitly addressed in the article by [<reflink idref="bib9" id="ref3">9</reflink>]) there is one crucial issue that remains: equivalence of different DCMs, within or across different classes of DCMs. As DCMs claim to encode cognitive psychological theories, it is important that competing theories give rise to different models. That is, it should be possible to distinguish between competing theories on the basis of infinite observations. To the best of our knowledge this issue has not yet been addressed. The situation is similar to that with the linear logistic test model (LLTM, [<reflink idref="bib10" id="ref4">10</reflink>]; [<reflink idref="bib5" id="ref5">5</reflink>]) and the item response model with internal restrictions on item difficulty (MIRID, [<reflink idref="bib3" id="ref6">3</reflink>]; [<reflink idref="bib4" id="ref7">4</reflink>]) that similarly try to encode substantive theories, and to the rotational invariance problem known from factor analysis. Since different, yet equivalent, models may lead to different diagnoses, and subsequently, different treatments, the problem is essential. It is the purpose of this commentary to make a start with addressing the equivalence problem.</p> <p>As an aside, observe that the MIRID is a generalization of the LLTM, in which some of the entries in the <emph>Q</emph>-matrix are considered to be parameters rather than known constants. In this way we can implement partial cognitive theories. A similar approach can be adopted with CDMs as a basis for both implementing partial cognitive theories and for developing statistical tests focused on single entries in the <emph>Q</emph>-matrix.</p> <hd id="AN0036819613-2">EQUIVALENT DCMs</hd> <p>In order to make progress, it is important to formally state what it means if two models are equivalent. Formally, two models are equivalent if, given parameters for the one, we can <emph>always</emph> find parameters for the other such that both models represent the same distribution over the observations. We will illustrate with an example that the problem of equivalent DCMs exist, and relate some of the problems to the literature that is available for LLTMs ([<reflink idref="bib1" id="ref8">1</reflink>]; [<reflink idref="bib2" id="ref9">2</reflink>]) and MIRIDs ([<reflink idref="bib7" id="ref10">7</reflink>]). We illustrate the problem with the noisy inputs, deterministic "or" (NIDO) model, as it is presented in Table 3 of Rupp and Templin:</p> <p>Graph</p> <p>It will prove convenient to express the model in terms of the log-odds:</p> <p>Graph</p> <p>because these can easily be cast in matrix form:</p> <p>(<reflink idref="bib1" id="ref11">1</reflink>)</p> <p>Graph</p> <p>where</p> <p></p> <ulist> <item> • <emph>Q<subs>jk</subs></emph> = <emph>q<subs>jk</subs></emph></item> <p></p> <item> • <emph>B<subs>k</subs></emph> = β<subs><emph>k</emph></subs></item> <p></p> <item> • <emph>A<subs>ki</subs></emph> = α<subs><emph>ik</emph></subs></item> <p></p> <item> • <emph>G<subs>k,k</subs></emph> = γ<subs><emph>k</emph></subs> and <emph>G<subs>k,l</subs></emph> = 0 for <emph>l</emph> ≠ <emph>k</emph></item> </ulist> <p>It is readily found, using Theorem 1.5 in [<reflink idref="bib8" id="ref12">8</reflink>], which is the key Theorem 1 of [<reflink idref="bib7" id="ref13">7</reflink>] that if O is any matrix such that[<reflink idref="bib1" id="ref14">1</reflink>] OO<sups><emph>g</emph>1</sups> Q = Q:</p> <p>Graph</p> <p>Both the design matrix Q and O yield the exact same model, and no amount of data can ever allow us to distinguish between the two. However, both models may encode very different substantive theories and yield very different γ and β parameter values.</p> <p>In order to further illustrate this problem, we consider a small artificial example consisting of the following six items:</p> <p></p> <ulist> <item> 1. 1 + 2 = ?</item> <p></p> <item> 2. 3−2 = ?</item> <p></p> <item> 3. You already have four apples and are given two additional ones by your mother. Your brother however eats one of them. How many apples do you have left?</item> <p></p> <item> 4. 2 × 3 = ?</item> <p></p> <item> 5. 8/2 = ?</item> <p></p> <item> 6. Jan is three times as old as John, and John is half as old as Mary. Mary is 8. How old is Jan?</item> </ulist> <p>Our substantive cognitive theory is reflected in the following <emph>Q</emph>-matrix:</p> <p>Graph</p> <p>We see that items 1, 2, 4, and 5 refer to the elementary operations of addition, subtraction, multiplication, and division, whereas items 3 and 6 combine two elementary operations and additionaly depend on reading ability (the fifth skill needed).</p> <p>A second researcher looks at these same items and develops an alternative theory. He agrees with the first researcher that items one and two refer to the <emph>abstract</emph> skill of addition and subtraction. However, children can solve the third item without knowing the mathematical notation used in the first two items, because knowing the symbols for addition and subtraction are not needed to solve the item. So, our researcher concludes that a separate skill is needed for this item. The fourth and fifth item similarly, refer to the <emph>abstract</emph> skills of multiplication and division. For the sixth item, the researcher is not so sure. Because its formulation is much closer to the mathematical notation used in items 4 and 5, he decides to leave its structure undefined. So, he ends up with the following <emph>Q</emph>-matrix:</p> <p>Graph</p> <p>where <emph>a</emph>, <emph>b</emph>, <emph>c</emph>, and <emph>d</emph> are parameters that need to be estimated. Through data analysis, he finds that, in fact, the following structure applies to these items:</p> <p>Graph</p> <p>Our researcher may be somewhat puzzled by the values he has estimated, but concludes that his theory holds and that items such as 3 tap different skills than do items 1 and 2.</p> <p>The reader may expect, and formally check using the results presented above, that the original <emph>Q</emph>-matrix and the estimated <emph>Q</emph>-matrix O lead to equivalent models. Hence, on the basis of these items it is not possible to distinguish between the different theories on the basis of the responses children give to these items.</p> <p>Observe that the interpretation of the profile α<subs><emph>i</emph></subs>, which assumes the same numerical value with both models, is very different. For instance, consider a student for whom the probability that α<subs>1</subs> equals 1 is 1/10 and for whom the probability that α<subs>5</subs> equals 1 is 9/10. Our first researcher would reach the conclusion that this student can't add numbers, but masters reading; the second researcher would reach the opposite conclusion. Thus, different but equivalent models may indeed lead to very different conclusions.</p> <p>It is readily found that O<sups><emph>g</emph>1</sups> Q has the following form for this example:</p> <p>Graph</p> <p>And hence the original β and γ values (in the form of the matrices B and G need to be premultiplied with this matrix.</p> <hd id="AN0036819613-3">CONCLUSION</hd> <p>We have illustrated that the problems with model equivalence found in more traditional psychometric models that attempt to encode substantive theories (i.e., the LLTM and MIRID) also occur with DCMs. A somewhat contrived example was presented to illustrate the problem in the context of one particular DCM (i.e., the NIDO model), and to illustrate how the methods developed for dealing with equivalent models in the context of the MIRID model can directly be applied to the NIDO model. However, within the general class of DCMs the problem is probably much more complicated. In closing, we believe that <emph>if</emph> the issues raised by Rupp and Templin, as well as those raised in this commentary are properly addressed, DCMs may find their way to the toolkit of the psychometric practitioner.</p> <ref id="AN0036819613-4"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref8" type="bt">1</bibl> <bibtext> Bechger, T. M., Verhelst, N. D. and Verstralen, H. H. F. M.2001. Identifiability of nonlinear logistic test models. Psychometrika, 66: 357–372.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref9" type="bt">2</bibl> <bibtext> Bechger, T. M., Verstralen, H. H. F. M. and Verhelst, N. D.2002. Equivalent linear logistic test models. Psychometrika, 67: 123–136.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref6" type="bt">3</bibl> <bibtext> Butter, R.1994. Item response model with internal restrictions on item difficulty, KU Leuven: Unpublished doctoral dissertation.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref7" type="bt">4</bibl> <bibtext> Butter, R., De Boeck, P. and Verhelst, N. D.1998. An item response model with internal restrictions on item difficulty. Psychometrika, 63: 47–63.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref5" type="bt">5</bibl> <bibtext> Fischer, G. H.1995. "The linear logistic test model". In Rasch models: Foundations, recent developments and applications, Edited by: Fischer, G. H. and Molenaar, I. W.131–155. Berlin: Springer.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref2" type="bt">6</bibl> <bibtext> Maris, E.1999. Estimating multiple classification latent class models. Psychometrika, 64(2): 187–212.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref10" type="bt">7</bibl> <bibtext> Maris, G. and Bechger, T. M.2004. Equivalent MIRID models. Psychometrika, 69: 627–639.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref12" type="bt">8</bibl> <bibtext> Pringle, R. M. and Rayner, A. A.1971. Generalized inverse matrices with applications to statistics, London: Charles Griffin & Co.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref1" type="bt">9</bibl> <bibtext> Rupp, A. A. and Templin, J. L.2008. Unique Characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement, 6(4): 219–262.</bibtext> </blist> <blist> <bibtext> Schleiblechner, H.1972. Das Lernen and Lösen komplexer Denkaufgaben [The learning and solving of complex reasoning items]. Zeitschrift für Experimentelle and Angewandte Psychologie, 3: 456–506.</bibtext> </blist> </ref> <ref id="AN0036819613-5"> <title> Footnotes </title> <blist> <bibtext> <sups>1</sups> Observe that O<sups><emph>g</emph>1</sups> = (O<sups><emph>T</emph></sups> O)<sups>–1</sups> O<sups><emph>T</emph></sups> provides an easy expression for a <emph>g</emph>1 inverse of the matrix O.</bibtext> </blist> </ref> <aug> <p>By Gunter Maris and Timo Bechger</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref4"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ831122
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Equivalent Diagnostic Classification Models
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Maris%2C+Gunter%22">Maris, Gunter</searchLink><br /><searchLink fieldCode="AR" term="%22Bechger%2C+Timo%22">Bechger, Timo</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Measurement%3A+Interdisciplinary+Research+and+Perspectives%22"><i>Measurement: Interdisciplinary Research and Perspectives</i></searchLink>. Jan 2009 7(1):41-46.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Psychology Press. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 6
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2009
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Opinion Papers
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Factor+Analysis%22">Factor Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink><br /><searchLink fieldCode="DE" term="%22Psychometrics%22">Psychometrics</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Problems%22">Evaluation Problems</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence%22">Evidence</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement%22">Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Criterion+Referenced+Tests%22">Criterion Referenced Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Diagnostic+Tests%22">Diagnostic Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement+Techniques%22">Measurement Techniques</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Assessment%22">Educational Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Testing%22">Educational Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22State+of+the+Art+Reviews%22">State of the Art Reviews</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/15366360802715478
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1536-6367
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Rupp and Templin (2008) do a good job at describing the ever expanding landscape of Diagnostic Classification Models (DCM). In many ways, their review article clearly points to some of the questions that need to be answered before DCMs can become part of the psychometric practitioners toolkit. Apart from the issues mentioned in this article that are explicitly addressed in the article by Rupp and Templin (2008) there is one crucial issue that remains: equivalence of different DCMs, within or across different classes of DCMs. As DCMs claim to encode cognitive psychological theories, it is important that competing theories give rise to different models. That is, it should be possible to distinguish between competing theories on the basis of infinite observations. To the best of the authors' knowledge this issue has not yet been addressed. The situation is similar to that with the linear logistic test model (LLTM, Schleiblechner, 1972; Fischer, 1995) and the item response model with internal restrictions on item difficulty (MIRID, Butter, 1994; Butter, De Boeck, & Verhelst, 1998) that similarly try to encode substantive theories, and to the rotational invariance problem known from factor analysis. Since different, yet equivalent, models may lead to different diagnoses, and subsequently, different treatments, the problem is essential. This commentary addresses the equivalence problem. (Contains 1 footnote.)
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: ERIC
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 10
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2009
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ831122
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ831122
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/15366360802715478
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 6
        StartPage: 41
    Subjects:
      – SubjectFull: Factor Analysis
        Type: general
      – SubjectFull: Classification
        Type: general
      – SubjectFull: Psychometrics
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Evaluation Problems
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Evidence
        Type: general
      – SubjectFull: Measurement
        Type: general
      – SubjectFull: Criterion Referenced Tests
        Type: general
      – SubjectFull: Diagnostic Tests
        Type: general
      – SubjectFull: Measurement Techniques
        Type: general
      – SubjectFull: Educational Assessment
        Type: general
      – SubjectFull: Educational Testing
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: State of the Art Reviews
        Type: general
    Titles:
      – TitleFull: Equivalent Diagnostic Classification Models
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Maris, Gunter
      – PersonEntity:
          Name:
            NameFull: Bechger, Timo
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2009
          Identifiers:
            – Type: issn-print
              Value: 1536-6367
          Numbering:
            – Type: volume
              Value: 7
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Measurement: Interdisciplinary Research and Perspectives
              Type: main
ResultId 1