CLAIRE: clustering evaluation based on item response theory and model agreement.
Saved in:
| Title: | CLAIRE: clustering evaluation based on item response theory and model agreement. |
|---|---|
| Authors: | Ferreira-Junior, Manuel1 (AUTHOR) mfj@cin.ufpe.br, Lima Neto, Eufrasio A.2 (AUTHOR) eufrasio@de.ufpb.br, Ferreira, Marcelo R. P.2 (AUTHOR) marcelo.ferreira@academico.ufpb.br, Silva Filho, Telmo M.3 (AUTHOR) telmo.silvafilho@bristol.ac.uk, Prudêncio, Ricardo B. C.1 (AUTHOR) rbcp@cin.ufpe.br |
| Source: | Machine Learning. Nov2025, Vol. 114 Issue 11, p1-39. 39p. |
| Abstract: | Clustering evaluation is a complex task. External measures, such as the Rand index, are often used for benchmarking, but they are not applicable in real, unsupervised scenarios due to the lack of ground truth. Thus, we often turn to internal measures for model evaluation, e.g. silhouette, Dunn, and Davies-Bouldin. These indexes have the advantage of evaluating models based on the clustered data points themselves, however, they rely on a chosen distance, so they are only meaningful for models that use the same distance. Additionally, they fail if all instances are assigned to a single cluster and they aim to evaluate separation and cohesion instead of quantifying a model’s ability to recover any underlying classes. Thus, internal measures are not suited to compare models that are estimated differently. In this paper, we propose CLAIRE (CLuster Agreement-based Item REsponses), a method for global evaluation of clustering models, by assuming that good models agree on whether pairs of instances should be clustered together or not. We leverage Item Response Theory to estimate model ability and instance difficulty, using response matrices obtained by measuring the agreement between models. Experiments were carried out using diverse sets of clustering methods and datasets with different numbers of clusters and varying shapes and levels of overlapping and noise. Results show that CLAIRE is robust to the presence of random partitions in the pool of models and correctly ranks models across the many tested scenarios with a surprisingly high correlation with external measures of clustering quality, meaning it also indirectly evaluates the recovery of underlying classes. [ABSTRACT FROM AUTHOR] |
| Copyright of Machine Learning is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 188772791 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: CLAIRE: clustering evaluation based on item response theory and model agreement. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Ferreira-Junior%2C+Manuel%22">Ferreira-Junior, Manuel</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> mfj@cin.ufpe.br</i><br /><searchLink fieldCode="AR" term="%22Lima+Neto%2C+Eufrasio+A%2E%22">Lima Neto, Eufrasio A.</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> eufrasio@de.ufpb.br</i><br /><searchLink fieldCode="AR" term="%22Ferreira%2C+Marcelo+R%2E+P%2E%22">Ferreira, Marcelo R. P.</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> marcelo.ferreira@academico.ufpb.br</i><br /><searchLink fieldCode="AR" term="%22Silva+Filho%2C+Telmo+M%2E%22">Silva Filho, Telmo M.</searchLink><relatesTo>3</relatesTo> (AUTHOR)<i> telmo.silvafilho@bristol.ac.uk</i><br /><searchLink fieldCode="AR" term="%22Prudêncio%2C+Ricardo+B%2E+C%2E%22">Prudêncio, Ricardo B. C.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> rbcp@cin.ufpe.br</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Machine+Learning%22">Machine Learning</searchLink>. Nov2025, Vol. 114 Issue 11, p1-39. 39p. – Name: Abstract Label: Abstract Group: Ab Data: Clustering evaluation is a complex task. External measures, such as the Rand index, are often used for benchmarking, but they are not applicable in real, unsupervised scenarios due to the lack of ground truth. Thus, we often turn to internal measures for model evaluation, e.g. silhouette, Dunn, and Davies-Bouldin. These indexes have the advantage of evaluating models based on the clustered data points themselves, however, they rely on a chosen distance, so they are only meaningful for models that use the same distance. Additionally, they fail if all instances are assigned to a single cluster and they aim to evaluate separation and cohesion instead of quantifying a model’s ability to recover any underlying classes. Thus, internal measures are not suited to compare models that are estimated differently. In this paper, we propose CLAIRE (CLuster Agreement-based Item REsponses), a method for global evaluation of clustering models, by assuming that good models agree on whether pairs of instances should be clustered together or not. We leverage Item Response Theory to estimate model ability and instance difficulty, using response matrices obtained by measuring the agreement between models. Experiments were carried out using diverse sets of clustering methods and datasets with different numbers of clusters and varying shapes and levels of overlapping and noise. Results show that CLAIRE is robust to the presence of random partitions in the pool of models and correctly ranks models across the many tested scenarios with a surprisingly high correlation with external measures of clustering quality, meaning it also indirectly evaluates the recovery of underlying classes. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Machine Learning is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=188772791 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s10994-025-06911-0 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 39 StartPage: 1 Titles: – TitleFull: CLAIRE: clustering evaluation based on item response theory and model agreement. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Ferreira-Junior, Manuel – PersonEntity: Name: NameFull: Lima Neto, Eufrasio A. – PersonEntity: Name: NameFull: Ferreira, Marcelo R. P. – PersonEntity: Name: NameFull: Silva Filho, Telmo M. – PersonEntity: Name: NameFull: Prudêncio, Ricardo B. C. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 11 Text: Nov2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 08856125 Numbering: – Type: volume Value: 114 – Type: issue Value: 11 Titles: – TitleFull: Machine Learning Type: main |
| ResultId | 1 |
