View in EDS

CLAIRE: clustering evaluation based on item response theory and model agreement.

Saved in:
Bibliographic Details
Title: CLAIRE: clustering evaluation based on item response theory and model agreement.
Authors: Ferreira-Junior, Manuel1 (AUTHOR) mfj@cin.ufpe.br, Lima Neto, Eufrasio A.2 (AUTHOR) eufrasio@de.ufpb.br, Ferreira, Marcelo R. P.2 (AUTHOR) marcelo.ferreira@academico.ufpb.br, Silva Filho, Telmo M.3 (AUTHOR) telmo.silvafilho@bristol.ac.uk, Prudêncio, Ricardo B. C.1 (AUTHOR) rbcp@cin.ufpe.br
Source: Machine Learning. Nov2025, Vol. 114 Issue 11, p1-39. 39p.
Abstract: Clustering evaluation is a complex task. External measures, such as the Rand index, are often used for benchmarking, but they are not applicable in real, unsupervised scenarios due to the lack of ground truth. Thus, we often turn to internal measures for model evaluation, e.g. silhouette, Dunn, and Davies-Bouldin. These indexes have the advantage of evaluating models based on the clustered data points themselves, however, they rely on a chosen distance, so they are only meaningful for models that use the same distance. Additionally, they fail if all instances are assigned to a single cluster and they aim to evaluate separation and cohesion instead of quantifying a model’s ability to recover any underlying classes. Thus, internal measures are not suited to compare models that are estimated differently. In this paper, we propose CLAIRE (CLuster Agreement-based Item REsponses), a method for global evaluation of clustering models, by assuming that good models agree on whether pairs of instances should be clustered together or not. We leverage Item Response Theory to estimate model ability and instance difficulty, using response matrices obtained by measuring the agreement between models. Experiments were carried out using diverse sets of clustering methods and datasets with different numbers of clusters and varying shapes and levels of overlapping and noise. Results show that CLAIRE is robust to the presence of random partitions in the pool of models and correctly ranks models across the many tested scenarios with a surprisingly high correlation with external measures of clustering quality, meaning it also indirectly evaluates the recovery of underlying classes. [ABSTRACT FROM AUTHOR]
Copyright of Machine Learning is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Description
Abstract:Clustering evaluation is a complex task. External measures, such as the Rand index, are often used for benchmarking, but they are not applicable in real, unsupervised scenarios due to the lack of ground truth. Thus, we often turn to internal measures for model evaluation, e.g. silhouette, Dunn, and Davies-Bouldin. These indexes have the advantage of evaluating models based on the clustered data points themselves, however, they rely on a chosen distance, so they are only meaningful for models that use the same distance. Additionally, they fail if all instances are assigned to a single cluster and they aim to evaluate separation and cohesion instead of quantifying a model’s ability to recover any underlying classes. Thus, internal measures are not suited to compare models that are estimated differently. In this paper, we propose CLAIRE (CLuster Agreement-based Item REsponses), a method for global evaluation of clustering models, by assuming that good models agree on whether pairs of instances should be clustered together or not. We leverage Item Response Theory to estimate model ability and instance difficulty, using response matrices obtained by measuring the agreement between models. Experiments were carried out using diverse sets of clustering methods and datasets with different numbers of clusters and varying shapes and levels of overlapping and noise. Results show that CLAIRE is robust to the presence of random partitions in the pool of models and correctly ranks models across the many tested scenarios with a surprisingly high correlation with external measures of clustering quality, meaning it also indirectly evaluates the recovery of underlying classes. [ABSTRACT FROM AUTHOR]
ISSN:08856125
DOI:10.1007/s10994-025-06911-0