View in EDS HTML Full Text PDF Full Text

Reproducibility of methodological radiomics score (METRICS): an intra- and inter-rater reliability study endorsed by EuSoMII.

Saved in:
Bibliographic Details
Title: Reproducibility of methodological radiomics score (METRICS): an intra- and inter-rater reliability study endorsed by EuSoMII.
Authors: Akinci D'Antonoli, Tugba1 (AUTHOR) tugba.akincidantonoli@unibas.ch, Cavallo, Armando Ugo2 (AUTHOR), Kocak, Burak3 (AUTHOR), Borgheresi, Alessandra4,5 (AUTHOR), Ponsiglione, Andrea6 (AUTHOR), Stanzione, Arnaldo6 (AUTHOR), Koltsakis, Emmanouil7,8 (AUTHOR), Doniselli, Fabio Martino9 (AUTHOR), Vernuccio, Federica10 (AUTHOR), Ugga, Lorenzo6 (AUTHOR), Triantafyllou, Matthaios11 (AUTHOR), Huisman, Merel12 (AUTHOR), Klontzas, Michail E.8,13,14 (AUTHOR), Trotta, Romina15 (AUTHOR), Cannella, Roberto10 (AUTHOR), Fanni, Salvatore Claudio16 (AUTHOR), Cuocolo, Renato17 (AUTHOR)
Source: European Radiology. Aug2025, Vol. 35 Issue 8, p4533-4545. 13p.
Subjects: Radiomics, Inter-observer reliability, Statistical reliability, Standards, Reproducible research
Abstract: Objectives: To investigate the intra- and inter-rater reliability of the total methodological radiomics score (METRICS) and its items through a multi-reader analysis. Materials and methods: A total of 12 raters with different backgrounds and experience levels were recruited for the study. Based on their level of expertise, raters were randomly assigned to the following groups: two inter-rater reliability groups, and two intra-rater reliability groups, where each group included one group with and one group without a preliminary training session on the use of METRICS. Inter-rater reliability groups assessed all 34 papers, while intra-rater reliability groups completed the assessment of 17 papers twice within 21 days each time, and a "wash out" period of 60 days in between. Results: Inter-rater reliability was poor to moderate between raters of group 1 (without training; ICC = 0.393; 95% CI = 0.115–0.630; p = 0.002), and between raters of group 2 (with training; ICC = 0.433; 95% CI = 0.127–0.671; p = 0.002). The intra-rater analysis was excellent for raters 9 and 12, good to excellent for raters 8 and 10, moderate to excellent for rater 7, and poor to good for rater 11. Conclusion: The intra-rater reliability of the METRICS score was relatively good, while the inter-rater reliability was relatively low. This highlights the need for further efforts to achieve a common understanding of METRICS items, as well as resources consisting of explanations, elaborations, and examples to improve reproducibility and enhance their usability and robustness. Key Points: QuestionsGuidelines and scoring tools are necessary to improve the quality of radiomics research; however, the application of these tools is challenging for less experienced raters. FindingsIntra-rater reliability was high across all raters regardless of experience level or previous training, and inter-rater reliability was generally poor to moderate across raters. Clinical relevanceGuidelines and scoring tools are necessary for proper reporting in radiomics research and for closing the gap between research and clinical implementation. There is a need for further resources offering explanations, elaborations, and examples to enhance the usability and robustness of these guidelines. [ABSTRACT FROM AUTHOR]
Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 186373274
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Reproducibility of methodological radiomics score (METRICS): an intra- and inter-rater reliability study endorsed by EuSoMII.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Akinci+D'Antonoli%2C+Tugba%22">Akinci D'Antonoli, Tugba</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> tugba.akincidantonoli@unibas.ch</i><br /><searchLink fieldCode="AR" term="%22Cavallo%2C+Armando+Ugo%22">Cavallo, Armando Ugo</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Kocak%2C+Burak%22">Kocak, Burak</searchLink><relatesTo>3</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Borgheresi%2C+Alessandra%22">Borgheresi, Alessandra</searchLink><relatesTo>4,5</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Ponsiglione%2C+Andrea%22">Ponsiglione, Andrea</searchLink><relatesTo>6</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Stanzione%2C+Arnaldo%22">Stanzione, Arnaldo</searchLink><relatesTo>6</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Koltsakis%2C+Emmanouil%22">Koltsakis, Emmanouil</searchLink><relatesTo>7,8</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Doniselli%2C+Fabio+Martino%22">Doniselli, Fabio Martino</searchLink><relatesTo>9</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Vernuccio%2C+Federica%22">Vernuccio, Federica</searchLink><relatesTo>10</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Ugga%2C+Lorenzo%22">Ugga, Lorenzo</searchLink><relatesTo>6</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Triantafyllou%2C+Matthaios%22">Triantafyllou, Matthaios</searchLink><relatesTo>11</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Huisman%2C+Merel%22">Huisman, Merel</searchLink><relatesTo>12</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Klontzas%2C+Michail+E%2E%22">Klontzas, Michail E.</searchLink><relatesTo>8,13,14</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Trotta%2C+Romina%22">Trotta, Romina</searchLink><relatesTo>15</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Cannella%2C+Roberto%22">Cannella, Roberto</searchLink><relatesTo>10</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Fanni%2C+Salvatore+Claudio%22">Fanni, Salvatore Claudio</searchLink><relatesTo>16</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Cuocolo%2C+Renato%22">Cuocolo, Renato</searchLink><relatesTo>17</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22European+Radiology%22">European Radiology</searchLink>. Aug2025, Vol. 35 Issue 8, p4533-4545. 13p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Radiomics%22">Radiomics</searchLink><br /><searchLink fieldCode="DE" term="%22Inter-observer+reliability%22">Inter-observer reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+reliability%22">Statistical reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Standards%22">Standards</searchLink><br /><searchLink fieldCode="DE" term="%22Reproducible+research%22">Reproducible research</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objectives: To investigate the intra- and inter-rater reliability of the total methodological radiomics score (METRICS) and its items through a multi-reader analysis. Materials and methods: A total of 12 raters with different backgrounds and experience levels were recruited for the study. Based on their level of expertise, raters were randomly assigned to the following groups: two inter-rater reliability groups, and two intra-rater reliability groups, where each group included one group with and one group without a preliminary training session on the use of METRICS. Inter-rater reliability groups assessed all 34 papers, while intra-rater reliability groups completed the assessment of 17 papers twice within 21 days each time, and a "wash out" period of 60 days in between. Results: Inter-rater reliability was poor to moderate between raters of group 1 (without training; ICC = 0.393; 95% CI = 0.115–0.630; p = 0.002), and between raters of group 2 (with training; ICC = 0.433; 95% CI = 0.127–0.671; p = 0.002). The intra-rater analysis was excellent for raters 9 and 12, good to excellent for raters 8 and 10, moderate to excellent for rater 7, and poor to good for rater 11. Conclusion: The intra-rater reliability of the METRICS score was relatively good, while the inter-rater reliability was relatively low. This highlights the need for further efforts to achieve a common understanding of METRICS items, as well as resources consisting of explanations, elaborations, and examples to improve reproducibility and enhance their usability and robustness. Key Points: QuestionsGuidelines and scoring tools are necessary to improve the quality of radiomics research; however, the application of these tools is challenging for less experienced raters. FindingsIntra-rater reliability was high across all raters regardless of experience level or previous training, and inter-rater reliability was generally poor to moderate across raters. Clinical relevanceGuidelines and scoring tools are necessary for proper reporting in radiomics research and for closing the gap between research and clinical implementation. There is a need for further resources offering explanations, elaborations, and examples to enhance the usability and robustness of these guidelines. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=186373274
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s00330-025-11443-1
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 4533
    Subjects:
      – SubjectFull: Radiomics
        Type: general
      – SubjectFull: Inter-observer reliability
        Type: general
      – SubjectFull: Statistical reliability
        Type: general
      – SubjectFull: Standards
        Type: general
      – SubjectFull: Reproducible research
        Type: general
    Titles:
      – TitleFull: Reproducibility of methodological radiomics score (METRICS): an intra- and inter-rater reliability study endorsed by EuSoMII.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Akinci D'Antonoli, Tugba
      – PersonEntity:
          Name:
            NameFull: Cavallo, Armando Ugo
      – PersonEntity:
          Name:
            NameFull: Kocak, Burak
      – PersonEntity:
          Name:
            NameFull: Borgheresi, Alessandra
      – PersonEntity:
          Name:
            NameFull: Ponsiglione, Andrea
      – PersonEntity:
          Name:
            NameFull: Stanzione, Arnaldo
      – PersonEntity:
          Name:
            NameFull: Koltsakis, Emmanouil
      – PersonEntity:
          Name:
            NameFull: Doniselli, Fabio Martino
      – PersonEntity:
          Name:
            NameFull: Vernuccio, Federica
      – PersonEntity:
          Name:
            NameFull: Ugga, Lorenzo
      – PersonEntity:
          Name:
            NameFull: Triantafyllou, Matthaios
      – PersonEntity:
          Name:
            NameFull: Huisman, Merel
      – PersonEntity:
          Name:
            NameFull: Klontzas, Michail E.
      – PersonEntity:
          Name:
            NameFull: Trotta, Romina
      – PersonEntity:
          Name:
            NameFull: Cannella, Roberto
      – PersonEntity:
          Name:
            NameFull: Fanni, Salvatore Claudio
      – PersonEntity:
          Name:
            NameFull: Cuocolo, Renato
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Text: Aug2025
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 09387994
          Numbering:
            – Type: volume
              Value: 35
            – Type: issue
              Value: 8
          Titles:
            – TitleFull: European Radiology
              Type: main
ResultId 1