View in EDS HTML Full Text PDF Full Text

A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction.

Saved in:
Bibliographic Details
Title: A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction.
Authors: Alamoodi, A. H.1,2,3 alamoodi.abdullah91@gmail.com, Zughoul, Omar4, David, Dianese5, Garfan, Salem5, Pamucar, Dragan6,7,8, Albahri, O. S.9,10, Albahri, A. S.11,12, Yussof, Salman1,13, Sharaf, Iman Mohamad14
Source: Journal of Medical Systems. 8/31/2024, Vol. 48 Issue 1, p1-12. 12p.
Subjects: Research funding, Clinical decision support systems, Natural language processing, Decision making in clinical medicine, Problem solving, Uncertainty, Logic, Conceptual structures, Mathematical models, Case studies, Theory, Medical practice, Evaluation
Geographic Terms: United States
Abstract: Artificial intelligence (AI) has become a crucial element of modern technology, especially in the healthcare sector, which is apparent given the continuous development of large language models (LLMs), which are utilized in various domains, including medical beings. However, when it comes to using these LLMs for the medical domain, there's a need for an evaluation platform to determine their suitability and drive future development efforts. Towards that end, this study aims to address this concern by developing a comprehensive Multi-Criteria Decision Making (MCDM) approach that is specifically designed to evaluate medical LLMs. The success of AI, particularly LLMs, in the healthcare domain, depends on their efficacy, safety, and ethical compliance. Therefore, it is essential to have a robust evaluation framework for their integration into medical contexts. This study proposes using the Fuzzy-Weighted Zero-InConsistency (FWZIC) method extended to p, q-quasirung orthopair fuzzy set (p, q-QROFS) for weighing evaluation criteria. This extension enables the handling of uncertainties inherent in medical decision-making processes. The approach accommodates the imprecise and multifaceted nature of real-world medical data and criteria by incorporating fuzzy logic principles. The MultiAtributive Ideal-Real Comparative Analysis (MAIRCA) method is employed for the assessment of medical LLMs utilized in the case study of this research. The results of this research revealed that "Medical Relation Extraction" criteria with its sub-levels had more importance with (0.504) than "Clinical Concept Extraction" with (0.495). For the LLMs evaluated, out of 6 alternatives, ( A 4 ) "GatorTron S 10B" had the 1st rank as compared to ( A 1 ) "GatorTron 90B" had the 6th rank. The implications of this study extend beyond academic discourse, directly impacting healthcare practices and patient outcomes. The proposed framework can help healthcare professionals make more informed decisions regarding the adoption and utilization of LLMs in medical settings. [ABSTRACT FROM AUTHOR]
Copyright of Journal of Medical Systems is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 179604257
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Alamoodi%2C+A%2E+H%2E%22">Alamoodi, A. H.</searchLink><relatesTo>1,2,3</relatesTo><i> alamoodi.abdullah91@gmail.com</i><br /><searchLink fieldCode="AR" term="%22Zughoul%2C+Omar%22">Zughoul, Omar</searchLink><relatesTo>4</relatesTo><br /><searchLink fieldCode="AR" term="%22David%2C+Dianese%22">David, Dianese</searchLink><relatesTo>5</relatesTo><br /><searchLink fieldCode="AR" term="%22Garfan%2C+Salem%22">Garfan, Salem</searchLink><relatesTo>5</relatesTo><br /><searchLink fieldCode="AR" term="%22Pamucar%2C+Dragan%22">Pamucar, Dragan</searchLink><relatesTo>6,7,8</relatesTo><br /><searchLink fieldCode="AR" term="%22Albahri%2C+O%2E+S%2E%22">Albahri, O. S.</searchLink><relatesTo>9,10</relatesTo><br /><searchLink fieldCode="AR" term="%22Albahri%2C+A%2E+S%2E%22">Albahri, A. S.</searchLink><relatesTo>11,12</relatesTo><br /><searchLink fieldCode="AR" term="%22Yussof%2C+Salman%22">Yussof, Salman</searchLink><relatesTo>1,13</relatesTo><br /><searchLink fieldCode="AR" term="%22Sharaf%2C+Iman+Mohamad%22">Sharaf, Iman Mohamad</searchLink><relatesTo>14</relatesTo>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Journal+of+Medical+Systems%22">Journal of Medical Systems</searchLink>. 8/31/2024, Vol. 48 Issue 1, p1-12. 12p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Research+funding%22">Research funding</searchLink><br /><searchLink fieldCode="DE" term="%22Clinical+decision+support+systems%22">Clinical decision support systems</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+language+processing%22">Natural language processing</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+making+in+clinical+medicine%22">Decision making in clinical medicine</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+solving%22">Problem solving</searchLink><br /><searchLink fieldCode="DE" term="%22Uncertainty%22">Uncertainty</searchLink><br /><searchLink fieldCode="DE" term="%22Logic%22">Logic</searchLink><br /><searchLink fieldCode="DE" term="%22Conceptual+structures%22">Conceptual structures</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+models%22">Mathematical models</searchLink><br /><searchLink fieldCode="DE" term="%22Case+studies%22">Case studies</searchLink><br /><searchLink fieldCode="DE" term="%22Theory%22">Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Medical+practice%22">Medical practice</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation%22">Evaluation</searchLink>
– Name: SubjectGeographic
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22United+States%22">United States</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Artificial intelligence (AI) has become a crucial element of modern technology, especially in the healthcare sector, which is apparent given the continuous development of large language models (LLMs), which are utilized in various domains, including medical beings. However, when it comes to using these LLMs for the medical domain, there's a need for an evaluation platform to determine their suitability and drive future development efforts. Towards that end, this study aims to address this concern by developing a comprehensive Multi-Criteria Decision Making (MCDM) approach that is specifically designed to evaluate medical LLMs. The success of AI, particularly LLMs, in the healthcare domain, depends on their efficacy, safety, and ethical compliance. Therefore, it is essential to have a robust evaluation framework for their integration into medical contexts. This study proposes using the Fuzzy-Weighted Zero-InConsistency (FWZIC) method extended to p, q-quasirung orthopair fuzzy set (p, q-QROFS) for weighing evaluation criteria. This extension enables the handling of uncertainties inherent in medical decision-making processes. The approach accommodates the imprecise and multifaceted nature of real-world medical data and criteria by incorporating fuzzy logic principles. The MultiAtributive Ideal-Real Comparative Analysis (MAIRCA) method is employed for the assessment of medical LLMs utilized in the case study of this research. The results of this research revealed that "Medical Relation Extraction" criteria with its sub-levels had more importance with (0.504) than "Clinical Concept Extraction" with (0.495). For the LLMs evaluated, out of 6 alternatives, ( A 4 ) "GatorTron S 10B" had the 1st rank as compared to ( A 1 ) "GatorTron 90B" had the 6th rank. The implications of this study extend beyond academic discourse, directly impacting healthcare practices and patient outcomes. The proposed framework can help healthcare professionals make more informed decisions regarding the adoption and utilization of LLMs in medical settings. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Journal of Medical Systems is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=179604257
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10916-024-02090-y
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 12
        StartPage: 1
    Subjects:
      – SubjectFull: Research funding
        Type: general
      – SubjectFull: Clinical decision support systems
        Type: general
      – SubjectFull: Natural language processing
        Type: general
      – SubjectFull: Decision making in clinical medicine
        Type: general
      – SubjectFull: Problem solving
        Type: general
      – SubjectFull: Uncertainty
        Type: general
      – SubjectFull: Logic
        Type: general
      – SubjectFull: Conceptual structures
        Type: general
      – SubjectFull: Mathematical models
        Type: general
      – SubjectFull: Case studies
        Type: general
      – SubjectFull: Theory
        Type: general
      – SubjectFull: Medical practice
        Type: general
      – SubjectFull: Evaluation
        Type: general
      – SubjectFull: United States
        Type: general
    Titles:
      – TitleFull: A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Alamoodi, A. H.
      – PersonEntity:
          Name:
            NameFull: Zughoul, Omar
      – PersonEntity:
          Name:
            NameFull: David, Dianese
      – PersonEntity:
          Name:
            NameFull: Garfan, Salem
      – PersonEntity:
          Name:
            NameFull: Pamucar, Dragan
      – PersonEntity:
          Name:
            NameFull: Albahri, O. S.
      – PersonEntity:
          Name:
            NameFull: Albahri, A. S.
      – PersonEntity:
          Name:
            NameFull: Yussof, Salman
      – PersonEntity:
          Name:
            NameFull: Sharaf, Iman Mohamad
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 31
              M: 08
              Text: 8/31/2024
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 01485598
          Numbering:
            – Type: volume
              Value: 48
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Medical Systems
              Type: main
ResultId 1