View in EDS

Novel Techniques to Assess Predictive Systems and Reduce Their Alarm Burden.

Saved in:
Bibliographic Details
Title: Novel Techniques to Assess Predictive Systems and Reduce Their Alarm Burden.
Authors: Handler, Jonathan A.1 (AUTHOR) jhandler@gmail.com, Feied, Craig F.2 (AUTHOR) craig.feied@asatte.net, Gillam, Michael T.3 (AUTHOR) mike@healthlab.com
Source: IEEE Journal of Biomedical & Health Informatics. Oct2022, Vol. 26 Issue 10, p5267-5278. 12p.
Subjects: Failure mode & effects analysis, Machine learning, Utility functions, Cost effectiveness, Alarms, False positive error
Abstract: Machine prediction algorithms (e.g., binary classifiers) often are adopted on the basis of claimed performance using classic metrics such as precision and recall. However, classifier performance depends heavily upon the context (workflow) in which the classifier operates. Classic metrics do not reflect the realized performance of a predictor unless certain implicit assumptions are met, and these assumptions cannot be met in many common clinical scenarios. This often results in suboptimal implementations and in disappointment when expected outcomes are not achieved. One common failure mode for classic metrics arises when multiple predictions can be made for the same event, particularly when redundant true positive predictions produce little additional value. This describes many clinical alerting systems. We explain why classic metrics cannot correctly represent predictor performance in such contexts, and introduce an improved performance assessment technique using utility functions to score predictions based on their utility in a specific workflow context. The resulting utility metrics (u-metrics) explicitly account for the effects of temporal relationships and other sources of variability in prediction utility. Compared to traditional measures, u-metrics more accurately reflect the real-world costs and benefits of a predictor operating in a realized context. The improvement can be significant. We also describe a formal approach to snoozing, a mitigation strategy in which some predictions are suppressed to improve predictor performance by reducing false positives while retaining event capture. Snoozing is especially useful for predictors that generate interruptive alarms. U-metrics correctly measure and predict the performance benefits of snoozing, whereas traditional metrics do not. [ABSTRACT FROM AUTHOR]
Copyright of IEEE Journal of Biomedical & Health Informatics is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 160690401
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Novel Techniques to Assess Predictive Systems and Reduce Their Alarm Burden.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Handler%2C+Jonathan+A%2E%22">Handler, Jonathan A.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> jhandler@gmail.com</i><br /><searchLink fieldCode="AR" term="%22Feied%2C+Craig+F%2E%22">Feied, Craig F.</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> craig.feied@asatte.net</i><br /><searchLink fieldCode="AR" term="%22Gillam%2C+Michael+T%2E%22">Gillam, Michael T.</searchLink><relatesTo>3</relatesTo> (AUTHOR)<i> mike@healthlab.com</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22IEEE+Journal+of+Biomedical+%26+Health+Informatics%22">IEEE Journal of Biomedical & Health Informatics</searchLink>. Oct2022, Vol. 26 Issue 10, p5267-5278. 12p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Failure+mode+%26+effects+analysis%22">Failure mode & effects analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink><br /><searchLink fieldCode="DE" term="%22Utility+functions%22">Utility functions</searchLink><br /><searchLink fieldCode="DE" term="%22Cost+effectiveness%22">Cost effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Alarms%22">Alarms</searchLink><br /><searchLink fieldCode="DE" term="%22False+positive+error%22">False positive error</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Machine prediction algorithms (e.g., binary classifiers) often are adopted on the basis of claimed performance using classic metrics such as precision and recall. However, classifier performance depends heavily upon the context (workflow) in which the classifier operates. Classic metrics do not reflect the realized performance of a predictor unless certain implicit assumptions are met, and these assumptions cannot be met in many common clinical scenarios. This often results in suboptimal implementations and in disappointment when expected outcomes are not achieved. One common failure mode for classic metrics arises when multiple predictions can be made for the same event, particularly when redundant true positive predictions produce little additional value. This describes many clinical alerting systems. We explain why classic metrics cannot correctly represent predictor performance in such contexts, and introduce an improved performance assessment technique using utility functions to score predictions based on their utility in a specific workflow context. The resulting utility metrics (u-metrics) explicitly account for the effects of temporal relationships and other sources of variability in prediction utility. Compared to traditional measures, u-metrics more accurately reflect the real-world costs and benefits of a predictor operating in a realized context. The improvement can be significant. We also describe a formal approach to snoozing, a mitigation strategy in which some predictions are suppressed to improve predictor performance by reducing false positives while retaining event capture. Snoozing is especially useful for predictors that generate interruptive alarms. U-metrics correctly measure and predict the performance benefits of snoozing, whereas traditional metrics do not. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of IEEE Journal of Biomedical & Health Informatics is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=160690401
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1109/JBHI.2022.3189312
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 12
        StartPage: 5267
    Subjects:
      – SubjectFull: Failure mode & effects analysis
        Type: general
      – SubjectFull: Machine learning
        Type: general
      – SubjectFull: Utility functions
        Type: general
      – SubjectFull: Cost effectiveness
        Type: general
      – SubjectFull: Alarms
        Type: general
      – SubjectFull: False positive error
        Type: general
    Titles:
      – TitleFull: Novel Techniques to Assess Predictive Systems and Reduce Their Alarm Burden.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Handler, Jonathan A.
      – PersonEntity:
          Name:
            NameFull: Feied, Craig F.
      – PersonEntity:
          Name:
            NameFull: Gillam, Michael T.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: Oct2022
              Type: published
              Y: 2022
          Identifiers:
            – Type: issn-print
              Value: 21682194
          Numbering:
            – Type: volume
              Value: 26
            – Type: issue
              Value: 10
          Titles:
            – TitleFull: IEEE Journal of Biomedical & Health Informatics
              Type: main
ResultId 1