View in EDS

A randomized prospective study of a hybrid rule- and data-driven virtual patient.

Saved in:
Bibliographic Details
Title: A randomized prospective study of a hybrid rule- and data-driven virtual patient.
Authors: Stiff, Adam1 stiff.4@osu.edu, White, Michael2, Fosler-Lussier, Eric1, Lifeng Jin2, Jaffe, Evan2, Danforth, Douglas3
Source: Natural Language Engineering. Jan2024, Vol. 30 Issue 1, p31-72. 42p.
Subjects: Simulated patients, Convolutional neural networks, Longitudinal method, Hybrid systems, Machine learning, Logistic regression analysis
Abstract: Randomized prospective studies represent the gold standard for experimental design. In this paper, we present a randomized prospective study to validate the benefits of combining rule-based and data-driven natural language understandingmethods in a virtual patient dialogue system. The system uses a rule-based pattern matching approach together with a machine learning (ML) approach in the form of a text-based convolutional neural network, combining the two methods with a simple logistic regression model to choose between their predictions for each dialogue turn. In an earlier, retrospective study, the hybrid system yielded a nearly 50% error reduction on our initial data, in part due to the differential performance between the two methods as a function of label frequency. Given these gains, and considering that our hybrid approach is unique among virtual patient systems, we compare the hybrid system to the rule-based system by itself in a randomized prospective study. We evaluate 110 unique medical student subjects interacting with the system over 5,296 conversation turns, to verify whether similar gains are observed in a deployed system. This prospective study broadly confirms the findings from the earlier one but also highlights important deficits in our training data. The hybrid approach still improves over either rule-based or ML approaches individually, even handling unseen classes with some success. However, we observe that live subjects ask more out-of-scope questions than expected. To better handle such questions, we investigate several modifications to the system combination component. These show significant overall accuracy improvements and modest F1 improvements on out-of-scope queries in an offline evaluation. We provide further analysis to characterize the difficulty of the out-of-scope problem that we have identified, as well as to suggest future improvements over the baseline we establish here. [ABSTRACT FROM AUTHOR]
Copyright of Natural Language Engineering is the property of Cambridge University Press and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 176401420
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A randomized prospective study of a hybrid rule- and data-driven virtual patient.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Stiff%2C+Adam%22">Stiff, Adam</searchLink><relatesTo>1</relatesTo><i> stiff.4@osu.edu</i><br /><searchLink fieldCode="AR" term="%22White%2C+Michael%22">White, Michael</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Fosler-Lussier%2C+Eric%22">Fosler-Lussier, Eric</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Lifeng+Jin%22">Lifeng Jin</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Jaffe%2C+Evan%22">Jaffe, Evan</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Danforth%2C+Douglas%22">Danforth, Douglas</searchLink><relatesTo>3</relatesTo>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Natural+Language+Engineering%22">Natural Language Engineering</searchLink>. Jan2024, Vol. 30 Issue 1, p31-72. 42p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Simulated+patients%22">Simulated patients</searchLink><br /><searchLink fieldCode="DE" term="%22Convolutional+neural+networks%22">Convolutional neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Longitudinal+method%22">Longitudinal method</searchLink><br /><searchLink fieldCode="DE" term="%22Hybrid+systems%22">Hybrid systems</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink><br /><searchLink fieldCode="DE" term="%22Logistic+regression+analysis%22">Logistic regression analysis</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Randomized prospective studies represent the gold standard for experimental design. In this paper, we present a randomized prospective study to validate the benefits of combining rule-based and data-driven natural language understandingmethods in a virtual patient dialogue system. The system uses a rule-based pattern matching approach together with a machine learning (ML) approach in the form of a text-based convolutional neural network, combining the two methods with a simple logistic regression model to choose between their predictions for each dialogue turn. In an earlier, retrospective study, the hybrid system yielded a nearly 50% error reduction on our initial data, in part due to the differential performance between the two methods as a function of label frequency. Given these gains, and considering that our hybrid approach is unique among virtual patient systems, we compare the hybrid system to the rule-based system by itself in a randomized prospective study. We evaluate 110 unique medical student subjects interacting with the system over 5,296 conversation turns, to verify whether similar gains are observed in a deployed system. This prospective study broadly confirms the findings from the earlier one but also highlights important deficits in our training data. The hybrid approach still improves over either rule-based or ML approaches individually, even handling unseen classes with some success. However, we observe that live subjects ask more out-of-scope questions than expected. To better handle such questions, we investigate several modifications to the system combination component. These show significant overall accuracy improvements and modest F1 improvements on out-of-scope queries in an offline evaluation. We provide further analysis to characterize the difficulty of the out-of-scope problem that we have identified, as well as to suggest future improvements over the baseline we establish here. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Natural Language Engineering is the property of Cambridge University Press and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=176401420
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1017/S1351324922000420
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 42
        StartPage: 31
    Subjects:
      – SubjectFull: Simulated patients
        Type: general
      – SubjectFull: Convolutional neural networks
        Type: general
      – SubjectFull: Longitudinal method
        Type: general
      – SubjectFull: Hybrid systems
        Type: general
      – SubjectFull: Machine learning
        Type: general
      – SubjectFull: Logistic regression analysis
        Type: general
    Titles:
      – TitleFull: A randomized prospective study of a hybrid rule- and data-driven virtual patient.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Stiff, Adam
      – PersonEntity:
          Name:
            NameFull: White, Michael
      – PersonEntity:
          Name:
            NameFull: Fosler-Lussier, Eric
      – PersonEntity:
          Name:
            NameFull: Lifeng Jin
      – PersonEntity:
          Name:
            NameFull: Jaffe, Evan
      – PersonEntity:
          Name:
            NameFull: Danforth, Douglas
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Text: Jan2024
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 13513249
          Numbering:
            – Type: volume
              Value: 30
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Natural Language Engineering
              Type: main
ResultId 1