View in EDS PDF Full Text

A Rule-Based Approach for Automatically Extracting Data from Systematic Reviews and Their Updates to Model the Risk of Conclusion Change

Saved in:
Bibliographic Details
Title: A Rule-Based Approach for Automatically Extracting Data from Systematic Reviews and Their Updates to Model the Risk of Conclusion Change
Language: English
Authors: Bashir, Rabia (ORCID 0000-0002-9613-8957), Dunn, Adam G. (ORCID 0000-0002-1720-8209), Surian, Didi (ORCID 0000-0003-2299-2971)
Source: Research Synthesis Methods. Mar 2021 12(2):216-225.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 10
Publication Date: 2021
Document Type: Journal Articles
Reports - Research
Descriptors: Literature Reviews, Data, Automation, Statistical Analysis, Regression (Statistics), Accuracy, Classification, Prediction, Risk
DOI: 10.1002/jrsm.1473
ISSN: 1759-2879
Abstract: Few data-driven approaches are available to estimate the risk of conclusion change in systematic review updates. We developed a rule-based approach to automatically extract information from reviews and updates to be used as features for modelling conclusion change risk. Rules were developed to extract relevant information from published Cochrane reviews and used to construct four features: the number of included trials and participants in the reviews, a measure based on the number of participants, and the time elapsed between the search dates. We compared the performance of random forest, decision tree, and logistic regression to predict the conclusion change risk. The performance was measured by accuracy, precision, recall, F[subscript 1]-score, and area under ROC (AU-ROC). One rule was developed to extract the conclusion change information (96% accuracy, 100 reviews), one for the search date (100% accuracy, 100 reviews), one for the number of included clinical trials (100% accuracy, 100 reviews), and 22 for the number of participants (97.3% accuracy, 200 reviews). For unseen reviews, the random forest classifier showed the highest accuracy (80.8%) and AU-ROC (0.80). All classifiers showed relatively similar performance with overlapping 95% confidence interval (CI). The coverage score was shown to be the most useful feature for predicting the conclusion change risk. Features mined from Cochrane reviews and updates can estimate conclusion change risk. If data from more published reviews and updates were made accessible, data-driven methods to predict the conclusion change risk may be a feasible way to support decisions about updating reviews.
Abstractor: As Provided
Entry Date: 2021
Accession Number: EJ1288307
Database: ERIC
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFLJA_rdoeP3z4YHOOwCFhLAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDHODPPQwcveyL5NA5QIBEICBm6Lv8TVVSk-ibrmSVrbV1K571SJdhBqJ2oXdyznm0ZNre5MUfwaUjGKu8vV7uy8-lFNigOijkJCgqGyLUTWFNeeXv58a3cNMrZaA1-fjUdkEsYb6v_OJIQ5GEaWB0Kli2mscVR_QRkkkTCObYh9VjNd4eU9eIh5NH_VmOjRlQVuyUs3TpfH-XVIQEmp67g2ZpGSky_RQfmuayCBo
Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1288307
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Rule-Based Approach for Automatically Extracting Data from Systematic Reviews and Their Updates to Model the Risk of Conclusion Change
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Bashir%2C+Rabia%22">Bashir, Rabia</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-9613-8957">0000-0002-9613-8957</externalLink>)<br /><searchLink fieldCode="AR" term="%22Dunn%2C+Adam+G%2E%22">Dunn, Adam G.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-1720-8209">0000-0002-1720-8209</externalLink>)<br /><searchLink fieldCode="AR" term="%22Surian%2C+Didi%22">Surian, Didi</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2299-2971">0000-0003-2299-2971</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Synthesis+Methods%22"><i>Research Synthesis Methods</i></searchLink>. Mar 2021 12(2):216-225.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 10
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2021
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Literature+Reviews%22">Literature Reviews</searchLink><br /><searchLink fieldCode="DE" term="%22Data%22">Data</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Regression+%28Statistics%29%22">Regression (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Risk%22">Risk</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jrsm.1473
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1759-2879
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Few data-driven approaches are available to estimate the risk of conclusion change in systematic review updates. We developed a rule-based approach to automatically extract information from reviews and updates to be used as features for modelling conclusion change risk. Rules were developed to extract relevant information from published Cochrane reviews and used to construct four features: the number of included trials and participants in the reviews, a measure based on the number of participants, and the time elapsed between the search dates. We compared the performance of random forest, decision tree, and logistic regression to predict the conclusion change risk. The performance was measured by accuracy, precision, recall, F[subscript 1]-score, and area under ROC (AU-ROC). One rule was developed to extract the conclusion change information (96% accuracy, 100 reviews), one for the search date (100% accuracy, 100 reviews), one for the number of included clinical trials (100% accuracy, 100 reviews), and 22 for the number of participants (97.3% accuracy, 200 reviews). For unseen reviews, the random forest classifier showed the highest accuracy (80.8%) and AU-ROC (0.80). All classifiers showed relatively similar performance with overlapping 95% confidence interval (CI). The coverage score was shown to be the most useful feature for predicting the conclusion change risk. Features mined from Cochrane reviews and updates can estimate conclusion change risk. If data from more published reviews and updates were made accessible, data-driven methods to predict the conclusion change risk may be a feasible way to support decisions about updating reviews.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1288307
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1288307
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jrsm.1473
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 10
        StartPage: 216
    Subjects:
      – SubjectFull: Literature Reviews
        Type: general
      – SubjectFull: Data
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Regression (Statistics)
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Classification
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Risk
        Type: general
    Titles:
      – TitleFull: A Rule-Based Approach for Automatically Extracting Data from Systematic Reviews and Their Updates to Model the Risk of Conclusion Change
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Bashir, Rabia
      – PersonEntity:
          Name:
            NameFull: Dunn, Adam G.
      – PersonEntity:
          Name:
            NameFull: Surian, Didi
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-print
              Value: 1759-2879
          Numbering:
            – Type: volume
              Value: 12
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Research Synthesis Methods
              Type: main
ResultId 1