Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks.
Saved in:
| Title: | Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. |
|---|---|
| Authors: | Al Hajj, Hassan1, Lamard, Mathieu1,2, Conze, Pierre-Henri1,3, Cochener, Béatrice1,2,4, Quellec, Gwenolé1 gwenole.quellec@inserm.fr |
| Source: | Medical Image Analysis. Jul2018, Vol. 47, p203-218. 16p. |
| Subjects: | Video recording in medicine, Recurrent neural networks, Decision support systems, Cholecystectomy, Digestive organ surgery |
| Abstract: | This paper investigates the automatic monitoring of tool usage during a surgery, with potential applications in report generation, surgical training and real-time decision support. Two surgeries are considered: cataract surgery, the most common surgical procedure, and cholecystectomy, one of the most common digestive surgeries. Tool usage is monitored in videos recorded either through a microscope (cataract surgery) or an endoscope (cholecystectomy). Following state-of-the-art video analysis solutions, each frame of the video is analyzed by convolutional neural networks (CNNs) whose outputs are fed to recurrent neural networks (RNNs) in order to take temporal relationships between events into account. Novelty lies in the way those CNNs and RNNs are trained. Computational complexity prevents the end-to-end training of “CNN+RNN” systems. Therefore, CNNs are usually trained first, independently from the RNNs. This approach is clearly suboptimal for surgical tool analysis: many tools are very similar to one another, but they can generally be differentiated based on past events. CNNs should be trained to extract the most useful visual features in combination with the temporal context. A novel boosting strategy is proposed to achieve this goal: the CNN and RNN parts of the system are simultaneously enriched by progressively adding weak classifiers (either CNNs or RNNs) trained to improve the overall classification accuracy. Experiments were performed in a dataset of 50 cataract surgery videos, where the usage of 21 surgical tools was manually annotated, and a dataset of 80 cholecystectomy videos, where the usage of 7 tools was manually annotated. Very good classification performance are achieved in both datasets: tool usage could be labeled with an average area under the ROC curve of A z = 0.9961 and A z = 0.9939 , respectively, in offline mode (using past, present and future information), and A z = 0.9957 and A z = 0.9936 , respectively, in online mode (using past and present information only). [ABSTRACT FROM AUTHOR] |
| Copyright of Medical Image Analysis is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 129908645 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Al+Hajj%2C+Hassan%22">Al Hajj, Hassan</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Lamard%2C+Mathieu%22">Lamard, Mathieu</searchLink><relatesTo>1,2</relatesTo><br /><searchLink fieldCode="AR" term="%22Conze%2C+Pierre-Henri%22">Conze, Pierre-Henri</searchLink><relatesTo>1,3</relatesTo><br /><searchLink fieldCode="AR" term="%22Cochener%2C+Béatrice%22">Cochener, Béatrice</searchLink><relatesTo>1,2,4</relatesTo><br /><searchLink fieldCode="AR" term="%22Quellec%2C+Gwenolé%22">Quellec, Gwenolé</searchLink><relatesTo>1</relatesTo><i> gwenole.quellec@inserm.fr</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Medical+Image+Analysis%22">Medical Image Analysis</searchLink>. Jul2018, Vol. 47, p203-218. 16p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Video+recording+in+medicine%22">Video recording in medicine</searchLink><br /><searchLink fieldCode="DE" term="%22Recurrent+neural+networks%22">Recurrent neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+support+systems%22">Decision support systems</searchLink><br /><searchLink fieldCode="DE" term="%22Cholecystectomy%22">Cholecystectomy</searchLink><br /><searchLink fieldCode="DE" term="%22Digestive+organ+surgery%22">Digestive organ surgery</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: This paper investigates the automatic monitoring of tool usage during a surgery, with potential applications in report generation, surgical training and real-time decision support. Two surgeries are considered: cataract surgery, the most common surgical procedure, and cholecystectomy, one of the most common digestive surgeries. Tool usage is monitored in videos recorded either through a microscope (cataract surgery) or an endoscope (cholecystectomy). Following state-of-the-art video analysis solutions, each frame of the video is analyzed by convolutional neural networks (CNNs) whose outputs are fed to recurrent neural networks (RNNs) in order to take temporal relationships between events into account. Novelty lies in the way those CNNs and RNNs are trained. Computational complexity prevents the end-to-end training of “CNN+RNN” systems. Therefore, CNNs are usually trained first, independently from the RNNs. This approach is clearly suboptimal for surgical tool analysis: many tools are very similar to one another, but they can generally be differentiated based on past events. CNNs should be trained to extract the most useful visual features in combination with the temporal context. A novel boosting strategy is proposed to achieve this goal: the CNN and RNN parts of the system are simultaneously enriched by progressively adding weak classifiers (either CNNs or RNNs) trained to improve the overall classification accuracy. Experiments were performed in a dataset of 50 cataract surgery videos, where the usage of 21 surgical tools was manually annotated, and a dataset of 80 cholecystectomy videos, where the usage of 7 tools was manually annotated. Very good classification performance are achieved in both datasets: tool usage could be labeled with an average area under the ROC curve of A z = 0.9961 and A z = 0.9939 , respectively, in offline mode (using past, present and future information), and A z = 0.9957 and A z = 0.9936 , respectively, in online mode (using past and present information only). [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Medical Image Analysis is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=129908645 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1016/j.media.2018.05.001 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 16 StartPage: 203 Subjects: – SubjectFull: Video recording in medicine Type: general – SubjectFull: Recurrent neural networks Type: general – SubjectFull: Decision support systems Type: general – SubjectFull: Cholecystectomy Type: general – SubjectFull: Digestive organ surgery Type: general Titles: – TitleFull: Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Al Hajj, Hassan – PersonEntity: Name: NameFull: Lamard, Mathieu – PersonEntity: Name: NameFull: Conze, Pierre-Henri – PersonEntity: Name: NameFull: Cochener, Béatrice – PersonEntity: Name: NameFull: Quellec, Gwenolé IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 07 Text: Jul2018 Type: published Y: 2018 Identifiers: – Type: issn-print Value: 13618415 Numbering: – Type: volume Value: 47 Titles: – TitleFull: Medical Image Analysis Type: main |
| ResultId | 1 |
