View in EDS HTML Full Text PDF Full Text

Fine-grained activity classification in assembly based on multi-visual modalities.

Saved in:
Bibliographic Details
Title: Fine-grained activity classification in assembly based on multi-visual modalities.
Authors: Chen, Haodong1 (AUTHOR) h.chen@mst.edu, Zendehdel, Niloofar1 (AUTHOR), Leu, Ming C.1 (AUTHOR), Yin, Zhaozheng2 (AUTHOR)
Source: Journal of Intelligent Manufacturing. Jun2024, Vol. 35 Issue 5, p2215-2233. 19p.
Subjects: Factory safety, Recurrent neural networks, Feature extraction, Classification, Quality control, Prediction models
Abstract: Assembly activity recognition and prediction help to improve productivity, quality control, and safety measures in smart factories. This study aims to sense, recognize, and predict a worker's continuous fine-grained assembly activities in a manufacturing platform. We propose a two-stage network for workers' fine-grained activity classification by leveraging scene-level and temporal-level activity features. The first stage is a feature awareness block that extracts scene-level features from multi-visual modalities, including red–green–blue (RGB) and hand skeleton frames. We use the transfer learning method in the first stage and compare three different pre-trained feature extraction models. Then, we transmit the feature information from the first stage to the second stage to learn the temporal-level features of activities. The second stage consists of the Recurrent Neural Network (RNN) layers and a final classifier. We compare the performance of two different RNNs in the second stage, including the Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU). The partial video observation method is used in the prediction of fine-grained activities. In the experiments using the trimmed activity videos, our model achieves an accuracy of > 99% on our dataset and > 98% on the public dataset UCF 101, outperforming the state-of-the-art models. The prediction model achieves an accuracy of > 97% in predicting activity labels using 50% of the onset activity video information. In the experiments using an untrimmed video with continuous assembly activities, we combine our recognition and prediction models and achieve an accuracy of > 91% in real time, surpassing the state-of-the-art models for the recognition of continuous assembly activities. [ABSTRACT FROM AUTHOR]
Copyright of Journal of Intelligent Manufacturing is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 177538579
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Fine-grained activity classification in assembly based on multi-visual modalities.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Chen%2C+Haodong%22">Chen, Haodong</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> h.chen@mst.edu</i><br /><searchLink fieldCode="AR" term="%22Zendehdel%2C+Niloofar%22">Zendehdel, Niloofar</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Leu%2C+Ming+C%2E%22">Leu, Ming C.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Yin%2C+Zhaozheng%22">Yin, Zhaozheng</searchLink><relatesTo>2</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Journal+of+Intelligent+Manufacturing%22">Journal of Intelligent Manufacturing</searchLink>. Jun2024, Vol. 35 Issue 5, p2215-2233. 19p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Factory+safety%22">Factory safety</searchLink><br /><searchLink fieldCode="DE" term="%22Recurrent+neural+networks%22">Recurrent neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Feature+extraction%22">Feature extraction</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink><br /><searchLink fieldCode="DE" term="%22Quality+control%22">Quality control</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction+models%22">Prediction models</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Assembly activity recognition and prediction help to improve productivity, quality control, and safety measures in smart factories. This study aims to sense, recognize, and predict a worker's continuous fine-grained assembly activities in a manufacturing platform. We propose a two-stage network for workers' fine-grained activity classification by leveraging scene-level and temporal-level activity features. The first stage is a feature awareness block that extracts scene-level features from multi-visual modalities, including red–green–blue (RGB) and hand skeleton frames. We use the transfer learning method in the first stage and compare three different pre-trained feature extraction models. Then, we transmit the feature information from the first stage to the second stage to learn the temporal-level features of activities. The second stage consists of the Recurrent Neural Network (RNN) layers and a final classifier. We compare the performance of two different RNNs in the second stage, including the Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU). The partial video observation method is used in the prediction of fine-grained activities. In the experiments using the trimmed activity videos, our model achieves an accuracy of > 99% on our dataset and > 98% on the public dataset UCF 101, outperforming the state-of-the-art models. The prediction model achieves an accuracy of > 97% in predicting activity labels using 50% of the onset activity video information. In the experiments using an untrimmed video with continuous assembly activities, we combine our recognition and prediction models and achieve an accuracy of > 91% in real time, surpassing the state-of-the-art models for the recognition of continuous assembly activities. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Journal of Intelligent Manufacturing is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=177538579
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10845-023-02152-x
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 19
        StartPage: 2215
    Subjects:
      – SubjectFull: Factory safety
        Type: general
      – SubjectFull: Recurrent neural networks
        Type: general
      – SubjectFull: Feature extraction
        Type: general
      – SubjectFull: Classification
        Type: general
      – SubjectFull: Quality control
        Type: general
      – SubjectFull: Prediction models
        Type: general
    Titles:
      – TitleFull: Fine-grained activity classification in assembly based on multi-visual modalities.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Chen, Haodong
      – PersonEntity:
          Name:
            NameFull: Zendehdel, Niloofar
      – PersonEntity:
          Name:
            NameFull: Leu, Ming C.
      – PersonEntity:
          Name:
            NameFull: Yin, Zhaozheng
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Text: Jun2024
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 09565515
          Numbering:
            – Type: volume
              Value: 35
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Journal of Intelligent Manufacturing
              Type: main
ResultId 1