Visual-semantic graph neural network with pose-position attentive learning for group activity recognition.
Saved in:
| Title: | Visual-semantic graph neural network with pose-position attentive learning for group activity recognition. |
|---|---|
| Authors: | Liu, Tianshan1 (AUTHOR) tianshan.liu@connect.polyu.hk, Zhao, Rui1 (AUTHOR), Lam, Kin-Man1 (AUTHOR), Kong, Jun2 (AUTHOR) |
| Source: | Neurocomputing. Jun2022, Vol. 491, p217-231. 15p. |
| Subjects: | Temporal lobe, Information measurement |
| Abstract: | • A pose-position attention strategy is proposed to update the bi-modal visual graph. • A linguistic-embedding-based semantic graph is presented to model label relations. • A semantic-preserving loss is designed to maintain the semantics consistency. • Both visual and semantic information are fused for group activity recognition. Video-based group activities typically contain interactive contexts among diverse visual modalities between multiple persons, and semantic relationships between individual actions. Nevertheless, majority of the existing methods for recognizing group activity either captures the relationships among different persons by utilizing a solely RGB modality or neglect to exploit the label hierarchies between individual actions and the group activity. To tackle these issues, we propose a visual-semantic graph neural network, with pose-position attentive learning (VSGNN-PAL), for group activity recognition. Specifically, we first extract the individual-level appearance and motion representations from RGB and optical-flow inputs, to build a bi-modal visual graph. Two attentive aggregators are further proposed to integrate both the pose and position information to measure the relevance scores between persons, and dynamically refine the representation of each visual node from both modality-specific and cross-modal perspectives. To model a semantic hierarchy from a label space, we construct a semantic graph based on the linguistic embeddings of individual actions and group activity labels. We further employ a bi-directional mapping learning scheme, to integrate the label-relation-aware semantic context into the visual representations. Besides, a global reasoning module is introduced to progressively generate the group-level representations with the scene description maintained. Furthermore, we formulate a semantic-preserving loss, to maintain the consistency between the learned high-level representations and the semantics of the ground-truth labels. Experimental results on three group activity benchmarks demonstrate that the proposed method achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR] |
| Copyright of Neurocomputing is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 156588613 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Visual-semantic graph neural network with pose-position attentive learning for group activity recognition. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Liu%2C+Tianshan%22">Liu, Tianshan</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> tianshan.liu@connect.polyu.hk</i><br /><searchLink fieldCode="AR" term="%22Zhao%2C+Rui%22">Zhao, Rui</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Lam%2C+Kin-Man%22">Lam, Kin-Man</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Kong%2C+Jun%22">Kong, Jun</searchLink><relatesTo>2</relatesTo> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Neurocomputing%22">Neurocomputing</searchLink>. Jun2022, Vol. 491, p217-231. 15p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Temporal+lobe%22">Temporal lobe</searchLink><br /><searchLink fieldCode="DE" term="%22Information+measurement%22">Information measurement</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: • A pose-position attention strategy is proposed to update the bi-modal visual graph. • A linguistic-embedding-based semantic graph is presented to model label relations. • A semantic-preserving loss is designed to maintain the semantics consistency. • Both visual and semantic information are fused for group activity recognition. Video-based group activities typically contain interactive contexts among diverse visual modalities between multiple persons, and semantic relationships between individual actions. Nevertheless, majority of the existing methods for recognizing group activity either captures the relationships among different persons by utilizing a solely RGB modality or neglect to exploit the label hierarchies between individual actions and the group activity. To tackle these issues, we propose a visual-semantic graph neural network, with pose-position attentive learning (VSGNN-PAL), for group activity recognition. Specifically, we first extract the individual-level appearance and motion representations from RGB and optical-flow inputs, to build a bi-modal visual graph. Two attentive aggregators are further proposed to integrate both the pose and position information to measure the relevance scores between persons, and dynamically refine the representation of each visual node from both modality-specific and cross-modal perspectives. To model a semantic hierarchy from a label space, we construct a semantic graph based on the linguistic embeddings of individual actions and group activity labels. We further employ a bi-directional mapping learning scheme, to integrate the label-relation-aware semantic context into the visual representations. Besides, a global reasoning module is introduced to progressively generate the group-level representations with the scene description maintained. Furthermore, we formulate a semantic-preserving loss, to maintain the consistency between the learned high-level representations and the semantics of the ground-truth labels. Experimental results on three group activity benchmarks demonstrate that the proposed method achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Neurocomputing is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=156588613 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1016/j.neucom.2022.03.066 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 15 StartPage: 217 Subjects: – SubjectFull: Temporal lobe Type: general – SubjectFull: Information measurement Type: general Titles: – TitleFull: Visual-semantic graph neural network with pose-position attentive learning for group activity recognition. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Liu, Tianshan – PersonEntity: Name: NameFull: Zhao, Rui – PersonEntity: Name: NameFull: Lam, Kin-Man – PersonEntity: Name: NameFull: Kong, Jun IsPartOfRelationships: – BibEntity: Dates: – D: 28 M: 06 Text: Jun2022 Type: published Y: 2022 Identifiers: – Type: issn-print Value: 09252312 Numbering: – Type: volume Value: 491 Titles: – TitleFull: Neurocomputing Type: main |
| ResultId | 1 |
