Adapting independent large-scale pretrained models for human action recognition.
Saved in:
| Title: | Adapting independent large-scale pretrained models for human action recognition. |
|---|---|
| Authors: | PEHLİVAN, SELEN1 selen.pehlivantort@vtt.fi |
| Source: | Turkish Journal of Electrical Engineering & Computer Sciences. 2026, Vol. 34 Issue 3, p437-452. 17p. |
| Subject Terms: | *Human activity recognition, *Machine learning |
| Abstract: | Transferring knowledge from large-scale, independently pretrained image and text models to video understanding requires addressing several challenges, including maintaining generalization capabilities of models, integrating them into multimodal architectures, and fine-tuning with temporal dynamics. This study evaluates the effectiveness of parameter-efficient fine-tuning (PEFT) techniques in transferring pretrained knowledge from two independent models for video action recognition within a simple, streamlined multimodal fusion pipeline. Specifically, we adapt CLIP as the text branch and DINOv2 as the image branch, keeping both backbones frozen to preserve their pretrained robustness, while introducing lightweight, task-specific modules to adapt and fuse the branches with temporal dynamics. A simple fusion transformer combines the image and text branches, enabling their efficient integration with minimal training cost. We systematically evaluate the framework on widely-recognized midscale video benchmark datasets, comparing prompt-based and adapter-based PEFT techniques across different data regimes. Our results demonstrate that this combination achieves competitive performance, highlights the transferability and scalability of independent pretrained models for a targeted task, and provides practical insights for adapting large models using midscale, task specific video datasets. In particular, adaptations of the DINOv2 image encoder and CLIP text encoder improve recognition accuracy over the frozen baseline up to an average absolute gains of 3.47% across K5--KAll. Moreover, the proposed DoRA DINOv2 combined with an adapter-based CLIP text encoder achieves competitive state-of-the-art performance on UCF101, HMDB51, and DIVING48, consistently outperforming prior methods in few-shot scenarios and reaching up to 82.0% accuracy with K2 training examples. [ABSTRACT FROM AUTHOR] |
| Database: | Energy & Power Source |
| FullText | Links: – Type: pdflink Text: Availability: 0 |
|---|---|
| Header | DbId: enr DbLabel: Energy & Power Source An: 194007412 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Adapting independent large-scale pretrained models for human action recognition. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22PEHLİVAN%2C+SELEN%22">PEHLİVAN, SELEN</searchLink><relatesTo>1</relatesTo><i> selen.pehlivantort@vtt.fi</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Turkish+Journal+of+Electrical+Engineering+%26+Computer+Sciences%22">Turkish Journal of Electrical Engineering & Computer Sciences</searchLink>. 2026, Vol. 34 Issue 3, p437-452. 17p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Human+activity+recognition%22">Human activity recognition</searchLink><br />*<searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Transferring knowledge from large-scale, independently pretrained image and text models to video understanding requires addressing several challenges, including maintaining generalization capabilities of models, integrating them into multimodal architectures, and fine-tuning with temporal dynamics. This study evaluates the effectiveness of parameter-efficient fine-tuning (PEFT) techniques in transferring pretrained knowledge from two independent models for video action recognition within a simple, streamlined multimodal fusion pipeline. Specifically, we adapt CLIP as the text branch and DINOv2 as the image branch, keeping both backbones frozen to preserve their pretrained robustness, while introducing lightweight, task-specific modules to adapt and fuse the branches with temporal dynamics. A simple fusion transformer combines the image and text branches, enabling their efficient integration with minimal training cost. We systematically evaluate the framework on widely-recognized midscale video benchmark datasets, comparing prompt-based and adapter-based PEFT techniques across different data regimes. Our results demonstrate that this combination achieves competitive performance, highlights the transferability and scalability of independent pretrained models for a targeted task, and provides practical insights for adapting large models using midscale, task specific video datasets. In particular, adaptations of the DINOv2 image encoder and CLIP text encoder improve recognition accuracy over the frozen baseline up to an average absolute gains of 3.47% across K5--KAll. Moreover, the proposed DoRA DINOv2 combined with an adapter-based CLIP text encoder achieves competitive state-of-the-art performance on UCF101, HMDB51, and DIVING48, consistently outperforming prior methods in few-shot scenarios and reaching up to 82.0% accuracy with K2 training examples. [ABSTRACT FROM AUTHOR] |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=enr&AN=194007412 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.55730/1300-0632.4184 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 17 StartPage: 437 Subjects: – SubjectFull: Human activity recognition Type: general – SubjectFull: Machine learning Type: general Titles: – TitleFull: Adapting independent large-scale pretrained models for human action recognition. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: PEHLİVAN, SELEN IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: 2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 13000632 Numbering: – Type: volume Value: 34 – Type: issue Value: 3 Titles: – TitleFull: Turkish Journal of Electrical Engineering & Computer Sciences Type: main |
| ResultId | 1 |
