Regional Dynamic Sign Language Recognition using Multimodal Conformer Architecture.
Saved in:
| Title: | Regional Dynamic Sign Language Recognition using Multimodal Conformer Architecture. |
|---|---|
| Authors: | Hugar, Gurusiddappa1 gurusidda.h@gmail.com, Kagalkar, Ramesh M.2 |
| Source: | Sakarya University Journal of Computer & Information Sciences (SAUCIS). Jun2026, Vol. 9 Issue 2, p436-450. 15p. |
| Subjects: | Sign language, Artificial neural networks, Convolutional neural networks, K-nearest neighbor classification |
| Abstract: | This paper proposes a multimodal Conformer architecture for dynamic sign language recognition in Kannada Sign Language (KSL). The model incorporates visual features extracted by EfficientNet-B0 together with 3D hand key points obtained from MediaPipe. The paper further proposes an Adaptive Confidence Correction (ACC) strategy, supported by k-NN classification, when the confidence scores for the hand key points of the signs are low and inconsistent. Finally, the dataset included 1,180 video samples covering 11 dynamic signs. Our tests demonstrate that the presented method achieves a state-of-the-art accuracy of 98.63% with a low inference time of 58ms while outperforming baselines including CNN-LSTM, Transformer, I3D, and SlowFast. Cross-validation tests and statistical analyses further support the robustness of the presented work. This work makes the following key contributions (1) the development of a newly collected in-house dataset Kannada Sign Language (KSL) dataset addressing the data scarcity of underrepresented regional sign languages, (2) the first adaptation of the Conformer architecture for sign language recognition, validated through ablation and cross-validation studies, and (3) a deployable, low-latency framework designed for mobile-edge integration and privacy-aware deployment. The dataset will be released upon acceptance and with a valid research request. By addressing accessibility challenges in human-computer interaction and offering a reproducible benchmark for regional sign-language technologies, this work supports future advances in cross-lingual generalization and low-resource optimization. [ABSTRACT FROM AUTHOR] |
| Copyright of Sakarya University Journal of Computer & Information Sciences (SAUCIS) is the property of Sakarya University Journal of Computer & Information Sciences (SAUCIS) and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 195616452 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Regional Dynamic Sign Language Recognition using Multimodal Conformer Architecture. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Hugar%2C+Gurusiddappa%22">Hugar, Gurusiddappa</searchLink><relatesTo>1</relatesTo><i> gurusidda.h@gmail.com</i><br /><searchLink fieldCode="AR" term="%22Kagalkar%2C+Ramesh+M%2E%22">Kagalkar, Ramesh M.</searchLink><relatesTo>2</relatesTo> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Sakarya+University+Journal+of+Computer+%26+Information+Sciences+%28SAUCIS%29%22">Sakarya University Journal of Computer & Information Sciences (SAUCIS)</searchLink>. Jun2026, Vol. 9 Issue 2, p436-450. 15p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Sign+language%22">Sign language</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+neural+networks%22">Artificial neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Convolutional+neural+networks%22">Convolutional neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22K-nearest+neighbor+classification%22">K-nearest neighbor classification</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: This paper proposes a multimodal Conformer architecture for dynamic sign language recognition in Kannada Sign Language (KSL). The model incorporates visual features extracted by EfficientNet-B0 together with 3D hand key points obtained from MediaPipe. The paper further proposes an Adaptive Confidence Correction (ACC) strategy, supported by k-NN classification, when the confidence scores for the hand key points of the signs are low and inconsistent. Finally, the dataset included 1,180 video samples covering 11 dynamic signs. Our tests demonstrate that the presented method achieves a state-of-the-art accuracy of 98.63% with a low inference time of 58ms while outperforming baselines including CNN-LSTM, Transformer, I3D, and SlowFast. Cross-validation tests and statistical analyses further support the robustness of the presented work. This work makes the following key contributions (1) the development of a newly collected in-house dataset Kannada Sign Language (KSL) dataset addressing the data scarcity of underrepresented regional sign languages, (2) the first adaptation of the Conformer architecture for sign language recognition, validated through ablation and cross-validation studies, and (3) a deployable, low-latency framework designed for mobile-edge integration and privacy-aware deployment. The dataset will be released upon acceptance and with a valid research request. By addressing accessibility challenges in human-computer interaction and offering a reproducible benchmark for regional sign-language technologies, this work supports future advances in cross-lingual generalization and low-resource optimization. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Sakarya University Journal of Computer & Information Sciences (SAUCIS) is the property of Sakarya University Journal of Computer & Information Sciences (SAUCIS) and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=195616452 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.35377/saucis...1747386 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 15 StartPage: 436 Subjects: – SubjectFull: Sign language Type: general – SubjectFull: Artificial neural networks Type: general – SubjectFull: Convolutional neural networks Type: general – SubjectFull: K-nearest neighbor classification Type: general Titles: – TitleFull: Regional Dynamic Sign Language Recognition using Multimodal Conformer Architecture. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Hugar, Gurusiddappa – PersonEntity: Name: NameFull: Kagalkar, Ramesh M. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 06 Text: Jun2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 26368129 Numbering: – Type: volume Value: 9 – Type: issue Value: 2 Titles: – TitleFull: Sakarya University Journal of Computer & Information Sciences (SAUCIS) Type: main |
| ResultId | 1 |
