View in EDS

Regional Dynamic Sign Language Recognition using Multimodal Conformer Architecture.

Saved in:
Bibliographic Details
Title: Regional Dynamic Sign Language Recognition using Multimodal Conformer Architecture.
Authors: Hugar, Gurusiddappa1 gurusidda.h@gmail.com, Kagalkar, Ramesh M.2
Source: Sakarya University Journal of Computer & Information Sciences (SAUCIS). Jun2026, Vol. 9 Issue 2, p436-450. 15p.
Subjects: Sign language, Artificial neural networks, Convolutional neural networks, K-nearest neighbor classification
Abstract: This paper proposes a multimodal Conformer architecture for dynamic sign language recognition in Kannada Sign Language (KSL). The model incorporates visual features extracted by EfficientNet-B0 together with 3D hand key points obtained from MediaPipe. The paper further proposes an Adaptive Confidence Correction (ACC) strategy, supported by k-NN classification, when the confidence scores for the hand key points of the signs are low and inconsistent. Finally, the dataset included 1,180 video samples covering 11 dynamic signs. Our tests demonstrate that the presented method achieves a state-of-the-art accuracy of 98.63% with a low inference time of 58ms while outperforming baselines including CNN-LSTM, Transformer, I3D, and SlowFast. Cross-validation tests and statistical analyses further support the robustness of the presented work. This work makes the following key contributions (1) the development of a newly collected in-house dataset Kannada Sign Language (KSL) dataset addressing the data scarcity of underrepresented regional sign languages, (2) the first adaptation of the Conformer architecture for sign language recognition, validated through ablation and cross-validation studies, and (3) a deployable, low-latency framework designed for mobile-edge integration and privacy-aware deployment. The dataset will be released upon acceptance and with a valid research request. By addressing accessibility challenges in human-computer interaction and offering a reproducible benchmark for regional sign-language technologies, this work supports future advances in cross-lingual generalization and low-resource optimization. [ABSTRACT FROM AUTHOR]
Copyright of Sakarya University Journal of Computer & Information Sciences (SAUCIS) is the property of Sakarya University Journal of Computer & Information Sciences (SAUCIS) and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Description
Abstract:This paper proposes a multimodal Conformer architecture for dynamic sign language recognition in Kannada Sign Language (KSL). The model incorporates visual features extracted by EfficientNet-B0 together with 3D hand key points obtained from MediaPipe. The paper further proposes an Adaptive Confidence Correction (ACC) strategy, supported by k-NN classification, when the confidence scores for the hand key points of the signs are low and inconsistent. Finally, the dataset included 1,180 video samples covering 11 dynamic signs. Our tests demonstrate that the presented method achieves a state-of-the-art accuracy of 98.63% with a low inference time of 58ms while outperforming baselines including CNN-LSTM, Transformer, I3D, and SlowFast. Cross-validation tests and statistical analyses further support the robustness of the presented work. This work makes the following key contributions (1) the development of a newly collected in-house dataset Kannada Sign Language (KSL) dataset addressing the data scarcity of underrepresented regional sign languages, (2) the first adaptation of the Conformer architecture for sign language recognition, validated through ablation and cross-validation studies, and (3) a deployable, low-latency framework designed for mobile-edge integration and privacy-aware deployment. The dataset will be released upon acceptance and with a valid research request. By addressing accessibility challenges in human-computer interaction and offering a reproducible benchmark for regional sign-language technologies, this work supports future advances in cross-lingual generalization and low-resource optimization. [ABSTRACT FROM AUTHOR]
ISSN:26368129
DOI:10.35377/saucis...1747386