ABSTRACT: SignConnect is a browser-based application that turns American Sign Language (ASL) alphabet signs and selected gestures into text and speech. MediaPipe extracts hand landmarks, a dense neural network recognizes static alphabet poses, and a Long Short-Term Memory (LSTM) network handles gesture sequences. The dense model takes 63 hand-landmark features, and the LSTM takes 30 frames of 149 features each. Besides HELLO, THANK YOU, YES, NO, PLEASE, SORRY, HELP, I LOVE YOU, GOOD and BYE, the vocabulary includes J, Z, a SPACE control gesture and a NONE background class. The alphabet model was trained on prepared data supplemented with webcam samples, while the motion model was trained on selfrecorded sequences. TensorFlow.js runs inference in the browser. Recognized outputs build up an editable text that comes with predefined suggestions and speech synthesis. The alphabet classifier reached 97.14% accuracy on 454 original-data test samples, and the LSTM reached 87.14% accuracy on 70 development clips across 14 classes. These results support a limited-vocabulary prototype, but evaluation with unseen signers and varied recording conditions is still needed.
KEYWORDS: American Sign Language, dense neural network, LSTM, gesture recognition, MediaPipe, TensorFlow.js, speech synthesis.
SIGNCONNECT: CONVERTING ASL INTO TEXT AND SPEECH
KINJAL SUNIL DAKWE, SHRAWNI MILIND CHAVAN, MS. SEEMA MURKAR