Date of Award

Summer 2022

Document Type

Thesis

Supervisor

Dr. Vladimir Milosavljevic

Abstract

This study has proposed an alternative to promote the learning and enhancement of Lámh language for communication partners that support current users by creating a real time detection tool to recognise 20 chosen Lámh signs based on existing studies in the field. This implementation was carried out by generating primary data composed by MediaPipe landmark numpy arrays of 40 frames and 45 repetitions per sign. The Neural Networks were built using the Python library Keras and the applied SVM models were built with the library sklearn. The real time detection was carried out by integrating the mentioned elements with the library OpenCV. Neural Networks with different architectures with Long Short-Term Memory (LSTM) and 1D Convolutional Neural Network (CNN) were compared with SVM classifications applied with cross-validations to achieve the optimal hyperparameters in order to determine the most appropriate model.

The final chosen model after the assessment of the training and testing accuracy and loss was the two 1-D CNN layers with 32 and 64 nodes respectively, a dropout of 0.2 followed by two LSTM layers with 32 and 64 nodes respectively and a dense layer of 32 nodes. The training accuracy was 99.86%, the testing accuracy was 93.33%, the training loss was 0.0035 and the testing loss was 0.1791. This was the model which performed better in a real-time detection environment, easily detecting 8 different Lámh signs and detecting other 6 with reservations.

For future work, some skeletal motion signs should be captured again and other data augmentation strategies should be adopted, like capturing hips and legs landmarks alongside the signs and explore the augmentation of the data by promoting offset measures of the landmark coordinates of the skeletons captured by MediaPipe. Once the corrections of the methodology achieve better real time results, works toward tool accessibility and user experience should be investigated in order to generate a Lámh language real-time detection tool that could potentially promote Lámh and become a learning alternative for communication partners.

Comments

MSc in Data Analytics

Included in

Education Commons

Share

COinS