Document Type
Thesis
Date of Award
5-31-2026
Degree Name
Master of Science in Artificial Intelligence - (M.S.)
Department
Data Science
First Advisor
Huiran Jin
Second Advisor
Lijing Wang
Third Advisor
Alisha Pradhan
Abstract
A real time multimodal smart home control system deployed on a Raspberry Pi 5 is presented. The system combines hand gestures, short voice cues, and proximity aware interaction to execute household commands such as light brightness control, fan speed adjustment, and stop or kill switch actions. Lightweight gesture and keyword spotting voice classifiers were trained offline and exported to TensorFlow Lite for efficient on device inference. For more natural spoken phrases, the system additionally integrates a locally deployed pretrained offline ASR component rather than a speech recognizer trained from scratch. Using a USB camera and microphone, the system operates fully on the edge and translates accepted commands into GPIO(General Purpose Input/Output) based actuator control.
To improve robustness under changing real-world conditions, the decision layer uses adaptive fusion that begins with balanced weighting and shifts trust toward the modality that appears more reliable at runtime. Reliability is estimated using confidence and quality measures, including an audio signal to noise ratio-based score for voice and an image sharpness measure derived from Laplacian variance for gesture input. A physical feedback button provides live supervision to adjust fusion bias over time. A hybrid voice pathway also combines keyword spotting for fast short commands with an offline ASR fallback for more natural spoken phrases.
Evaluations under multiple conditions showed that the keyword spotting based multimodal pathway remained more stable across environments, while offline ASR performed very well in quiet settings but degraded significantly under noise.
Recommended Citation
Bora, Vaibhav, "Adaptive multimodal smart home control on a raspberry pi 5 using hand gestures, voice cues, and user feedback" (2026). Theses. 3549.
https://digitalcommons.njit.edu/theses/3549
