Passive Vital Sign Monitoring via Facial Vibrations Leveraging AR/VR Headsets

Document Type

Conference Proceeding

Publication Date

6-18-2023

Abstract

Vital signs (e.g., breathing and heart rates) and personal identities are essential information for personalized medicine and healthcare. The popularity of augmented reality/virtual reality (AR/VR) provides an excellent opportunity for enabling long-term health monitoring in a broad range of scenarios, including virtual entertainment, education, and telemedicine. However, commercial-off-the-shelf AR/VR devices do not have dedicated biosensors for providing vital signs and personal identities. In this work, we propose a novel framework that can generate fine-grained vital sign signals and other personalized health information of an AR/VR user through passive sensing on AR/VR devices. In particular, we find that the user's minute facial vibrations induced by breathing and heart beating can impact the readily available motion sensors on AR/VR headsets, which encode rich vital sign patterns and unique biometrics. The proposed framework further estimates the breathing and heartbeat rates, detects the gender and identity, and derives the body fat percentage of the user. To mitigate the impacts of body movement, we design an adaptive filtering scheme to cancel the spontaneous and non-spontaneous motion artifacts. We also develop unique facial vibration features and deep learning techniques to facilitate vital sign signal reconstruction and user identification. Extensive experiments demonstrate that our framework can achieve a low error of vital sign signal reconstruction and rate measurement, along with 95.51% and 93.33% accuracy on identity and gender recognition.

Identifier

85169450343 (Scopus)

ISBN

[9798400701108]

Publication Title

Mobisys 2023 Proceedings of the 21st Annual International Conference on Mobile Systems Applications and Services

External Full Text Location

https://doi.org/10.1145/3581791.3596848

First Page

96

Last Page

109

Grant

CNS2201465

Fund Ref

National Science Foundation

This document is currently not available here.

Share

COinS