Incorporating AI-assisted Sensing into the Metaverse: Opportunities for Interactions, eSports, and Security Enhancement

With the rapid growth and development of Virtual Reality (VR) and Augmented Reality (AR), extensive research has been carried out in the domain of the Metaverse, including immersive gaming, human-computer interaction, eSports, and the associated security & privacy concerns.

My research explores the potential of incorporating Artificial Intelligence (AI)-assisted sensing technologies to facilitate a more immersive, convenient, authentic, and secure virtual experience. This dissertation mainly focus on the following topics: (1) how to perform facial expression tracking to improve the users' awareness in the Metaverse; (2) fitness tracking for immersive eCycling; (3) running gait analysis for immersive indoor running, and (4) the potential privacy leakages through unrestricted sensors in the Metaverse.

We first propose a single-earpiece lightweight biosensing system, BioFace-3D, that can unobtrusively, continuously, and reliably sense the entire facial movements, track 2D facial landmarks, and further render 3D facial animations. In addition, to revolutionize the current state of eCycling and address a series of deficiencies related to the user’s virtual experience, we propose SmarCyPad, an innovative smart seat pad that can continuously and unobtrusively track five cycling-specific metrics. Moreover, we design a running gait analysis system by utilizing the acoustic sensors to reconstruct the runner's cadence, ground contact time, pressure distribution, and center of pressure. Finally, we conduct a comprehensive study to assess the trustworthiness of the embedded sensors on VR, which embed various forms of sensitive data that may put users’ privacy at risk. We validate this vulnerability through developing malware programs and malicious websites and specifically explore to what extent it exposes the user’s information in the context of keystroke snooping. By exploring these opportunities, this dissertation aim to facilitate a more immersive, convenient, authentic, and secure virtual experience in the Metaverse.