Protecting visual information in augmented reality from malicious application developers

Visual applications - those that use camera frames as part of the application - allows for a rich, context-aware experience. The continuing development of mixed and augmented reality (MR/AR) on head-mounted displays (HMDs) furthers the richness of this experience by providing users a continuous vision experience, where visual information continuously provides context, and the real world is augmented by the virtual. However, these visual applications raise serious privacy concerns because they can capture private user information. To understand user privacy concerns in continuous vision computing environments, we study three MR/AR applications (augmented markers, augmented faces, and text capture). We show that in modern mobile visual applications, typical users are exposed to potential mass collection of sensitive information. To address such deficiencies, we develop a framework that provides resource isolation between user information contained in camera frames and application access to the network. We implement the design as a proof of concept on the Android operating system and demonstrate its performance and usability with a modern state-of-the-art augmented reality library and several augmented reality applications. By comparing the applications from our case study with modified versions which better protect user privacy, results show that our design efficiently protects users against data collection in MR/AR applications with less than 0.7% performance overhead.