Understanding and Recognizing Bystanders in Images for Privacy Protection

Summary

Privacy violations of bystanders in photographs taken in public places is a long-studied problem. Proposed solutions rely on bystanders to be proactive and use tools and techniques to protect their privacy [1–9]. These tools require them to share sensitive information, such as location, facial features, and privacy pref- erences with other users and cloud servers, which are themselves violations of privacy. Here, we attempt to detect bystanders in images automatically using computer vision and machine learning. Upon detection, any privacy-preserving action could be taken (e.g., obfuscation), enforcing a privacy-by-default policy without placing the burden on the bystanders or sharing any sensitive informa- tion. We begin by understanding what rationales and concepts humans use to distin- guish between subject and bystander in images, since these concepts are nuanced and context-specific. In a study, we asked participants to label people in images as bystanders or subjects, provide justification for their labels, and rate each person for (presumably) relevant concepts, e.g., whether the person was posing for and comfortable being in the image, can be replaced by another random person and so on. Our correlation and regression analyses revealed significant association among these features and the most common reasons humans use to classify subject/bystander. Using factor analysis, we identified two underlying constructs humans use to identify bystanders: visual-appearance and importance of the person for the image. We experimented with several classification models for automatic detection. The best performing model (mean accuracy 85% for 10-fold cross-validation) is a two-step prediction pipeline based on our hypoth- esis on how humans do it. First, we predict the relevant concepts using features extracted from the images (such as body-pose [10], facial expression [11], and the location of a person) using regression models. These predicted values were then used to classify subject/bystander. Detailed study methodology and (additional) findings are presented in the poster.