Analysis of user preferences using photos and videos from mobile device based on object detection and neural networks

In this paper we focus on the problem of user preferences prediction using the gallery of his mobile device. We consider such categories of interests as interior items, food, transport and sport equipment. The novel two-phased method has been proposed. At the first stage, the facial regions are detected on all photos and videos, and the feature vectors are extracted using deep convolutional neural networks. These feature vectors are grouped using known agglomerative clustering techniques. Finally, we select public photos and videos which do not contain faces from the large clusters. At the second stage, these public images are processed on the remote server using high precision Faster R-CNN object detectors. Objects from other images (personal images) are detected on mobile device in offline mode using SSDLite and MobileNet. In the experimental study several neural network-based detectors have been trained using the united training sample from MS Coco, ImageNet and Open Images datasets. Their comparative analysis demonstrated that the Faster R-CNN-based models are characterized with 30 % higher recall when compared to the SSDLite detectors. However, the latter models process each image 3—9-times faster. Finally, we presented the experimental results of facial clustering with GFW (Grouping Faces in the Wild) dataset using either existing feature descriptors (VGGFace, VGGFace2) or the preliminarily trained MobileNet. The latter model with average link hierarchical clustering achieved the highest B-cubed F-measure.