A Day in the Life

RM 215

The goal of this research project is to create a model of the human visual system with anatomical and experiential constraints. The anatomical constraints implemented in a model include a foveated retina, the log-polar transform between the retina and V1, and the division between the central and peripheral pathways in the visual system (Wang & Cottrell, 2017). The experiential constraint consists of a realistic training set that models the human visual experience. Currently, the dataset most often used for training deep networks is ImageNet, a highly unrealistic dataset of 1.2M images of 1,000 categories (i.e., images of aquatic animals, herbivorous species, dog breeds, or common household objects). Thus, any network trained on any of these existing categories either becomes a dog, aquatic or herbivorous expert, which is only true of a small subset of the human population. Thus, the purpose of this "Day in the Life" project is to collect a more realistic dataset of what humans observe and fixate upon in daily life. Through the use of a wearable eye-tracker with an Intel Realsense scene camera that gives depth information, we are recording data from subjects as they go about their day. We then use a deep network to segment and label the objects that were fixated on. With this new data, our final goal is to develop a training set that is faithful to the distribution of what individuals actually look at in terms of frequency, dwell time, and distance. Training a visual system model with this data should result in representations that more closely mimic those developed in visual cortex. Moreover, this data will be useful in vision science, as frequency, probably the most important variable in psycholinguistics, has not typically been manipulated in human visual processing experiments for lack of norms. Here we report some initial results from this project.