Evaluating Object Recognition Behavioral Consistency on Out-of-Distribution Stimuli

Towards understanding the human brain, neuroscientists have taken to reverse-engineering the brain using artificial neural networks (ANN). As vision models approach state of the art, neuroscientists are interested in evaluating their behavior consistency with human behavior. Geirhos et al found ANN are at chance on predicting human choices given out-of-distribution (OOD) data (data not used for training the model). We examined this claim through its choice of readout and error consistency metric.

We trained a logistic regression model on various vision models to predict the class from a OOD image. We found that this decoder readout reflects higher behavior consistency than reported in Geirhos et al, which used the Wordnet decoder. While there is weak behaviorial consistency, it is not “at chance”.