Material classification by man and machine: Understanding human perception using psychophysics and convolutional neural networks

We are surrounded by various materials (e.g., metal, cloth, plastic or glass), which we easily identify from sight alone. Importantly, the particular material impression typically does not collapse when the material is viewed from a different direction or under different lighting. We aim to understand the mechanisms underlying this perceptual stability using recently developed deep-learning algorithms. Networks trained using human perceptual classifications of materials are expected to show similar behavioral performance to human observers, replicating the same successes and errors. We then interrogate the internal structure of the networks to understand how the classifications are made. Finally, we test whether human performance can be (i) disrupted by modifying or removing information that the network suggests is important and (ii) preserved when the edits are unimportant. Understanding the mechanisms that support robust material perception may be useful for online shopping and improving safety-critical decisions such as road wetness by autonomous cars.