Deep Learning and Statistics: Bridging the Gap with Probabilistic Structure

Eric Nalisnick 
University of Cambridge

Deep neural networks have demonstrated impressive performance in predictive tasks. However, these models have been found to be opaque, brittle, and data-hungry---which frustrates their use for scientific, medical, and safety-critical applications. In this talk, I describe how imposing additional probabilistic structure on the network makes it more amenable to the best practices of traditional statistical modeling. For instance, I show that the deep learning regularization strategy known as “dropout” can be interpreted as a Bayesian structured shrinkage prior. Taking this perspective better illuminates modeling assumptions as well as improves performance in small-data settings. For a second example, I show how to constrain the deep neural network to encode only bijective functions. Under this constraint, the network can then be interpreted as a reparameterized linear model, which in turn improves the tractability of inference and criticism. I close the talk by arguing that this latter example presents a modest but clear step towards next-generation probabilistic models. Ideally, these models should be capable of making complex, autonomous decisions, but at the same time, can be probed and criticized within the traditional statistical modeling workflow.