The wild success of Deep Neural Network (DNN) models in a variety of  domains has created considerable excitement in the machine learning community. Despite this success, a deep understanding  of why DNNs perform so well, and whether their performance is somehow brittle, has been lacking.