Analysis of over-parameterized neural networks (NN) has drawn significant attention in recent years. It was shown that these systems behave like convex systems under various restricted settings, such as in two-level neural networks, and when learning is restricted locally in the so-called neural tangent kernel space around specialized random initializations; However, there are no theoretical techniques that can analyze fully trained deep neural networks encountered in practice.
This talk presents a solution to this problem. We introduce a new technique called neural feature repopulation, and show that under suitable representations, over parameterized deep neural networks are inherently convex, and when optimized, the system can learn effective feature representations suitable for the underlying learning task.
This highly unexpected result can be used to explain the empirical success of deep neural networks that are fully trained. In particular, it explains why they do not tend to stuck in bad local minima, despite the common perception of being highly "nonconvex", and it provides theoretical insights on how do neural networks learn effective feature representations in practice.