Analysis of over-parameterized neural networks (NN) has drawn significant attention in recent years. It was shown that these systems behave like convex systems under various restricted settings, such as in two-level neural networks, and when learning is restricted locally in the so-called neural tangent kernel space around specialized random initializations; However, there are no theoretical techniques that can analyze fully trained deep neural networks encountered in practice.
This talk presents a solution to this problem. We introduce a new technique called neural feature repopulation, and show that under suitable representations, over parameterized deep neural networks are inherently convex, and when optimized, the system can learn effective feature representations suitable for the underlying learning task.
This highly unexpected result can be used to explain the empirical success of deep neural networks that are fully trained. In particular, it explains why they do not tend to stuck in bad local minima, despite the common perception of being highly "nonconvex", and it provides theoretical insights on how do neural networks learn effective feature representations in practice.

Tong Zhang is a professor of Computer Science and Mathematics at The Hong Kong University of Science and Technology. Tong Zhang received a B.A. in mathematics and computer science from Cornell University and a Ph.D. in Computer Science from Stanford University.
Previously, he was a professor at Rutgers university, and worked at IBM, Yahoo, Baidu, and Tencent. Tong Zhang's research interests include machine learning algorithms and theory, statistical methods for big data and their applications. He is a fellow of ASA and IMS, and he has been in the editorial boards of leading machine learning journals like JMLR、Machine Learning Journal and program committees of top machine learning conferences, such as NIPS, ICML and COLT.