A Convex Formulation of Deep Neural Networks

主讲人 Speaker:张潼
时间 Time: 周五10:30-11:30,2019-10-18
地点 Venue:清华大学近春园西楼三层报告厅

摘要 Abstract

Analysis of over-parameterized neural networks (NN) has drawn significant attention in recent years. It was shown that these systems behave like convex systems under various restricted settings, such as in two-level neural networks, and when learning is restricted locally in the so-called neural tangent kernel space around specialized random initializations; However, there are no theoretical techniques that can analyze fully trained deep neural networks encountered in practice.
This talk presents a solution to this problem. We introduce a new technique called neural feature repopulation, and show that under suitable representations, over parameterized deep neural networks are inherently convex, and when optimized, the system can learn effective feature representations suitable for the underlying learning task.
This highly unexpected result can be used to explain the empirical success of deep neural networks that are fully trained. In particular, it explains why they do not tend to stuck in bad local minima, despite the common perception of being highly "nonconvex", and it provides theoretical insights on how do neural networks learn effective feature representations in practice.

报告人简介 Profile

张潼博士是机器学习领域的国际著名专家,拥有美国康奈尔大学数学和计算机双学士学位,以及斯坦福大学计算机硕士和博士学位,目前在香港科技大学数学系和计算机系任教。他曾经担任美国新泽西州立大学终身教授、IBM 研究院研究员和雅虎研究院主任科学家、百度研究院副院长和大数据实验室负责人以及腾讯 AI Lab 主任。
同时,张潼博士还曾参加美国国家科学院大数据专家委员会,负责过多个美国国家科学基金资助的大数据研究项目,并兼任美国统计学会和国际数理统计学会院士,神经信息处理系统进展大会(NIPS)、国际机器学习大会(ICML)及学习理论大会(COLT)等国际顶级机器学习会议主席或领域主席,以及机器学习研究期刊(JMLR)和 机器学习期刊(Machine Learning Journal)等国际一流人工智能期刊编委。