Introduction to Convex Optimization

12 units · Graduate

Intermittent: 12 units As machine learning grows in prominence, so also has optimization become a mainstay for machine learning, particularly techniques for convex optimization. Most learning problems are formulated as optimization of some objective function, sometimes subject to constraints. This course explores the optimization algorithms used to solve these machine learning problems. We characterize the properties of the optimization problems that enable these techniques to be efficient (e.g. convexity, smoothness, linearity, separability) as well as properties that inhibit efficient optimization (e.g. nonconvexity). Core topics include first order methods (gradient descent, subgradient methods, proximal and stochastic gradient descent), duality and linear programming, and second-order/quasi-Newton methods. We also consider advanced techniques ranging from those that have spurred the growth of deep learning (e.g. adaptive gradient methods, momentum) and those that enable large-scale distributed optimization. The course will focus both on theory and practical applications, frequently drawing motivation from examples in machine learning. The course is designed so that a machine learning (ML) course could be taken after or before this one; ML is not a prerequisite. Students will gain the tools to both implement and analyze modern optimization techniques. Prerequisites: ( 21-241 Min. grade C or 21-240 Min. grade C) and ( 15-359 Min. grade C or 36-217 Min. grade C or 36-219 Min. grade C or 21-325 Min. grade C) and ( 10-315 Min. grade C or 21-254 Min. grade C or 21-256 Min. grade C or 21-259 Min. grade C or 10-301 Min. grade C or 11-485 Min. grade C) and ( 15-151 Min. grade C or 21-127 Min. grade C) and 15-122 Min. grade C Course Website: https://www.cs.cmu.edu/~mgormley/courses/10425