Course Description
Introduction of optimization algorithms suitable for solving large- scale problems, with a focus on exploring recent advances in the context of machine learning. Students will learn several algorithms for solving smooth and non-smooth problems, compare the efficacy of those methods, and discuss the trade-offs in terms of statistical accuracy, scalability, and algorithmic complexity.
Athena Title
Large-Scale Opt for Mach Learn
Prerequisite
CSCI 4380/6380 or CSCI 4360/6360 or permission of department
Semester Course Offered
Not offered on a regular basis.
Grading System
A - F (Traditional)
Course Objectives
The goal of this course is to introduce efficient optimization algorithms for solving large-scale problems in modern data science and machine-learning applications. The course will cover fundamental concepts in optimization theory and explore recent algorithmic advances in machine-learning applications. Upon successful completion of the course, students will be able to: - understand the basic theory of optimization, - identify the structure of the problem, such as convexity and smoothness, - select an appropriate method that exploits the known structure of the given optimization problem, - compare the advantage and disadvantage of one method over another, and - implement optimization algorithms using python.
Topical Outline
1. Foundation for optimization - Convex sets and functions - Basics of optimization - Convex problems 2. First-order methods - Gradient descent - Subgradient methods - Proximal gradient methods - Stochastic gradient descent - Acceleration: momentum and Nesterov's method 3. Optimality - Largrange duality - KKT conditions 4. Second-order methods - Newton's method - Primal-dual interior point methods 5. Advanced Topics - Stochastic Variance Reduced Gradient - ADMM - Coordinate descent - Mirror descent - Neural networks