Mark Davenport
Email: mdav (at) gatech (dot) edu
Office Hours: TBD. Also by appointment.

This course is an introduction to the fundamentals of optimization with a focus on algorithms and applications in signal processing, control systems, machine learning, and robotics. The central theme of the course is the use of linear algebra and optimization in posing and solving practical information processing problems.

Download the syllabus.

Introductory courses in linear algebra and multivariable calculus are the main pre-requisites. Most of the course will use the language of matrices and vectors. Students should be comfortable with the use of matrices to represent systems of equations and the notion of taking a gradient of a function of many variables. Familiarity with eigenvalues, eigenvectors, and eigenvalue decompositions will be extremely helpful. Finally, students should also have basic Python programming skills.

There is no required text. Below is a list of books that I have found helpful over the years for learning (and teaching) the material in this class.

Linear algebra

Linear Algebra and its Applications by Strang. (amazon)

Numerical Linear Algebra by Trefethen and Bau. (amazon)

Matrix Analysis by Horn and Johnson. (amazon)

Optimization

Convex Optimization by Boyd and Vanderberghe. (Available as a free pdf from author's website)

Optimization Models by Calafiore and El Ghaoui. (amazon)

Numerical Optimization by Nocedal and Wright. (amazon)

Optimization by Vector Space Methods by Luenberger. (amazon)

Machine learning

Machine Learning Refined by Watt, Borhani, and Katsaggelos. (amazon)

Linear Algebra and Learning from Data by Strang. (amazon)

The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman. (Available as a free pdf from author's website)

Learning from Data by Abu-Mostafa, Magdon-Ismail, and Lin. (amazon)

Potpourri

The Master Algorithm by Domingos. (amazon)

Data and Goliath by Schneier. (amazon)

Weapons of Math Destruction by O'Neil. (amazon)

Additional online resources

The Matrix Cookbook

A short, useful introduction to matrix calculus

If you find anything else useful, let me know and I will post it here.

• The method of least squares
  • Applications and formulation: Regression and interpolation
  • Solving least squares problems (Review of multivariable calculus)
  • Understanding least squares problems (Review of linear algebra)
  • Computing least squares solutions
• Unconstrained optimization
  • Convex optimization
  • Gradient descent
  • Conjugate gradients
  • Acceleration: The heavy ball method and Nesterov's optimal method
  • Newton's method and quasi-Newton methods
  • Non-smooth optimization
  • Stochastic gradient descent
  • Applications: Approximation, filter design, tracking, logistic regression, neural networks
• Constrained optimization
  • Lagrange duality
  • The KKT conditions
  • Algorithms for constrained optimization
  • Linear programming
  • The simplex algorithm
  • Second order cone programs
  • Semidefinite programs
  • Applications: Support vector machines, portfolio optimization, feature selection, optimal power flow, recommendation systems
• Beyond convex optimization
  • Integer programming
  • Dynamic programming
  • Optimization on graphs
  • Optimization in game theory
  • Applications: Error correction, optimal control, reinforcement learning, generative adversarial networks