Topics in machine learning

Introduction (supervised learning, risk, error decomposition, over-fitting and capacity control + cross-validation, Bayes predictor for classification and regression)

Differentiable programming (backpropagation algorithm) and theoretical challenges posed by modern methods (large deep neural networks)

Statistical analysis of Empirical Risk Minimization (learning theory, from finite number of hypotheses to Rademacher / covering numbers)

First-order methods for optimization (gradient descent, stochastic gradient descent).

Kernel methods (positive-definite kernels and Reproducing Kernel Hilbert Spaces)

Algorithmic regularization of gradient descent (reparameterized models, least-squares, mirror descent, logistic loss and max-margin)

Dynamics of large neural networks (parameterizations, neural tangent kernel and feature-learning limits, infinite width and infinite depth limits)

This is *not* a course for student who want to learn deep learning practice. This course is intended for the theory-inclined students who want to discover the mathematical tools that shed light on some aspects of deep learning. The course is up to date with recent theoretical research.