Machine learning for physicists

* Examples and types of problems that machine learning can solve.

* Linear regression in matrix notation. The concept of prediction, estimation. Least squares method. High-dimensional underdetermined problems and the concept of regularization aka ridge regression. Polynomial regression. The concept of bias and variance trade-off and overfitting. Usage of train, validation and test sets.

* Reminder of key concept from probability theory. Bayesian inference, maximum likelihood and maximum a posteriori estimation.

* Least-squares as maximum likelihood of probabilistic model with additive Gaussian noise. Regularization as a prior. Relation with inverse problems in signal processing. Generalized linear model.

* Robust regression, sparse regression, LASSO. Role of sparsity for variable selection. Compressed sensing.

* Gradient descent and stochastic gradient descent.

* Linear classification. Examples of classification losses. Logistic regression and its probabilistic interpretation. Multi-class classification, one-hot-encoding of classes, cross-entropy loss.Data that are not linearly separable. K-nearest neighbours. Curse of dimensionality.

* Unsupervised learning, dimensionality reduction, low-rank approximation. Singular values decomposition (SVD) and principal component analysis (PCA).

Examples: Recommender systems, reconstruction of Europe geography from human genome, spin-glass card game aka planted spin glass model.

* Analogy between learning/inference in high dimension and statistical mechanics. Maximum a posteriori estimation as search for the ground state.

Minimum mean squared error estimator. Bayesian inference as sampling from the Boltzmann measure.

* Monte Carlo Markov Chains and their basic principles. Metropolis-Hastings update rule.

Gibbs sampling aka heat bath. Simulated annealing.

Bayesian learning of hyper-parameters, expectation maximization algorithm.

* Clustering. The k-means algorithm. Gaussian mixture model.

* Non-linear regression as linear regression in feature space. Representor theorem. Kernel methods as infinite-dimensional feature spaces. Kernel ridge regression. Examples of kernels and their feature spaces. Kernels as universal approximations. Classification with kernels, support vector machines.

* Random feature regression as approximation of kernels. One hidden-layer neural networks as features learning machines. Neural networks as universal approximators. Worst case computational hardness of training. Multi-layer neural networks as learning features of features.

* Deep learning for regression and classifications. Terminology of multi-layer feed-forward neural networks. Training with stochastic gradients descent aka the back-propagation algorithm. Discussion of hyper-parameters to be set when using neural networks. Historical notes and comments on performance of deep learning.

Importance of locality and translational symmetry. Convolutional neural networks for image classification. Design and terminology of convolutional and pooling layers.

Modus operandi of deep neural networks. Over-parametrization and lack of overfitting. Double descent behaviour replaced the bias-variance trade-off. Interpolation of the training set and its consequences for training, implicit regularization.

Concept of transfer learning, adversarial examples, data augmentation.

* Self-supervised learning. Data generative models. The principle of auto-encoder, its training and usage.Boltzmann machine. Maximum entropy principle. Training algorithm for the Boltzmann machine. Flow and diffusion-based generative models.

* Attention mechanism and attention layers. Bacisc of transformer architectures.