Empirical processes
Summary
From prototypical examples of estimators used by statisticians, to more complex nonparametric models, methods and theorems will be taught to study their (non)asymptotic behavior, when defined as mappings depending on an empirical measure based on a sample of random observations.
Content
This course provides central results and methods to understand and derive generalization guarantees of modern learning algorithms. The main topics are:
- Fundamental limit theorems for classes of empirical processes.
- Basic probabilistic inequalities: methods and results.
- Symmetrization and chaining methods.
- How to measure the complexity of classes of functions: from combinatorial and entropic considerations to Rademacher averages.
- Concentration-of-measure phenomenon: maximal inequalities and concentration inequalities of suprema of empirical processes.
The exposed methods and results will be illustrated with statistical applications related to machine learning and hypothesis testing throughout the lecture and exercise sessions.
Keywords
Empirical processes, limit theorems, concentration of measure, probabilistic inequalities, generalization properties, learning algorthims.
Learning Prerequisites
Important concepts to start the course
Previous courseworks in analysis, mathematical statistics and probability are highly required, with interests in statistical learning theory and algorithms.
Learning Outcomes
By the end of the course, the student must be able to:
- Formulate the fundamental framework related to empirical processes
- Manipulate probabilistic inequalities to empirical estimators
- Describe the concept of concentration-of-measure phenomenon for empirical processes
- Apply concentration inequalities to derive the performance of statistical learning procedures
Teaching methods
The semester will be composed of:
- Lectures using slides and blackboard (for some proofs and examples in particular).
- Exercise sessions.
Expected student activities
Student are expected to:
- Attend to all lectures.
- Participate to exercise weekly sessions.
Assessment methods
Written exams (midterm and final).
Resources
Bibliography
- Stéphane Boucheron, Gábor Lugosi, and Pascal Massart, Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, Oxford Academic, 2013.
- Sara van de Geer, Empirical Processes in M-Estimation. Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press, 2009.
- Aad W. Vaart , Jon A. Wellner, Weak Convergence and Empirical Processes. With Applications to Statistics. Springer Series in Statistics (SSS), Springer New York, 1996.
- Víctor H. Peña , Evarist Giné, Decoupling. From Dependence to Independence. Probability and Its Applications (PIA), Springer New York, 1999.
Moodle Link
In the programs
- Semester: Spring
- Exam form: Written (summer session)
- Subject examined: Empirical processes
- Courses: 2 Hour(s) per week x 14 weeks
- Exercises: 2 Hour(s) per week x 14 weeks
- Type: optional
- Semester: Spring
- Exam form: Written (summer session)
- Subject examined: Empirical processes
- Courses: 2 Hour(s) per week x 14 weeks
- Exercises: 2 Hour(s) per week x 14 weeks
- Type: optional
- Semester: Spring
- Exam form: Written (summer session)
- Subject examined: Empirical processes
- Courses: 2 Hour(s) per week x 14 weeks
- Exercises: 2 Hour(s) per week x 14 weeks
- Type: optional
- Semester: Spring
- Exam form: Written (summer session)
- Subject examined: Empirical processes
- Courses: 2 Hour(s) per week x 14 weeks
- Exercises: 2 Hour(s) per week x 14 weeks
- Type: optional
- Semester: Spring
- Exam form: Written (summer session)
- Subject examined: Empirical processes
- Courses: 2 Hour(s) per week x 14 weeks
- Exercises: 2 Hour(s) per week x 14 weeks
- Type: optional