Deep reinforcement learning

Summary

This course provides an overview and introduces modern methods for reinforcement learning (RL.) The course starts with the fundamentals of RL, such as Q-learning, and delves into commonly used approaches, like PPO and DQN. The course will introduce students to practical applications of RL.

Content

Keywords

Deep learning, reinforcement learning, TD learning, SARSA, Actor-Critic Networks, policy gradients, alphago, alphastar, planning, alignment, RLHF, PPO

Learning Prerequisites

Required courses

• Analysis I, II
• Linear Algebra
• Probability and statistics (MATH-232)
• Algorithms I (CS-250)

Recommended courses

• Introduction to machine learning (CS-233)
• Machine learning (CS-433)

Important concepts to start the course

• Regularization in machine learning,
  
• Gradient descent. Stochastic gradient descent.
• Expectation, statistics
• Linear algebra and probabilities
• programming

Learning Outcomes

By the end of the course, the student must be able to:

• Apply Understand and define basic problems and tasks in reinforcement learning (like Markov decision process, model-based and model-free RL, on-policy vs off-policy RL)
• Assess / Evaluate Formulate a real-world problem as an RL setting to apply the approaches taught in the class.
• Elaborate Implement standard deep RL algorithms.
• Judge Understand the failure modes of these models and learning algorithms.
• Propose Read and review academic papers to understand their contributions and learn how to evaluate them critically.
• Apply Students gain the skills and knowledge necessary to tackle complex problems in autonomous robotics, game-playing, and other domains through lectures, hands-on coding exercises, practical applications, and course projects.

Transversal skills

• Continue to work through difficulties or initial failure to find optimal solutions.
• Access and evaluate appropriate sources of information.
• Write a scientific or technical report.
• Manage priorities.

Teaching methods

• Lectures
• Lab sessions
• Individual course projects
• Paper reading
• Group projects

Expected student activities

• Work on miniproject
• Solve all exercises
• Attend all lectures and take notes during lecture, participate in quizzes.
• If you cannot attend a lecture, then you must read the recommended book chapters
• Work on a project

Assessment methods

• Written final exam (25%)
• Assignments (25%)
• Course project (50%)

Supervision

Resources

Bibliography

• Textbook: Reinforcement Learning by Sutton and Barto (MIT Press). Pdfs of the preprint version  of the book are availble online

Ressources en bibliothèque

• Reinforcement Learning / Sutton

Moodle Link

• https://go.epfl.ch/CS-456