Artificial neural networks/reinforcement learning

CS-456 / 5 credits

Language: English

Summary

Since 2010 approaches in deep learning have revolutionized fields as diverse as computer vision, machine learning, or artificial intelligence. This course gives a systematic introduction into influential models of deep artificial neural networks, with a focus on Reinforcement Learning.

Content

General Introduction: Deep Networks and Reinforcement Learning

Reinforcement Learning 1: Bellman equation and SARSA
Reinforcement Learning 2: variants of SARSA, Q-learning, n-step-TD learning
Reinforcement Learning 3: Policy gradient

Deep Networks 1: BackProp, Multilayer Perceptrons, automatic differentiation
Deep Networks 2: Tricks of the Trade in deep learning
Deep Networks 3: Loss landscape and optimization methods for deep networks

Deep reinforcement learning 1: DeepQ and Actor-Critic, Inductive Bias
Deep reinforcement learning 2: Eligibility traces from Policy Gradient, Model-free
Deep reinforcement learning 3: Atari games, Replay buffer, and robotics
Deep reinforcement learning 4: Model-Based Deep RL
Deep reinforcement learning 5: Exploration: novelty, surprise, information gain
Biology and reinforcement learning: Three-factor learning rules
Hardware, energy consumption, and three-factor learning rules

Keywords

Deep learning, artificial neural networks, reinforcement learning, TD learning, SARSA, Actor-Critic Networks

Learning Prerequisites

Required courses

CS 433 Machine Learning (or equivalent)

Calculus, Linear Algebra (at the level equivalent to first 2 years of EPFL in STI or IC, such as Computer Science, Physics or Electrical Engineering)

Recommended courses

stochastic processes

optimization

Important concepts to start the course

Regularization in machine learning,
Training base versus Test base, cross validation.
Gradient descent. Stochastic gradient descent.
Expectation, Poisson Process, Bernoulli Process.

Learning Outcomes

By the end of the course, the student must be able to:

Apply learning in deep networks to real data
Assess / Evaluate performance of learning algorithms
Elaborate relations between different mathematical algorithms of learning
Judge limitations of algorithms
Propose algoriothms and models for learning from experience
Apply Reinforcement Learning

Transversal skills

Continue to work through difficulties or initial failure to find optimal solutions.
Access and evaluate appropriate sources of information.
Write a scientific or technical report.
Manage priorities.

Teaching methods

Ex cathedra lectures and miniproject. Every week the ex cathedra lectures are interrupted for at least one in-class exercise which is then discussed in classroom before the lecture continues. Additional exercises are given as homework or can be disussed in the second exercise hour.

Expected student activities

Work on miniproject

Solve all exercises

Attend all lectures and take notes during lecture, participate in quizzes.

If you cannot attend a lecture, then you must read the recommended book chapters

Assessment methods

Written exam (70 percent) and miniproject (30 percent)

Supervision

Office hours	Yes
Assistants	Yes
Forum	Yes
Others

Resources

Bibliography

Textbook: Deep Learning by Goodfellow, Bengio, Courville (MIT Press)
Textbook: Reinforcement Learning by Sutton and Barto (MIT Press)

Pdfs of the preprint version for both books are availble online

Ressources en bibliothèque

Websites

Videos

http://yes, for most session.

In the programs

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Semester: Spring
Exam form: Written (summer session)
Subject examined: Artificial neural networks/reinforcement learning
Lecture: 2 Hour(s) per week x 14 weeks
Exercises: 2 Hour(s) per week x 14 weeks

Reference week

	Mo	Tu	We	Th	Fr
8-9
9-10
10-11
11-12		CO3
12-13		CO3
13-14
14-15		CO3
15-16		CO3
16-17
17-18
18-19
19-20
20-21
21-22

Légendes:

Lecture

Exercise, TP

Project, other