Block Course, 04.10.2023 - 13.10.2023, 9:00-18:00
Lecturers: Prof. Dr. Joschka Boedecker (Uni Freiburg), Prof. Dr. Moritz Diehl (Uni Freiburg) and Prof. Dr. Sebastien Gros (NTNU Trondheim)
Exercises: Andrea Ghezzi and Jasper Hoffmann
- Faculty of Engineering from Oct 4 to 6: HS 0-26, Georges-Köhler-Allee 101, 79110 Freiburg, Google Maps
- Historical University from Oct 9 to 13: Kollegiengebäude I, HS 1015, Platz der Universität 3, 79098 Freiburg, Google Maps
- Aperitif at Waldsee: Waldseestraße 84, 79117 Freiburg im Breisgau, Google Maps
- Dinner at Dattler: Am Schlossberg 1, 79104 Freiburg im Breisgau, Google Maps
Freiburg mobility services:
Freiburg has a reliable tram services and many bike roads.
- You can buy tickets on the tram/bus or from the mobile app of VAG (local transportation company) GooglePlay, AppStore
- You can rent Frelo bikes at the Frelo stations dislocated around the city and you can ride your bike for 30 mins and bring it back to a Frelo station (this service for 6 Euro per month) Frelo website.
This comprehensive course spans 8 days, divided into two weeks. During the first week (3 days), we will provide a solid foundation in MPC (Model Predictive Control) and RL (Reinforcement Learning). In the second week (5 days), we will delve into advanced methods. We will explore nonlinear MPC (NMPC), transformers, policy gradient methods, and techniques for combining MPC and RL.
Lectures will be supported with intensive exercise/programming sessions.
In addition, in the last three days participants will work on their own project in the domain of MPC / RL helped by the professors and the tutors. The project work can be an exciting opportunity to share ideas and collaborate with other participants.
Note that the schedule might be subject to change!
(JB: Joschka Boedecker, MD: Moritz Diehl, SG: Sebastien Gros, JH: Jasper Hoffmann, HH: Hannes Homburger)
|Week 1 at Faculty of Engineering (HS 0-26 - Buil. 101)||Week 2 at Historical University (HS 1015 - Buil. KG I)|
|Wed 4-Oct||Thu 5-Oct||Fri 6-Oct||Mon 9-Oct||Tue 10-Oct||Wed 11-Oct||Thu 12-Oct||Fri 13-Oct|
Dynamic Systems and Simulation (MD)
Dynamic Programming and LQR (MD)
Monte Carlo RL, Temporal Difference and Q-Learning (JB)
Real-Time Algorithms for NMPC (MD)
Policy gradient and Actor-Critic methods (JB)
Introduction MPC and RL Framework (SG)
When to use RL in MPC? (SG)
|Past project presentations +
|10:30-11:00||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break|
Dynamic Systems and Simulation
Dynamic Programming and Optimal Control
MPC-based value functions, policies, and their sensitivities
RL with MPC as function approximators
|Project work /
|12:20-12:30||Project Brainstorming||Project Brainstorming||Project Brainstorming||Project Brainstorming|
Numerical Optimization (MD)
Deep Learning (JB)
RL with Function Approximation (JB)
Safety and Stability in MPCRL (SG)
|Q & A Session + Project work||Project presentations|
|15:30-16:00||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break||Coffee Break|
Transformer in practice (forecasting)
|Project brainstorming and kick-off||Project work|
|Break||Break||From 17 to 18: Short Research Announcements||Break|
|19:00-||Social Gathering||Aperitif at Waldsee||Workshop Dinner|
- Lecture 0: Introduction
- Lecture 1: Dynamic Systems and Simulation
- Lecture 2: Nonlinear Optimization
- Lecture 3: Dynamic Programming and LQR
- Lecture 4: Deep Learning
- Lecture 5: Monte Carlo RL, Temporal Difference and Q-learning
- Lecture 6: RL with Function Approximation
- Lecture 7: Real-Time Algorithms for Nonlinear MPC
- Lecture 8: Transformer
- Lecture 9: Policy Gradient and Actor-Critic Methods
- Lecture 10: MPPI
- Lecture 11: Introduction to MPC and RL Framework
- Lecture 12: Safety and Stability in MPCRL
- Lecture 13: Model Predictive Control for Markov Decision Processes
- Overview of the exercises as pdf: 1st week
All exercises of first week: download as zip file
- Exercise 1: zip folder with material | solution
- Exercise 2: zip folder with material | solution
- Exercise 3: zip folder with material | solution
- Exercise 4: zip folder with material | solution
- Exercise 5: zip folder with material | solution
- Exercise 6: zip folder with material | solution
- Exercise 7: zip folder with material | solution | acados presentation
- Exercise 8: zip folder with material | solution
- Exercise 9: zip folder with material | solution
- Exercise10: zip folder with material | solution
- Exercise11: zip folder with material | solution
- Exercise12: zip folder with material | solution
- Projects: project instructions
Commit for a project by filling out this form.
Additional links for the project work:
- StableBaselines3: A good RL library that works out of the box for gym interfaces.
- minimalRL: A minimalist RL library that has each algorithm in one file.
- Tips&Tricks: A guide on important design choices while doing reinforcement learning experiments.
- Acados Examples: A collection of simple problems solved with Acados.
- J. B. Rawlings, D. Q. Mayne, M. Diehl. Model Predictive Control. Nobhill Publishing, 2017 (free PDF here)
- R. S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018 (free PDF here)
- D. P. Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, 2019
Literature references from S. Gros:
Possible lunch locations for 1st week:
- Fraunhofer IPM Canteen (7 euro, 8 mins walk) GMaps
- Westarkaden (several food options, 13 mins walk | 3 mins by tram, every 8 mins) GMaps
- XXL Döner and pizza (around 7 euro, 7 mins walk) GMaps
- Backery (sandwiches and more, 7 mins walk) GMaps
Possible lunch locations for 2nd week:
- Mensa Rempartstraße (payment only with mensa card which can is issued at the entrance of the mensa, if you can prove to be a student you will have student prices, approx., 4 Euro per meal)
- Veggie Liebe (falafel), Blauer Fuchs (restaurant)
- Several diners (salads, burgers, kebab, backery) in Niemenstr. GMaps
Registration is closed!
The registration is now closed but if you want to join there might be still few places available, for information reach out Andrea
Registration: within September 20, 2023, until the limit of 60 participants is reached (first come first served). The registration is recorded after the fee has been transferred and received.
Participation fee: 400 EUR (free of charge for master’s students from the University of Freiburg). The fee includes a welcome reception and a dinner with the participants.
Cancellation policy: no refund possible.
This block course is intended for master students and PhD students from engineering, computer science, mathematics, physics, and other mathematical sciences.
For interested Master students:
- We accept registration only from master student from the University of Freiburg
- We strongly recommend the students to have taken: (Numerical Optimal or Numerical Optimal Control) or (Reinforcement Learning) courses
- The evaluation of the course will be based on the exercise sessions and the project works. Further details on evaluation will be published soon!
Relevant only for students of the university of Freiburg.
In order to receive 3 ECTS for this course, students need to pass all of the following:
- Studienleistung (SL, ungraded)
- Participation in the exercise session
- Prüfungsleistung (PL, graded)
- Project report
Every student from the University of Freiburg needs to fill the registration form.
Please also read the project instructions from above!
On the first day (October 4th), students further need to decide whether they want to commit themselves to do the PL. The registration will take via the PL registration form below.
|This course has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 953348.|