Model Predictive Control and Reinforcement Learning

Block Course, 04.10.2023 - 13.10.2023, 9:00-18:00

Lecturers: Prof. Dr. Joschka Boedecker (Uni Freiburg), Prof. Dr. Moritz Diehl (Uni Freiburg) and Prof. Dr. Sebastien Gros (NTNU Trondheim)

Exercises: Andrea Ghezzi and Jasper Hoffmann

Contacts: for any questions feel free to contact  Andrea or Jasper

Locations:

  • Faculty of Engineering from Oct 4 to 6: HS 0-26, Georges-Köhler-Allee 101, 79110 Freiburg, Google Maps
  • Historical University from Oct 9 to 13: Kollegiengebäude I, HS 1015, Platz der Universität 3, 79098 Freiburg, Google Maps
  • Aperitif at Waldsee: Waldseestraße 84, 79117 Freiburg im Breisgau, Google Maps
  • Dinner at Dattler: Am Schlossberg 1, 79104 Freiburg im Breisgau, Google Maps

Freiburg mobility services:

Freiburg has a reliable tram services and many bike roads.

  • You can buy tickets on the tram/bus or from the mobile app of VAG (local transportation company)  GooglePlay, AppStore
  • You can rent Frelo bikes at the Frelo stations dislocated around the city and you can ride your bike for 30 mins and bring it back to a Frelo station (this service for 6 Euro per month) Frelo website.

We are pleased to announce the third edition of this block course, building on the success of the previous editions (MPCRL22, MPCRL21)!

This comprehensive course spans 8 days, divided into two weeks. During the first week (3 days), we will provide a solid foundation in MPC (Model Predictive Control) and RL (Reinforcement Learning). In the second week (5 days), we will delve into advanced methods. We will explore nonlinear MPC (NMPC), transformers, policy gradient methods, and techniques for combining MPC and RL.

Lectures will be supported with intensive exercise/programming sessions. 

In addition, in the last three days participants will work on their own project in the domain of MPC / RL helped by the professors and the tutors. The project work can be an exciting opportunity to share ideas and collaborate with other participants.

Checkout our flyer!

Preliminary schedule

Note that the schedule might be subject to change!

(JB: Joschka Boedecker, MD: Moritz Diehl, SG: Sebastien Gros, JH: Jasper Hoffmann, HH: Hannes Homburger)

 

  Week 1 at Faculty of Engineering (HS 0-26 - Buil. 101)   Week 2 at Historical University (HS 1015 - Buil. KG I)
  Wed 4-Oct Thu 5-Oct Fri 6-Oct   Mon 9-Oct Tue 10-Oct Wed 11-Oct Thu 12-Oct Fri 13-Oct
09:00-10:30 Welcome +
Lecture 1
 Dynamic Systems and Simulation (MD)		 Lecture 3
Dynamic Programming and LQR (MD)		 Lecture 5
Monte Carlo RL, Temporal Difference and Q-Learning (JB) 		   Motivation +
 Lecture 7
Real-Time Algorithms for NMPC (MD)		 Lecture 9
Policy gradient and Actor-Critic methods (JB)		 Lecture 11
Introduction MPC and RL Framework (SG)		 Lecture 13
When to use RL in MPC? (SG)		 Past project presentations + 
Project work
10:30-11:00 Coffee Break Coffee Break Coffee Break   Coffee Break Coffee Break Coffee Break Coffee Break Coffee Break
11:00-12:20 Exercise 1
Dynamic Systems  and Simulation		 Exercise 3
 Dynamic Programming and Optimal Control		 Exercise 5
 Q-learning		   Exercise 7
 NMPC (Acados)

Exercise 9
Policy Gradient

 Exercise 11
MPC-based value functions, policies, and their sensitivities		 Exercise 12
RL with MPC as function approximators		 Project work /
Project presentations
12:20-12:30 Project Brainstorming Project Brainstorming   Project Brainstorming Project Brainstorming
12:30-14:00 Lunch Lunch Lunch   Lunch Lunch Lunch Lunch Lunch
14:00-15:30 Lecture 2
Numerical Optimization (MD)		 Lecture 4
Deep Learning (JB)		 Lecture 6
RL with Function Approximation (JB)		   Lecture 8
Transformers (JH)		 Lecture  10
MPPI (HH)		 Lecture 12
Safety and Stability in MPCRL (SG)		 Q & A Session + Project work Project presentations
15:30-16:00 Coffee Break Coffee Break Coffee Break   Coffee Break Coffee Break Coffee Break Coffee Break Coffee Break
16:00-17:30 Exercise 2
Numerical Optimization		 Exercise 4
 PyTorch		 Exercise 6
DQN		   Exercise 8
Transformer in practice (forecasting)		 Exercise 10
MPPI		 Project brainstorming and kick-off Project work  
     Break      Break From 17 to 18:     Short Research Announcements  Break    
19:00-   Social Gathering     Aperitif at Waldsee   Workshop Dinner    

Teaching Resources

Lecture slides:

Exercise sessions:

Project work:

Commit for a project by filling out this form.

Further Resources

Additional links for the project work:

  • StableBaselines3: A good RL library that works out of the box for gym interfaces.
  • minimalRL: A minimalist RL library that has each algorithm in one file.
  • Tips&Tricks: A guide on important design choices while doing reinforcement learning experiments.
  • Acados Examples: A collection of simple problems solved with Acados.

Books:

  • J. B. Rawlings, D. Q. Mayne, M. Diehl. Model Predictive Control. Nobhill Publishing, 2017 (free PDF here)
  • R. S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018 (free PDF here)
  • D. P. Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, 2019

Literature references from S. Gros:

Possible lunch locations for 1st week:

  • Fraunhofer IPM Canteen (7 euro, 8 mins walk) GMaps
  • Westarkaden (several food options, 13 mins walk | 3 mins by tram, every 8 mins) GMaps
  • XXL Döner and pizza (around 7 euro, 7 mins walk) GMaps 
  • Backery (sandwiches and more, 7 mins walk) GMaps

Possible lunch locations for 2nd week:

  • Mensa Rempartstraße (payment only with mensa card which can is issued at the entrance of the mensa, if you can prove to be a student you will have student prices, approx., 4 Euro per meal)
  • Veggie Liebe (falafel), Blauer Fuchs (restaurant)
  • Several diners (salads, burgers, kebab, backery) in Niemenstr. GMaps

Registration is closed!

The registration is now closed but if you want to join there might be still few places available, for information reach out Andrea

Registration: within September 20, 2023, until the limit of 60 participants is reached (first come first served). The registration is recorded after the fee has been transferred and received.

Participation fee: 400 EUR (free of charge for master’s students from the University of Freiburg). The fee includes a welcome reception and a dinner with the participants. 

Cancellation policy: no refund possible.

Targeted audience

This block course is intended for master students and PhD students from engineering, computer science, mathematics, physics, and other mathematical sciences. 

For interested Master students:

  • We accept registration only from master student from the University of Freiburg
  • We strongly recommend the students to have taken: (Numerical Optimal or Numerical Optimal Control) or (Reinforcement Learning) courses
  • The evaluation of the course will be based on the exercise sessions and the project works. Further details on evaluation will be published soon!

Formal requirements

Relevant only for students of the university of Freiburg.

In order to receive 3 ECTS for this course, students need to pass all of the following:

  • Studienleistung (SL, ungraded)
    • Participation in the exercise session
  • Prüfungsleistung (PL, graded)
    • Project report

Every student from the University of Freiburg needs to fill the registration form.

Please also read the project instructions from above!

On the first day (October 4th), students further need to decide whether they want to commit themselves to do the PL. The registration will take via the PL registration form below.

Registration link for PL

 

   This course has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 953348.