Model Predictive Control and Reinforcement Learning

Freiburg, July 26 to August 4, 2021 


This block course is intended for master and PhD students from engineering, computer science, mathematics, physics, and other mathematical sciences. The aim is that participants understand the main concepts of model predictive control (MPC) and reinforcement learning (RL) as well the similarities and differences between the two approaches. In hands-on exercises and project work they learn to apply the methods to practical optimal control problems from science and engineering. 

The course consists of lectures, exercises, and project work. Most lectures will be given by Prof. Dr. Joschka Boedecker and Prof. Dr. Moritz Diehl from the University of Freiburg. Towards the end of the course, invited guest lectures will be given by external experts in the field.

Covered topics include

  • Optimal Control Problem (OCP) formulations - constrained, infinite horizon, discrete time, stochastic, robust
  • Dynamic Programming (DP) concepts and algorithms - value iteration and policy iteration
  • Linear Quadratic Regulator (LQR) and Riccati equations
  • convexity considerations in DP for constrained linear systems
  • Model predictive control (MPC) formulations and stability guarantees
  • MPC algorithms - quadratic programming, direct multiple shooting, Gauss-Newton, real-time iterations
  • Differential Dynamic Programming (DDP) for the solution of unconstrained MPC problems 
  • Reinforcement Learning (RL) formulations and approaches - model-free and model-based 
  • RL algorithms - Monte Carlo, temporal differences, model learning, direct policy search
  • similarities and differences between MPC and RL

Note on the chosen notation convention: we will focus on discrete time systems with time index k and denote system states by sk, control actions by ak, and we will minimize a stage cost c(sk,ak) rather than maximize a reward. A deterministic system would be denoted by sk+1= f(sk, ak).

The course will be conducted as a mixed virtual and real event if the Corona situation in the summer of 2021 allows, and otherwise be fully virtual. Each course day starts at 9:00 and ends at 17:00 (5 p.m.) Freiburg time (MET). Lectures are typically followed by computer exercises in Python. Towards the end of the course, participants will work on application projects which apply at least one of the MPC and RL methods to self-chosen application problems from any area of science or engineering. The results of the projects, that can be performed in teams, will be presented in a public presentation on the last day of the course and a short report to be submitted two weeks after the course. The report will determine the final grade of the course.

The number of exercise and project participants is limited to 60 due to limited supervision capabilities. All lecture videos and exercises will be made openly available such that an even larger number of participants can follow the contents of the course. The maximum number of physical participants in Freiburg is expected to be either 0 or 25.

Registration will open in May 2021. An expression of interest in form of a preregistration is possible here.

Any questions on the course can be addressed to Katrin Baumgärtner (katrin.baumgaertner@imtek.uni-freiburg.de).