Sequential Decision

ESIEE Paris, 5th year (work-study program), 2026-2027.

A system learns sequentially when its decisions determine the data it observes next: recommending an article, setting a price, choosing the answer of an assistant. The course starts with multi-armed bandits, where the whole problem is the trade-off between exploration and exploitation, then moves to reinforcement learning, and ends with the alignment of language models from human preferences. It consists of 10 sessions of 3 hours, each combining a lecture, a tutorial and a lab. All materials are in English.

Key information

  • Next session: see the table of materials.
  • Projects: pairs and topic by email by November 4; task 1 done for session 8 (November 25); notebook due on December 8; defenses at session 10 (December 9). Subject, starter notebooks.
  • Lab setup: setup archive (current seqlearn: version 2026.10.30), instructions.
  • Assessment: project and written exam; details to come.
  • Contact: igor.colin@esiee.fr.
  • Latest updates: 30 Oct, the project notebooks check the version of seqlearn in their first cell; 29 Oct, materials of session 7 and the projects; 28 Oct, materials of session 6.

Materials

SessionDateRoomTopicSlidesHandoutTutorialLab
1PER 5403The K-armed bandit: regret, Hoeffding’s inequality, greedy, epsilon-greedy, explore-then-commitpdfpdfpdfwarm-up · Colab
2PER 5309Optimism: UCB (Labs 1 and 2)pdfpdfpdfipynb · Colab
3PER 5401Thompson sampling and linear banditspdfpdfpdfipynb · Colab
4PER 5401Recap of session 3, Lab 3 and a bandit challenge (Lab 4)pdfpdfipynb · Colab
5PER 5401Contextual bandits, exponential weights and off-policy evaluationpdfpdfpdfipynb · Colab
6PER 5403Pure exploration and learning from preferencespdfpdfpdfipynb · Colab
7PER 5309Markov decision processes and Q-learning; presentation of the projectspdfpdfpdfipynb · Colab
8PER 5403Policy gradient
9PER 5403Preferences and RLHF
10PER 5403Project defenses

Slides and tutorial sheets are added after each session, lab notebooks on the day of the session. The handout is the slides without the animation steps, for printing.

Labs

The labs are Jupyter notebooks in Python. They run on a laptop CPU, or online with Google Colab.

Setup

To run the labs on your laptop:

  1. Download the setup archive once and unzip it. It contains SETUP.pdf, seqlearn/ (the Python package used in the labs), environment.yml and check_install.py.
  2. Install the environment as described in SETUP.pdf, and run check_install.py.
  3. For each lab, download the notebook (column Lab of the table above) and save it in the same folder, next to seqlearn/.

Google Colab

Nothing needs to be installed: upload the Colab version of the notebook (File > Upload notebook), and run the cells of its section “Framework” first.

Troubleshooting

A notebook stops with a ModuleNotFoundError on a part of seqlearn

Most labs need a new version of seqlearn (for instance seqlearn.challenge for Lab 4). Download the setup archive again and replace your seqlearn/ folder with the new one. Your own notebooks are not modified. The project starter notebooks do this themselves in their first cell.

Lab 3 stops on ipywidgets

If you installed the course environment before Lab 3, install the package ipywidgets:

  • with Miniforge (option A of SETUP.pdf): conda install -n esiee-sequential -c conda-forge ipywidgets
  • with a virtual environment (option B): activate .venv as in SETUP.pdf, then run pip install ipywidgets

Then restart Jupyter from the course folder.

Projects

The project is done in pairs, on one of the five topics below. The subject describes the topics, what is expected and the calendar: pairs and topic by email by November 4, task 1 done for session 8 (November 25), notebook due on December 8, defenses at session 10 (December 9). Each topic has a starter notebook, to save next to seqlearn/ (download the setup archive again first: it contains the simulators of the projects, in seqlearn/projects/).

#TopicStarter notebook
1Recommendation under driftipynb · Colab
2Bayesian optimizationipynb · Colab
3Exploration in reinforcement learningipynb · Colab
4Model arenaipynb · Colab
5Rating the players of a team gameipynb · Colab

References

  • T. Lattimore, C. Szepesvári, Bandit Algorithms, Cambridge University Press, 2020.
  • A. Slivkins, Introduction to Multi-Armed Bandits, Foundations and Trends in Machine Learning, 2019.
  • R. Sutton, A. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018.
  • N. Lambert, Reinforcement Learning from Human Feedback, 2025.