ESIEE Paris, 5th year (work-study program), 2026-2027.
A system learns sequentially when its decisions determine the data it observes next: recommending an article, setting a price, choosing the answer of an assistant. The course starts with multi-armed bandits, where the whole problem is the trade-off between exploration and exploitation, then moves to reinforcement learning, and ends with the alignment of language models from human preferences. It consists of 10 sessions of 3 hours, each combining a lecture, a tutorial and a lab. All materials are in English.
seqlearn: version 2026.10.30), instructions.seqlearn
in their first cell; 29 Oct, materials of session 7 and the projects; 28 Oct,
materials of session 6.| Session | Date | Room | Topic | Slides | Handout | Tutorial | Lab |
|---|---|---|---|---|---|---|---|
| 1 | PER 5403 | The K-armed bandit: regret, Hoeffding’s inequality, greedy, epsilon-greedy, explore-then-commit | warm-up · Colab | ||||
| 2 | PER 5309 | Optimism: UCB (Labs 1 and 2) | ipynb · Colab | ||||
| 3 | PER 5401 | Thompson sampling and linear bandits | ipynb · Colab | ||||
| 4 | PER 5401 | Recap of session 3, Lab 3 and a bandit challenge (Lab 4) | ipynb · Colab | ||||
| 5 | PER 5401 | Contextual bandits, exponential weights and off-policy evaluation | ipynb · Colab | ||||
| 6 | PER 5403 | Pure exploration and learning from preferences | ipynb · Colab | ||||
| 7 | PER 5309 | Markov decision processes and Q-learning; presentation of the projects | ipynb · Colab | ||||
| 8 | PER 5403 | Policy gradient | |||||
| 9 | PER 5403 | Preferences and RLHF | |||||
| 10 | PER 5403 | Project defenses |
Slides and tutorial sheets are added after each session, lab notebooks on the day of the session. The handout is the slides without the animation steps, for printing.
The labs are Jupyter notebooks in Python. They run on a laptop CPU, or online with Google Colab.
To run the labs on your laptop:
SETUP.pdf, seqlearn/ (the Python package used
in the labs), environment.yml and check_install.py.SETUP.pdf, and run check_install.py.seqlearn/.Nothing needs to be installed: upload the Colab version of the notebook (File > Upload notebook), and run the cells of its section “Framework” first.
ModuleNotFoundError on a part of seqlearnMost labs need a new version of seqlearn (for instance seqlearn.challenge for Lab 4).
Download the setup archive again and replace your seqlearn/ folder with the new one.
Your own notebooks are not modified. The project starter notebooks do this
themselves in their first cell.
ipywidgetsIf you installed the course environment before Lab 3, install the package ipywidgets:
SETUP.pdf):
conda install -n esiee-sequential -c conda-forge ipywidgets.venv as in SETUP.pdf, then
run pip install ipywidgetsThen restart Jupyter from the course folder.
The project is done in pairs, on one of the five topics below. The
subject describes the topics, what is
expected and the calendar: pairs and topic by email by November 4, task 1 done for
session 8 (November 25), notebook due on December 8, defenses at session 10
(December 9). Each topic has a starter notebook, to save next to seqlearn/
(download the setup archive again first: it contains the simulators of the projects,
in seqlearn/projects/).
| # | Topic | Starter notebook |
|---|---|---|
| 1 | Recommendation under drift | ipynb · Colab |
| 2 | Bayesian optimization | ipynb · Colab |
| 3 | Exploration in reinforcement learning | ipynb · Colab |
| 4 | Model arena | ipynb · Colab |
| 5 | Rating the players of a team game | ipynb · Colab |