SCIENCES PO
INTRODUCTORY AI COURSE · SESSION 5
From Raw Completion to Helpful Assistant
How does a model trained to predict next word become an assistant?
PRESENTED BY
Evan Dufraisse
9 August 2026
FROM LLM TO CHATBOT
SESSION 5
USER PROMPT
Can you tell me what is the capital of France?
GPT-2
Base language model - continues plausible text
ASSISTANT
Instruction-tuned model - answers the request
"It is called 'Neuverre'?" "I have no idea," replied Mr. Boughen, as if he had been discussing with the whip in the palace, and the truth of the matter was not.
"The capital of France is Paris."
FROM LLM TO CHATBOT
SESSION 5
CENTRAL QUESTION
Central Question
How a model trained to only complete text is turned into an assistant that follows requests?
FROM LLM TO CHATBOT
SESSION 5
Historical Transitions
FROM LLM TO CHATBOT
SESSION 5
The Earlier Fine-Tuning Paradigm (2018-2020)
FROM LLM TO CHATBOT
SESSION 5
The Earlier Fine-Tuning Paradigm (2018-2020)
FROM LLM TO CHATBOT SESSION 5 The Earlier Fine-Tuning Paradigm (2018-2020)
FROM LLM TO CHATBOT SESSION 5 Translate French to English French : Le chat dort sur le canapé. English : The cat is sleeping on the sofa. French : Nous partirons demain matin. English : We will leave tomorrow morning. French : Elle ne trouve plus ses clés . English : She can no longer find her keys. French : Il pleuvait lorsque je suis arrivé . English : <model completes> Informatics is the automated processing of data. Few-shot Few-shot learning is the ability of a model to perform a task after seeing only a small number of labeled examples. With few shots in its context , GPT-2 could learn some task without updating its parameters ! GPT-3 (2020) Demonstrate Strong Few-Shot Learning Abilities
FROM LLM TO CHATBOT
SESSION 5
FLAN & T0 (2021) Generalizing Beyond Seen Tasks
LLM
FROM LLM TO CHATBOT
SESSION 5
The Recipe of An Assistant: InstructGPT, ChatGPT (2022)
FROM LLM TO CHATBOT
SESSION 5
The Training Cake Recipe (2022)
Yann LeCun’s Cake Analogy (2016)
FROM LLM TO CHATBOT
SESSION 5
Pretraining: Predict the Next Token (The Cake)
FROM LLM TO CHATBOT
SESSION 5
Pretraining: Where the Data Come From?
Informatics is the automated processing of data.
Common Crawl
Common Crawl is a nonprofit project that regularly collects public webpages and releases them as large, open datasets.
FROM LLM TO CHATBOT
SESSION 5
Pretraining: Where the Data Come From?
FROM LLM TO CHATBOT
SESSION 5
Pretraining: Filtering the Data
Keep coherent, substantive text, remove pages that are malformed, repetitive, too short, etc…
FROM LLM TO CHATBOT
SESSION 5
Pretraining: Data Mixture
Sources are not used in their raw proportions: each is deliberately weighted to shape what the model learns.
FROM LLM TO CHATBOT
SESSION 5
Pretraining: How Much Text Is 300 Billion Tokens?
FROM LLM TO CHATBOT
SESSION 5
Pretraining: Evolution Across Years
FROM LLM TO CHATBOT
SESSION 5
Pretraining: Memorization… Overfitting?
NYT: How the U.S. Lost Out on iPhone Work (2012)
FROM LLM TO CHATBOT SESSION 5 Supervised Learning (The Icing) During supervised fine-tuning , we train the model on conversations containing instructions and appropriate answers . Each conversation follows a predefined chat template that separates the different messages. The template includes a system prompt defining the model’s intended behavior and role markers identifying the user and assistant messages. The model learns to interpret this structure and generate the assistant’s response.
FROM LLM TO CHATBOT SESSION 5 Supervised Learning (The Icing) During supervised fine-tuning , we train the model on conversations containing instructions and appropriate answers . Each conversation follows a predefined chat template that separates the different messages. The template includes a system prompt defining the model’s intended behavior and role markers identifying the user and assistant messages. The model learns to interpret this structure and generate the assistant’s response. Instruction Tuning represents only 0.01-0.05% of the total training tokens! ~1M conversations
FROM LLM TO CHATBOT SESSION 5
FROM LLM TO CHATBOT SESSION 5 This data comes from real user-assistant conversations, human created datasets or LLM-generated conversations.
FROM LLM TO CHATBOT SESSION 5 Supervised Learning (The Icing) DEMO
FROM LLM TO CHATBOT SESSION 5 LLM Weakness: Cannot Build Persistent Memories by Itself
FROM LLM TO CHATBOT SESSION 5 LLM Weakness: Cannot Build Persistent Memories by Itself An LLM’s episodic memory is the conversation context available to it . To simulate long-term memory , relevant information must be stored in an external database. The model cannot update its weights.
FROM LLM TO CHATBOT
SESSION 5
Reinforcement Learning Stage (The Cherry)
Informatics is the automated processing of data.
Reinforcement Learning
Reinforcement learning further aligns the model’s behavior with human intentions and preferences.
FROM LLM TO CHATBOT
SESSION 5
Reinforcement Learning: Humans Compare the Answers
Reinforcement Learning: Using an another LLM as a Judge
FROM LLM TO CHATBOT
SESSION 5
FROM LLM TO CHATBOT
SESSION 5
Reinforcement Learning: Learning a Reward Model
FROM LLM TO CHATBOT
SESSION 5
Reinforcement Learning: Optimizing the Language Model
Reward hacking: when a bias becomes the objective PROMPT Why do leaves change color in autumn? Answer in two sentences. INTENDED BEHAVIOR Concise and correct As daylight decreases, trees stop producing chlorophyll, so the green color fades. Yellow, orange and red pigments then become visible. Reward-model score 7 / 10 Actual quality: HIGH Preference data slightly favors longer answers RL optimization pushes the score higher LEARNED SHORTCUT Longer looks better A comprehensive explanation of autumn leaves ……….. Reward-model score 10 / 10 Actual quality: LOW The model maximizes what the reward model recognizes not what humans intended. FROM LLM TO CHATBOT SESSION 5
FROM LLM TO CHATBOT
SESSION 5
Reinforcement Learning: Verifiable Rewards
FROM LLM TO CHATBOT
SESSION 5
Hallucinations
FROM LLM TO CHATBOT
SESSION 5
Syncophancy – Excess Agreeableness
FROM LLM TO CHATBOT
SESSION 5
Compute Across Training Phases
Pretraining still dominates the compute budget. Instruction tuning and reinforcement learning usually represent less than a few percent, although this share is growing for reasoning models.
FROM LLM TO CHATBOT
SESSION 5
Estimated Electricity Cost of Training
FROM LLM TO CHATBOT
SESSION 5
Secret and Expensive Recipes…
Most training recipes remain secret
For most models, especially flagship models, we do not know exactly what data they were trained on or how they were trained.
This makes innovations difficult to reproduce or verify for two reasons:
•Replicating the training process is prohibitively expensive.
•The exact training recipes are not publicly available.
FROM LLM TO CHATBOT
SESSION 5
O
OPEN SOURCE / FULLY OPEN
The full stack is open
Study, reproduce, modify and share the model.
WHAT YOU GET
✓
Weights under open terms
✓
Training and inference code
✓
Data or detailed data information
✓
Reproducible training recipe
EXAMPLE
K2-Think, OLMo 2
Fully open release
W
OPEN WEIGHTS
Weights are released
Download, run and fine-tune—but the recipe may stay private.
WHAT YOU GET
✓
Model weights are downloadable
✓
Local inference and fine-tuning
?
Full training pipeline may be missing
?
Custom license may restrict use
EXAMPLE
Meta Llama 3.1
Community license
C
CLOSED SOURCE
Provider keeps control
Use the model through a product or API; you cannot host it.
WHAT YOU GET
✓
Access through an app or API
×
Weights are not available
×
Training code is not available
×
Training data is not disclosed
EXAMPLE
OpenAI GPT-4.1
Hosted API access
Open-source, Open-weights, Closed-source
FROM LLM TO CHATBOT
SESSION 5
Open-source, Open-weights, Closed-source
A platform on which AI actors share model weights and datasets.
FROM LLM TO CHATBOT
SESSION 5
FROM LLM TO CHATBOT
SESSION 5
Bigger Is Not the Whole Story
FROM LLM TO CHATBOT
SESSION 5
Efficiency in Architectures
Phase | Estimated electricity |
One GPT‑4o-scale training run | ≈43–54 GWh |
One average ChatGPT query | ≈0.3–0.34 Wh |
Inference at 1 billion queries/day | ≈110–124 GWh/year |
Estimated electricity consumption of ChatGPT per phases
With time, inference can become the greatest source of consumption
FROM LLM TO CHATBOT
SESSION 5
Efficiency in Architectures
FROM LLM TO CHATBOT
SESSION 5
Efficiency in Architectures
FROM LLM TO CHATBOT
SESSION 5
Running LLMs Locally on Your Laptop
FROM LLM TO CHATBOT
SESSION 5
How Do We Evaluate LLMs? Benchmarks
Example: a multi-benchmark model comparison
01 · WHAT A BENCHMARK DOES
A standardized test for model capability
Like a test set, it is meant to score a model on tasks that were not part of the evaluation itself.
02 · WHY ONE SCORE IS NOT ENOUGH
General-purpose models need a portfolio of tests
Benchmarks probe different capabilities: knowledge, culture, mathematics, common-sense reasoning, coding—and more.
READ SCORES AS A PROFILE, NOT A SINGLE RANKING.
Where Benchmarks Fall Short
01 · GOODHART’S LAW
When a measure becomes a target, it stops being a good measure.
At release, a benchmark can reveal real differences in underlying model performance.
Over time, teams patch weaknesses exposed by the test. The score rises—even when broader capability does not.
This behavior is often called “benchmaxxing.”
02 · DATA CONTAMINATION
A model may have seen the test before evaluation.
Pretraining uses enormous amounts of web data. Benchmark items can enter the corpus inadvertently.
A high score may then reflect memorization rather than generalization.
This concern has appeared in evaluations of GPT-3 and GPT-4o.
BENCHMARKS ARE EVIDENCE—NOT PROOF—OF BROAD CAPABILITY.
FROM LLM TO CHATBOT
SESSION 5
FROM LLM TO CHATBOT
SESSION 5
Wrapping-up
TRAINING PIPELINE
SYSTEMS & EVALUATION
PRETRAINING BUILDS THE BASE MODEL
Next-token prediction over filtered, mixed data learns broad language patterns.
SUPERVISED TUNING TEACHES THE ASSISTANT FORMAT
Curated instruction-response conversations teach the model how to follow requests.
PREFERENCE OPTIMIZATION SHAPES BEHAVIOUR
Human comparisons and reward signals steer helpfulness—but can introduce shortcuts.
MEMORY LIVES OUTSIDE THE WEIGHTS
Persistent state requires context management, databases, or other external systems.
OPENNESS AND EFFICIENCY ARE DESIGN CHOICES
Access, reproducibility, compute, and serving costs vary across model releases.
BENCHMARKS ARE EVIDENCE, NOT PROOF
Use multiple tests and check for gaming, contamination, and real-world performance.