SCIENCES PO
INTRODUCTORY AI COURSE · SESSION 3
AI for the Language
How does text become computation?
PRESENTED BY
Evan Dufraisse
27 August 2026
LANGUAGE AND LLMS
SESSION 4
What is a vector?
Informatics is the automated processing of data.
A Vector
A vector is an ordered table of numbers, that can be seen as coordinates into a space.
LANGUAGE AND LLMS
SESSION 4
What is an embedding?
Informatics is the automated processing of data.
An Embedding
An embedding is a vector of numbers representing an object.
Each dimension captures one of its characteristics.
Each dimension captures one of its characteristics.
LANGUAGE AND LLMS
SESSION 4
Vector: Data or Learnable parameters?
Vectors/Embeddings are just tables of numbers, those numbers can be learned or not, it doesn’t matter.
LANGUAGE AND LLMS
SESSION 4
Vector: Data or Learnable parameters?
Vectors/Embeddings are just tables of numbers, those numbers can be learned or not, it doesn’t matter.
LANGUAGE AND LLMS
SESSION 4
Quiz - Embeddings of Adjectives (NRC VAD Lexicon)
LANGUAGE AND LLMS
SESSION 4
Quiz - Embeddings of Adjectives (NRC VAD Lexicon)
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013) - Can we discover automatically semantic dimensions?
LANGUAGE AND LLMS
SESSION 4
•Embedding size: 100-300 dimensions
•Training corpus: 5M to 100B words
•Vocabulary: thousands to millions of distinct words
Word2Vec (2013) - Can we discover automatically semantic dimensions?
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
How do machines learn? Several Paradigms
Human learning | Machine learning |
Someone shows you examples of what are cats and dogs. | Supervised learning: learn from examples paired with the correct answer. |
Without being taught their names, you notice that birds seem to form different groups. | Unsupervised learning: discover patterns or groups in unlabeled data. |
You notice that coffee after 4 p.m. harms your sleep, so you stop drinking it. | Reinforcement learning: learn which actions work through their consequences. |
Knowing tennis helps you learn padel or badminton faster. | Transfer learning: reuse knowledge learned for one task on a related task. |
You hide a word in a sentence and try to reconstruct it from context. | Self-supervised learning: create the learning task from the data itself. |
LANGUAGE AND LLMS
SESSION 4
Self-Supervised Learning
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013) – Semantic and Syntactic Directions
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013) – Biases
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
Word2Vec (2013) – Limits
Limits:
-Polysemy (.i.e “I sat by the river bank”)
-Out-of-vocabulary (can’t get an embedding for a word not in the model)
-Unordered words: Poor handling of sentences, negation, etc…
LANGUAGE AND LLMS
SESSION 4
From Word2Vec to a Language Model
LANGUAGE AND LLMS
SESSION 4
What is a Language Model ?
Informatics is the automated processing of data.
A Language Model
A language model is a system that assigns probabilities to sequences of words, or equivalently, predicts how likely each next word is given the words before it.
LANGUAGE AND LLMS SESSION 4 Word2Vec (2013)
LANGUAGE AND LLMS
SESSION 4
Predict the next word/token with a language model
LANGUAGE AND LLMS
SESSION 4
Predict the next word/token with a language model
LANGUAGE AND LLMS
SESSION 4
Predict the next word/token with a language model
LANGUAGE AND LLMS
SESSION 4
Predict the next word/token with a language model
LANGUAGE AND LLMS
SESSION 4
Predict the next word/token with a language model
LANGUAGE AND LLMS
SESSION 4
DEFINITION
What is tokenization?
Tokens have become the currency of modern AI models, the unit by which companies measure and charge for their use.
LANGUAGE AND LLMS SESSION 4 Tokenization changes the units a model sees WORD-LEVEL (Word2Vec) 3 words Treats each complete word as one unit. SUBWORD (OpenAI GPT3) 5 tokens Splits words into reusable pieces. He was unhelpful Subword tokenization trades a few extra units for a vocabulary that generalizes.
LANGUAGE AND LLMS SESSION 4 Subwords prevent the unknown-word problem “Ultracrepidarianism” WORD-LEVEL (Word2Vec) SUBWORD (OpenAI GPT3) <UNKNOWN WORD> The model reuses familiar pieces to encode a word it has never seen as a whole.
LANGUAGE AND LLMS
SESSION 4
DEFINITION
What is tokenization?
It allows the model to:
•Leverage shared stems, prefixes, and suffixes
•Handle rare or previously unseen words more effectively
Informatics is the automated processing of data.
DEFINITION
Tokenization
Tokenization is the process of breaking text into smaller units, called tokens, that a model can process and predict.
LANGUAGE AND LLMS
SESSION 4
DEFINITION
What is tokenization?
The length of a sentence is not necessarily indicative of the number of tokens
Informatics is the automated processing of data.
DEFINITION
Tokenization
Tokenization is the process of breaking text into smaller units, called tokens, that a model can process and predict.
LANGUAGE AND LLMS
SESSION 4
DEFINITION
What is tokenization?
• With GPT-3’s 50K-token vocabulary, French text can cost more than twice as much to process as equivalent English text.
• GPT-5’s 200K-token vocabulary significantly reduces this disparity.
LANGUAGE AND LLMS
SESSION 4
What is tokenization?
From a table of words with Word2Vec to a table of tokens
LANGUAGE AND LLMS
SESSION 4
What is tokenization?
From a table of words with Word2Vec to a table of tokens
LANGUAGE AND LLMS
SESSION 4
Attention is all you need (2017) – Only applied to automatic translation
https://arxiv.org/pdf/1706.03762
LANGUAGE AND LLMS
SESSION 4
Encoders can create strong representation of each token and of the sentence
Decoder are excellent in generating text based on previous words.
LANGUAGE AND LLMS
SESSION 4
The Transformer Encoder-Decoder was first presented in the context of machine translation.
Encoder: Good understanding of the input sentence
Decoder: Generate the translation
In practice we divided the architecture in its two parts.
LANGUAGE AND LLMS
SESSION 4
The Transformer Encoder-Decoder was first presented in the context of machine translation.
Encoder: Good understanding of the input sentence
Decoder: Generate the translation
LANGUAGE AND LLMS
SESSION 4
LANGUAGE AND LLMS
SESSION 4
Encoders can create strong representation of each token and of the sentence
Decoder are excellent in generating text based on previous words.
LANGUAGE AND LLMS
SESSION 4
Encoders can create strong representation of each token and of the sentence
Because of that, they are the adequate architecture to classify the words a sequence, or create embeddings of sentences.
Strengths of Encoders
LANGUAGE AND LLMS
SESSION 4
BERT (2018) - Encoder
Widely known Encoder
LANGUAGE AND LLMS
SESSION 4
GPT-1 (2018) - Decoder
LANGUAGE AND LLMS
SESSION 4
GPT-2 (2019) - Decoder
LANGUAGE AND LLMS SESSION 4 GPT-2 (2019) - Decoder Translate French to English French : Le chat dort sur le canapé. English : The cat is sleeping on the sofa. French : Nous partirons demain matin. English : We will leave tomorrow morning. French : Elle ne trouve plus ses clés . English : She can no longer find her keys. French : Il pleuvait lorsque je suis arrivé . English : <model completes> Informatics is the automated processing of data. Few-shot Few-shot learning is the ability of a model to perform a task after seeing only a small number of labeled examples. With few shots in its context , GPT-2 could learn some task without updating its parameters !
LANGUAGE AND LLMS
SESSION 4
How a Transformer Decoder Predicts the Next Token
LANGUAGE AND LLMS
SESSION 4
The Operations of the Attention Mechanism
LANGUAGE AND LLMS
SESSION 4
The Operations of the Attention Mechanism
LANGUAGE AND LLMS
SESSION 4
Attention Mechanisms - Internals
LANGUAGE AND LLMS
SESSION 4
GPT-2 Demo
LANGUAGE AND LLMS
SESSION 4
Temperature Sampling
Informatics is the automated processing of data.
Sampling Temperature
Sampling temperature is a parameter that controls randomness when selecting the LLM’s next token.
LANGUAGE AND LLMS
SESSION 4
Why was the Transformer a Game Changer? – Two Modes
•Training: The model processes an existing sentence and learns to predict its tokens.
•Inference: The model receives a prompt and generates a new sentence token by token.
LANGUAGE AND LLMS
SESSION 4
A simple 1-layer RNN (Recurrent Neural Network)
Main point: It’s not because generation of language is sequential that the training needs to be sequential!
Model` | Training | Inference |
RNN | Sequential | Sequential |
Transformer | Parallel | Sequential |
LANGUAGE AND LLMS
SESSION 4
Why was the Transformer a Game Changer?
LANGUAGE AND LLMS
SESSION 4
DEFINITION
What is tokenization?
Tokens have become the currency of modern AI models, the unit by which companies measure and charge for their use.
LANGUAGE AND LLMS
SESSION 4
A Game changer at what cost?
LANGUAGE AND LLMS
SESSION 4
A Game changer at what cost?
LANGUAGE AND LLMS
SESSION 4
Wrapping-up
REPRESENTATION
GENERATION & SCALE
EMBEDDINGS GIVE TOKENS GEOMETRY
Learned vectors place related meanings and directions near one another.
TOKENIZATION DEFINES THE UNITS
Subwords let finite vocabularies represent rare and unfamiliar words.
ATTENTION BUILDS CONTEXT
Each token combines information from the other tokens that matter.
ENCODERS REPRESENT; DECODERS GENERATE
Encoders build understanding; decoders produce output one token at a time.
GENERATION IS CONTROLLED SAMPLING
Next-token probabilities drive text; temperature changes diversity.
SCALE BRINGS CAPABILITY AND COST
Transformers improve with data and compute, but demand significant resources.