← LLMs & Transformers
02 · LLMs & Transformers

What is Pre-training?

Pre-training is the broad initial training stage that teaches a model general patterns from large amounts of data.

What is it?

Pre-training is the broad initial training stage that teaches a model general patterns from large amounts of data.

For a language model, this often means predicting missing or next tokens repeatedly. The result is a base model that can later be adapted.

Think of it like...

Think of it like a general education before specialist training. A student studies many subjects before choosing one job.

General education before specialist training
Broad data creates a reusable starting point.

How it works

Large datasets become training examples. Prediction errors update parameters until a base model emerges.

1 · Training dataCollect broad materialText, code, images, or other data provide examples.
2 · TokenizationMake unitsLanguage is converted into tokens the model can process.
3 · PredictionGuess the next partThe model predicts a missing or next token.
4 · ErrorCompare with targetThe prediction is measured against the training example.
5 · UpdateAdjust parametersThe model changes internal values and repeats the loop.
6 · Base modelFinish the broad stageThe result can be prompted or adapted for applications.

Real-world examples

Language training · Next token

Input: The trader closed the position because it was

System: The model predicts likely next tokens and compares them

Output: A continuation such as risky

General capability

Input: A large mixed dataset

System: Repeated prediction teaches broad patterns

Output: A reusable base model

Model building

Input: A saved checkpoint

System: Training preserves learned parameter values

Output: A model ready for adaptation

What it is NOT

Pre-training ≠ Fine-tuning

Pre-training: Broad initial training

Fine-tuning: Later targeted training

Pre-training ≠ Prompting

Pre-training: Changes model parameters through training

Prompting: Changes runtime instructions

Pre-training ≠ RAG

Pre-training: Builds learned capability before deployment

RAG: Retrieves outside information at query time

Remember this

Pre-training teaches broad patterns from large amounts of data before task-specific adaptation.

Independent video · Azure Neural voice

Video

visual explainer
broad data · prediction · base modelMP4 · H.264 + AAC
↑