03 · Tokens, Context & Inference · Topic 03

What is Latency?

Latency is the time between an AI request and the result becoming available.

What is it?

Latency is the time an AI system takes to return a result.

It can include time to receive the request, prepare context, run the model, generate tokens, and send the response. A system can have low latency even when its total work is large if the work is well optimized.

Think of it like...

Think of latency like waiting for a reply on a phone call.

The first pause is like time to first token. The full conversation is like the time until the complete response arrives. The analogy covers waiting time, not the technical causes.

How it works

1 · RequestSend the inputThe application sends a prompt and context.

2 · PrepareBuild the inputThe system loads context and prepares tokens.

3 · StartReturn first outputThe model begins generating a result.

4 · GenerateProduce more tokensThe response grows token by token.

5 · FinishReturn the resultThe application receives the complete response.

Real-world examples

Everyday example · Chat reply

Input: a short user question

System: uses a small context and streams output

Output: the first words appear quickly

Business example · Long document

Input: a question plus a large document

System: processes more context before generation

Output: a longer wait before the answer starts

What it is NOT

Latency ≠ Throughput

Latency: time for one request or result.

Throughput: work completed over a period of time.

Latency ≠ Model Quality

Latency: how long the system takes.

Model quality: how useful or accurate the result is.

Latency ≠ Only Model Speed

Latency: includes system, network, context, and generation time.

Model speed: only one part of the total wait.

Remember this

Latency is the time users wait for an AI result, from request to response.

Independent explainer

Video

visual explainer