03 · Tokens, Context & Inference · Topic 04

What are Tokens per Second?

Tokens per second measures how quickly a model generates output tokens.

What is it?

Tokens per second, or TPS, is the number of output tokens a model generates in one second.

It is a generation-speed measure. Token counts are not the same as words, so TPS cannot be read as words per second. Actual user experience also depends on time to first token and other system delays.

Think of it like...

Think of tokens per second like a printer's pages per minute.

A faster printer produces pages sooner, but the time to start, the document size, and the connection also matter. TPS describes generation speed, not the whole request experience.

How it works

1 · InputReceive the requestThe model receives instructions and context.

2 · PredictChoose the next tokenThe model calculates the next output unit.

3 · EmitReturn one tokenThe application receives generated output.

4 · RepeatContinue generationThe loop runs for each next token.

5 · MeasureCount output speedGenerated tokens are divided by generation time.

Real-world examples

Everyday example · Short answer

Input: a simple question

System: generates a short response at 40 TPS

Output: the answer appears and finishes quickly

Business example · Long report

Input: a request for a detailed report

System: generates many tokens at 20 TPS

Output: the report takes longer even with steady speed

What it is NOT

TPS ≠ Words per Second

TPS: counts model tokens.

Words per second: counts words, which do not map directly to tokens.

TPS ≠ Time to First Token

TPS: measures generation after output starts.

Time to first token: measures the initial wait.

TPS ≠ Total Response Time

TPS: is one generation-speed metric.

Total response time: also includes network, context, and setup time.

Remember this

Tokens per second measures generation speed, not the whole waiting time.

Independent explainer

Video

visual explainer