KV cache stores attention information from earlier tokens so generation can continue more efficiently.
KV cache is temporary memory used during generation to reuse attention information from earlier tokens.
In a Transformer, attention uses key and value states. Keeping those states avoids recomputing the same earlier context for every new token. The cache uses memory and grows as the sequence grows.
Think of KV cache like keeping notes while reading a long document.
When you continue reading, you use the notes instead of rereading every earlier page. The analogy explains reuse during one generation, not permanent memory across separate requests.
Input: a conversation that keeps growing
System: reuses earlier attention states during generation
Output: less repeated computation for each new token
Input: a prompt that needs a long response
System: updates the cache as tokens are generated
Output: continued generation with lower repeated work
KV cache: runtime data used to speed up attention during generation.
Context window: the amount of information the model can work with.
KV cache: usually belongs to one active generation.
Long-term memory: information stored for later use across interactions.
KV cache: temporary request-specific states.
Model parameters: learned values that are part of the model.
Explore next
KV cache reuses earlier attention states to make ongoing generation more efficient.