Attention is a mechanism that lets a neural network weigh which pieces of information are most relevant to one another.
For each part of the input, attention calculates how strongly it should use information from other parts.
The result is a context-aware representation: each token can carry information about relationships that matter for the current task.
When reading “The trader closed the position because it was risky,” you look at other words to interpret “it” and decide which relationship matters most.
Attention does this using mathematical weights, not human awareness.
Connect “it” with the risky position in a sentence.
Relate words across a sentence while producing a translation.
Use relevant earlier tokens when generating the next token.
Relate image patches to understand the whole image.
It is a mathematical operation over representations, not awareness or focus.
Attention is a mechanism. A Transformer is an architecture that uses attention and other components.
Attention weighs relationships among available tokens. A context window is the amount of input a model can process.
Attention is a key mechanism inside Transformer layers, which build contextual representations from token relationships.
Explore next
Attention mathematically weighs relationships between pieces of information.
No verified video asset is available for this topic yet.