What is it?
Chunking divides content into smaller units called chunks.
Chunks may be paragraphs, sections, fixed token ranges, or semantic blocks. The purpose is to make retrieval more manageable and relevant.
08 · Embeddings, Search & RAG · Topic 01
What is Chunking?
Chunking splits large documents or data into smaller pieces that are easier to search and retrieve.
Think of it like...
Think of cutting a large textbook into index cards.
When someone asks a question, you can retrieve only the cards that contain the useful section instead of carrying the whole textbook into the answer.
How it works
1 · DocumentStart with contentA document or other source is ready to prepare.
2 · StrategyChoose boundariesThe system chooses fixed, structural, or semantic boundaries.
3 · SplitCreate chunksThe content is divided into smaller retrievable pieces.
4 · MetadataKeep useful contextTitles, source names, positions, or other metadata may be attached.
5 · IndexPrepare for searchChunks can be embedded and added to a search index.
6 · RetrieveFind relevant piecesA search system returns chunks related to the user's question.
Real-world examples
Everyday example · Textbook
Input: a long textbook
System: splits it by useful sections
Output: a few relevant sections for a question
Business example · Policy search
Input: a company handbook
System: chunks policies with titles and page metadata
Output: the relevant policy passage for an employee question
What it is NOT
Chunk ≠ Token
Chunk: a larger retrieval unit such as a paragraph or section.
Token: a model-readable text unit.
Chunking ≠ Embedding
Chunking: divides content.
Embedding: represents content as numbers for comparison.
Chunking ≠ Summarization
Chunking: preserves pieces of the source.
Summarization: rewrites content in a shorter form.
Related concepts
Document → Chunking → Embeddings → Index → Retrieval → Answer
Explore next
Remember this
Chunking breaks large content into retrievable pieces before search or indexing.
Video
Video
A video for this topic is not available yet.