RAG chunk¶

Chunking splits raw documents from the ingest stage into bounded units of text that the embed stage can vectorise. Helix ships ten chunking strategies plus a set of nine document-genre templates. Each strategy implements chunk.Strategy.Chunk(*rag.Document) []*rag.Chunk. All strategies are dispatchable via chunk.GetStrategy(name, chunkSize, overlap), which is what the DSL processor rag-chunk calls.

The pipeline shape is ingest -> chunk -> embed. Every *rag.Chunk produced here is later fed to an embedder (pkg/rag/embed) and stored in a vector backend (pkg/rag/store). Empty chunks are dropped by ChunkProcessor because downstream embedders reject empty input.

Strategies¶

fixed — split by fixed character count¶

rag:
  chunk:
    type: fixed
    size: 512
    overlap: 50

Slices the text on byte offsets — fastest and most predictable. Use it when you need uniform input sizes and don't care about respecting word or sentence boundaries (e.g. embedding logs, dense plain-text). Source: pkg/rag/chunk/chunk.go.

sentence — group whole sentences up to a size cap¶

rag:
  chunk:
    type: sentence
    size: 1000
    overlap: 100

Splits at .!? followed by whitespace, then groups whole sentences. Always breaks at sentence boundaries — never mid-word. There are two implementations: chunk.go's SentenceStrategy (used by GetStrategy) plus sentence.go's SentenceChunker, which adds paragraph-aware grouping (ParagraphSep, default "\n\n") and trims whitespace. Use this for prose, articles, transcripts. Source: pkg/rag/chunk/chunk.go, pkg/rag/chunk/sentence.go.

recursive — split on a hierarchy of separators¶

rag:
  chunk:
    type: recursive
    size: 1000
    overlap: 100

Tries separators in order: "\n\n", "\n", ". ", " ", "". Falls back to the next-finer separator only if a chunk is still over size. The default — sensible for almost any prose, code, or mixed input. There are two variants: chunk.go's RecursiveStrategy (no overlap) and recursive.go's RecursiveChunker (post-hoc overlap pass). Source: pkg/rag/chunk/chunk.go, pkg/rag/chunk/recursive.go.

token — split by estimated token budget¶

rag:
  chunk:
    type: token
    size: 512
    overlap: 50

Estimates token count as words * 1.3 (rough English approximation, no tokenizer dependency). Use this when chunks must fit a model context budget — picks a size matching the LLM's per-call window. Faster than shipping a real tokenizer. Source: pkg/rag/chunk/token.go.

semantic — split where vocabulary shifts¶

rag:
  chunk:
    type: semantic
    size: 1024
    similarity_threshold: 0.3

Groups consecutive sentences whose word sets share enough vocabulary, breaks when similarity drops. The default in GetStrategy uses Jaccard similarity (no embedder, fast). A separate SemanticChunker in semantic.go uses real embeddings (cosine similarity, configurable embedder) — pass it to chunk.SemanticChunker.ChunkWithContext. Use when topic boundaries matter more than uniform size. Source: pkg/rag/chunk/chunk.go, pkg/rag/chunk/semantic.go.

code — split on language-aware declaration boundaries¶

rag:
  chunk:
    type: code
    size: 1500
    overlap: 100

Default language is go (via GetStrategy). Other supported: python, javascript/js/typescript/ts, java, rust. Splits at language-specific declaration keywords (func, def, class, function, fn, public, etc.). Each declaration stays as one block; oversized blocks fall back to recursive splitting. Use for source-code ingestion (best to also set language via the YAML). Source: pkg/rag/chunk/code.go.

markdown — split by heading hierarchy¶

rag:
  chunk:
    type: markdown
    size: 1000
    overlap: 100
    retain_headers: true

Parses ATX-style headings (# … ######) into a heading stack. Each section becomes a chunk (or, if oversized, is paragraph-split first then hard-split). Use for technical docs, README files, knowledge bases — retain_headers: true gives chunks self-contained context. Source: pkg/rag/chunk/markdown.go.

html — split on block-level element boundaries¶

rag:
  chunk:
    type: html
    size: 1000

Splits at opening tags <div>, <section>, <article>, <p>, <h1>-<h6>, <li>, <tr>, <blockquote>, <pre>. Pure-regex, no external HTML parser. Use for scraped HTML where structure matters but markdown conversion isn't desirable. Source: pkg/rag/chunk/html.go.

sliding_window — overlapping fixed-stride windows¶

rag:
  chunk:
    type: sliding_window
    size: 500
    stride: 250
    unit: chars

Generates windows at fixed offsets, similar to fixed but with explicit stride. Default in GetStrategy uses chunkSize/2 stride (50% overlap) and char units. Use when retrieval should be robust to where a relevant span sits inside a chunk. Source: pkg/rag/chunk/sliding_window.go.

hierarchical — multi-level parent-child chunks¶

rag:
  chunk:
    type: hierarchical
    levels: [2048, 512, 128]
    overlap: 50

Produces chunks at every level with chunk_level, parent_id, and children metadata. Each level is fixed-size split of the original; a child is mapped to a parent by text containment. Designed for auto-merging retrieval (retrieve.AutoMergingRetriever, retrieve.RetrieverAutoMerging): you index children, retrieve children that match, then collapse to the parent for context. There's also a ParentChildStrategy in chunk.go for the simpler two-level case (parent + child). Source: pkg/rag/chunk/hierarchical.go, pkg/rag/chunk/chunk.go.

Document-genre templates¶

Beyond the ten general strategies, chunk.GetTemplate(name, maxSize, overlap) returns nine domain-aware strategies that split on genre-specific boundaries and tag each chunk with chunk_template:

Source: pkg/rag/chunk/templates.go.

Choosing between them¶

When the document is recognisably one of the nine genres above, prefer the template — it preserves natural-unit metadata that the retriever and synthesizer can exploit.

See also¶

• ingest — produces the *rag.Document that chunk consumes.
• embed — consumes the []*rag.Chunk this stage emits.
• retrieve — auto_merging / parent retrievers rely on the parent/child metadata produced by hierarchical and parent-child.
• RAG guide — patterns, recipes.