Skip to content

RAG Pipeline Guide

Give your AI agents access to real knowledge. Helix provides a complete Retrieval-Augmented Generation (RAG) pipeline as declarative YAML steps — from document ingestion through chunking, embedding, storage, retrieval, postprocessing, and synthesis. Each stage is a separate step that can be composed, customized, and extended independently.

Table of Contents

Pipeline Overview

A full RAG pipeline flows through these stages:

Ingest → Chunk → Embed → Store → Retrieve → Postprocess → Synthesize

Additional stages for advanced use cases:

Extract (metadata) ─┐
Cache (embeddings)  ─┤
Graph Build/Query   ─┤
Eval (quality)      ─┘

Top-Level pipelines: Primitive

RAG flows can also be defined as top-level pipelines: primitives, which provide a simplified syntax focused on data processing:

# Top-level pipelines: primitive (recommended for standalone RAG/ETL)
pipelines:
  - id: doc-search
    source: file://./docs
    steps:
      - rag-chunk:
          strategy: semantic
          threshold: 0.5
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: pgvector
          connection: "postgres://user:pass@db:5432/rag"
      - rag-retrieve:
          strategy: hybrid
          top_k: 10
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o
    output: text

Use pipelines: when the flow is a self-contained data processing pipeline. Use workflows: with RAG steps when the RAG pipeline is part of a larger integration (e.g., triggered by Kafka, with results routed to another system).

RAG as a tools: Primitive

For RAG endpoints that agents or MCP clients call, use tools: — this adds mandatory input/output schemas and input validation:

tools:
  - id: kb-search
    description: "Search the knowledge base"
    input:
      query: { type: string, description: "Search query", required: true }
      top_k: { type: integer, description: "Max results", default: 5 }
    output:
      answer: { type: string, description: "Synthesized answer" }
      sources: { type: array, description: "Source documents" }
    steps:
      - rag-retrieve:
          strategy: hybrid
          top_k: 5
      - rag-postprocess:
          steps:
            - type: cohere-rerank
              params:
                top_n: "3"
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o-mini

Agents reference this tool by ID: tools: [{ name: kb-search, type: tool, tool_id: kb-search }]

Minimal RAG Pipeline

Using the traditional workflows: syntax:

workflows:
  - id: simple-rag
    trigger:
      uri: http-server://0.0.0.0:8080/api/ask
    steps:
      - rag-ingest:
          parser: text
      - rag-chunk:
          strategy: fixed
          chunk_size: 512
          overlap: 50
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memory
          collection: docs
      - rag-retrieve:
          strategy: simple
          top_k: 5
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o

Stage 1: Ingestion

The rag-ingest step parses raw data into documents. Helix includes 10 readers covering text, structured data, markup, and binary formats.

Reader Types

Reader Extensions Description
text .txt Plain text passthrough
json .json JSON document parsing
pdf .pdf PDF text extraction (pure Go, no external deps)
html .html, .htm HTML tag stripping with structure preservation
csv .csv, .tsv Tabular data with header support
markdown .md Markdown with frontmatter and heading-based splitting
xml .xml XML element extraction
docx .docx Microsoft Word document parsing
starlang .star Starlang script-based parsing
directory (folder) Auto-detect reader by file extension

Plain Text

- rag-ingest:
    parser: text

PDF Documents

- rag-ingest:
    reader: pdf

Extracts text from PDF binary streams. Metadata includes pdf_version, stream_count, image_count, and title when available.

HTML Content

- rag-ingest:
    reader: html

Strips <script>, <style>, <nav>, and <footer> tags. Converts headings to markdown-like formatting. Decodes HTML entities. Extracts <title> into metadata.

CSV / TSV Files

- rag-ingest:
    reader: csv
    delimiter: ","
    has_header: true
    rows_per_chunk: 100
Parameter Default Description
delimiter , Field delimiter character
has_header true First row contains column names
rows_per_chunk 0 Rows per document (0 = all in one)

With has_header: true, each row is formatted as column: value pairs. With rows_per_chunk: 100, large CSVs are split into multiple documents.

Markdown

- rag-ingest:
    reader: markdown
    split_by_heading: true
    heading_level: 2
    strip_formatting: false
Parameter Default Description
split_by_heading false Create separate documents per heading section
heading_level 1 Heading level to split at (1-6)
strip_formatting false Remove markdown formatting from text

When split_by_heading: true, each section becomes a separate document with the heading as metadata. YAML frontmatter (between --- delimiters) is extracted into document metadata.

XML

- rag-ingest:
    reader: xml
    row_tag: item
    text_tags:
      - title
      - description
Parameter Default Description
row_tag (none) Repeating element tag; each match becomes a document
text_tags (all) Extract only these tags' text content

Directory Reader

- rag-ingest:
    reader: directory
    path: ./documents
    recursive: true
    extensions:
      - .md
      - .txt
      - .pdf
Parameter Default Description
recursive false Walk subdirectories
extensions (all supported) Filter by file extensions

The directory reader auto-detects the file format by extension and dispatches to the appropriate reader (text, JSON, PDF, HTML, CSV, Markdown, XML, DOCX).

Stage 2: Chunking

The rag-chunk step splits documents into smaller chunks for embedding. Helix provides 10 chunking strategies ranging from simple fixed-size to embedding-aware semantic splitting.

Strategy Overview

Strategy Best For Key Feature
fixed General purpose Consistent chunk sizes
token Token-aware splitting Respects token boundaries
sentence Natural language Preserves sentence boundaries
recursive Mixed content Hierarchical separator fallback
semantic High-quality RAG Embedding-based breakpoints
code Source code Language-aware function/class boundaries
markdown Documentation Heading hierarchy awareness
html Web content Block element boundaries
sliding_window Overlapping coverage Fixed stride with overlap
hierarchical Auto-merging retrieval Multi-level parent-child chunks

Fixed Size (Default)

- rag-chunk:
    strategy: fixed
    chunk_size: 512
    overlap: 50

Splits text at character boundaries with configurable overlap.

Token-Based

- rag-chunk:
    strategy: token
    chunk_size: 256
    overlap: 25

Splits respecting token (word) boundaries rather than raw characters.

Sentence Splitter

- rag-chunk:
    strategy: sentence
    chunk_size: 1000
    overlap: 100

Detects sentence boundaries (., !, ? followed by whitespace) and groups sentences until the chunk size is reached. Respects paragraph boundaries (\n\n) as natural break points.

Parameter Default Description
chunk_size 1000 Target chunk size in characters
overlap 0 Overlap between chunks (max: chunk_size/4)

Recursive Character Splitter

- rag-chunk:
    strategy: recursive
    chunk_size: 1000
    overlap: 100
    separators:
      - "\n\n"
      - "\n"
      - ". "
      - " "
      - ""

Tries splitting by each separator in order. If a chunk is still too large after splitting by the first separator, it recursively splits with the next separator. This preserves the most meaningful boundaries possible.

Parameter Default Description
chunk_size 1000 Target chunk size
overlap 0 Overlap (max: chunk_size/4)
separators ["\n\n", "\n", ". ", " ", ""] Separator hierarchy (tried in order)

Semantic Chunker

- rag-chunk:
    strategy: semantic
    threshold: 0.5
    chunk_size: 2048
    overlap: 0

Uses an embedding model to compute cosine similarity between adjacent sentences. Creates chunk breaks where similarity drops below the threshold. Produces chunks with natural semantic boundaries.

Parameter Default Description
threshold 0.5 Similarity threshold (0.0-1.0); lower = larger chunks
chunk_size 2048 Maximum chunk size

Requires an embedder to be configured (uses the workflow's rag-embed provider). Falls back to Jaccard similarity if embedding fails.

Code Splitter

- rag-chunk:
    strategy: code
    language: go
    chunk_size: 1500
    overlap: 100

Splits source code at language-specific declaration boundaries:

Language Split Points
go func, type ... struct, type ... interface
python def, class, async def
javascript function, class, const ... = (, export
java public, private, protected, class, interface
rust fn, pub fn, struct, impl, enum, trait
generic Blank lines + indentation changes

Falls back to recursive character splitting if individual code blocks exceed chunk_size.

Markdown Splitter

- rag-chunk:
    strategy: markdown
    chunk_size: 1000
    overlap: 50
    retain_headers: true

Splits at heading boundaries (# through ######), respecting heading hierarchy. Small subsections under the same parent heading stay together.

Parameter Default Description
chunk_size 1000 Maximum chunk size
overlap 0 Overlap (max: chunk_size/4)
retain_headers false Prepend parent headings to each chunk for context

When retain_headers: true, each chunk includes its parent heading chain:

# User Guide
## Authentication
### JWT

This section covers JWT setup...

Becomes a chunk with content: # User Guide\n## Authentication\n### JWT\nThis section covers JWT setup...

HTML Splitter

- rag-chunk:
    strategy: html
    chunk_size: 1000
    tags:
      - div
      - section
      - article
      - p

Splits at block-level HTML tag boundaries using regex-based tag matching (no external dependencies).

Parameter Default Description
chunk_size 1000 Maximum chunk size
tags div, section, article, p, h1-h6, li, tr, blockquote, pre Tags to split on

Sliding Window

- rag-chunk:
    strategy: sliding_window
    window_size: 500
    stride: 250
    unit: words

Produces overlapping fixed-size windows. Every position in the text is covered by at least one chunk.

Parameter Default Description
window_size 500 Window size
stride window_size/2 Step size between windows
unit chars Unit of measurement: chars or words

Hierarchical Chunker

- rag-chunk:
    strategy: hierarchical
    levels:
      - 2048
      - 512
      - 128
    overlap: 50

Creates chunks at multiple granularity levels with parent-child relationships. Used together with the auto_merging retrieval strategy -- if enough child chunks match a query, the parent chunk is returned instead for better context.

Parameter Default Description
levels (required) Chunk sizes per level, descending (e.g., [2048, 512, 128])
overlap 0 Overlap between chunks at each level

Each chunk's metadata includes chunk_level, parent_id, and children for tree traversal.

Stage 3: Embedding

The rag-embed step converts text chunks into vector embeddings.

Supported Providers

Provider Default Model Dimensions Batch Size
openai text-embedding-3-small 1536 2048
ollama nomic-embed-text 768 1
mistral mistral-embed 1024 512
vertex / gemini text-embedding-004 768 100
bedrock amazon.titan-embed-text-v2:0 1024 1
jina jina-embeddings-v3 1024 500
huggingface BAAI/bge-large-en-v1.5 1024 64
nomic nomic-embed-text-v1.5 768 400
cohere embed-v4.0 1024 96
voyage voyage-3 1024 128

Basic Usage

- rag-embed:
    provider: openai
    model: text-embedding-3-small

Custom Endpoint

- rag-embed:
    provider: openai
    model: text-embedding-3-large
    api_key: "${env:OPENAI_API_KEY}"
    base_url: "https://my-proxy.example.com/v1"

Local Embeddings

- rag-embed:
    provider: ollama
    model: nomic-embed-text
    base_url: "http://localhost:11434"

Stage 4: Vector Storage

The rag-store step persists embeddings in a vector database. Helix supports 10 vector store backends.

Store Overview

Store Type Best For
memory In-memory Testing, small datasets (<100K vectors)
pgvector PostgreSQL extension Teams already on PostgreSQL
redis Redis module Low-latency, caching-oriented
chromadb Dedicated vector DB Simple setup, local development
opensearch Search engine AWS ecosystem, full-text + vector
milvus Dedicated vector DB Large-scale production
pinecone Managed SaaS Zero-ops, serverless
qdrant Dedicated vector DB Advanced filtering, Rust performance
elasticsearch Search engine Existing Elastic infrastructure
weaviate Dedicated vector DB Multi-modal, GraphQL API
memx Unified search+SQL Dual-mode (ES API + PostgreSQL), hybrid search, high throughput

In-Memory Store

- rag-store:
    store: memory
    collection: my-docs

Pure Go brute-force cosine similarity. No external dependencies. Thread-safe. Ideal for testing and prototyping.

PostgreSQL (pgvector)

- rag-store:
    store: pgvector
    connection: "postgres://${DB_USER}:${DB_PASS}@localhost:5432/mydb"
    collection: documents
    dimensions: 1536
Parameter Default Description
connection (required) PostgreSQL connection string
collection vector_store Table name
dimensions 1536 Vector dimensions

Auto-creates the table and enables the vector extension. Uses <=> (cosine distance) for search. Batch upserts with ON CONFLICT handling.

- rag-store:
    store: redis
    connection: "redis://localhost:6379"
    collection: doc-vectors
    dimensions: 1536
Parameter Default Description
connection (required) Redis connection string
collection vector_idx Index name
dimensions 1536 Vector dimensions

Requires the RediSearch module. Supports FLAT (exact) and HNSW (approximate) algorithms. Vectors stored as float32 little-endian bytes.

ChromaDB

- rag-store:
    store: chromadb
    connection: "http://localhost:8000"
    collection: my-collection
Parameter Default Description
connection http://localhost:8000 ChromaDB server URL
collection (required) Collection name

Pure HTTP REST client. Auto-creates the collection. Distance scores converted to similarity scores.

OpenSearch

- rag-store:
    store: opensearch
    connection: "https://opensearch.example.com:9200"
    collection: documents
    dimensions: 1536
    username: admin
    password: "${env:OPENSEARCH_PASSWORD}"
Parameter Default Description
connection http://localhost:9200 OpenSearch URL
collection (required) Index name
dimensions 1536 Vector dimensions
username (none) Basic auth username
password (none) Basic auth password

Uses the k-NN plugin with HNSW algorithm. NDJSON bulk operations for efficient upsert/delete.

Milvus

- rag-store:
    store: milvus
    connection: "localhost:19530"
    collection: documents
    dimensions: 1536

Pinecone

- rag-store:
    store: pinecone
    connection: "https://my-index-abc123.svc.pinecone.io"
    collection: my-namespace

Requires api_key set via environment or component config.

Qdrant

- rag-store:
    store: qdrant
    connection: "http://localhost:6333"
    collection: documents

Elasticsearch

- rag-store:
    store: elasticsearch
    connection: "http://localhost:9200"
    collection: documents
    dimensions: 1536

Weaviate

- rag-store:
    store: weaviate
    connection: "http://localhost:8080"
    collection: Documents

MemX

MemX provides a unified search backend that combines Elasticsearch-speed full-text search with PostgreSQL-grade SQL joins behind one service. Two connection modes are available:

  • http://host:port -- ES-compatible HTTP API (fastest, recommended for search workloads)
  • postgres://user:pass@host:port/db -- PostgreSQL wire protocol (for SQL joins and transactions)

Auto-Discovery: When connecting to an existing memx table, column names are auto-detected from table indexes -- no manual column configuration needed:

  • HNSW/IVF/DiskANN index on a column --> auto-detected as the embedding column for vector search
  • FULLTEXT index on a column --> auto-detected as the content column for BM25 text search
  • Column with _id suffix or EMBEDDING type --> auto-detected as ID or embedding column

This means id_column, embedding_column, and content_column configuration is optional. Helix reads the table schema and indexes at startup and resolves them automatically. You can still provide explicit column overrides as escape hatches when auto-discovery does not match your schema.

database: field: Both rag-store and rag-retrieve support the database: field to target a specific memx database:

# Minimal (auto-discovery) -- recommended
- rag-store:
    store: memx
    connection: "http://localhost:8080"
    database: my_database
    collection: products

# With explicit column overrides (optional escape hatch)
- rag-store:
    store: memx
    connection: "http://localhost:8080"
    database: my_database
    collection: products
    id_column: product_id
    embedding_column: product_vec
    content_column: description
    dimensions: 1536

# PostgreSQL mode -- for SQL joins
- rag-store:
    store: memx
    connection: "postgres://user:pass@localhost:5432/mydb"
    collection: documents
    dimensions: 1536

Raw Results: memx retrieve returns raw []map[string]any containing all columns from the table (embedding column excluded, _score field included). There is no forced ID/Content/Metadata mapping -- the agent receives the raw JSON with all columns exactly as stored. Use Starlang transforms downstream if you need to reshape the data.

Performance: MemX achieves 1549 embeddings/sec ingest throughput and sub-300us vector search latency on commodity hardware.

Stage 5: Retrieval

The rag-retrieve step searches the vector store and returns relevant chunks. Helix provides 9 retrieval strategies.

Strategy Overview

Strategy Description Best For
simple Direct vector similarity search Default, most use cases
bm25 Keyword-based BM25 scoring Exact keyword matching
fusion Reciprocal Rank Fusion of vector + BM25 Balanced relevance
hybrid Weighted combination of two retrievers Fine-tuned relevance
mmr Maximal Marginal Relevance Diversity in results
ensemble Multiple retrievers with RRF Combining specialized retrievers
auto_merging Merges child chunks into parents Hierarchical chunking
router Routes queries to the best retriever Multi-domain systems
parent Returns parent chunks for matching children Context preservation
memx MemX dense vector search MemX backend, fast similarity
memx_text MemX BM25 keyword search MemX backend, exact keyword matching
memx_hybrid MemX fusion (vector + BM25) MemX backend, balanced relevance

Simple (Default)

- rag-retrieve:
    strategy: simple
    top_k: 10

Direct cosine similarity search against the vector store.

- rag-retrieve:
    strategy: bm25
    top_k: 10

Term frequency-inverse document frequency scoring. Best for queries with specific keywords or terminology.

Fusion (Vector + BM25)

- rag-retrieve:
    strategy: fusion
    top_k: 10

Runs both vector and BM25 retrievers, combines results using Reciprocal Rank Fusion (RRF). Balances semantic understanding with keyword precision.

Hybrid Retrieval

- rag-retrieve:
    strategy: hybrid
    top_k: 10
    vector_weight: 0.7
    keyword_weight: 0.3
Parameter Default Description
top_k 5 Number of results to return
vector_weight 0.7 Weight for vector similarity scores
keyword_weight 0.3 Weight for keyword match scores

Runs vector and keyword retrievers in parallel, normalizes scores to [0,1], combines with configurable weights, and deduplicates by chunk ID.

Maximal Marginal Relevance (MMR)

- rag-retrieve:
    strategy: mmr
    top_k: 5
    lambda: 0.5
    fetch_k: 20
Parameter Default Description
top_k 5 Final number of results
lambda 0.5 Diversity factor (0.0 = max diversity, 1.0 = max relevance)
fetch_k top_k * 3 Initial candidate pool size

Iteratively selects results that balance relevance to the query with diversity from already-selected results. Prevents returning near-duplicate chunks.

Ensemble Retrieval

- rag-retrieve:
    strategy: ensemble
    top_k: 10

Runs multiple retrievers in parallel and combines results using weighted Reciprocal Rank Fusion:

score(doc) = sum(weight_i / (K + rank_i(doc)))

where K defaults to 60 (the RRF constant). Deduplicates and returns top-k by combined score.

Auto-Merging Retrieval

- rag-retrieve:
    strategy: auto_merging
    top_k: 10
    merge_threshold: 0.5
Parameter Default Description
top_k 5 Number of results
merge_threshold 0.5 Fraction of children needed to trigger merge

Designed for use with the hierarchical chunking strategy. Groups retrieved child chunks by their parent_id metadata. If the fraction of a parent's children that appear in results exceeds the threshold, replaces them with the full parent chunk for better context.

Example: Hierarchical + Auto-Merging Pipeline

workflows:
  - id: hierarchical-rag
    trigger:
      uri: http-server://0.0.0.0:8080/api/ask
    steps:
      - rag-ingest:
          parser: text
      - rag-chunk:
          strategy: hierarchical
          levels: [2048, 512, 128]
          overlap: 50
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memory
          collection: docs
      - rag-retrieve:
          strategy: auto_merging
          top_k: 10
          merge_threshold: 0.4
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o

Router Retrieval

- rag-retrieve:
    strategy: router
    top_k: 10

Routes queries to the best-suited retriever based on query analysis. The KeywordRouteSelector matches query text against keyword patterns to select the retriever.

# Minimal (auto-discovery detects columns from table indexes)
- rag-retrieve:
    strategy: memx
    store: memx
    connection: http://localhost:8080
    database: products_db
    collection: products
    top_k: 10

# With optional column overrides
- rag-retrieve:
    strategy: memx
    store: memx
    connection: http://localhost:8080
    database: products_db
    collection: products
    id_column: product_id
    embedding_column: product_vec
    content_column: description
    top_k: 10

Dense vector similarity search using the MemX backend. Auto-discovery reads table indexes to detect the embedding column (HNSW/IVF/DiskANN) and content column (FULLTEXT) automatically. Results are returned as raw []map[string]any with all table columns (embedding excluded, _score included).

MemX Text Search (BM25)

- rag-retrieve:
    strategy: memx_text
    store: memx
    connection: http://localhost:8080
    database: products_db
    collection: products
    top_k: 20

BM25 keyword-based search using the MemX full-text index. No embeddings needed for queries -- pure keyword matching. The content column is auto-detected from FULLTEXT indexes.

- rag-retrieve:
    strategy: memx_hybrid
    store: memx
    connection: http://localhost:8080
    database: products_db
    collection: products
    top_k: 10
    vector_weight: 0.7
    keyword_weight: 0.3

Reciprocal rank fusion of MemX dense vector search and BM25 text search. The vector_weight and keyword_weight parameters control the balance between semantic and keyword relevance.

MemX End-to-End Pipeline Example

A complete RAG pipeline using MemX for storage and retrieval with auto-discovery (no column config needed):

workflows:
  # Ingest documents into MemX
  - id: ingest
    trigger:
      uri: http-server://0.0.0.0:8080/api/ingest
    steps:
      - rag-ingest:
          reader: directory
          path: ./docs
          recursive: true
      - rag-chunk:
          strategy: semantic
          threshold: 0.5
          chunk_size: 512
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memx
          connection: "http://localhost:8080"
          database: docs_db
          collection: knowledge

  # Query with hybrid search
  - id: ask
    trigger:
      uri: http-server://0.0.0.0:8080/api/ask
    steps:
      - rag-retrieve:
          strategy: memx_hybrid
          store: memx
          connection: "http://localhost:8080"
          database: docs_db
          collection: knowledge
          top_k: 10
          vector_weight: 0.7
          keyword_weight: 0.3
      - rag-postprocess:
          steps:
            - type: similarity-filter
              params:
                threshold: "0.6"
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o

MemX Agent + Pipeline (E2E Verified)

A verified end-to-end pipeline where a ReAct agent uses a memx-backed pipeline as a tool:

pipelines:
  - id: product-search
    steps:
      - rag-retrieve:
          strategy: memx
          store: memx
          connection: http://localhost:8080
          database: products_db
          collection: products
          top_k: 3

agents:
  - id: product-agent
    type: react
    model: { provider: openai, model: gpt-4o, api_key: ${OPENAI_API_KEY} }
    system_prompt: Use the search tool to find products.
    tools:
      - name: search-products
        type: tool
        tool_id: product-search
        description: Search the product catalog
        parameters:
          type: object
          properties:
            query: { type: string, description: What to search for }
    trigger: agui://0.0.0.0:4500/products

The agent receives raw JSON results with all table columns (embedding excluded, _score included). No Chunk wrapping -- the data is passed through as []map[string]any.

Stage 6: Postprocessing

The rag-postprocess step refines retrieval results before synthesis. Multiple postprocessors can be chained.

Postprocessor Types

Type Description
similarity-filter Remove chunks below a score threshold
top-k Keep only top K results
keyword-filter Filter by keyword presence
metadata-filter Filter by metadata key-value conditions
multi-metadata-filter Structured multi-condition metadata filter
dedup Remove duplicate content
pii-mask Mask PII (email, phone, SSN, etc.)
recency Boost recent documents
length-filter Filter by content length
diversity Limit results per source document
normalize Normalize scores to [0,1]
keyword-boost Boost scores for keyword matches
cohere-rerank Rerank via Cohere API
jina-rerank Rerank via Jina API
long-context-reorder Reorder for LLM attention optimization

Similarity Threshold

- rag-postprocess:
    steps:
      - type: similarity-filter
        params:
          min_score: "0.7"

Removes chunks with similarity scores below the threshold.

Metadata Filter

- rag-postprocess:
    steps:
      - type: metadata-filter
        params:
          key: source
          value: documentation
          op: eq

Operators: eq, ne, contains, exists.

Multi-Condition Metadata Filter

- rag-postprocess:
    steps:
      - type: multi-metadata-filter
        mode: all
        filters:
          - key: source
            operator: eq
            value: "docs"
          - key: version
            operator: gt
            value: "2.0"
          - key: language
            operator: contains
            value: "en"
          - key: reviewed
            operator: exists
            value: ""
Parameter Default Description
mode all all = all conditions must match; any = at least one
filters (required) List of conditions

Each condition supports: eq, neq, gt, lt, contains, exists.

PII Masking

- rag-postprocess:
    steps:
      - type: pii-masker

Masks: email addresses, phone numbers, SSNs, credit card numbers, IP addresses.

Cohere Reranker

- rag-postprocess:
    steps:
      - type: cohere-rerank
        params:
          api_key: "${env:COHERE_API_KEY}"
          model: rerank-v3.5
          top_n: "5"
Parameter Default Description
api_key (required) Cohere API key
model rerank-v3.5 Reranking model
top_n (all) Maximum results to return

Calls the Cohere /v2/rerank API to reorder results by neural relevance scoring. Significantly improves result quality compared to vector similarity alone.

Jina Reranker

- rag-postprocess:
    steps:
      - type: jina-rerank
        params:
          api_key: "${env:JINA_API_KEY}"
          model: jina-reranker-v2-base-multilingual
          top_n: "5"
Parameter Default Description
api_key (required) Jina API key
model jina-reranker-v2-base-multilingual Reranking model
top_n (all) Maximum results to return

Long Context Reorder

- rag-postprocess:
    steps:
      - type: long-context-reorder

Reorders chunks based on the "Lost in the Middle" research paper. Places the most relevant results at the beginning and end of the context, with less relevant results in the middle. This matches LLM attention patterns -- models pay more attention to the start and end of their context window.

Given ranked results [1, 2, 3, 4, 5, 6], reorders to: [1, 3, 5, 6, 4, 2].

Chaining Postprocessors

Multiple postprocessors execute in sequence:

- rag-postprocess:
    steps:
      - type: similarity-filter
        params:
          min_score: "0.5"
      - type: cohere-rerank
        params:
          api_key: "${env:COHERE_API_KEY}"
          top_n: "10"
      - type: diversity-filter
        params:
          max_per_document: "2"
      - type: long-context-reorder

This chain: filters low-quality results, reranks with Cohere, limits duplicates per source, then optimizes ordering for LLM attention.

Stage 7: Response Synthesis

The rag-synthesize step generates the final answer from retrieved context. Helix provides 5 synthesis modes optimized for different context sizes and quality requirements.

Mode Overview

Mode LLM Calls Best For
compact 1 (usually) Default; context fits in one call
refine N (per chunk) Large context; iterative improvement
tree-summarize log(N) Very large context; recursive summarization
accumulate N + 1 Independent per-chunk answers; comprehensive coverage
simple-summarize 1 Simple single-shot generation

Compact (Default)

- rag-synthesize:
    mode: compact
    model:
      provider: openai
      model: gpt-4o
    max_tokens: 4000
Parameter Default Description
max_tokens 4000 Maximum context tokens (estimated as chars/4)

Concatenates as many chunks as possible into a single LLM context, separated by \n\n---\n\n. If chunks exceed the token limit, overflows to refine mode for remaining chunks. Minimizes LLM calls while maximizing context.

Refine

- rag-synthesize:
    mode: refine
    model:
      provider: groq
      model: llama-3.3-70b-versatile
    max_chunks: 10
Parameter Default Description
max_chunks 0 (all) Maximum chunks to process
  1. Generates an initial answer from the first chunk
  2. For each subsequent chunk, asks the LLM to refine the answer given new context
  3. Each refinement can correct, expand, or improve the answer

Best for large context sets where quality matters more than speed.

Tree Summarize

- rag-synthesize:
    mode: tree-summarize
    model:
      provider: anthropic
      model: claude-sonnet-4-20250514
    chunks_per_level: 3
Parameter Default Description
chunks_per_level 3 Chunks combined per summarization step
  1. Groups chunks into batches of chunks_per_level
  2. Summarizes each batch with the LLM
  3. If multiple summaries remain, recursively summarizes them
  4. Continues until a single final answer

Scales logarithmically with context size. Ideal for very large document collections.

Accumulate

- rag-synthesize:
    mode: accumulate
    model:
      provider: openai
      model: gpt-4o
    parallel: true
Parameter Default Description
parallel false Generate per-chunk answers concurrently
  1. Generates an independent answer from each chunk (can run in parallel)
  2. Combines all per-chunk answers into a final response using a combine prompt

Best when each chunk may contain a distinct, valid answer. Parallel mode improves throughput.

Simple Summarize (Single-Shot)

- rag-synthesize:
    mode: simple-summarize
    model:
      provider: ollama
      model: llama3.1

Simple single-shot generation from the assembled context. No advanced chunking or iteration.

Model Configuration

All synthesis modes accept a model block:

model:
  provider: groq          # LLM provider
  model: llama-3.3-70b-versatile  # Model name
  api_key: "${env:GROQ_API_KEY}"   # API key (optional if set globally)
  base_url: "https://..."          # Custom endpoint (optional)
  temperature: 0.7                 # Sampling temperature (optional)
  max_tokens: 2048                 # Max response tokens (optional)

See the LLM Providers page for all 16 supported providers.

Advanced Stages

Metadata Extraction

The rag-extract step extracts metadata from documents using rule-based or LLM-powered methods.

- rag-extract:
    extractors:
      - title
      - summary
      - keywords
      - keywords-llm
      - questions
      - entities
      - language
    model:
      provider: openai
      model: gpt-4o-mini
Extractor Method Description
title LLM Extracts or generates a concise title
summary LLM Generates a summary of the document
keywords Rule-based Extracts significant terms by frequency
keywords-llm LLM LLM-generated keywords
questions LLM Generates questions the document answers
entities LLM Named entity extraction
language Rule-based Detects document language

LLM-powered extractors (summary, keywords-llm, questions, entities) require a model block.

Evaluation

The rag-eval step measures RAG pipeline quality.

- rag-eval:
    metrics:
      - faithfulness
      - answer_relevancy
      - context_relevancy
      - hit_rate
      - mrr
      - ndcg
    model:
      provider: openai
      model: gpt-4o
Metric Type Description
faithfulness LLM-judged Is the answer supported by the context?
answer_relevancy LLM-judged Does the answer address the query?
context_relevancy LLM-judged Is the retrieved context relevant?
correctness LLM-judged Is the answer factually correct?
hit_rate Statistical Fraction of queries with a relevant result in top-k
mrr Statistical Mean Reciprocal Rank of first relevant result
ndcg Statistical Normalized Discounted Cumulative Gain
map Statistical Mean Average Precision

LLM-judged metrics require a model block.

Caching

The rag-cache step caches embeddings and retrieval results.

- rag-cache:
    enabled: true
    max_size: 10000
Parameter Default Description
enabled true Enable caching
max_size 1000 Maximum cache entries
transform (none) Cache key transformation

Knowledge Graph RAG

Build and query knowledge graphs from documents.

Build:

- rag-graph-build:
    store: in-memory
    model:
      provider: openai
      model: gpt-4o

Extracts entities and relationships from text using an LLM, building a queryable knowledge graph.

Query:

- rag-graph-query:
    hops: 2
    top_k: 10
Parameter Default Description
hops 2 Graph traversal depth
top_k 10 Maximum results

Traverses the knowledge graph from query entities, following relationships up to hops edges deep.

Complete Examples

Production RAG API

A complete production pipeline with Jina embeddings, pgvector storage, hybrid retrieval, Cohere reranking, and Groq synthesis:

workflows:
  - id: production-rag
    trigger:
      uri: http-server://0.0.0.0:8080/api/ask
    auth:
      jwt:
        secret: "${env:JWT_SECRET}"
    steps:
      - rag-ingest:
          reader: directory
          path: ./knowledge-base
          recursive: true
          extensions: [".md", ".txt", ".pdf"]
      - rag-chunk:
          strategy: recursive
          chunk_size: 512
          overlap: 50
          separators: ["\n\n", "\n", ". ", " "]
      - rag-extract:
          extractors: [title, keywords]
      - rag-embed:
          provider: jina
          model: jina-embeddings-v3
          api_key: "${env:JINA_API_KEY}"
      - rag-store:
          store: pgvector
          connection: "postgres://${DB_USER}:${DB_PASS}@db:5432/rag"
          collection: knowledge
          dimensions: 1024
      - rag-retrieve:
          strategy: hybrid
          top_k: 20
          vector_weight: 0.7
          keyword_weight: 0.3
      - rag-postprocess:
          steps:
            - type: similarity-filter
              params:
                min_score: "0.4"
            - type: cohere-rerank
              params:
                api_key: "${env:COHERE_API_KEY}"
                model: rerank-v3.5
                top_n: "8"
            - type: long-context-reorder
      - rag-cache:
          enabled: true
          max_size: 50000
      - rag-synthesize:
          mode: compact
          model:
            provider: groq
            model: llama-3.3-70b-versatile
            api_key: "${env:GROQ_API_KEY}"
          max_tokens: 6000
      - log:
          message: "RAG query completed"

A pipeline optimized for searching source code and technical documentation:

workflows:
  - id: code-search
    trigger:
      uri: http-server://0.0.0.0:8080/api/code-search
    steps:
      - rag-ingest:
          reader: directory
          path: ./src
          recursive: true
          extensions: [".go", ".py", ".js", ".md"]
      - rag-chunk:
          strategy: code
          language: go
          chunk_size: 1500
          overlap: 100
      - rag-extract:
          extractors: [title, keywords]
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: qdrant
          connection: "http://localhost:6333"
          collection: code-docs
      - rag-retrieve:
          strategy: mmr
          top_k: 10
          lambda: 0.6
          fetch_k: 30
      - rag-postprocess:
          steps:
            - type: diversity-filter
              params:
                max_per_document: "3"
      - rag-synthesize:
          mode: refine
          model:
            provider: anthropic
            model: claude-sonnet-4-20250514
          max_chunks: 8

Multi-Source Knowledge Base

A pipeline that ingests from multiple formats with different chunking per format, metadata filtering, and graph-enhanced retrieval:

workflows:
  # Ingest markdown documentation
  - id: ingest-docs
    trigger:
      uri: timer://ingest?period=1h
    steps:
      - rag-ingest:
          reader: directory
          path: ./docs
          recursive: true
          extensions: [".md"]
      - rag-chunk:
          strategy: markdown
          chunk_size: 800
          retain_headers: true
      - rag-embed:
          provider: mistral
          model: mistral-embed
      - rag-store:
          store: chromadb
          connection: "http://localhost:8000"
          collection: knowledge

  # Ingest CSV data
  - id: ingest-data
    trigger:
      uri: timer://ingest-csv?period=1h
    steps:
      - rag-ingest:
          reader: csv
          delimiter: ","
          has_header: true
          rows_per_chunk: 50
      - rag-chunk:
          strategy: fixed
          chunk_size: 1000
      - rag-embed:
          provider: mistral
          model: mistral-embed
      - rag-store:
          store: chromadb
          connection: "http://localhost:8000"
          collection: knowledge

  # Query endpoint
  - id: query
    trigger:
      uri: http-server://0.0.0.0:8080/api/query
    steps:
      - rag-retrieve:
          strategy: fusion
          top_k: 15
      - rag-postprocess:
          steps:
            - type: multi-metadata-filter
              mode: any
              filters:
                - key: source
                  operator: contains
                  value: "docs"
                - key: source
                  operator: contains
                  value: "data"
            - type: jina-rerank
              params:
                api_key: "${env:JINA_API_KEY}"
                top_n: "8"
      - rag-synthesize:
          mode: tree-summarize
          model:
            provider: groq
            model: llama-3.3-70b-versatile
          chunks_per_level: 3

Quality Evaluation Pipeline

Evaluate your RAG pipeline quality with multiple metrics:

workflows:
  - id: rag-with-eval
    trigger:
      uri: http-server://0.0.0.0:8080/api/ask
    steps:
      - rag-ingest:
          parser: text
      - rag-chunk:
          strategy: semantic
          threshold: 0.5
          chunk_size: 1024
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memory
          collection: eval-docs
      - rag-retrieve:
          strategy: simple
          top_k: 10
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o
      - rag-eval:
          metrics:
            - faithfulness
            - answer_relevancy
            - context_relevancy
          model:
            provider: openai
            model: gpt-4o
      - log:
          message: "Eval scores: ${header.HelixRAGEvalScores}"

Graph-Enhanced RAG

Combine vector retrieval with knowledge graph traversal:

workflows:
  - id: graph-rag
    trigger:
      uri: http-server://0.0.0.0:8080/api/graph-ask
    steps:
      - rag-ingest:
          parser: text
      - rag-chunk:
          strategy: sentence
          chunk_size: 500
      - rag-graph-build:
          store: in-memory
          model:
            provider: openai
            model: gpt-4o-mini
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memory
          collection: graph-docs
      - rag-retrieve:
          strategy: simple
          top_k: 10
      - rag-graph-query:
          hops: 2
          top_k: 5
      - rag-synthesize:
          mode: refine
          model:
            provider: openai
            model: gpt-4o

Strategy Selection Guide

Choosing a Chunking Strategy

Scenario Recommended Strategy
General text documents recursive or sentence
High-quality RAG (slower) semantic
Source code repositories code with language specified
Technical documentation (Markdown) markdown with retain_headers: true
Web-scraped HTML content html
Need overlapping coverage sliding_window
Auto-merging retrieval hierarchical with auto_merging retriever
Simple, predictable chunks fixed

Choosing a Retrieval Strategy

Scenario Recommended Strategy
Simple similarity search simple
Keyword-heavy queries bm25
Balanced keyword + semantic fusion or hybrid
Diverse results (avoid duplicates) mmr with lambda: 0.5
Multiple specialized retrievers ensemble
Hierarchical chunks auto_merging
Multi-domain knowledge base router

Choosing a Synthesis Mode

Scenario Recommended Mode
Context fits in one LLM call compact (default)
Large context, quality matters refine
Very large context tree-summarize
Each chunk has independent answers accumulate
Simple, fast response simple-summarize

Choosing a Vector Store

Scenario Recommended Store
Testing / prototyping memory
Already using PostgreSQL pgvector
Need low latency + caching redis
Simple local development chromadb
AWS ecosystem opensearch
Large-scale production milvus or pinecone
Advanced filtering needs qdrant
Existing Elastic infrastructure elasticsearch

OCR & Document Intelligence

Helix integrates with 8 OCR backends to extract text from scanned PDFs, images, and complex document layouts. Each backend runs as an external service — Helix communicates via HTTP REST APIs (no cgo required).

Supported Backends

Backend Service Best For
docling IBM Docling High-quality PDF/doc conversion with tables
mineru MinerU Academic papers, complex layouts
marker Marker Fast, high-fidelity PDF to markdown
tesseract Tesseract OCR Traditional OCR, scanned images
surya Surya Multilingual OCR, handwriting
unstructured Unstructured.io Document parsing with element classification
azure-doc-intelligence Azure Document Intelligence Enterprise, tables, forms, invoices
vision-llm Any vision LLM (GPT-4o, Claude, Gemini) Flexible, no external OCR service needed

OCR with Docling

workflows:
  - id: ocr-ingest
    trigger:
      uri: http-server://0.0.0.0:8080/api/ingest
    steps:
      - rag-ingest:
          parser: ocr
          ocr:
            backend: docling
            endpoint: "http://localhost:5000"
            output_format: markdown
      - rag-chunk:
          strategy: markdown
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: pgvector
          collection: scanned_docs

OCR with MinerU

      - rag-ingest:
          parser: ocr
          ocr:
            backend: mineru
            endpoint: "http://localhost:8888"
            language: en

OCR with Marker

      - rag-ingest:
          parser: ocr
          ocr:
            backend: marker
            endpoint: "http://localhost:8001"

OCR with Tesseract

      - rag-ingest:
          parser: ocr
          ocr:
            backend: tesseract
            endpoint: "http://localhost:8884"
            language: eng

OCR with Surya

      - rag-ingest:
          parser: ocr
          ocr:
            backend: surya
            endpoint: "http://localhost:8885"
            language: en

OCR with Unstructured.io

      - rag-ingest:
          parser: ocr
          ocr:
            backend: unstructured
            endpoint: "https://api.unstructured.io"
            api_key: "${env:UNSTRUCTURED_API_KEY}"

OCR with Azure Document Intelligence

      - rag-ingest:
          parser: ocr
          ocr:
            backend: azure-doc-intelligence
            endpoint: "https://my-resource.cognitiveservices.azure.com"
            api_key: "${env:AZURE_DOC_KEY}"
            model: prebuilt-layout

Azure supports async processing — Helix automatically polls the Operation-Location header until the analysis is complete.

OCR with Vision LLM

No external OCR service needed — uses any vision-capable LLM to extract text from images:

      - rag-ingest:
          parser: ocr
          ocr:
            backend: vision-llm
            model:
              provider: openai
              model: gpt-4o
            output_format: markdown

Table Extraction

The TableExtractor can isolate markdown tables from OCR output for structured data processing:

workflows:
  - id: table-extract
    trigger:
      uri: http-server://0.0.0.0:8080/api/extract-tables
    steps:
      - rag-ingest:
          parser: ocr
          ocr:
            backend: azure-doc-intelligence
            endpoint: "${env:AZURE_ENDPOINT}"
            api_key: "${env:AZURE_DOC_KEY}"
          extract_tables: true

OCR Configuration Reference

Parameter Description Default
backend OCR backend to use (required)
endpoint Backend service URL (required for non-vision)
api_key API key for the service (optional)
model Model name (Azure: prebuilt-layout, prebuilt-invoice, etc.) backend-specific
language Language hint for OCR (optional)
output_format markdown or text markdown
timeout HTTP timeout 120s

Large File OCR (Streaming)

For large scanned PDFs (100MB+), use the streaming OCR reader which avoids loading the entire file in memory:

workflows:
  - id: large-pdf-ocr
    trigger:
      uri: http-server://0.0.0.0:8080/api/ingest-large
    steps:
      - rag-ingest:
          parser: streaming-ocr
          ocr:
            backend: docling
            endpoint: "http://localhost:5000"
          streaming:
            max_memory_bytes: 52428800   # 50MB — files larger spill to temp disk
            page_split: true             # Split PDFs into pages, OCR each independently
            concurrency: 4               # Parallel page OCR workers
            max_file_size: 1073741824    # 1GB hard limit
      - rag-chunk:
          strategy: markdown
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: pgvector
          collection: large_docs

How it works:

  1. Small files (< max_memory_bytes): Buffered in memory, sent to OCR as-is
  2. Large files: Spilled to a temp file on disk, then streamed to the OCR backend via io.Pipe multipart upload (constant memory)
  3. Page splitting (PDFs only): Large PDFs are split at /Type /Page boundaries, each page OCR-ed independently with bounded concurrency, results concatenated with <!-- Page N --> markers
  4. Unknown-size streams: Reads up to threshold, spills to temp file if exceeded
  5. Progress tracking: Optional callback for monitoring long-running OCR jobs

Streaming OCR Configuration:

Parameter Description Default
max_memory_bytes Memory threshold before spilling to temp file 50MB
page_split Split PDFs into pages for parallel OCR false
concurrency Number of parallel page workers 4
max_file_size Maximum allowed file size (0 = unlimited) 0

Partial failure handling: When page_split is enabled, individual page failures don't abort the entire document. Successfully OCR-ed pages are combined, and the failed_pages metadata field reports how many pages failed.

Query Transformation

Query transformation improves retrieval quality by rewriting or expanding the user's query before embedding and searching. Helix supports 4 strategies.

HyDE (Hypothetical Document Embeddings)

Generates a hypothetical answer to the query, then embeds that answer for retrieval. Works best when the expected answer has different vocabulary than the query.

workflows:
  - id: hyde-rag
    trigger:
      uri: http-server://0.0.0.0:8080/api/ask
    steps:
      - rag-retrieve:
          strategy: hyde
          top_k: 10
          hyde:
            model:
              provider: openai
              model: gpt-4o
          embedding:
            provider: openai
            model: text-embedding-3-small
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o

MultiQuery

Generates N alternative phrasings of the query, retrieves for each, and merges results using Reciprocal Rank Fusion.

      - rag-retrieve:
          strategy: multi_query
          top_k: 10
          multi_query:
            model:
              provider: openai
              model: gpt-4o
            num_queries: 3
          embedding:
            provider: openai
            model: text-embedding-3-small

StepBack Prompting

Generates a more abstract, broader query alongside the original. Retrieves for both and merges results. Good for specific or narrow queries.

      - rag-retrieve:
          strategy: step_back
          top_k: 10
          step_back:
            model:
              provider: openai
              model: gpt-4o
          embedding:
            provider: openai
            model: text-embedding-3-small

SubQuestion Decomposition

Breaks complex queries into simpler sub-questions, retrieves for each independently, and merges all results.

      - rag-retrieve:
          strategy: sub_question
          top_k: 10
          sub_question:
            model:
              provider: openai
              model: gpt-4o
          embedding:
            provider: openai
            model: text-embedding-3-small

Choosing a Query Transformation Strategy

Scenario Recommended Strategy
Query uses different words than documents hyde
Query is ambiguous or has multiple interpretations multi_query
Query is too specific or narrow step_back
Query is complex with multiple aspects sub_question
Simple, direct queries No transformation needed (use simple retrieval)