RAG postprocess¶

Postprocessors transform []*rag.SearchResult after retrieval and before generation: filter out noise, rerank with a real model, mask PII, decay by age, deduplicate, or rearrange to fight "lost in the middle". All implementations satisfy:

type Postprocessor interface {
    Process(ctx, query string, results []*rag.SearchResult) ([]*rag.SearchResult, error)
}

The DSL processor rag-postprocess chains steps via postprocess.NewChain(steps...). Each step is resolved by postprocess.GetPostprocessor(name, params). A given pipeline can chain any number of postprocessors.

Implementations¶

similarity-filter — drop results below a score threshold¶

rag:
  postprocess:
    - type: similarity-filter
      min_score: 0.5

Use as the first step to discard low-confidence matches. Source: pkg/rag/postprocess/postprocess.go (SimilarityFilter).

threshold-similarity-filter — alias with separate registration¶

rag:
  postprocess:
    - type: threshold-similarity-filter
      min_score: 0.5

Functionally identical to similarity-filter, registered separately so external tooling can address the distinction. Source: pkg/rag/postprocess/similarity_filter.go.

top-k — keep only the top K (re-sorts by score)¶

rag:
  postprocess:
    - type: top-k
      k: 5

Source: pkg/rag/postprocess/postprocess.go (TopKFilter).

keyword-filter — keep results containing keywords¶

rag:
  postprocess:
    - type: keyword-filter
      keywords: helix,esb,rag
      mode: any

Source: pkg/rag/postprocess/postprocess.go (KeywordFilter).

metadata-filter — single-condition metadata gate¶

rag:
  postprocess:
    - type: metadata-filter
      key: source
      value: confluence
      op: eq

Source: pkg/rag/postprocess/postprocess.go (MetadataFilter).

multi-metadata-filter — multiple conditions (AND/OR)¶

rag:
  postprocess:
    - type: multi-metadata-filter
      mode: and
      filters:
        - { key: source, value: docs, op: eq }
        - { key: lang,   value: en,   op: eq }

Use when retrieval needs structured filtering (tenant, language, date ranges, etc.) before generation. Source: pkg/rag/postprocess/metadata_filter.go.

dedup — drop duplicates by chunk ID¶

rag:
  postprocess:
    - type: dedup

No fields. Source: pkg/rag/postprocess/postprocess.go (DedupFilter).

pii-mask — basic email / SSN / phone redaction¶

rag:
  postprocess:
    - type: pii-mask

Replaces email addresses, US SSN-shaped strings, and phone-shaped strings with [REDACTED]. Use for compliance when retrieved chunks must not surface PII. Source: pkg/rag/postprocess/postprocess.go (PIIMasker). An extended variant AdvancedPIIMasker with more patterns lives in advanced.go.

recency — boost recent documents¶

rag:
  postprocess:
    - type: recency
      timestamp_key: created_at
      boost_factor: 1.5

A complementary AdvancedRecencyFilter in advanced.go supports exponential decay (DecayFactor) and hard max-age cutoff (MaxAge). Source: pkg/rag/postprocess/postprocess.go, pkg/rag/postprocess/advanced.go.

length-filter — drop very short / very long chunks¶

rag:
  postprocess:
    - type: length-filter
      min_length: 50
      max_length: 5000

Source: pkg/rag/postprocess/postprocess.go (LengthFilter).

diversity — cap chunks per document¶

rag:
  postprocess:
    - type: diversity
      max_per_document: 3

Caps how many chunks may come from any single source document. There is also an AdvancedDiversityFilter (advanced.go) doing Jaccard-on- word-sets similarity instead of doc-ID grouping. Source: pkg/rag/postprocess/postprocess.go, pkg/rag/postprocess/advanced.go.

normalize — rescale scores to [0,1]¶

rag:
  postprocess:
    - type: normalize

Min-max normalisation across the current result set. Useful before fusion-style merges. An AdvancedScoreNormalizer (z-score) is also provided in advanced.go. Source: pkg/rag/postprocess/postprocess.go.

keyword-boost — boost scores when query terms appear¶

rag:
  postprocess:
    - type: keyword-boost
      boost_factor: 1.2

Multiplies scores for chunks whose content contains query terms. Source: pkg/rag/postprocess/postprocess.go (KeywordBooster).

long-context-reorder — fight "lost in the middle"¶

rag:
  postprocess:
    - type: long-context-reorder

Given [1st, 2nd, 3rd, 4th, 5th, 6th] produces [1st, 3rd, 5th, 6th, 4th, 2nd] — high-relevance results at the start and end of the prompt, lower-relevance in the middle (based on the "Lost in the Middle" paper). Use as the last step before the synthesizer when LLM context windows are long. Source: pkg/rag/postprocess/long_context_reorder.go.

cohere-rerank — Cohere Rerank API¶

rag:
  postprocess:
    - type: cohere-rerank
      api_key: ${COHERE_API_KEY}
      top_n: 5

Calls the rerank-v3.5 model. Strong general-purpose cross-encoder. Source: pkg/rag/postprocess/rerank_cohere.go.

jina-rerank — Jina Rerank API¶

rag:
  postprocess:
    - type: jina-rerank
      api_key: ${JINA_API_KEY}
      top_n: 5

Uses jina-reranker-v2-base-multilingual. Source: pkg/rag/postprocess/rerank_jina.go.

cross-encoder-rerank — self-hosted cross-encoder¶

AdvancedReranker POSTs (query, documents) to a configured HTTP endpoint and expects {scores: [...]} back. Use when you run your own cross-encoder service (e.g. sentence-transformers/ms-marco-MiniLM-L-6-v2). Source: pkg/rag/postprocess/advanced.go (CrossEncoderReranker).

colbert-rerank — ColBERT late-interaction reranker¶

ColBERTReranker POSTs (query, documents) to a ColBERT service endpoint and expects {scores: [...]} back. Use for state-of-the-art late-interaction reranking when you can run a ColBERT model. Source: pkg/rag/postprocess/advanced.go (ColBERTReranker).

Choosing between them¶

A common production chain: similarity-filter → dedup → cohere-rerank → top-k → pii-mask → long-context-reorder.

See also¶

• retrieve — produces the []*rag.SearchResult consumed here.
• generate — consumes postprocessed results.
• eval — measure whether postprocessing helps end-to-end.