RAG ingest¶

Ingest is the first stage of the RAG pipeline. It turns raw input bytes ([]byte) plus metadata (filename, content-type, source URI) into a *rag.Document ready for the chunk stage. Each format-specific implementation satisfies:

type Parser interface {
    Parse(ctx, data []byte, metadata map[string]string) (*rag.Document, error)
    SupportedTypes() []string
}

The DSL processor rag-ingest looks up the parser by name via ingest.GetParser(parserType) or, for path-based dispatch, ingest.DetectParser(path). There is also a Reader interface (Read(ctx, data, source) ([]*rag.Document, error)) for sources that produce many docs (HTML, CSV, directory).

Built-in parsers¶

These eight parsers are in ingest.ParserRegistry and dispatched by GetParser / DetectParser.

text — plain text¶

rag:
  ingest:
    type: text

Pass-through: bytes become Document.Content verbatim. Default for unknown extensions in DetectParser. Source: pkg/rag/ingest/ingest.go (TextParser).

markdown — markdown files¶

rag:
  ingest:
    type: markdown

Same as text — markdown is fed through unchanged so the chunk: markdown strategy can parse headings. For structured reader use MarkdownReader (markdown_reader.go). Source: pkg/rag/ingest/ingest.go (MarkdownParser), pkg/rag/ingest/markdown_reader.go (MarkdownReader).

html — strip tags, return text¶

rag:
  ingest:
    type: html

Strips <...> runs and collapses whitespace. For structured ingestion with metadata extraction use the Reader-style HTMLReader (html_reader.go). Source: pkg/rag/ingest/ingest.go (HTMLParser), pkg/rag/ingest/html_reader.go (HTMLReader).

pdf — pure-Go PDF text extraction¶

rag:
  ingest:
    type: pdf

Parses the PDF cross-reference table, decompresses FlateDecode content streams, and extracts text from PDF operators (Tj, TJ, ', "). Pure-stdlib — no pdfcpu / unipdf dependency. Works well for text-based PDFs; scanned PDFs need the ocr parser. Source: pkg/rag/ingest/ingest.go (PDFParser), pkg/rag/ingest/pdf.go (PDFReader).

docx — Microsoft Word¶

rag:
  ingest:
    type: docx

Unzips the .docx package and extracts text from word/document.xml. Source: pkg/rag/ingest/docx.go (DOCXParser).

csv — comma-separated values¶

rag:
  ingest:
    type: csv

Each row becomes part of the document content. For per-row documents use the CSVReader (csv_reader.go). Source: pkg/rag/ingest/docx.go (CSVParser), pkg/rag/ingest/csv_reader.go (CSVReader).

json — JSON files¶

rag:
  ingest:
    type: json

Flattens JSON into a textual representation suitable for chunking. Source: pkg/rag/ingest/json.go.

xml — XML files¶

rag:
  ingest:
    type: xml

Extracts text content from XML elements. For metadata-aware reading use XMLReader (xml_reader.go). Source: pkg/rag/ingest/xml.go, pkg/rag/ingest/xml_reader.go.

starlang — Starlang script source¶

rag:
  ingest:
    type: starlang

For ingesting .star files alongside other source code. Treats the content as plain text but with .star as the canonical extension in DetectParser. Source: pkg/rag/ingest/starlang.go.

Multi-document readers¶

These satisfy the Reader interface and produce many *rag.Documents from a single input. Use them where one source maps to many documents.

directory — walk a directory tree¶

rag:
  ingest:
    type: directory
    path: ./docs
    recursive: true
    extensions: [".md", ".txt"]

Auto-selects the appropriate parser per file extension via DetectParser. Source: pkg/rag/ingest/directory.go.

image — vision-LLM image description¶

rag:
  ingest:
    type: image
    description_model: ${VISION_LLM_REF}
    include_base64: false
    description_prompt: "Describe this image in detail."

Supports .jpg, .jpeg, .png, .gif, .webp, .bmp, .svg. Without a vision model only metadata is extracted; with one a text description is generated and used as the document content. Source: pkg/rag/ingest/image.go.

audio — audio-LLM transcription¶

rag:
  ingest:
    type: audio
    transcription_model: ${AUDIO_LLM_REF}
    transcription_prompt: "Transcribe the audio verbatim."

Supports .wav, .mp3, .ogg, .flac, .m4a. Source: pkg/rag/ingest/audio.go.

ocr — pluggable OCR backend¶

rag:
  ingest:
    type: ocr
    backend: docling
    endpoint: http://localhost:5005
    api_key: ${OCR_KEY}
    model: docling-default
    language: en
    output_format: markdown
    timeout: 120s

Used for scanned PDFs and images where the text isn't directly extractable. The docling backend is the default recommendation; the vision-llm backend uses any vision-capable LLM (e.g. GPT-4o). Pairs with StreamingOCRReader (ocr_streaming.go) for very large PDFs and MemoryPageCheckpoint (checkpoint.go) for crash-recoverable page-by-page jobs. Source: pkg/rag/ingest/ocr.go, pkg/rag/ingest/ocr_pdfsplit.go, pkg/rag/ingest/ocr_streaming.go, pkg/rag/ingest/checkpoint.go.

Choosing between them¶

For PDFs specifically: try pdf first (cheap, no external deps); if the extracted text is empty or garbled, the document was a scanned image — switch to ocr.

See also¶

• chunk — consumes the *rag.Document produced here.
• extract — adds extracted metadata (title, summary, keywords) to the document.
• Multimodal guide — image and audio ingestion patterns.
• RAG guide — end-to-end recipes.