Skip to content

Multimodal AI Guide

Build agents that see, hear, and understand. Helix agents and pipelines can process text, images, audio, and video natively. This guide covers how each data type flows through the platform — from ingestion to LLM processing to tool responses — enabling vision agents, audio transcription, multimodal RAG, and rich content analysis.

Table of Contents

Architecture Overview

Multimodal data flows through three layers:

Input (HTTP, file, exchange)
  ↓
Content Parts (LLM messages / Exchange body)
  ↓
Provider-specific format (OpenAI vision, Anthropic image, Gemini inlineData)

The key abstraction is ContentPart -- a union type that carries text, images, audio, or video alongside metadata (MIME type, URL, binary data). Providers automatically convert ContentParts to their native multimodal format.

Content Types

ContentPart

Every piece of multimodal content is represented as a ContentPart:

Field Type Description
Type ContentType text, image, audio, video, file
Text string Text content (for text type)
MIMEType string Media type (e.g., image/png, audio/wav)
Data []byte Raw binary data (base64-decoded)
URL string URL reference (image URL, audio URL)
Detail string Resolution hint: low, high, auto (images only)

Supported Media Types

Content Type Supported Formats Providers
Text Any text All providers
Image JPEG, PNG, GIF, WebP, BMP, SVG OpenAI, Anthropic, Vertex/Gemini, Ollama
Audio WAV, MP3, OGG, FLAC, M4A OpenAI (gpt-4o-audio), Gemini
Video MP4, WebM Gemini
File Any binary Via tool/exchange passthrough

Agents with Multimodal Input

Text + Image (Vision)

The most common multimodal pattern -- send an image with a text prompt:

agents:
  - id: image-analyzer
    type: react
    trigger: http-server://0.0.0.0:8080/api/analyze
    model:
      provider: openai
      model: gpt-4o
    system_prompt: "You are an image analysis assistant."

From a Go client, use RunWithImage:

agent := adk.New(adk.AgentConfig{
    Type:         adk.TypeReAct,
    Model:        model, // OpenAI gpt-4o, Anthropic Claude, Gemini, etc.
    SystemPrompt: "You are an image analysis assistant.",
})

// Send text + image URL
result, err := agent.RunWithImage(ctx,
    "What objects are in this image?",
    "https://example.com/photo.jpg",
)
fmt.Println(result.Content) // "The image shows a cat sitting on a windowsill..."

Arbitrary Multimodal Parts

For complex inputs with multiple images, audio, or mixed media:

result, err := agent.RunWithParts(ctx, []llm.ContentPart{
    llm.TextPart("Compare these two images:"),
    llm.ImageURLPart("https://example.com/before.jpg", "high"),
    llm.ImageURLPart("https://example.com/after.jpg", "high"),
})

Base64-Encoded Images

For images from file uploads or binary sources:

imageData, _ := os.ReadFile("photo.png")
result, err := agent.RunWithParts(ctx, []llm.ContentPart{
    llm.TextPart("Describe this photo"),
    llm.ImageDataPart("image/png", imageData),
})

Audio Input

For models that support audio (OpenAI gpt-4o-audio, Gemini):

audioData, _ := os.ReadFile("recording.wav")
result, err := agent.RunWithParts(ctx, []llm.ContentPart{
    llm.TextPart("Transcribe this audio"),
    llm.AudioDataPart("audio/wav", audioData),
})

Provider-Specific Formats

Each provider receives ContentParts in its native format. This is automatic -- you write ContentParts once and they work across all vision-capable providers.

OpenAI (GPT-4o, GPT-4o-mini)

ContentParts are converted to OpenAI's content array format:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,...", "detail": "high"}}
  ]
}

Audio inputs use input_audio blocks:

{"type": "input_audio", "input_audio": {"data": "base64...", "format": "wav"}}

Anthropic (Claude Sonnet/Opus)

Images use Anthropic's source format:

{
  "type": "image",
  "source": {"type": "base64", "media_type": "image/png", "data": "base64..."}
}

URL images are fetched and sent as base64 (Anthropic requires base64).

Vertex / Gemini

Images use inline data:

{"inlineData": {"mimeType": "image/png", "data": "base64..."}}

Gemini natively supports images, audio, and video in inline data blocks.

Ollama

Images use Ollama's native images field (array of base64 strings). Models like llava, bakllava, and moondream support vision.

OpenAI-Compatible Providers

Groq, Mistral, Together, Fireworks, DeepInfra, xAI, and vLLM all use the same OpenAI content array format. Vision support depends on the model chosen.

Multimodal Exchanges

For pipeline routing (non-agent flows), exchanges carry multimodal content via MultipartContent:

Setting Multimodal Body

import "github.com/arvenza/helix/pkg/exchange"

// Create multipart content
exchange.SetMultipartBody(ex,
    exchange.NewTextContent("Process this image"),
    exchange.NewImageURLContent("https://example.com/photo.jpg"),
)

// Check if multipart
if exchange.IsMultipart(ex) {
    content := exchange.GetMultipartBody(ex)
    for _, part := range content.Parts {
        fmt.Printf("Type: %s, MIME: %s\n", part.Type, part.MIMEType)
    }
}

Multimodal Exchange Headers

Header Description
HelixContentType MIME type of body content
HelixContentEncoding Content encoding (e.g., base64)
HelixMediaType High-level type: text, image, audio, video
HelixImageURL Direct image URL for vision processing
HelixAudioURL Direct audio URL
HelixVideoURL Direct video URL
HelixMultipart "true" if body is MultipartContent

HTTP Workflow with Image Processing

agents:
  - id: vision-analyze
    type: react
    trigger: direct://vision-analyze
    model:
      provider: openai
      model: gpt-4o
    system_prompt: "Analyze the provided image and describe what you see."

workflows:
  - id: image-pipeline
    trigger:
      uri: http-server://0.0.0.0:8080/api/process-image
    steps:
      - transform:
          starlang: |
            # Image data arrives as binary body
            # Set content type header
            headers["HelixMediaType"] = "image"
            headers["HelixContentType"] = "image/jpeg"
            result = payload
      - target: direct://vision-analyze

RAG with Multimodal Documents

Image Ingestion

The ImageReader processes image files for RAG pipelines:

workflows:
  - id: image-rag
    trigger:
      uri: timer://ingest?period=1h
    steps:
      - rag-ingest:
          reader: image
      - rag-chunk:
          strategy: fixed
          chunk_size: 500
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memory
          collection: images

Without a vision model: Generates metadata descriptions (filename, format, dimensions).

With a vision model: Sends the image to a vision-capable LLM for rich text descriptions:

reader := &ingest.ImageReader{
    DescriptionModel: visionModel, // e.g., OpenAI gpt-4o
    IncludeBase64:    false,
}
doc, err := reader.Parse(ctx, imageBytes, metadata)
// doc.Content = "A photograph showing a golden retriever sitting in a park..."

Supported formats: .jpg, .jpeg, .png, .gif, .webp, .bmp, .svg

Audio Ingestion

The AudioReader handles audio files:

workflows:
  - id: audio-rag
    trigger:
      uri: timer://ingest?period=1h
    steps:
      - rag-ingest:
          reader: audio
      - rag-chunk:
          strategy: sentence
          chunk_size: 1000
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memory
          collection: audio

Without a model: Metadata descriptions only.

With a transcription model: Full audio-to-text transcription.

Supported formats: .wav, .mp3, .ogg, .flac, .m4a

Multimodal Documents

The MultimodalDocument type extends the standard Document with media attachments:

import "github.com/arvenza/helix/pkg/rag"

doc := &rag.MultimodalDocument{
    Document: rag.Document{
        ID:      "doc-1",
        Content: "Product photo showing the new widget",
        Source:  "catalog/widget.jpg",
    },
}

// Attach media
doc.AddMedia(rag.MediaContent{
    Type:     rag.MediaImage,
    MIMEType: "image/jpeg",
    Data:     imageBytes,
    Caption:  "Front view of the widget in blue colorway",
})

// Check media
if doc.HasMedia() {
    images := doc.MediaOfType(rag.MediaImage)
    fmt.Printf("Document has %d images\n", len(images))
}

// Convert to plain document (includes captions in content)
plainDoc := doc.ToDocument(true) // true = include captions

Multimodal Embeddings (CLIP)

For image-based similarity search, use the CLIPEmbedder:

import "github.com/arvenza/helix/pkg/rag/embed"

clipEmbedder := &embed.CLIPEmbedder{
    BaseURL:    "http://localhost:8000",
    APIKey:     "",
    Model:      "clip-vit-large-patch14",
    Dimensions: 768,
}

// Embed text
textVec, _ := clipEmbedder.EmbedSingle(ctx, "a photo of a cat")

// Embed image
imageVec, _ := clipEmbedder.EmbedImage(ctx, imageBytes, "image/jpeg")

// Both vectors are in the same embedding space -- you can search text→image or image→image

The MultimodalEmbedder interface extends the standard Embedder:

type MultimodalEmbedder interface {
    Embedder
    EmbedImage(ctx context.Context, imageData []byte, mimeType string) ([]float32, error)
    EmbedImages(ctx context.Context, images []ImageInput) ([][]float32, error)
}

Multimodal Tools

Agents can use tools that return multimodal content:

import (
    "github.com/arvenza/helix/pkg/agent/llm"
    "github.com/arvenza/helix/pkg/agent/tool"
)

// A tool that generates images
type ImageGenTool struct{}

func (t *ImageGenTool) Info() *tool.ToolInfo {
    return &tool.ToolInfo{
        Name:        "generate_image",
        Description: "Generate an image from a text description",
        Parameters: map[string]any{
            "type": "object",
            "properties": map[string]any{
                "prompt": map[string]any{"type": "string"},
            },
        },
    }
}

func (t *ImageGenTool) Invoke(ctx context.Context, args string) (string, error) {
    // Fallback text response
    return "Image generated successfully", nil
}

func (t *ImageGenTool) InvokeMultimodal(ctx context.Context, args string) ([]llm.ContentPart, error) {
    // Parse args, call image generation API...
    imageData := generateImage(args)
    return []llm.ContentPart{
        llm.TextPart("Here is the generated image:"),
        llm.ImageDataPart("image/png", imageData),
    }, nil
}

The MultimodalTool interface:

type MultimodalTool interface {
    Tool
    InvokeMultimodal(ctx context.Context, arguments string) ([]llm.ContentPart, error)
}

When the agent framework detects a MultimodalTool, it calls InvokeMultimodal and includes the returned content parts in the conversation.

Complete Examples

Image Q&A API

Accept image uploads via HTTP and answer questions about them:

workflows:
  - id: image-qa
    trigger:
      uri: http-server://0.0.0.0:8080/api/image-qa
    steps:
      - transform:
          starlang: |
            # Extract question from query parameter
            question = headers.question
            if question == "":
              question = "What is in this image?"
            headers["HelixMediaType"] = "image"
            result = payload
      - agent:
          type: react
          model:
            provider: openai
            model: gpt-4o
          system_prompt: |
            You are a visual assistant. Analyze images and answer questions about them.
            Be specific and detailed in your descriptions.

Test with curl:

curl -X POST http://localhost:8080/api/image-qa \
  -H "Content-Type: image/jpeg" \
  -H "question: How many people are in this photo?" \
  --data-binary @photo.jpg

Multi-Provider Vision Comparison

Send the same image to multiple vision models and compare responses:

agents:
  - id: openai-vision
    type: react
    trigger: direct://openai-vision
    model:
      provider: openai
      model: gpt-4o
    system_prompt: "Describe this image in one paragraph."

  - id: anthropic-vision
    type: react
    trigger: direct://anthropic-vision
    model:
      provider: anthropic
      model: claude-sonnet-4-20250514
    system_prompt: "Describe this image in one paragraph."

  - id: gemini-vision
    type: react
    trigger: direct://gemini-vision
    model:
      provider: gemini
      model: gemini-2.0-flash
    system_prompt: "Describe this image in one paragraph."

workflows:
  - id: vision-compare
    trigger:
      uri: http-server://0.0.0.0:8080/api/compare-vision
    steps:
      - multicast:
          uris:
            - direct://openai-vision
            - direct://anthropic-vision
            - direct://gemini-vision
      - aggregate:
          strategy: collect

Image + Document RAG

Build a knowledge base from both text documents and images:

workflows:
  # Ingest text documents
  - id: ingest-text
    trigger:
      uri: timer://ingest-text?period=1h
    steps:
      - rag-ingest:
          reader: directory
          path: ./docs
          recursive: true
          extensions: [".md", ".txt"]
      - rag-chunk:
          strategy: markdown
          chunk_size: 512
          retain_headers: true
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: pgvector
          connection: "postgres://user:pass@db:5432/rag"
          collection: knowledge
          dimensions: 1536

  # Ingest images (with vision-generated descriptions)
  - id: ingest-images
    trigger:
      uri: timer://ingest-images?period=1h
    steps:
      - rag-ingest:
          reader: image
      - rag-chunk:
          strategy: sentence
          chunk_size: 500
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: pgvector
          connection: "postgres://user:pass@db:5432/rag"
          collection: knowledge
          dimensions: 1536

  # Unified query endpoint
  - id: query
    trigger:
      uri: http-server://0.0.0.0:8080/api/ask
    steps:
      - rag-retrieve:
          strategy: hybrid
          top_k: 10
          vector_weight: 0.7
          keyword_weight: 0.3
      - rag-postprocess:
          steps:
            - type: cohere-rerank
              params:
                api_key: "${env:COHERE_API_KEY}"
                top_n: "5"
      - rag-synthesize:
          mode: compact
          model:
            provider: openai
            model: gpt-4o

Audio Transcription Pipeline

Transcribe audio files and make them searchable:

workflows:
  - id: audio-pipeline
    trigger:
      uri: http-server://0.0.0.0:8080/api/transcribe
    steps:
      - rag-ingest:
          reader: audio
      - rag-chunk:
          strategy: sentence
          chunk_size: 1000
      - rag-embed:
          provider: openai
          model: text-embedding-3-small
      - rag-store:
          store: memory
          collection: transcripts
      - log:
          message: "Audio transcribed and indexed"

Interactive Multimodal Input

Beyond programmatic APIs, Helix agents support interactive multimodal input from frontends via the AG-UI protocol. These features enable real-time voice, file upload, and document processing in chat-style interfaces.

Voice Input (Speech-to-Text)

Agents exposed via AG-UI can receive voice input. The frontend captures audio via the browser's MediaRecorder API and sends it as a binary audio content part. The agent pipeline handles speech-to-text conversion before LLM processing.

agents:
  - id: voice-assistant
    type: react
    trigger: agui://0.0.0.0:4100/agent
    model:
      provider: openai
      model: gpt-4o
    system_prompt: "You are a voice-enabled assistant. Respond concisely."

When audio arrives without accompanying text, the agent automatically transcribes the audio using the configured STT provider before sending the transcript to the LLM. Supported audio formats: WAV, WebM, MP3, OGG, M4A.

File Upload (Images, PDF, CSV, Text)

AG-UI frontends can upload files as content parts. The agent detects the file type and processes accordingly:

File Type Processing
Images (JPEG, PNG, GIF, WebP) Sent directly to vision-capable LLMs as image content parts
PDF Text extracted via built-in reader or OCR (if scanned)
CSV Parsed into structured text with column headers
Plain text (.txt, .md) Passed through as text content 
agents:
  - id: file-processor
    type: react
    trigger: agui://0.0.0.0:4100/agent
    model:
      provider: openai
      model: gpt-4o
    system_prompt: |
      You can process uploaded files. For images, describe what you see.
      For documents, summarize the content. For data files, analyze the data.

PDF OCR via Docling

For scanned PDFs and image-heavy documents, configure the OCR backend to extract text with layout awareness:

helix:
  ocr:
    backend: docling
    endpoint: http://localhost:5001

agents:
  - id: document-reader
    type: react
    trigger: agui://0.0.0.0:4100/agent
    model:
      provider: openai
      model: gpt-4o
    system_prompt: |
      You process uploaded documents. Extract key information,
      summarize tables, and answer questions about the content.

When a PDF is uploaded, the pipeline: 1. Detects the PDF content type 2. Sends the PDF to the Docling endpoint for OCR processing 3. Receives structured markdown with tables, headings, and text 4. Passes the extracted text to the LLM as the user message

Auto-Prompt for Multimodal Input

When a user sends a file (image, PDF, audio) without accompanying text, the agent generates an appropriate default prompt based on the content type:

Content Type Auto-Prompt
Image "Describe this image in detail."
PDF / Document "Summarize this document."
Audio "Transcribe and summarize this audio."
CSV / Data "Analyze this data and provide key insights."

This means users can simply drag-and-drop a file into the chat without typing anything, and the agent will respond meaningfully.

Base64 Image Stripping for Large OCR Output

When OCR processes multi-page documents, the raw output can contain embedded base64 images (e.g., figures, charts extracted from PDFs). These base64 strings can be extremely large and waste LLM context tokens. Helix automatically strips base64 image data from OCR output before sending it to the LLM, replacing it with a placeholder reference:

[Image: figure_1.png (stripped, 2.3MB base64)]

This optimization can reduce token usage by 10-100x for image-heavy documents. The original images remain available as separate content parts if the agent needs to reference them via a vision model.

Content Part Reference

Constructors

Constructor Description
llm.TextPart(text) Plain text content
llm.ImageURLPart(url, detail) Image from URL (detail: low, high, auto)
llm.ImageDataPart(mimeType, data) Image from binary data
llm.AudioURLPart(url) Audio from URL
llm.AudioDataPart(mimeType, data) Audio from binary data
llm.VideoPart(url) Video from URL
llm.FilePart(mimeType, data) Generic file attachment

Agent Methods

Method Description
agent.Run(ctx, text) Text-only input (existing)
agent.RunWithImage(ctx, text, imageURL) Text + single image URL
agent.RunWithParts(ctx, parts) Arbitrary multimodal input

Message Helpers

Method Description
msg.HasParts() True if message contains ContentParts
msg.TextContent() Extracts text (from Parts or Content field)
part.DataURI() Returns data:mime;base64,... URI for binary parts

Exchange Helpers

Function Description
exchange.SetMultipartBody(ex, parts...) Set multipart content as body
exchange.GetMultipartBody(ex) Extract multipart content from body
exchange.IsMultipart(ex) Check if exchange has multipart content
exchange.NewTextContent(text) Create text content part
exchange.NewImageContent(mime, data) Create image content part
exchange.NewImageURLContent(url) Create image URL content part
exchange.NewAudioContent(mime, data) Create audio content part

Provider Vision Model Support

Provider Vision Models Image Audio Video
OpenAI gpt-4o, gpt-4o-mini, gpt-4-turbo Yes Yes (gpt-4o-audio) No
Anthropic claude-sonnet-4-20250514, claude-opus-4-20250514 Yes No No
Vertex/Gemini gemini-2.0-flash, gemini-1.5-pro Yes Yes Yes
Ollama llava, bakllava, moondream Yes No No
Groq llama-3.2-90b-vision-preview Yes No No
Together Llama-3.2-90B-Vision-Instruct-Turbo Yes No No
Fireworks Model dependent Yes No No
Azure gpt-4o (deployed) Yes Yes No
Bedrock anthropic.claude-3-sonnet Yes No No