LLM & Embedding Providers¶

Power your agents and RAG pipelines with any LLM. Helix supports 17 LLM providers (plus a gemini alias for Vertex) and 10 embedding providers. Each can be used in agent steps, RAG synthesis, metadata extraction, and as direct route components.

For general agent and RAG configuration, see the User Guide.

Table of Contents¶

• LLM Providers
• OpenAI
• Anthropic
• Ollama
• Qwen (Alibaba Cloud DashScope)
• Groq
• Mistral
• Together AI
• Fireworks AI
• DeepInfra
• xAI (Grok)
• vLLM (Self-hosted)
• Azure OpenAI
• AWS Bedrock
• Google Vertex AI / Gemini
• Cohere
• HuggingFace
• Replicate
• Reasoning, FinishReason & Audio I/O
• reasoning_effort (o1/o3/gpt-5 + qwen-thinking)
• reasoning_content (Qwen3 / DeepSeek-R1)
• FinishReason everywhere
• Audio output (Qwen-Omni, GPT-4o-audio)
• ExtraBody escape hatch
• Embedding Providers
• OpenAI Embeddings
• Ollama Embeddings
• Cohere Embeddings
• Voyage AI Embeddings
• Mistral Embeddings
• Google Vertex / Gemini Embeddings
• AWS Bedrock Embeddings
• Jina Embeddings
• HuggingFace Embeddings
• Nomic Embeddings
• Component URIs
• Common Model Configuration Fields
• Provider Selection Guide

LLM Providers¶

OpenAI¶

• Provider name: openai
• Default model: gpt-4o
• Auth: Bearer token (OPENAI_API_KEY)
• API endpoint: https://api.openai.com/v1/chat/completions

# In agent step
- agent:
    type: react
    model:
      provider: openai
      model: gpt-4o
      api_key: "${OPENAI_API_KEY}"
      temperature: 0.7
      max_tokens: 4096

# As route component
workflows:
  - id: openai-chat
    trigger:
      uri: direct://input
    steps:
      - target: openai://chat?model=gpt-4o&apiKey=${OPENAI_API_KEY}

Anthropic¶

• Provider name: anthropic
• Default model: claude-sonnet-4-20250514
• Auth: x-api-key header (ANTHROPIC_API_KEY)
• API endpoint: https://api.anthropic.com/v1/messages

- agent:
    type: react
    model:
      provider: anthropic
      model: claude-sonnet-4-20250514
      api_key: "${ANTHROPIC_API_KEY}"

Ollama¶

• Provider name: ollama
• Default model: llama3
• Auth: None (local)
• Base URL: http://localhost:11434 (configurable)

- agent:
    type: react
    model:
      provider: ollama
      model: llama3
      base_url: "http://localhost:11434"

Qwen (Alibaba Cloud DashScope)¶

• Provider name: qwen
• Default model: qwen-plus
• Auth: Bearer token (DASHSCOPE_API_KEY)
• API endpoint: https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions

Qwen is registered as a named OpenAI-compatible provider — DashScope's compatible-mode/v1 endpoint speaks the OpenAI chat-completions schema. Qwen-specific extras are wired through the shared ExtraBody escape hatch:

- agent:
    type: react
    model:
      provider: qwen
      model: qwen3-32b
      api_key: "${DASHSCOPE_API_KEY}"
      # Optional — point at a self-hosted Qwen instance instead of DashScope
      # base_url: "http://localhost:8080/v1"

Thinking mode (Qwen3). Toggle deep-thinking via the request extra_body. The model emits reasoning_content deltas before the final content; both surface on Message.ReasoningContent and Message.Content respectively. See reasoning_content below.

Audio I/O (Qwen-Omni). Use WithAudioOutput to ask Qwen-Omni for a wav voice response in one of Cherry / Serena / Ethan / Chelsie; the returned Message.Parts carries the decoded audio. Send audio TO the model by attaching AudioDataPart(mimeType, bytes) to the user message's parts.

Groq¶

• Provider name: groq
• Default model: llama-3.3-70b-versatile
• Auth: Bearer token (GROQ_API_KEY)
• API endpoint: https://api.groq.com/openai/v1/chat/completions

- agent:
    type: react
    model:
      provider: groq
      model: llama-3.3-70b-versatile
      api_key: "${GROQ_API_KEY}"

# As route component
- target: groq://chat?model=llama-3.3-70b-versatile&apiKey=${GROQ_API_KEY}

Mistral¶

• Provider name: mistral
• Default model: mistral-large-latest
• Auth: Bearer token (MISTRAL_API_KEY)
• API endpoint: https://api.mistral.ai/v1/chat/completions

- agent:
    type: react
    model:
      provider: mistral
      model: mistral-large-latest
      api_key: "${MISTRAL_API_KEY}"

Together AI¶

• Provider name: together
• Default model: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
• Auth: Bearer token (TOGETHER_API_KEY)
• API endpoint: https://api.together.xyz/v1/chat/completions

- agent:
    type: react
    model:
      provider: together
      model: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
      api_key: "${TOGETHER_API_KEY}"

Fireworks AI¶

• Provider name: fireworks
• Default model: accounts/fireworks/models/llama-v3p1-70b-instruct
• Auth: Bearer token (FIREWORKS_API_KEY)
• API endpoint: https://api.fireworks.ai/inference/v1/chat/completions

- agent:
    type: react
    model:
      provider: fireworks
      model: accounts/fireworks/models/llama-v3p1-70b-instruct
      api_key: "${FIREWORKS_API_KEY}"

DeepInfra¶

• Provider name: deepinfra
• Default model: meta-llama/Meta-Llama-3.1-70B-Instruct
• Auth: Bearer token (DEEPINFRA_API_KEY)
• API endpoint: https://api.deepinfra.com/v1/openai/chat/completions

- agent:
    type: react
    model:
      provider: deepinfra
      model: meta-llama/Meta-Llama-3.1-70B-Instruct
      api_key: "${DEEPINFRA_API_KEY}"

xAI (Grok)¶

• Provider name: xai
• Default model: grok-2-latest
• Auth: Bearer token (XAI_API_KEY)
• API endpoint: https://api.x.ai/v1/chat/completions

- agent:
    type: react
    model:
      provider: xai
      model: grok-2-latest
      api_key: "${XAI_API_KEY}"

vLLM (Self-hosted)¶

• Provider name: vllm
• Default model: default
• Auth: Optional Bearer token
• Base URL: http://localhost:8000/v1 (configurable)

- agent:
    type: react
    model:
      provider: vllm
      model: my-fine-tuned-model
      base_url: "http://gpu-server:8000/v1"

Azure OpenAI¶

• Provider name: azure
• Default model: gpt-4o (deployment name)
• Auth: api-key header
• Base URL: Required -- your Azure resource endpoint

- agent:
    type: react
    model:
      provider: azure
      model: gpt-4o-deployment     # Azure deployment name
      api_key: "${AZURE_API_KEY}"
      base_url: "https://myresource.openai.azure.com"

# As route component (apiVersion param available)
- target: azure://chat?model=gpt-4o-deployment&apiKey=${AZURE_API_KEY}&baseURL=https://myresource.openai.azure.com&apiVersion=2024-10-21

AWS Bedrock¶

• Provider name: bedrock
• Default model: (none -- must specify a model ARN or ID)
• Auth: AWS credentials as accessKeyId:secretAccessKey[:sessionToken]
• Base URL: AWS region (default: us-east-1)

- agent:
    type: react
    model:
      provider: bedrock
      model: anthropic.claude-3-sonnet-20240229-v1:0
      api_key: "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}"
      base_url: us-east-1   # region

# As route component
- target: bedrock://chat?model=anthropic.claude-3-sonnet-20240229-v1:0&region=us-east-1&accessKey=${AWS_ACCESS_KEY_ID}&secretKey=${AWS_SECRET_ACCESS_KEY}

Google Vertex AI / Gemini¶

• Provider name: vertex or gemini (aliases)
• Default model: gemini-2.0-flash
• Auth: API key as query parameter
• API endpoint: https://generativelanguage.googleapis.com/v1beta/models

- agent:
    type: react
    model:
      provider: gemini
      model: gemini-2.0-flash
      api_key: "${GOOGLE_API_KEY}"

# As route component
- target: vertex://chat?model=gemini-2.0-flash&apiKey=${GOOGLE_API_KEY}

Cohere¶

• Provider name: cohere
• Default model: command-r-plus
• Auth: Bearer token (COHERE_API_KEY)
• API endpoint: https://api.cohere.com/v2/chat

- agent:
    type: react
    model:
      provider: cohere
      model: command-r-plus
      api_key: "${COHERE_API_KEY}"

HuggingFace¶

• Provider name: huggingface
• Default model: meta-llama/Meta-Llama-3.1-70B-Instruct
• Auth: Bearer token (HF_API_KEY)
• API endpoint: https://api-inference.huggingface.co/models/{model}/v1/chat/completions

- agent:
    type: react
    model:
      provider: huggingface
      model: meta-llama/Meta-Llama-3.1-70B-Instruct
      api_key: "${HF_API_KEY}"

Replicate¶

• Provider name: replicate
• Default model: meta/meta-llama-3.1-405b-instruct
• Auth: Bearer token (REPLICATE_API_TOKEN)
• API endpoint: https://api.replicate.com/v1/predictions
• Note: Uses async prediction API with polling

- agent:
    type: react
    model:
      provider: replicate
      model: meta/meta-llama-3.1-405b-instruct
      api_key: "${REPLICATE_API_TOKEN}"

Reasoning, FinishReason & Audio I/O¶

These surfaces are wired uniformly across the chat-completions providers (anything riding the openai-compat layer — that's openai, azure, huggingface, plus the seven openai-compat carriers + qwen). Where a vendor doesn't recognise a field the request still goes out cleanly; the response just lacks that surface.

reasoning_effort (o1/o3/gpt-5 + qwen-thinking)¶

Top-level knob for how many reasoning tokens the model spends before producing the final answer. Plumbed into the request body via ExtraBody.

import "github.com/arvenza/helix/pkg/agent/llm"

resp, err := model.Generate(ctx, messages,
    llm.WithReasoningEffort(llm.ReasoningEffortHigh))

Custom strings are also accepted (llm.WithReasoningEffort("ultra")) for vendor-specific tiers. No-op on providers that don't recognise the field.

reasoning_content (Qwen3 / DeepSeek-R1)¶

Some openai-compat providers stream a separate reasoning_content field carrying the model's thinking trace. Helix surfaces it on the Message:

resp, _ := model.Generate(ctx, messages,
    llm.WithEnableThinking(true),     // qwen-specific
    llm.WithReasoningEffort("high"),  // openai/qwen3-thinking
)

fmt.Println("trace:",   resp.Message.ReasoningContent)
fmt.Println("answer:",  resp.Message.Content)
fmt.Println("ended:",   resp.Message.FinishReason)

For streaming (StreamGenerate), each StreamChunk exposes the same ReasoningContent and FinishReason fields; CollectMessage concatenates the deltas. Use WithPreserveThinking(true) to keep the trace in the multi-turn history sent on the next call (Qwen3 drops it by default).

FinishReason everywhere¶

Every Message returned by Generate and every StreamChunk returned by StreamGenerate carries FinishReason — the provider's reported reason for ending generation. Stable values across providers:

Drives downstream behaviour:

if resp.Message.FinishReason == "length" {
    // bump max_tokens and retry, or trim memory
}
if resp.Message.FinishReason == "content_filter" {
    // surface a polite refusal in your UI
}

Audio output (Qwen-Omni, GPT-4o-audio)¶

WithAudioOutput(format, voice) sets the top-level modalities=["text","audio"] and audio={format, voice} fields on the request body via ExtraBody. Works against every openai-compat provider whose API recognises these fields.

import "github.com/arvenza/helix/pkg/agent/llm"

resp, _ := qwenOmni.Generate(ctx, messages,
    llm.WithAudioOutput(llm.QwenAudioFormatWav, llm.QwenVoiceCherry))

// resp.Message.Content    — the spoken transcript
// resp.Message.Parts[N]   — Type=ContentAudio carrying the WAV bytes

Qwen-Omni voices: QwenVoiceCherry, QwenVoiceSerena, QwenVoiceEthan, QwenVoiceChelsie. Audio format: QwenAudioFormatWav (DashScope currently only supports WAV for assistant output).

Sending audio to the model: attach AudioDataPart(mimeType, bytes) to the user message's Parts. The shared multimodal converter emits the input_audio content block; format is derived from the MIME type (audio/wav → wav).

audio, _ := os.ReadFile("question.wav")
msg := llm.Message{
    Role: "user",
    Parts: []llm.ContentPart{
        llm.AudioDataPart("audio/wav", audio),
        llm.TextPart("Transcribe this and reply in audio."),
    },
}

ExtraBody escape hatch¶

Any chat-completions field a vendor exposes — even one Helix doesn't model — can be plumbed through GenerateOptions.ExtraBody:

resp, _ := model.Generate(ctx, messages,
    llm.WithExtraBodyField("custom_safety", map[string]any{"strict": true}),
    llm.WithExtraBodyField("logprobs", true),
)

ExtraBody is map[string]any; keys are merged at the top of the JSON request body just before marshalChatRequestWithExtra writes it out. The Qwen-specific helpers (WithEnableThinking, WithPreserveThinking, WithAudioOutput) and the universal WithReasoningEffort all set fields through this path.

Embedding Providers¶

Embedding providers are used in rag-embed steps and in vector memory backends.

OpenAI Embeddings¶

• Provider name: openai
• Default model: text-embedding-3-small (1536 dimensions)
• Batch size: 2048

- rag-embed:
    provider: openai
    model: text-embedding-3-small
    api_key: "${OPENAI_API_KEY}"

Ollama Embeddings¶

• Provider name: ollama
• Default model: nomic-embed-text (768 dimensions)
• Batch size: N/A (local)

- rag-embed:
    provider: ollama
    model: nomic-embed-text
    base_url: "http://localhost:11434"

Cohere Embeddings¶

• Provider name: cohere
• Default model: embed-v4.0 (1024 dimensions)
• Batch size: 96

- rag-embed:
    provider: cohere
    model: embed-v4.0
    api_key: "${COHERE_API_KEY}"

Voyage AI Embeddings¶

• Provider name: voyage or anthropic
• Default model: voyage-3 (1024 dimensions)
• Batch size: 128

- rag-embed:
    provider: voyage
    model: voyage-3
    api_key: "${VOYAGE_API_KEY}"

Mistral Embeddings¶

• Provider name: mistral
• Default model: mistral-embed (1024 dimensions)
• Batch size: 512

- rag-embed:
    provider: mistral
    model: mistral-embed
    api_key: "${MISTRAL_API_KEY}"

Google Vertex / Gemini Embeddings¶

• Provider name: vertex or gemini
• Default model: text-embedding-004 (768 dimensions)
• Batch size: 100

- rag-embed:
    provider: gemini
    model: text-embedding-004
    api_key: "${GOOGLE_API_KEY}"

AWS Bedrock Embeddings¶

• Provider name: bedrock
• Default model: amazon.titan-embed-text-v2:0 (1024 dimensions)
• Batch size: N/A (single-request)

- rag-embed:
    provider: bedrock
    model: amazon.titan-embed-text-v2:0
    api_key: "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}"
    base_url: us-east-1

Jina Embeddings¶

• Provider name: jina
• Default model: jina-embeddings-v3 (1024 dimensions)
• Batch size: 500

- rag-embed:
    provider: jina
    model: jina-embeddings-v3
    api_key: "${JINA_API_KEY}"

HuggingFace Embeddings¶

• Provider name: huggingface
• Default model: BAAI/bge-large-en-v1.5 (1024 dimensions)
• Batch size: 64

- rag-embed:
    provider: huggingface
    model: BAAI/bge-large-en-v1.5
    api_key: "${HF_API_KEY}"

Nomic Embeddings¶

• Provider name: nomic
• Default model: nomic-embed-text-v1.5 (768 dimensions)
• Batch size: 400

- rag-embed:
    provider: nomic
    model: nomic-embed-text-v1.5
    api_key: "${NOMIC_API_KEY}"

Component URIs¶

Most LLM providers are also available as Helix route components with URI syntax (the agent layer reaches all 17 providers via provider: in YAML; the :// component scheme is currently wired for the 16 providers below — qwen is only reachable from the agent layer):

{scheme}://chat?model={model}&apiKey={key}&temperature={temp}&maxTokens={max}

Common Model Configuration Fields¶

All providers accept the same YAML configuration fields within model: blocks:

Provider Selection Guide¶