Skip to content

Models

A LanguageModel configures LLM access for a LanguageCluster. The operator reads all LanguageModel resources in the namespace and registers them with the cluster's shared LiteLLM gateway. Agents on gateway-backed runtimes never hold provider credentials or connect to providers directly: they call the gateway with their own gateway key.

The exception is vendor-key runtimes — Claude Code and Cursor — which talk to their vendor directly with the vendor's own key and do not use the gateway or LanguageModels at all.

How It Works

One LiteLLM proxy (gateway) runs per LanguageCluster. When you add or remove a LanguageModel, the gateway restarts with the updated model list — no agent redeploy required.

Credential Management

API keys are never injected into agent pods. Store them in a Secret:

kubectl create secret generic anthropic-credentials \
  --from-literal=api-key=sk-ant-your-key-here

Reference the Secret from the model spec:

apiVersion: langop.io/v1alpha1
kind: LanguageModel
metadata:
  name: claude-sonnet
spec:
  provider: anthropic
  modelName: claude-sonnet-4-5
  apiKeySecretRef:
    name: anthropic-credentials
    key: api-key

The gateway pod mounts the Secret and presents a single OpenAI-compatible endpoint to agents. Rotating a key is a kubectl create secret operation — the gateway restarts, agents are unaffected.

Agent Integration

The operator injects two environment variables into every agent container:

Variable Value
MODEL_ENDPOINT http://gateway.<namespace>.svc.cluster.local:8000
LLM_MODEL Comma-separated list of model names from spec.models[].name

Both are also available through /etc/agent/config.yaml under the models: key:

models:
  claude-sonnet:
    role: primary
    provider: anthropic
    model: claude-sonnet-4-5
    endpoint: http://gateway.my-cluster.svc.cluster.local:8000

Agents call the gateway with the model name they want. The gateway routes to the correct upstream provider.

Gateway Authentication

The gateway only serves requests carrying a valid key. Every agent gets its own:

  • The operator injects MODEL_API_KEY into each agent's containers, next to MODEL_ENDPOINT. Agents send it as the bearer key (Authorization: Bearer $MODEL_API_KEY); the bundled gateway-backed runtimes do this for you.
  • A key is sk-langop-<agent UUID>.<signature>, where the signature is an HMAC of the agent UUID under the cluster's secret. The gateway checks it without a key store, and records the agent UUID as the request's LiteLLM user_id, so usage can be attributed per agent.
  • An agent key works on model API routes only: chat completions, responses, embeddings, /v1/messages, the Gemini /v1beta/models/... routes, /v1/models and /model/info (LiteLLM's own list of LLM routes). Admin routes such as /model/new, /key/*, /user/*, /team/*, spend and config get 403; they need the master key.
  • A request without a valid key gets 401. Custom agent images must send MODEL_API_KEY; one that sends a fixed placeholder is rejected.

The HMAC secret lives in the Secret gateway-auth (key hmac-secret) in the cluster's namespace. The operator generates it on first reconcile; create it yourself beforehand to choose the value. Rotating it (edit the value, or delete the Secret to have a new one generated) restarts the gateway and replaces every agent's key, and running agents are restarted with their new key.

For access outside an agent — an admin, a script, a client using the gateway's external ingress — set a master key on the gateway, which the gateway also accepts:

apiVersion: langop.io/v1alpha1
kind: LanguageCluster
spec:
  gateway:
    deployment:
      env:
        - name: LITELLM_MASTER_KEY
          valueFrom:
            secretKeyRef:
              name: gateway-master-key
              key: key

The master key must be at least 32 characters, and must not be sk-langop-proxy. That value was the default key in older adapter images and appears in public repositories; it was never a credential. The gateway refuses to start with either, naming the problem in its log, and its auth hook ignores such a key too. The check covers LITELLM_MASTER_KEY and a general_settings.master_key set through LANGOP_GATEWAY_EXTRA_CONFIG. Agents never need the master key: they authenticate with their own MODEL_API_KEY. Without a master key, every request needs a valid agent key. To create a strong one:

kubectl create secret generic gateway-master-key -n <cluster-namespace> \
  --from-literal=key="sk-$(openssl rand -hex 32)"

Supported Providers

Provider Value
Anthropic provider: anthropic
OpenAI provider: openai
Google AI Studio (Gemini API) provider: gemini
Azure OpenAI provider: azure (+ endpoint, apiVersion)
AWS Bedrock provider: bedrock (+ region)
Google Vertex AI provider: vertex (+ project, location)
Any OpenAI chat-completions API (Ollama, vLLM, LM Studio…) provider: openai-compatible (+ endpoint)
Any other LiteLLM provider (DeepSeek, Qwen, xAI, Mistral, Groq, OpenRouter…) litellmProvider: <LiteLLM prefix>

custom is deprecated and behaves exactly like openai-compatible. Providers that need more than one credential (Bedrock access keys, a Vertex service account, an Azure AD app) take a whole Secret through credentialsSecretRef, applied to that model only. See LanguageModel for each provider's fields and examples.

Other providers

Anything LiteLLM supports is reachable with litellmProvider, its LiteLLM provider prefix, plus the provider's API key. LiteLLM knows each one's base URL, so endpoint is only needed to override it:

Provider litellmProvider
DeepSeek deepseek
Z.ai (GLM) zai
Moonshot (Kimi) moonshot
MiniMax minimax
xAI (Grok) xai
Mistral mistral
Groq groq
Together AI together_ai
Fireworks AI fireworks_ai
OpenRouter openrouter
Alibaba Qwen (DashScope) dashscope
Cerebras cerebras
Perplexity perplexity
spec:
  litellmProvider: deepseek
  modelName: deepseek-chat
  apiKeySecretRef:
    name: deepseek-credentials

See LiteLLM's provider list for the rest. Every example in the repository is a complete, CI-validated spec.

Self-hosted models (Ollama, vLLM)

spec:
  provider: openai-compatible
  modelName: llama3.2
  endpoint: http://ollama.default.svc.cluster.local:11434/v1

No apiKeySecretRef needed for unauthenticated endpoints. The same shape works for any server with an OpenAI-compatible API; their usual defaults:

Server endpoint (default port)
Ollama http://<host>:11434/v1
vLLM http://<host>:8000/v1
llama.cpp (llama-server) http://<host>:8080/v1
SGLang http://<host>:30000/v1
LM Studio http://<host>:1234/v1

The backend only needs /v1/chat/completions. Clients that speak the Responses API (/v1/responses, e.g. Codex) or the Anthropic Messages API (/v1/messages) still work: the gateway translates both to chat completions for these models.

Multiple models

Agents can reference multiple models. Each model is registered with the same gateway; the agent chooses which to call at runtime:

# LanguageAgent
spec:
  models:
    - name: claude-sonnet   # primary
    - name: llama3          # fallback / secondary

Wildcard models: a whole vendor catalogue

A LanguageModel with modelName: "*" stands for every model the provider offers, so one resource covers a vendor like OpenRouter:

apiVersion: langop.io/v1alpha1
kind: LanguageModel
metadata:
  name: openrouter
spec:
  provider: openai-compatible
  modelName: "*"
  endpoint: https://openrouter.ai/api/v1
  apiKeySecretRef:
    name: openrouter-credentials

The gateway registers it as openrouter/*. An agent picks a model by the vendor's own name with model, and calls the gateway with <LanguageModel name>/<model>; the gateway strips the prefix before calling the provider:

# LanguageAgent
spec:
  models:
    - name: openrouter
      model: anthropic/claude-sonnet-4.5   # the agent's LLM_MODEL is openrouter/anthropic/claude-sonnet-4.5

Each wildcard is addressed by its own resource name, so several vendors can coexist, and a name no LanguageModel serves is still rejected. Rate limits and the timeout apply to all of the vendor's models together. * is only valid on its own: partial patterns such as gpt-* are rejected.

Listing models

For a wildcard, the gateway's /v1/models does not return the vendor's catalogue. It lists LiteLLM's built-in table of known models for the provider type — for an OpenAI-compatible vendor like OpenRouter, that is OpenAI's model names. To discover what a vendor offers, use the vendor's own model list API.

Load balancing: models that share a modelName

Agents call the gateway by modelName, not by the LanguageModel's own name. When several LanguageModels in a cluster have the same modelName, the gateway serves them as one model and spreads requests across all of them. Use this to run one model on several endpoints:

apiVersion: langop.io/v1alpha1
kind: LanguageModel
metadata:
  name: llama-gpu-a
spec:
  provider: openai-compatible
  modelName: llama3.2
  endpoint: http://ollama-a.inference.svc.cluster.local:11434/v1
---
apiVersion: langop.io/v1alpha1
kind: LanguageModel
metadata:
  name: llama-gpu-b
spec:
  provider: openai-compatible
  modelName: llama3.2
  endpoint: http://ollama-b.inference.svc.cluster.local:11434/v1

Because the same thing happens by accident, for example two models with the same modelName but different providers or keys, creating or updating a LanguageModel whose modelName is already taken returns a warning naming the other models, and the gateway logs one on start. Give each model a distinct modelName if you want them addressed separately.

Model aliases: answering to a client's model names

Some clients ask for fixed model names of their own instead of the one you configured. spec.aliases lists extra names the gateway serves with a LanguageModel. A request for an alias goes to that model, on every API the gateway speaks (OpenAI, Anthropic and Gemini formats), and aliases appear in the gateway's /v1/models list.

apiVersion: langop.io/v1alpha1
kind: LanguageModel
metadata:
  name: claude-sonnet
spec:
  provider: anthropic
  modelName: claude-sonnet-4-5
  apiKeySecretRef:
    name: anthropic-credentials
  aliases:
    - gemini-3.1-pro-preview
    - gemini-3.1-flash-lite
  • A real modelName always wins over an alias: an alias equal to any LanguageModel's modelName is ignored, with a warning in the gateway log.
  • An alias belongs to one model. If two LanguageModels claim it, the first in name order keeps it and the gateway logs a warning. Load-balanced models that share a modelName may list the same alias.
  • A wildcard model (modelName: "*") cannot have aliases.
  • A name that is neither a modelName nor an alias gets an immediate 400 Invalid model name, so a missing alias shows up as a fast error, not a hang.

Gemini CLI

Gemini CLI speaks only the Gemini API, which the gateway serves at /v1beta/models/<model>:generateContent and :streamGenerateContent. Inside an agent, point it at the gateway with the agent's own key:

export GOOGLE_GEMINI_BASE_URL="$MODEL_ENDPOINT"
export GEMINI_API_KEY="$MODEL_API_KEY"
gemini -p "say hi"

The CLI calls several Gemini model IDs besides the one you choose: a classifier picks a model per turn, and helpers for summarising, prompt completion and edit correction use a flash-lite model. Each of those IDs must be an alias, or the call fails with 400. As of Gemini CLI v0.62.0 (packages/core/src/config/defaultModelConfigs.ts), the full set is:

  aliases:
    - gemini-2.5-pro
    - gemini-2.5-flash
    - gemini-2.5-flash-lite
    - gemini-3-pro-preview
    - gemini-3-flash-preview
    - gemini-3-flash
    - gemini-3.1-pro-preview
    - gemini-3.1-pro-preview-customtools
    - gemini-3.1-flash-lite
    - gemini-3.1-flash-lite-preview
    - gemini-3.5-flash
    - gemini-3.5-flash-lite
    - gemini-3.8-flash

The set changes between CLI releases, so check it against the CLI version you run. To send the CLI's main and helper calls to different models, split the list across two LanguageModels, for example the pro IDs on a large model and the flash/flash-lite IDs on a small one.

Advanced: Extra Gateway Configuration

For LiteLLM settings with no LanguageModel field (retries, fallbacks, caching, spend callbacks, or a hand-written model entry) set LANGOP_GATEWAY_EXTRA_CONFIG on the gateway. It is a YAML mapping deep-merged into the generated LiteLLM config: mappings merge recursively, lists and scalars replace.

apiVersion: langop.io/v1alpha1
kind: LanguageCluster
spec:
  gateway:
    deployment:
      env:
        - name: LANGOP_GATEWAY_EXTRA_CONFIG
          value: |
            litellm_settings:
              num_retries: 2
              request_timeout: 600

A list such as model_list replaces the generated one entirely, so prefer LanguageModels for models. A value that is not a mapping stops the gateway from starting.

Rate Limiting

spec:
  rateLimits:
    requestsPerMinute: 100
    tokensPerMinute: 50000

Limits are enforced by the shared gateway across all agents. Per-agent limits are not currently supported.