LiteLLM Fallback & Gateway Config

For routing strategy and cost optimization, see Model Routing and Cost & Adversarial Review.

Air-Gapped Fallback to LiteLLM

Not every deployment can reach the public model providers. In air-gapped or self-hosted environments, OpenFlows can fall back to a LiteLLM proxy that fronts locally hosted models (vLLM, Ollama, Text Generation Inference, or enterprise endpoints). When the Coder AI Gateway is unreachable, the orchestrator switches to the configured ai_gateway.fallback and passes the same provider / model pair to LiteLLM.

LiteLLM acts as a model router. It exposes an OpenAI-compatible endpoint and translates requests to whatever backend model you have configured. From OpenFlows' perspective, the fallback looks like a normal OpenAI provider call. The only difference is the base URL.

Configuring the Fallback

Set the fallback gateway in registry.json and provide the LiteLLM URL and key through the orchestrator environment (not the registry). The orchestrator process holds the LiteLLM credentials; worker workspaces never see them.

bash

# In the orchestrator environment (the machine running NEXUS / the controller)
export LITELLM_URL=http://litellm.internal:4000
export LITELLM_API_KEY=sk-litellm-...
# Only needed if LiteLLM proxies directly to a provider without holding keys itself:
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

Environment Variable	Purpose
`LITELLM_URL`	Base URL of the LiteLLM proxy (e.g. `http://litellm.internal:4000/v1`).
`LITELLM_API_KEY`	API key for the LiteLLM proxy. Stored in the orchestrator environment, not in worker workspaces.
`OPENAI_API_KEY`	Only needed if LiteLLM forwards directly to OpenAI without proxy-managed keys.
`ANTHROPIC_API_KEY`	Only needed if LiteLLM forwards directly to Anthropic without proxy-managed keys.

LiteLLM model aliases

LiteLLM maps incoming model names to backend models. Make sure your LiteLLM config defines aliases for the model identifiers used in registry.json, such as claude-sonnet-4-20250514 and gpt-4.1. If the alias is missing, LiteLLM returns a 404 and the orchestrator marks the request as failed.

Coder AI Gateway Configuration

The Coder AI Gateway is the primary routing path. It is part of your Coder deployment and is configured through the Coder admin interface, not through OpenFlows. OpenFlows only needs to know that the gateway exists and which model/provider slugs to pass to it.

Provider slugs. The value of agents.{role}.provider must match a provider slug configured in the Coder AI Gateway. Common slugs are anthropic, openai, google, fireworks, and groq. If you add a custom provider, add the same slug to registry.json.

Model identifiers. The value of agents.{role}.model is passed through to the gateway. The gateway may accept the raw model identifier (e.g. claude-sonnet-4-20250514) or a gateway alias. Check your gateway's model list to confirm.

Audit and cost. Because all calls route through the gateway, Coder can log which agent made which request, how many tokens were consumed, and what the cost was. This is the source of truth for usage attribution.

bash

# Verify the gateway can see a provider/model from the orchestrator host
curl -H "Authorization: Bearer $CODER_TOKEN"   https://coder.example.com/api/v2/ai-gateway/models

# Look for the provider and model slugs you intend to use.
# If the model is missing, the orchestrator will fail at request time with a 404.

Gateway availability is the critical path

If the Coder AI Gateway is down, the primary path fails. The fallback to LiteLLM only activates if ai_gateway.fallback is set and the LiteLLM URL is reachable. If neither is available, all agent work stops. Monitor gateway health as closely as you monitor Redis and Coder itself.