LLM Gateway on a self-hosted install

The LLM Gateway in the Hybrid or Air-Gapped chart: upgrade to it, publish its host on a public or private load balancer, size it, and open the egress it needs.

On Hybrid and Air-Gapped installs the gateway runs in your cluster. Model calls, provider keys and request records stay inside your environment; settings are managed in the Platform as on xpander cloud.

Always on

The gateway is always on, and agents send every model call through it. It cannot be turned off: since Hybrid chart 0.22.5 and Air-Gapped chart 0.17.2, the chart ignores llm-gw.enabled. To publish its host, set these values:

llm-gw:
  ingress:
    enabled: true      # publish llm-gw.<your domain>
    internal: false    # true: serve the host on a private load balancer

Then upgrade the release.

Upgrading an existing install

Every upgrade starts the gateway: install.sh upgrade and any helm upgrade, including --reuse-values. A llm-gw.enabled: false left in your values is ignored, so you can keep it or remove it. If your install had the gateway off, the upgrade starts the gateway pods and restarts agent-controller and api once.

Before you upgrade, mirror the llm-gw image if you mirror images, open the gateway’s network paths, and make sure every model provider your agents use has a key: one saved in the organization (Models) or one set on the install (secrets.static). A call to a provider with neither is refused with a message that names the missing key. Custom OpenAI-compatible endpoints keep working as configured.

New agent runs wait for a healthy gateway. If its pods cannot start, for example because the image cannot be pulled, new runs are refused with “The xpander AI gateway is unavailable right now” until they are ready. Check them after the upgrade:

kubectl -n <namespace> get pods -l app=llm-gw

A healthy gateway shows two llm-gw pods (one on a single-node cluster installed with install.sh), each 1/1 and Running.

On an Air-Gapped install:

• Mirror the gateway image first with the current image_sync.sh --manifest-out, which includes the llm-gw and tool-repo-seed images and pins them in your values. --icon-pack-tag no longer exists; remove it from your own scripts. If the gateway image is not pinned, the render stops and names the value to set.

Value	Default	What it does
llm-gw.ingress.enabled	false	Publishes llm-gw.<your domain>; needs the chart-wide ingress on
llm-gw.ingress.host	llm-gw.<your domain>	Another host name
llm-gw.ingress.internal	false	Serves the host on a private load balancer (internalClassName, internalAnnotations)
llm-gw.env		HTTPS_PROXY and NO_PROXY on a cluster that reaches the internet through a proxy
llm-gw.serviceAccountAnnotations		Workload identity for the gateway’s own service account

Agents in your cluster reach the gateway by its in-cluster service. This address is the Base URL the Models page shows when the host is not published. It is http://llm-gw:4000 on an Air-Gapped install, and http://xpander-llm-gw:4000 on a Hybrid install with the release name xpander (release names). The published host is for laptops, your apps and SDKs. Publishing it needs a DNS record and a certificate that covers the host (the chart reuses ingress.tls.secretName). While no gateway pod is ready, a small fallback pod answers the published host with a JSON 503 that coding tools and SDKs retry. The pod is llm-gw-unavailable on Air-Gapped, and xpander-llm-gw-unavailable on Hybrid with the release name xpander. Only the model paths are published (/v1/, /anthropic/, /gemini/, /openai/). Health, metrics and everything else stay inside the cluster.

On an Air-Gapped install, people point their coding tools at this host (Connect your tools). On a Hybrid install, people’s coding tools connect to xpander cloud instead (below), so publishing the host is needed only for your own apps and SDKs inside your network.

Long streams

A coding session can hold one response open for many minutes. The chart sets a 1500 second idle timeout on the ingress; set at least 1500 seconds on any load balancer in front of it (on an AWS Application Load Balancer, idle_timeout.timeout_seconds=1500).

Size

	Default
Servers	2 to 20, scaled on CPU (one on a single-node cluster installed with install.sh)
Each server	0.5 to 1 vCPU, 1 to 4 GiB memory
Database	None for the gateway itself; request records go to your install’s PostgreSQL

Plan the PostgreSQL storage by request volume and the retention you set in Platform > AI governance > Policies. See Sizing.

Keys and secrets

The chart generates the gateway’s internal key on install and keeps it on upgrade (Secret xpander-llm-gw on Hybrid with the release name xpander, xpander-llm-gw-keys on Air-Gapped). No caller ever uses it. To manage it yourself, set llm-gw.masterKey.existingSecret on Hybrid or create the Secret yourself with keygen off on Air-Gapped. On an ArgoCD install, either set it yourself or have ArgoCD ignore the generated Secret’s data, as in examples/argocd-application.yaml.

The chart also generates the key that encrypts saved prompt and response text. On Hybrid it is the Secret xpander-llm-gw-content with the release name xpander, or set llm-gw.contentKey.existingSecret. On Air-Gapped it is keygen.overrides.LLM_GW_CONTENT_KEY. Never delete or rotate it: saved text cannot be read after it changes. On Hybrid, both generated Secrets are kept on upgrade and on uninstall. On an ArgoCD install, ignore /data on both Secrets or set both existingSecret values.

No provider key is set on the gateway. Provider keys stay in your vault, or on the install in secrets.static, and are added to each call (Providers and keys).

Network

From the gateway to	Why
Your model providers, or your OpenAI-compatible endpoints	The calls themselves
The xpander services in your cluster, and Redis	Keys, settings and usage

With the chart’s network policy on, the gateway accepts traffic only on its serving port and can never reach the cloud metadata address. Egress is limited to DNS, agent-controller, Redis and public https addresses. List anything else it must reach in llm-gw.networkPolicy: providerCidrs for model endpoints on your own network (any port), cloudEndpointCidrs for private cloud endpoints such as Bedrock or Azure (port 443), and redisCidrs for an external Redis. Behind an outbound proxy, set HTTPS_PROXY and NO_PROXY in llm-gw.env; the gateway does not read global.env. A sealed install (no internet egress) uses only endpoints inside your network, and Bring your own key is the only license model.

If your network requires its own certificate authority for a gateway or proxy, the LLM Gateway trusts it the same way the other services do (Air-gapped).

Log destinations are sent from your install, so the collector or bucket you set only needs to be reachable from your cluster.

Hybrid specifics

• Settings. Policies, models, keys and budgets are managed in the Platform and reach your install within seconds.
• Agents. Agents on your install call your install’s gateway, on your stored keys and the credentials your install provides. xpander’s keys are never used.
• People’s coding tools. Claude Code, Codex, OpenCode and SDKs on your people’s computers connect to xpander cloud at https://llm-gw.xpander.ai, with their personal xpander key (Connect your tools). Those calls run on the person’s own Claude plan or a provider key you stored in xpander, never on xpander’s keys or on credentials inside your install. A model whose provider has no key stored in xpander shows in the person’s model list with No key for this provider is stored in xpander, so tools on your computer can’t use these models. Agents in your own environment may still use them. Their prompt and response text is never kept in xpander cloud.
• Records. Requests of agents on your install stay in your install: the Platform shows them when your browser can reach your environment directly, and totals for billing are sent with billing events. Requests from people’s coding tools are stored in your organization’s xpander cloud database and show in Logs.
• Budgets. See Budgets on self-hosted installs.