Hybrid deployment - xpander.ai

Hybrid is the self-hosted deployment where your data and agents run in your cluster and the control plane stays on xpander cloud. Execution, runtime data, credentials, and model keys stay inside your VPC; xpander cloud manages agent definitions, identity, and the skills catalog. The Agent Controller is the only cluster component that reaches Cloud (heartbeats, agent and skill pulls). This page is the complete install guide: from architecture review to a running, connected location.

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY>

Self-hosted locations are enabled per account: contact our team if the option isn’t available on yours.

Comparing deployments first? See Deployment options. The deployment that runs every component with no connection to xpander cloud at run time is Air-gapped deployment. Network boundary, PrivateLink, and exactly what leaves your VPC are on Hybrid Architecture: this page does not repeat those diagrams.

Architecture

The AWS network boundary, PrivateLink, authentication and runtime flows, and the Hybrid → Cloud sync surface (cluster identity, unpublished heartbeat schema, agent catalog, skill schema) are on Hybrid Security Architecture. This section is the operational view: what the chart deploys, how clients connect, and the residency table for an install review. Hybrid follows the control plane / data plane separation pattern, the same model used by AWS EKS, Datadog, and other infrastructure platforms:

Control plane (xpander Cloud): manages environment metadata, agent registration, skill definitions, and event logging. Admins and team members use Xpander Chat to assemble agents from approved skills and validate tasks. The control plane never touches runtime data.
Data plane (your VPC): runs all task execution, agent memory, skill credentials, model API keys, and user-facing services. The user’s browser connects directly to the data plane. Chat threads, activity views, and task results are served from the cluster: they do not pass through the cloud. Agent definitions and the skills catalog are authored in Cloud and pulled by the Agent Controller.

The data plane connects outbound only to the control plane over port 443 for metadata sync and heartbeats. The control plane never initiates connections into your cluster. MCP and external-system calls run from your cluster, and agent workspaces, the agents’ computers, run there too.

What Gets Deployed

Resource names on this page assume the release name xpander, as in every command here. With another release name, they start with <release>-xpander- instead, or with <release>- when the release name already contains xpander.

Component	Description
Agent Controller	Task scheduling, queue management, and dispatch
LLM Gateway (`llm-gw`)	One endpoint in front of every model provider; agents’ model calls go through it. Always on, see LLM Gateway
Execution tier (`agent-worker`)	Task execution runtime
API Proxy	API endpoint and request routing
API	REST API service
MCP	Accepts MCP client connections, generates tasks
Database	Agent memory, message history, and monitoring (PostgreSQL)
Redis	Cache and fast lookups
Chat UI	Chat interface with per-thread subdomains
Code Runner	Code execution runtime
AWS APIs Operator	AWS resource management

How It Works

Clients (Slack, the web UI, MCP clients, the SDK and API) connect directly to the cluster in your VPC. When using chat.xpander.ai, the cloud UI handles authentication and tells the client which self-hosted cluster to connect to; chat, tasks, and results then flow client-to-cluster. Inside the cluster, the Agent Controller schedules tasks and the execution tier (the agent-worker service) executes them: calling LLMs, persisting messages in PostgreSQL, and reaching the external systems behind skills through the Agent Controller’s skill relay. The only connection to xpander Cloud is the deployment manager sync: the Agent Controller authenticates with organization_id + environment_id + api_key, reports heartbeats, and pulls agent and skill (including MCP) configuration. The heartbeat JSON schema is not published. See Data that leaves the on-prem installation.

What Talks to What

The egress surface: the answers a security review needs. Inbound from xpander cloud: none; connections are outbound-only.

From your cluster	To	Port	Purpose
`agent-controller`	`deployment-manager.xpander.ai` (or the PrivateLink / staging endpoint)	443	Metadata sync, heartbeats, skill updates
`llm-gw` (the LLM Gateway)	Your LLM providers (OpenAI, Anthropic, Google, Bedrock, …)	443	Model calls
Skills (including MCP)	The external systems you connect	As configured	Run from your cluster, never from xpander cloud

Outbound connectivity options to the control plane:

Option	Target	Details
Global Accelerator	`15.197.85.80`, `166.117.85.46`	Public internet, encrypted over TLS
PrivateLink	`com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197`	AWS private backbone: setup below

Data Residency

Feature	Stays in your VPC?
Invoking agents (API to agent)	Yes: client connects directly to cluster
Chat threads and activity view	Yes: fetched from cluster, not cloud
Debugging agents (Session Manager)	Yes: all data served from cluster
Knowledge Base file uploads and queries	Yes: stored and queried in cluster
User identity and login	No
Agent metadata (agent list)	No

Data at rest stays in PostgreSQL and Redis inside your VPC; credentials and API keys are Kubernetes secrets; the control plane sync carries metadata and heartbeats only: user payloads never leave the VPC. Heartbeat field names are not published. Full field-level inventory: Data that leaves the on-prem installation.

Requirements

Sizing lives on one page: the minimum footprint, the sizing model and calculator, per-service requirements, the executor fleet and the data services are all on Sizing and requirements. The installer checks the same minimum numbers, published at charts.xpander.ai/requirements.json.

xpander container images are amd64 only. Do not use ARM/Graviton instances (t4g, m7g, c7g, etc.).

The release namespace must admit PodSecurity baseline (unlabelled is fine; restricted is refused), and the nodes that run the executor fleet need Landlock or fleet.landlock.enabled: false. See Node prerequisites. Need a cluster? Follow the EKS Cluster Setup guide. Bedrock needs your own credentials. Agents on a Hybrid install never use xpander’s keys (LLM Gateway). To run Claude Code on Bedrock here, add a Bedrock key or role under Platform > AI governance > Models, or give the cluster an AWS identity for Bedrock; see AI vendors.

Install

1. Get your one-liner from Xpander Chat

In Xpander Chat, open Platform > Agents > Runtime and choose the Locations tab (Connected runtimes and regions)
Click “Add Location”
Choose the Hybrid edition. The dialog shows the minimum footprint for the deployment you picked.
Optionally set a URL (your server’s FQDN, e.g. xpander.my-company.com) and the hostname style (dash-based subdomains if your DNS can’t do wildcards)
Copy the generated one-liner

The one-liner carries your identity, --org, --env, and --dm-api-key (plus --domain when you set a URL). That is how the cluster authenticates to the xpander control plane:

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid \
  --org <ORGANIZATION_UUID> \
  --env <ENVIRONMENT_UUID> \
  --dm-api-key <DEPLOYMENT_MANAGER_API_KEY> \
  --domain xpander.my-company.com

The one-liner deliberately carries no LLM provider credentials: the wizard prompts for those interactively (input hidden), or you pass them as flags. Secrets are written to a temporary values file (mode 600, removed on exit) and never appear on the helm command line.

2. Run the Installer

Run it from a terminal with kubectl access to the target cluster. The command downloads the installer from charts.xpander.ai and runs an interactive wizard: preflight checks, configuration, Helm install, then the connection back to xpander. Run bare (curl -fsSL https://charts.xpander.ai/install.sh | bash), it is the zero-knowledge entry point: the first question is which edition you are installing. The wizard walks through, in order:

Edition & requirements

If --edition wasn’t given, it asks. It then prints the minimum footprint for that edition (the same numbers as charts.xpander.ai/requirements.json) and asks you to confirm your cluster meets it, stopping cleanly if not.

Identity

Organization ID, Environment ID, and the Deployment Manager API key (input hidden): all from the install card in Xpander Chat, all skipped if passed as flags.

Ingress domain (optional)

Press Enter to skip: services then keep cluster-internal Services only, which is fine for a first install (port-forward reaches them; add ingress later). With a domain, it asks how to build hostnames: subdomain style (agent-controller.xpander.<DOMAIN>, needs wildcard DNS) or dash style (agent-controller-xpander.<DOMAIN>, individual A records).

Control plane reachability

Public internet (https://deployment-manager.xpander.ai), AWS PrivateLink (https://deployment-manager-privatelink.xpander.ai), or Staging. The installer auto-detects staging/local environments (a .stg. or local domain, or a kind/k3d/minikube/docker-desktop kube context) and leads with that option; any of --staging, --privatelink, or --dm-url overrides detection.

LLM providers (all optional)

OpenAI, Anthropic, Google/Gemini, Amazon Bedrock, then optionally Fireworks, Nebius, Helicone, and OpenRouter. Press Enter to skip any: the xpander installs and starts without keys, and model calls light up as soon as you add a provider key in the UI or with a helm upgrade.For Bedrock it asks how to authenticate: an API key / bearer token (AWS_BEARER_TOKEN_BEDROCK), or keyless IAM role / EKS pod identity. See Amazon Bedrock: keyless IAM or API key.

Namespace, context confirmation, summary

Namespace (default xpander, created if missing), an explicit confirmation of the kubectl context it is about to install into, and a masked configuration summary before anything changes.

Install Phases

The Hybrid install runs six phases, each announced as a stable [N/6] name: ... progress line (the same shape scripts can grep, see Automation & CI):

[1/6] preflight       helm ≥ 3.13, kubectl, cluster reachable, Kubernetes ≥ 1.28
[2/6] configuration   context confirmation + masked summary
[3/6] chart-source    helm repo add xpander https://charts.xpander.ai (or --chart-path)
[4/6] values          secrets written to a mode-600 temp file, removed on exit
[5/6] install         helm upgrade --install xpander xpander/xpander
[6/6] verify          waits for the agent-controller rollout (5-minute timeout)

The preflight accepts Helm 3.13, but the documented upgrade command (--reset-then-reuse-values) needs Helm 3.14 or later. Every failure prints a FIX: line naming the action that repairs it and a RERUN: line with your personalized non-interactive one-liner (secrets typed at prompts are placeholdered, never echoed). Fix the condition and paste the rerun line: the installer is idempotent.

Amazon Bedrock: keyless IAM or API key

--bedrock-auth iam is keyless: the pods sign Bedrock calls with their own AWS identity (EKS Pod Identity or IRSA) for chat, agent task executions and the summaries around them, supported since xpander-sdk 2.0.503 (images built after 27 August 2026). The role bound to the pods needs the Bedrock permissions itself (bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream, bedrock:Converse, bedrock:ConverseStream on *) and must be bound to both ServiceAccounts the LLM services run as: default and agent-controller. The chat and task paths do not assume a second role. --bedrock-auth key (default when --bedrock-key is given) stores a bearer token instead. Either way the account must be opted in to Bedrock’s data retention mode for the models you use. The “IAM role” method on a provider under Platform > AI governance > Models is not the path for chat and tasks; it applies to the Claude Code leg described next. If chat fails with a Bedrock entry configured that way, remove the entry and restart the agent-controller, agent-worker, agents, actions and functions deployments. Claude Code on Bedrock takes a different path. The CLI’s model calls go through the Agent Controller to the in-cluster LLM Gateway, and the controller resolves AWS auth itself, in this order: AWS_BEARER_TOKEN_BEDROCK, then BEDROCK_ASSUME_ROLE_ARN (STS, self-refreshing), then static keys, then the pod’s default credential chain (Pod Identity / IRSA). Set whichever you use as controller environment through the chart’s env values, with BEDROCK_REGION or AWS_REGION next to it. An organization that wants Claude Code on Bedrock adds its own Bedrock key under Platform > AI governance > Models; that key is preferred over the controller’s credential.

Fable- and Mythos-class Anthropic models on Bedrock need the AWS account’s data-retention opt-in, per region. With the default mode Bedrock answers 400 data retention mode 'default' is not available for this model. There is no console switch: PUT https://bedrock.<region>.amazonaws.com/data-retention with {"mode": "provider_data_share"} (or aws_review where offered), SigV4-signed with bedrock:PutAccountDataRetention, in every region a global. inference profile can route to, on whichever account signs the call (the controller’s, or the organization’s own Bedrock key account). Details on AI vendors.

3. Verify the Connection

When the install completes, a summary box prints the release, control-plane URL, ingress state, and configured providers. Your cluster registers with the control plane and the xpander console flips to “Connection established” in a minute or two.

kubectl get pods -n xpander

Expect one Running, 1/1-ready pod per application service: agent-controller, agent-worker, ai-gateway, api, aws-operator, chat, code-runner, and mcp. ai-gateway is a registration-only compatibility component; skill calls, including AWS ones, are served by agent-controller since chart 0.16.1. The LLM Gateway runs as two llm-gw pods (one on a single-node cluster installed with install.sh). You should also see the postgres and redis StatefulSets, and, at defaults, docker-registry and metrics-server. Test locally without ingress:

kubectl -n xpander port-forward service/xpander-agent-controller 9016:9016 &
kubectl -n xpander port-forward service/xpander-ai-gateway 9018:9018 &
kubectl -n xpander port-forward service/xpander-mcp 8081:8081 &

curl http://localhost:9016/health
curl http://localhost:9018/health
curl http://localhost:8081/health

In Xpander Chat, the sidebar ENVIRONMENTS switcher lists the installation next to “xpander.ai cloud - Managed”. Platform > Agents > Runtime, Locations tab, shows it under Connected runtimes and regions. Its Components tab shows heartbeat status per service, and its Configuration panel holds the Cloud Controls switches: Chat, Streaming and Scheduler.

PrivateLink

Your cluster must reach the xpander deployment manager. Two options:

Option	URL	Security
Public (TLS)	`https://deployment-manager.xpander.ai`	Encrypted over public internet. No setup needed; the installer’s default
Private (PrivateLink)	`https://deployment-manager-privatelink.xpander.ai`	Encrypted over AWS private backbone. This section

If public works for you, skip this section entirely: your cluster nodes just need outbound HTTPS (via NAT Gateway if in private subnets). If your security policy requires traffic to stay within the AWS network, set up PrivateLink, then tell the installer with --privatelink (or pick AWS PrivateLink at the wizard’s reachability question). --privatelink, --staging, and --dm-url are mutually exclusive: pass exactly one, or none for the public endpoint.

How PrivateLink Works

AWS PrivateLink is a two-sided connection. xpander exposes the deployment manager as a VPC Endpoint Service (provider). You create a VPC Endpoint (consumer) in your VPC, which provisions a private ENI with a 10.x.x.x IP address. Your pods connect to this private IP: traffic goes over the AWS backbone, never the internet.

Property	Value
Service Name	`com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197`
Service Region	`us-west-2`
Supported Regions	`us-west-2`, `us-west-1`, `eu-west-2`, `eu-central-1`, `ap-south-1`
Acceptance Required	No (auto-accepted for allowed accounts)
Private DNS Name	`deployment-manager-privatelink.xpander.ai`

Contact xpander to have your AWS account ID added as an allowed principal before creating the VPC endpoint.

Set It Up

Create the VPC endpoint

The xpander endpoint service is in us-west-2. If your cluster is in a different region, you must include the --service-region us-west-2 flag: this is the most common mistake (see Troubleshooting).

aws ec2 create-vpc-endpoint \
  --vpc-id <VPC_ID> \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  --subnet-ids <PRIVATE_SUBNET_A_ID> <PRIVATE_SUBNET_B_ID> \
  --no-private-dns-enabled \
  --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=xpander-privatelink}]' \
  --region <REGION> --profile <PROFILE>

Save from the output: VpcEndpointId, Groups[0].GroupId (the security group), DnsEntries[0].DnsName, and DnsEntries[0].HostedZoneId.

If your cluster is in us-west-2 (same region as the service), you can omit --service-region.

Allow HTTPS on the endpoint security group

The VPC endpoint is created with the VPC’s default security group, which only allows inbound traffic from itself. Allow HTTPS from your VPC CIDR:

aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp \
  --port 443 \
  --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>

Wait for the endpoint to become available

The endpoint transitions from pending to available in 1-3 minutes (auto-accepted):

aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids <VPCE_ID> \
  --query 'VpcEndpoints[0].State' \
  --region <REGION> --profile <PROFILE>

Create private DNS

Create a private hosted zone so deployment-manager-privatelink.xpander.ai resolves to the endpoint ENI inside your VPC:

aws route53 create-hosted-zone \
  --name deployment-manager-privatelink.xpander.ai \
  --caller-reference "xpander-privatelink-$(date +%s)" \
  --vpc VPCRegion=<REGION>,VPCId=<VPC_ID> \
  --hosted-zone-config PrivateZone=true \
  --profile <PROFILE>

Save the HostedZone.Id from the output, then create the alias record:

cat <<EOF > /tmp/privatelink-dns.json
{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "deployment-manager-privatelink.xpander.ai.",
      "Type": "A",
      "AliasTarget": {
        "DNSName": "<VPCE_DNS_NAME>",
        "HostedZoneId": "<VPCE_HOSTED_ZONE_ID>",
        "EvaluateTargetHealth": true
      }
    }
  }]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <PRIVATE_ZONE_ID> \
  --change-batch file:///tmp/privatelink-dns.json \
  --profile <PROFILE>

Verify connectivity

kubectl run curl-test --restart=Never --image=curlimages/curl:latest -n default \
  --command -- sh -c "curl -sk -o /dev/null -w '%{http_code}' \
  --connect-timeout 10 https://deployment-manager-privatelink.xpander.ai/health; echo"

sleep 15 && kubectl logs curl-test
# Expected output: 200

kubectl delete pod curl-test

A timeout or 000? See Troubleshooting below.

External Access & DNS

The xpander services communicate with each other internally via Kubernetes Services: no ingress is needed for that. You only need external access so that users can reach the chat UI, API, and other endpoints from their browser or client.

Already have an ingress controller? Give the installer a --domain (or set ingress.enabled=true in Helm values) and skip to DNS records.

Option A: VPN / Corporate Network Access

If your organization has a VPN that routes into the VPC, users can access xpander services directly: no ingress controller, NLB, ACM certificate, or public DNS needed. Point internal DNS records to the Kubernetes services:

chat.<DOMAIN>              → xpander-chat.xpander.svc.cluster.local
agent-controller.<DOMAIN>  → xpander-agent-controller.xpander.svc.cluster.local
ai-gateway.<DOMAIN>        → xpander-ai-gateway.xpander.svc.cluster.local

When installing, skip the domain prompt (ingress stays disabled, the default).

Option B: Public Access via Ingress + NLB

For public or internet-facing access, set up an ACM certificate, an nginx ingress controller, and an NLB, using the three steps below: SSL Certificate, Ingress Controller, and DNS Records.

SSL Certificate

Request an ACM certificate for your domain.

The certificate must include a wildcard for *.chat.<DOMAIN> because the xpander chat UI generates per-thread subdomains (e.g., moccasin-prawn.chat.<DOMAIN>). Standard wildcards only match one level, so *.<DOMAIN> does not cover these. If you omit *.chat.<DOMAIN>, the chat UI will show SSL certificate errors for thread URLs.

aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>

Add the DNS validation CNAME records to your Route 53 hosted zone. There will be two unique validation records, one for <DOMAIN> / *.<DOMAIN> (shared) and one for *.chat.<DOMAIN>:

aws acm describe-certificate --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE> \
  --query 'Certificate.DomainValidationOptions[0].ResourceRecord'

Wait for validation:

aws acm wait certificate-validated --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE>

Ingress Controller

Install the nginx ingress controller with AWS NLB and ACM SSL termination:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="<ACM_CERT_ARN>" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"="443" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"="tcp" \
  --set controller.service.targetPorts.https=http

Get the NLB hostname (you’ll need it for the DNS records):

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

DNS Records

Create wildcard CNAME records pointing *.<DOMAIN> and *.chat.<DOMAIN> to the NLB hostname. Both are required: the chat UI generates per-thread subdomains under chat.<DOMAIN>.

cat <<EOF > /tmp/dns-record.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.chat.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    }
  ]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <HOSTED_ZONE_ID> \
  --change-batch file:///tmp/dns-record.json \
  --profile <PROFILE>

This creates the following endpoints:

Service	URL
API	`https://api.<DOMAIN>`
Chat	`https://chat.<DOMAIN>`
AI Gateway	`https://ai-gateway.<DOMAIN>`
Agent Controller	`https://agent-controller.<DOMAIN>`
Execution tier (`agent-worker`)	`https://agent-worker.<DOMAIN>`
MCP	`https://mcp.<DOMAIN>`
Code Runner	`https://code-runner.<DOMAIN>`
AWS Operator	`https://aws-operator.<DOMAIN>`

Test it:

curl -sk https://api.<DOMAIN>/health

LLM Gateway

The chart runs the xpander LLM Gateway (llm-gw) in your cluster, and it is always on. Agents send every model call through it. It applies your organization’s allowed models, budgets and usage reporting. Agents reach it in-cluster at http://xpander-llm-gw:4000, the Base URL that Platform > AI governance > Models shows while the host is not published. Configuration, publishing and the Hybrid specifics are on LLM Gateway on a self-hosted install.

What the chart adds	Default
Gateway pods	2 (one on a single-node cluster installed with `install.sh`), HPA from 2 to 20 on CPU; each requests 0.5 vCPU / 1 GiB and may use up to 1 vCPU / 4 GiB. No database of its own
Secrets	`xpander-llm-gw` (master key) and `xpander-llm-gw-content` (seals saved prompt and response text), generated once and kept on upgrade and on uninstall. Bring your own with `llm-gw.masterKey.existingSecret` / `llm-gw.contentKey.existingSecret`
Image	`xpanderaihub/llm-gw`, configured like every other image (`global.imageRegistry`, `images.llmGw`, the chart’s pull secrets). Mirror it if you mirror images
Service account	`llm-gw-sa`; `llm-gw.serviceAccountAnnotations` carries workload identity (IRSA) for Bedrock or Vertex credentials

Never delete or rotate xpander-llm-gw-content: saved prompt and response text cannot be read without it. With Argo CD, ignore /data on both gateway Secrets, or set both existingSecret values:

spec:
  ignoreDifferences:
    - kind: Secret
      name: xpander-llm-gw
      jsonPointers: ["/data"]
    - kind: Secret
      name: xpander-llm-gw-content
      jsonPointers: ["/data"]

Model keys. Agents on a Hybrid install never use xpander’s keys: every provider your agents use needs a key saved in the organization (Platform > AI governance > Models) or set on the install (secrets.static), and a call to a provider with neither is refused. The refusal names the missing key; set install keys as shown below. Custom OpenAI-compatible endpoints keep working as configured. Health. New agent runs wait for a healthy gateway. If its pods cannot start (for example, the image cannot be pulled), new runs are refused with “The xpander AI gateway is unavailable right now” until the pods are ready:

kubectl -n <namespace> get pods -l app=llm-gw

A healthy gateway shows two llm-gw pods (one on a single-node cluster installed with install.sh), each 1/1 and Running. Network. With networkPolicy.enabled, the gateway gets its own policy: ingress on port 4000 only, egress to DNS, agent-controller, Redis and public model providers over HTTPS, never a private or link-local address. Add what else it must reach:

llm-gw:
  networkPolicy:
    providerCidrs: []        # model endpoints on your own network, any port
    cloudEndpointCidrs: []   # private cloud endpoints such as Bedrock or Azure, port 443
    redisCidrs: []           # an external Redis
  env:                       # the gateway does not read global.env
    HTTPS_PROXY: http://<proxy-host>:<port>
    NO_PROXY: <hosts that bypass the proxy>

Your own firewall must let the gateway reach the same model providers agent-controller reaches today. Publishing llm-gw.<domain>. Agents never need it. Only to serve coding clients on laptops and SDKs inside your network, set llm-gw.ingress.enabled: true (it renders only with the chart-wide ingress.enabled). You then need a DNS record for the host and a certificate that covers it (the chart reuses ingress.tls.secretName). Any load balancer in front needs an idle timeout of at least 1500 seconds (ALB: idle_timeout.timeout_seconds=1500). llm-gw.ingress.internal: true serves the host on a private load balancer (internalClassName, internalAnnotations). While no gateway pod is ready, a small xpander-llm-gw-unavailable pod answers the published host with a JSON 503 that coding clients and SDKs retry. Set llm-gw.unavailable.replicas: 2 to keep it through a node loss. It cannot be turned off. From chart 0.22.5 the chart ignores llm-gw.enabled; charts 0.22.0 to 0.22.4 still read it. You can leave the key in your values or remove it. Every helm upgrade, including --reuse-values, starts the gateway.

Managing LLM API Keys

Use helm upgrade with --reset-then-reuse-values (Helm 3.14 or later) to add or change API keys: it keeps the values you set and applies the chart’s defaults for everything else:

helm upgrade xpander xpander/xpander \
  --namespace xpander --reset-then-reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --timeout 5m

helm upgrade xpander xpander/xpander \
  --namespace xpander --reset-then-reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --set secrets.static.agentsOpenaiApiKey=<KEY> \
  --set secrets.static.fireworksApiKey=<KEY> \
  --timeout 5m

After the upgrade, pods that use the updated secrets will automatically restart. Verify with kubectl get pods -n xpander.

Your own keys and credits

Keys can also live in the product rather than the chart: an organization-level key per provider under Platform > AI governance > Models, or an agent-level credential on one agent. Whenever a model call runs on a key you brought (agent-level first, then the org-level key), it is never charged credits; xpander still measures the usage, so budgets, usage reports and the task’s token panel stay accurate. Which key a call needs is under LLM Gateway. Agents on Claude Code, Codex or OpenCode resolve keys the same way through the Agent Controller and the LLM Gateway, so the same rule applies to their turns.

Secret not updating after helm upgrade?

The xpander-static secret has a Helm resource keep policy, so helm upgrade may not update it on subsequent installs. If your API key isn’t being picked up after a helm upgrade, set it directly in the secret:

# Set the key directly in the Kubernetes secret
kubectl patch secret xpander-static -n xpander --type=merge \
  -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

# Restart the worker to pick up the new key
kubectl rollout restart deployment xpander-agent-worker -n xpander

Secret field name mapping:

Helm Value	Secret Field
`secrets.static.anthropicApiKey`	`ANTHROPIC_API_KEY`
`secrets.static.agentsOpenaiApiKey`	`AGENTS_OPENAI_API_KEY`
`secrets.static.fireworksApiKey`	`FIREWORKS_API_KEY`
`secrets.static.awsBedrockApiKey`	`AWS_BEARER_TOKEN_BEDROCK`
`secrets.static.googleApiKey`	`GOOGLE_API_KEY`
`secrets.static.nebiusApiKey`	`NEBIUS_API_KEY`
`secrets.static.heliconeApiKey`	`HELICONE_API_KEY`
`secrets.static.openrouterApiKey`	`OPENROUTER_API_KEY`
`secrets.static.deploymentManagerApiKey`	`DEPLOYMENT_MANAGER_API_KEY`

Key names read from an environment variable

An agent, a custom model provider or the organization’s default model can name its API key by an environment variable on agent-controller (agent-controller.env or global.env, for example MY_VLLM_TOKEN). Such a name never reads one of the install’s own credentials. These names read as no key:

names starting with AWS_ (except AWS_BEARER_TOKEN_BEDROCK), XPANDER_, FLEET_, LLM_GW, LLM_GATEWAY, LITELLM_, GITHUB_APP, SUPABASE, REDIS, OAUTH, JWT, LICENSE, MONGO, POSTGRES, PG_, ON_PREM_PG, SMTP, REGISTRY, SANDBOX_, HARNESS_ or KUBE
names ending with _SERVICE_API_KEY, _PASSWORD, _PRIVATE_KEY, _SECRET, _SECRET_KEY, _SESSION_TOKEN or _TOKEN_FILE
the platform’s own service keys (DEPLOYMENT_MANAGER_API_KEY, AGENT_CONTROLLER_API_KEY and the like)

The standard provider names (OPENAI_API_KEY, ANTHROPIC_API_KEY, AWS_BEARER_TOKEN_BEDROCK, GOOGLE_API_KEY, AZURE_OPENAI_API_KEY, OPENROUTER_API_KEY and the other provider keys) keep working, and so does any other name, such as VLLM_API_KEY. Rename a name that matches the list (for example VLLM_SECRET to VLLM_API_KEY) in both places: the agent-controller env that carries the value and the setting that names it. Model endpoints on metadata addresses are refused. A model endpoint set on an agent, a custom provider, the organization’s default model or an Azure key is refused when it resolves to a metadata or link-local address (169.254.0.0/16, fe80::/10, the cloud metadata hosts). The check runs when the endpoint is saved and again on every call, against the address actually dialled. Private addresses (a vLLM or model gateway on your own network) keep working.

Using Kubernetes Secrets (Recommended for Production)

kubectl create secret generic ai-service-keys \
  --namespace xpander \
  --from-literal=openai-api-key=sk-your-openai-key \
  --from-literal=anthropic-api-key=sk-ant-your-anthropic-key

# In your values file
agent-worker:
  envFromSecretKeys:
    AGENTS_OPENAI_API_KEY:
      secretName: "ai-service-keys"
      key: "openai-api-key"
    ANTHROPIC_API_KEY:
      secretName: "ai-service-keys"
      key: "anthropic-api-key"

Using the SDK with Self-Hosted

Set the base_url field on the xpander SDK’s Configuration object to point at your Agent Controller endpoint, as shown below.

Use an organization API key from Platform > Organization > Access > API keys. The controller also accepts the Agent Controller API key generated at install time. It is an internal service key with full access, so keep it for platform automation, not for SDK clients. To read the service key from the cluster:

kubectl -n xpander get secret xpander-generated -o jsonpath='{.data.AGENT_CONTROLLER_API_KEY}' | base64 -d

from xpander_sdk import Configuration

config = Configuration(
    api_key="your-org-api-key",  # Platform > Organization > Access > API keys
    organization_id="your-org-id",
    base_url="https://agent-controller.my-company.com"
)

Using the SDK against your install

Make sure your base_url points to the Agent Controller endpoint (e.g., https://agent-controller.{your-domain}), not the root domain.For more SDK examples, see the Self-Hosted SDK Configuration.

Upgrading

Upgrades work like the Datadog agent flow: watch the xpander releases feed (Releases page / Atom feed), then run the upgrade subcommand:

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade
# or, pinned:
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade --chart-version <X.Y.Z>

install.sh upgrade:

finds the release (default xpander in namespace xpander; override with --release / --namespace),
infers its edition from the installed chart: you never pass --edition to an upgrade,
reuses the values the release was installed with (helm get values), plus anything extra you pass (--set, --values),
shows current → target version with a link to the release notes, and asks before changing anything,
always runs with --atomic: a failed upgrade (including a failed hook) rolls back automatically and the previous revision keeps serving.

Manual-Helm equivalent (Helm 3.14 or later):

helm repo update
helm upgrade xpander xpander/xpander --namespace xpander --reset-then-reuse-values

--reset-then-reuse-values keeps the values you set and applies the new chart’s defaults for everything else. Plain --reuse-values replays the previous release’s values, including the old chart’s defaults, so it misses defaults that changed (see below).

Upgrading to the current chart

The current Hybrid chart is xpander 0.22.5 (release v2026.10.22). Database migrations run automatically. Upgrade with Helm 3.14 or later:

helm repo add xpander https://charts.xpander.ai && helm repo update xpander
helm upgrade xpander xpander/xpander --version 0.22.5 -n <namespace> --reset-then-reuse-values --wait --timeout 30m

Before you upgrade from a chart between 0.22.0 and 0.22.4: the changes below arrived across charts 0.22.1 to 0.22.5. Each one happens on the first upgrade to the chart that brings it, so skip the ones your current chart already has. On 0.22.3, for example, only the last two rows still apply.

Change	Chart that brings it
In-cluster Postgres on the pgvector build, with the pre-upgrade check	0.22.1
`BRAIN_INGEST_API_KEY` in the generated Secret	0.22.3
`envFromSecret` and `envFromConfigMap` render	0.22.4
The LLM Gateway is always on	0.22.5

In-cluster Postgres moves to the pgvector build (0.22.1). The postgres StatefulSet runs pgvector/pgvector:0.8.7-pg15-trixie instead of postgres:15: PostgreSQL 15.19, the same major version, on the same volume. If you use global.imageRegistry, mirror docker.io/pgvector/pgvector:0.8.7-pg15-trixie first. Take a volume snapshot of postgres-data-xpander-postgres-0, or set postgres.upgradeGuard.backup.pvcName to an existing PVC with room for a full dump. Postgres restarts once. If you already run 0.22.1 or later, Postgres is on pgvector and this step is done.
A pinned Postgres image stays on pgvector (0.22.1). If your values set images.postgres, keep it on a pgvector build of PostgreSQL 15. A pre-upgrade check refuses the upgrade when the major version would change.
A new key in the generated Secret (0.22.3). BRAIN_INGEST_API_KEY is added automatically and kept from then on. No action needed.
Environment from your own Secrets and ConfigMaps renders (0.22.4). global.envFromSecret, global.envFromConfigMap and the per-component envFromSecret and envFromConfigMap lists now render as the container’s envFrom. Earlier charts failed the render when they were set. Installs that never set them render the same as before.
The LLM Gateway is always on (0.22.5). The chart ignores llm-gw.enabled. If your install had llm-gw.enabled: false, mirror the llm-gw image if you mirror images, make sure every model provider your agents use has a key, and open the gateway’s network paths (LLM Gateway). The upgrade starts the gateway pods and restarts agent-controller and api once. Then check kubectl -n <namespace> get pods -l app=llm-gw. If the gateway was already on, nothing changes.
Per-agent MCP URLs move to the MCP host (release change, no chart change). MCP clients that used a per-agent MCP URL connect to https://mcp.<DOMAIN> instead; per-agent MCP URLs are no longer served. The API’s own MCP endpoint, https://api.<DOMAIN>/mcp/ (Model Context Protocol), does not change, so clients set up on it keep working. The chart has served mcp.<DOMAIN> from its ingress since 0.22.0; there is no value to set.

The Postgres upgrade, step by step. A pre-upgrade Job, xpander-postgres-upgrade-guard, refuses the upgrade when the running major version differs from the image’s, and lists every database. With postgres.upgradeGuard.backup.pvcName set, it first writes a pg_dumpall to that PVC, and refuses the upgrade if the dump cannot be taken in full. The dump must finish within postgres.upgradeGuard.timeoutSeconds (default 1500) and within the --timeout of the upgrade, so raise both for a large database. After the upgrade, the xpander-postgres-pgvector Job adds the vector extension to every database. It never fails the release, and its log ends with one status line:

kubectl -n <namespace> logs job/xpander-postgres-pgvector

To go back, helm rollback <release> <previous revision> returns the pod to postgres:15 on the same volume with all its data. Coming from a chart before 0.22.0, these changes also apply:

The LLM Gateway runs. Before you upgrade, mirror the llm-gw image if you mirror images, make sure every provider your agents use has a key, and open any network path the gateway needs (LLM Gateway). agent-controller and api restart once to pick up the gateway’s address. After the upgrade, check kubectl -n <namespace> get pods -l app=llm-gw (healthy result).
Argo CD. Add ignoreDifferences on /data for xpander-llm-gw and xpander-llm-gw-content, or set llm-gw.masterKey.existingSecret and llm-gw.contentKey.existingSecret.
Harness agents run only on the fleet executors. There is no pod per agent; when no executor is up, a turn waits in the queue. The chart always keeps at least one executor: fleet.replicas: 0 (or blank) renders one, and with fleet.keda.enabled, fleet.keda.maxReplicas: 0 holds one. agentContainers.harness (enabled, resources, storage, shm) is ignored and can be removed from your values; images.agentHarness stays, it is the image every executor runs.
The release Role reads pods but no longer writes them. agent-controller keeps get, list and watch on pods and still manages the Deployments, Services, volumes, Secrets and Jobs of agent workspaces. If your own automation used the release ServiceAccount to create, update or delete pods, add the rule back:
rbac: additionalRules: - apiGroups: [""] resources: ["pods"] verbs: ["create", "update", "patch", "delete"]
Fleet agents reach private networks like any other pod. fleet.networkPolicy.exceptCidrs now defaults to ["169.254.0.0/16"] and exceptCidrsV6 to ["fe80::/10"], so agents and the MCP endpoints they call reach 10/8, 172.16/12, 192.168/16 and in-cluster services; cloud metadata stays blocked. --reset-then-reuse-values picks this up. With --reuse-values, pass --set-json 'fleet.networkPolicy.exceptCidrs=["169.254.0.0/16"]' --set-json 'fleet.networkPolicy.exceptCidrsV6=["fe80::/10"]'. An install that set exceptCidrs itself keeps its own list.
Executor nodes need Landlock, or fleet.landlock.enabled: false. See Node prerequisites.
Model key names and endpoints. Environment-variable key names that match the install’s own credentials read as no key, and model endpoints on metadata addresses are refused (details). Check the key names on your agents, custom providers and the organization’s default model before you upgrade.
Custom-container agents move to the standard agent runtime during the upgrade.

--reuse-values with the new defaults:

helm upgrade xpander xpander/xpander --namespace xpander --reuse-values \
  --set-json 'fleet.networkPolicy.exceptCidrs=["169.254.0.0/16"]' \
  --set-json 'fleet.networkPolicy.exceptCidrsV6=["fe80::/10"]'

What else to expect from the current chart line:

PostgreSQL no longer restarts on every chart release. The config checksum hashes the ConfigMap data only, so a release whose Postgres configuration is byte-identical leaves the StatefulSet alone; a real postgres.* change still rolls it.
The fleet is always rendered, and --atomic waits for it. A running turn holds the roll for up to the drain window (fleet.executor.drainSeconds, 3600 s); the idle guard holds the executor at zero replicas on a cluster without its StorageClass so the upgrade completes anyway. If you drive helm upgrade --wait by hand, give --timeout at least the drain window or upgrade in a quiet period. Details on Sizing and requirements.

Installer Flags Reference

bash install.sh [flags]            # install (the wizard, then helm upgrade --install)
bash install.sh upgrade [flags]    # upgrade an existing release in place
bash install.sh status [flags]     # read-only health report (--json for one object)

Identity (Hybrid identifies with the three values shown in Xpander Chat; required non-interactively):

Flag	Description
`--edition hybrid\|airgap`	Edition. `hybrid` (default): execution tier here, control plane in xpander cloud. `airgap`: every component here, no connection to xpander cloud at run time
`--org UUID`	Organization ID
`--env UUID`	Environment ID
`--dm-api-key KEY`	Deployment manager API key

LLM providers (all optional, the wizard asks for these):

Flag	Description
`--openai-key KEY`	OpenAI
`--anthropic-key KEY`	Anthropic
`--google-key KEY`	Google / Gemini
`--bedrock-key KEY`	Bedrock bearer token (`AWS_BEARER_TOKEN_BEDROCK`)
`--bedrock-auth key\|iam`	`key` (default when `--bedrock-key` is given) or keyless IAM role / EKS pod identity, see Amazon Bedrock: keyless IAM or API key
`--fireworks-key KEY`	Fireworks AI
`--nebius-key KEY`	Nebius
`--helicone-key KEY`	Helicone observability
`--openrouter-key KEY`	OpenRouter

Hybrid connectivity & ingress:

Flag	Description
`--domain DOMAIN`	Enable ingress for this domain
`--dash-domains`	Dash-separated hostnames (`agent-controller-xpander.<DOMAIN>`)
`--privatelink`	Reach the deployment manager over AWS PrivateLink
`--staging`	Point at the xpander staging control plane (auto-detected for local/staging domains and local kube contexts)
`--dm-url URL`	Custom deployment manager URL. `--privatelink` / `--staging` / `--dm-url` are mutually exclusive

Chart and release:

Flag	Description
`--chart-path PATH`	Install from a local chart directory or `.tgz`
`--chart-version VERSION`	Pin the chart version
`--set KEY=VALUE`	Extra helm `--set` (repeatable)
`--values FILE`	Extra helm `-f` values file (repeatable)
`--namespace NAME`	Default: `xpander`
`--release NAME`	Default: `xpander`
`--kube-context NAME`	kubectl/helm context

Other:

Flag	Description
`--manual`	Print the whole install as copy-pasteable commands, personalized from the flags you passed, and do nothing. Both self-hosted deployments; secrets stay placeholders. See Advanced
`--values-out PATH`	Write the generated values file to PATH and exit: no cluster needed. Useful for GitOps: review the file, then `helm upgrade --install` it yourself
`--dry-run`	Render everything through helm, install nothing
`-y`, `--yes`	Accept every confirmation
`--non-interactive`	Never read the terminal, even if one exists. Implies `--yes`; missing required flags exit 2 instead of prompting
`-q`, `--quiet`	Phase lines and the outcome only
`--json`	`status` only: emit one JSON object
`--help` / `--version`	Usage text / installer version

Air-gap-only flags (license, mirroring, image tags, data tier, base domain, code execution): Air-gapped deployment.

Automation & CI

The installer is built to be driven by scripts, CI pipelines, and coding agents:

Every prompt has a flag. With --non-interactive nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies --yes.
Stable progress output. Progress is one line per transition in a grep-friendly shape, [3/6] chart-source: ok, and --quiet reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
status is the poll target. A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. --json emits one JSON object.

Exit codes:

Code	Meaning
`0`	Success
`1`	Install or upgrade failure (`status`: degraded)
`2`	Usage error, or a required flag missing with no terminal
`3`	`status`: no such release installed
`4`	Preflight failure (missing helm/kubectl, unreachable cluster, version too old)
`5`	External data-tier probe failed and `--on-probe-failure` was `abort` (air-gap external data tier)

Hybrid, end to end, no terminal:

bash install.sh --non-interactive --quiet \
  --edition hybrid --org <uuid> --env <uuid> --dm-api-key <key> \
  --domain xpander.example.com --openai-key <key>

Poll until healthy:

until bash install.sh status --quiet; do sleep 30; done

When a scripted run fails, the exit code plus the last FIX:/RERUN: lines tell you what to repair. See Troubleshooting.

Advanced: Review-First / Manual Install

Read the installer before running it

curl -fsSL https://charts.xpander.ai/install.sh | less
curl -fsSL https://charts.xpander.ai/install-v<version>.sh | less   # this exact version

The versioned URL is immutable: install-v<version>.sh is published once per release and never rewritten in place, so pin it in runbooks and CI (bash install.sh --version prints the version you’re holding).
The script is shellcheck-clean, enforced by CI on every push.
Secrets never touch the helm command line: they are written to a temporary values file (mode 600, removed on exit), so they don’t land in shell history or process listings.
Nothing runs against a cluster before the kube-context confirmation: the installer names the context and cluster it is about to install into and waits for your yes.

Do it yourself

Path	What you get
`--manual`	The whole install as copy-pasteable commands (repo setup, secrets, a complete values file, the helm command, verification) personalized from the flags you passed, executing nothing. Secrets stay placeholders
`--values-out PATH`	Just the generated values file (mode 600), plus the printed `helm upgrade --install` command to run yourself. No cluster needed
`--dry-run`	The full flow rendered through `helm --dry-run`, installing nothing

bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> --manual

Manual Helm install (no installer at all)

The installer is a wrapper around helm upgrade --install; everything it does can be done by hand. The cleanest middle path is --values-out:

bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> \
  --domain <DOMAIN> --values-out xpander-values.yaml
helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace -f xpander-values.yaml

Or fully by hand. Add the repository:

helm repo add xpander https://charts.xpander.ai
helm repo update

Set the deployment manager URL based on your connectivity choice (public https://deployment-manager.xpander.ai, or https://deployment-manager-privatelink.xpander.ai with PrivateLink) and install:

helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace \
  --set ingress.enabled=true \
  --set domain=<DOMAIN> \
  --set global.organizationId=<ORGANIZATION_ID> \
  --set global.environmentId=<ENVIRONMENT_ID> \
  --set secrets.static.deploymentManagerApiKey=<DEPLOYMENT_MANAGER_API_KEY> \
  --set urls.deploymentManager=<DEPLOYMENT_MANAGER_URL> \
  --timeout 10m

LLM provider keys are secrets.static.* values. The full list is in the key mapping table. Common optional parameters:

Parameter	Default	Description
`domain`	`""` (ingress disabled)	Base domain for ingress hostnames
`ingress.enabled`	`false`	Expose services via ingress
`ingress.tls.enabled`	`false`	Enable TLS on ingress
`ingress.tls.source`	`self-signed`	`self-signed`, `cert-manager`, or `external`
`redis.storage.size`	`8Gi`	Redis PVC size
`redis.storage.storageClass`	cluster default	Storage class for Redis PVC
`dockerRegistry.enabled`	`true`	In-cluster registry for agent container images. Disable it if your cluster provides one
`metricsServer.enabled`	`true`	metrics-server for autoscaling. Disable it if your cluster runs one

For production, use a values file instead of --set flags:

xpander-values.yaml

domain: "xpander.production.com"

global:
  organizationId: "your-org-id"
  environmentId: "your-env-id"
  env:
    LOG_LEVEL: "info"
    ENVIRONMENT: "production"

urls:
  deploymentManager: "https://deployment-manager.xpander.ai"

secrets:
  static:
    deploymentManagerApiKey: "your-key"
    agentsOpenaiApiKey: "sk-your-openai-key"
    anthropicApiKey: "sk-ant-your-anthropic-key"

ingress:
  enabled: true
  tls:
    enabled: true
    source: "cert-manager"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

resources:
  agentController:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "3000m"
      memory: "4Gi"

redis:
  storage:
    size: "32Gi"
    storageClass: "fast-ssd"

helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace \
  --values xpander-values.yaml

See Sizing and requirements for recommended resource values per service.

Troubleshooting

Every installer failure prints two lines: FIX:, the action that repairs the condition, and RERUN:, your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under ~/.xpander/logs/ (latest.log points at the most recent run). For scripted runs, the exit code names the failure class (Automation & CI), and bash install.sh status gives a read-only health verdict.

Installer

Release stuck in pending-install / pending-upgrade / failed

install.sh upgrade requires the release to be in deployed status: operating on a stuck release (pending-install, pending-upgrade, pending-rollback, failed, uninstalling) would strand it. Recover first:

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

Then paste your RERUN: line. A pending-install status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out: helm uninstall is the right recovery for it.

Release already runs the other deployment's chart

The two self-hosted deployments share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (--release / --namespace), or remove the existing release first (helm uninstall xpander -n xpander).

Piped run exits with 'these required flags are missing'

Piping the installer into bash leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as --non-interactive): Hybrid requires --org, --env, and --dm-api-key. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console, it carries the identity flags, or download the script and run bash install.sh for the interactive wizard.

Cluster and Pods

Pods stuck in Init

The init containers wait for agent-controller to be ready. Check its logs:

kubectl logs -n xpander deployment/xpander-agent-controller

Common cause: Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly: see PrivateLink.

Pods stuck in Pending

Cause: Storage class or PVC issues.

kubectl -n xpander get pvc
kubectl -n xpander describe pvc

Check that a default StorageClass exists and the EBS CSI driver is running:

kubectl get storageclass  # Should show a default (gp3)
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver  # Should show Running

If using EKS, ensure the node role has AmazonEBSCSIDriverPolicy attached.

exec format error

You are running on ARM/Graviton nodes. xpander images are amd64 only. Switch to x86 instance types (t3, m5, c5, etc.).

Insufficient CPU

The agent-worker pod requests 2 CPU by default. Two options:

Add more nodes, or use larger instances.
For non-production environments only, lower the CPU request directly on the xpander-agent-worker Deployment in the xpander namespace:

kubectl patch deployment xpander-agent-worker -n xpander \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]'

Health checks failing

Check application logs (skill calls, including AWS ones, log in agent-controller):

kubectl -n xpander logs deployment/xpander-agent-controller
kubectl -n xpander logs deployment/xpander-agent-worker
kubectl -n xpander logs deployment/xpander-mcp

PrivateLink

PrivateLink InvalidServiceName (cross-region)

When creating a VPC endpoint to the xpander service from any region other than us-west-2, you must include --service-region us-west-2:

aws ec2 create-vpc-endpoint \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  ...

Without --service-region, AWS looks for the service in your local region and fails with InvalidServiceName.

PrivateLink connection timeout (HTTP 000)

Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR:

aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp --port 443 --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>

Also verify the private DNS hosted zone and alias record were created correctly: see PrivateLink → Create private DNS.

Ingress and Networking

Ingress not accessible

Verify ingress configuration:

kubectl -n xpander describe ingress
kubectl -n xpander get ingress

Check that the NLB was provisioned and DNS CNAME records point to it:

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Load Balancer not provisioning

Symptom: the ingress controller’s Service stays <pending>. Check its events (kubectl -n ingress-nginx describe svc ingress-nginx-controller):

could not find any suitable subnets for creating the ELB: the VPC’s subnets are missing the ELB role tags. Tag public subnets kubernetes.io/role/elb=1 for an internet-facing NLB; for an internal one, tag the private subnets kubernetes.io/role/internal-elb=1 and annotate the Service service.beta.kubernetes.io/aws-load-balancer-internal: "true".
On EKS Auto Mode, ensure the cluster role trust policy includes sts:TagSession:

{
  "Effect": "Allow",
  "Principal": { "Service": "eks.amazonaws.com" },
  "Action": ["sts:AssumeRole", "sts:TagSession"]
}

SSL certificate errors on chat URLs

The ACM certificate must include *.chat.<DOMAIN> as a subject alternative name. The chat UI generates per-thread subdomains (e.g., moccasin-prawn.chat.<DOMAIN>) that are not covered by *.<DOMAIN>.Request a new certificate with:

aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>

API Keys

API key not being picked up after helm upgrade

The xpander-static secret has a Helm resource keep policy, so a helm upgrade does not update it on its own. Patch the key directly on the xpander-static Kubernetes secret in the xpander namespace:

kubectl patch secret xpander-static -n xpander --type=merge \
  -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

kubectl rollout restart deployment xpander-agent-worker -n xpander

See the secret field name mapping for all key names.

New runs refused: The xpander AI gateway is unavailable right now

New agent runs wait for a healthy LLM Gateway. Check its pods and the reason they are not ready (an image the cluster cannot pull is the usual one):

kubectl -n <namespace> get pods -l app=llm-gw
kubectl -n <namespace> describe pods -l app=llm-gw

See LLM Gateway for the healthy result.If you mirror images, mirror xpanderaihub/llm-gw and point images.llmGw or global.imageRegistry at it. The gateway cannot be turned off, so new runs wait until its pods are ready.

LLM calls fail with a missing API key

A call to a provider with no key, neither in the organization (Platform > AI governance > Models) nor on the install (secrets.static), is refused with a message that names the missing key. Add the key in either place. To check the install’s keys, verify they reach the agent-worker pod:

kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY

If missing, re-run the upgrade with the key set (--set secrets.static.<key>=...), or check that your secret is correctly mounted through envFromSecretKeys: see Managing LLM API Keys.

Debug Commands

# Get all resources
kubectl -n xpander get all

# Check events
kubectl -n xpander get events --sort-by=.metadata.creationTimestamp

# Describe problematic pods
kubectl -n xpander describe pod <pod-name>

# Check service endpoints
kubectl -n xpander get endpoints

Hybrid Security Architecture

Network boundary, PrivateLink, and what leaves your VPC

Self-Hosted Overview

Hybrid vs Air-Gapped comparison

Air-gapped deployment

Every component in your environment, no connection to xpander cloud at run time

Access Control

Identity by deployment, SSO, API keys, audit logs

Uninstalling

# Remove the Helm release
helm uninstall xpander --namespace xpander

# Clean up the namespace (optional)
kubectl delete namespace xpander

​Architecture

​What Gets Deployed

​How It Works

​What Talks to What

​Data Residency

​Requirements

​Install

​1. Get your one-liner from Xpander Chat

​2. Run the Installer

​Install Phases

​Amazon Bedrock: keyless IAM or API key

​3. Verify the Connection

​PrivateLink

​How PrivateLink Works

​Set It Up

​External Access & DNS

​Option A: VPN / Corporate Network Access

​Option B: Public Access via Ingress + NLB

​SSL Certificate

​Ingress Controller

​DNS Records

​LLM Gateway

​Managing LLM API Keys

​Your own keys and credits

​Key names read from an environment variable

​Using Kubernetes Secrets (Recommended for Production)

​Using the SDK with Self-Hosted

​Using the SDK against your install

​Upgrading

​Upgrading to the current chart

​Installer Flags Reference

​Automation & CI

​Advanced: Review-First / Manual Install

​Read the installer before running it

​Do it yourself

​Troubleshooting

​Installer

​Cluster and Pods

​PrivateLink

​Ingress and Networking

​API Keys

​Debug Commands

​Related

Hybrid Security Architecture

Self-Hosted Overview

Air-gapped deployment

Access Control

​Uninstalling

Architecture

What Gets Deployed

How It Works

What Talks to What

Data Residency

Requirements

Install

1. Get your one-liner from Xpander Chat

2. Run the Installer

Install Phases

Amazon Bedrock: keyless IAM or API key

3. Verify the Connection

PrivateLink

How PrivateLink Works

Set It Up

External Access & DNS

Option A: VPN / Corporate Network Access

Option B: Public Access via Ingress + NLB

SSL Certificate

Ingress Controller

DNS Records

LLM Gateway

Managing LLM API Keys

Your own keys and credits

Key names read from an environment variable

Using Kubernetes Secrets (Recommended for Production)

Using the SDK with Self-Hosted

Using the SDK against your install

Upgrading

Upgrading to the current chart

Installer Flags Reference

Automation & CI

Advanced: Review-First / Manual Install

Read the installer before running it

Do it yourself

Troubleshooting

Installer

Cluster and Pods

PrivateLink

Ingress and Networking

API Keys

Debug Commands

Related

Uninstalling