> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hybrid deployment

> Install xpander in your cluster with the control plane on xpander cloud.

Execution, runtime data, credentials and model keys run in your VPC. Agent definitions, identity and the skills catalog stay on xpander cloud. The Agent Controller is the only link, outbound only. Images are amd64 only.

![Diagram](https://docs.xpander.ai/images/diagrams/self-hosted-hybrid-70209eb3.svg)

Install with one command from **Platform > Agents > Runtime**, then an interactive wizard:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY>
```

> [!NOTE]
>
> Self-hosted locations are enabled per account: [contact our team](https://cal.com/team/xpander-ai/activate) if the option isn't available on yours.

Comparing deployments? See [Deployment options](https://docs.xpander.ai/self-hosted/index.md). The network boundary and exactly what leaves your VPC are on [Hybrid Architecture](https://docs.xpander.ai/resources/security-compliance/security-architecture-hybrid.md).

## Architecture

Hybrid separates a **control plane** from a **data plane**, the same model as AWS EKS or Datadog:

| Plane | Where | What it does |
| - | - | - |
| **Control plane** | xpander Cloud | Environment metadata, agent registration, skill definitions and event logging. It never touches runtime data. |
| **Data plane** | Your VPC | All task execution, agent memory, skill credentials, model API keys and user-facing services. Browsers connect to it directly, so chat threads, activity and results never pass through the cloud. The Agent Controller pulls agent definitions and the skills catalog from the cloud. |

The data plane connects **outbound only**, on port 443. The control plane never connects into your cluster. MCP calls, external-system calls and agent workspaces all run in your cluster.

### What gets deployed

Resource names on this page assume the release name `xpander`, as in every command here. With another release name, they start with `<release>-xpander-` instead, or with `<release>-` when the release name already contains `xpander`.

| Component | Description |
| - | - |
| **Agent Controller** | Task scheduling, queue management, and dispatch |
| **LLM Gateway** (`llm-gw`) | One endpoint in front of every model provider. Agents' model calls go through it. Always on, see [LLM Gateway](#llm-gateway) |
| **Execution tier** (`agent-worker`) | Task execution runtime |
| **API Proxy** | API endpoint and request routing |
| **API** | REST API service |
| **MCP** | Accepts MCP client connections, generates tasks |
| **Database** | Agent memory, message history, and monitoring (PostgreSQL) |
| **Redis** | Cache and fast lookups |
| **Chat UI** | Chat interface with per-thread subdomains |
| **Code Runner** | Code execution runtime |
| **AWS APIs Operator** | AWS resource management |

### How it works

Clients (Slack, the web UI, MCP clients, the API) connect to your cluster directly. On `chat.xpander.ai`, the cloud UI signs you in and points the client at your cluster. Inside, the **Agent Controller** schedules tasks and `agent-worker` runs them: model calls, messages in PostgreSQL, and skill calls through the Agent Controller's skill relay.

The only connection to xpander Cloud is the **deployment manager sync**: the Agent Controller authenticates with `organization_id` + `environment_id` + `api_key`, sends heartbeats (schema not published), and pulls agent and skill configuration, MCP included. See [Data that leaves the on-prem installation](https://docs.xpander.ai/resources/security-compliance/security-architecture-hybrid.md#data-that-leaves-the-on-prem-installation).

### What talks to what

Inbound from xpander cloud: **none**. Outbound only:

| From your cluster | To | Port | Purpose |
| - | - | - | - |
| `agent-controller` | `deployment-manager.xpander.ai` (or the [PrivateLink](#privatelink) / staging endpoint) | 443 | Metadata sync, heartbeats, skill updates |
| `llm-gw` (the LLM Gateway) | Your LLM providers (OpenAI, Anthropic, Google, Bedrock, ...) | 443 | Model calls |
| Skills (including MCP) | The external systems you connect | As configured | Run from your cluster, never from xpander cloud |

Outbound connectivity options to the control plane:

| Option | Target | Details |
| - | - | - |
| **Global Accelerator** | `15.197.85.80`, `166.117.85.46` | Public internet, encrypted over TLS |
| **PrivateLink** | `com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197` | AWS private backbone: [setup below](#privatelink) |

### Data residency

| Feature | Stays in your VPC? |
| - | :-: |
| Invoking agents (API to agent) | Yes: client connects directly to cluster |
| Chat threads and activity view | Yes: fetched from cluster, not cloud |
| Debugging agents (Session Manager) | Yes: all data served from cluster |
| Knowledge Base file uploads and queries | Yes: stored and queried in cluster |
| User identity and login | No |
| Agent metadata (agent list) | No |

Data at rest stays in PostgreSQL and Redis in your VPC, and credentials are Kubernetes secrets. The sync carries metadata and heartbeats only, never user payloads. Field-level inventory: [Data that leaves the on-prem installation](https://docs.xpander.ai/resources/security-compliance/security-architecture-hybrid.md#data-that-leaves-the-on-prem-installation).

## Requirements

Footprint, calculator and executor fleet: [Sizing and requirements](https://docs.xpander.ai/self-hosted/sizing.md). The installer checks the same minimums ([requirements.json](https://charts.xpander.ai/requirements.json)).

> [!WARNING]
>
> xpander container images are **amd64 only**. Do not use ARM/Graviton instances (`t4g`, `m7g`, `c7g`, etc.).

The release namespace must admit PodSecurity `baseline`. Unlabelled is fine, and `restricted` is refused. The nodes that run the executor fleet need Landlock or `fleet.landlock.enabled: false`. See [Node prerequisites](https://docs.xpander.ai/self-hosted/sizing.md#node-prerequisites).

Need a cluster? Follow the [EKS Cluster Setup](https://docs.xpander.ai/self-hosted/eks-setup.md) guide.

**Bedrock needs your own credentials.** Agents on a Hybrid install never use xpander's keys ([LLM Gateway](#llm-gateway)). To run Claude Code on Bedrock here, add a Bedrock key or role under **Platform > AI governance > Models**, or give the cluster an AWS identity for Bedrock. See [AI vendors](https://docs.xpander.ai/run/integrations/ai-vendors.md#anthropic-on-amazon-bedrock).

## Install

The install takes three steps:

| Step | What you do | Result |
| - | - | - |
| [1. Get your one-liner](#1-get-your-one-liner-from-xpander-chat) | Add a Hybrid location in xpander Chat and copy the command | A command that carries your identity |
| [2. Run the installer](#2-run-the-installer) | Paste it in a terminal with `kubectl` access and answer the wizard | Six install phases, each with a `FIX:` and `RERUN:` line on failure |
| [3. Verify the connection](#3-verify-the-connection) | Check the pods and the console | **"Connection established"** in a minute or two |

### 1. Get your one-liner from xpander Chat

1. In xpander Chat, open **Platform > Agents > Runtime** and choose the **Locations** tab (Connected runtimes and regions)
2. Click **"Add Location"**
3. Choose the **Hybrid** edition. The dialog shows the minimum footprint for the deployment you picked.
4. Optionally set a **URL** (your server's FQDN, e.g. `xpander.my-company.com`) and the hostname style (dash-based subdomains if your DNS can't do wildcards)
5. Copy the generated one-liner

The one-liner carries your identity, `--org`, `--env`, and `--dm-api-key` (plus `--domain` when you set a URL). That is how the cluster authenticates to the xpander control plane:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid \
  --org <ORGANIZATION_UUID> \
  --env <ENVIRONMENT_UUID> \
  --dm-api-key <DEPLOYMENT_MANAGER_API_KEY> \
  --domain xpander.my-company.com
```

> [!NOTE]
>
> The one-liner deliberately carries **no LLM provider credentials**: the wizard prompts for those interactively (input hidden), or you pass them as flags. Secrets are written to a temporary values file (mode 600, removed on exit) and never appear on the helm command line.

### 2. Run the installer

Run it from a terminal with `kubectl` access to the target cluster. The command downloads the installer from `charts.xpander.ai` and runs an interactive wizard: preflight checks, configuration, Helm install, then the connection back to xpander. Run bare (`curl -fsSL https://charts.xpander.ai/install.sh | bash`), it is the zero-knowledge entry point: the first question is which edition you are installing.

The wizard walks through, in order:

**1. Edition & requirements**

If `--edition` wasn't given, it asks. It then prints the minimum footprint for that edition (the same numbers as [charts.xpander.ai/requirements.json](https://charts.xpander.ai/requirements.json)) and asks you to confirm your cluster meets it, stopping cleanly if not.

**2. Identity**

Organization ID, Environment ID, and the Deployment Manager API key (input hidden): all from the install card in xpander Chat, all skipped if passed as flags.

**3. Ingress domain (optional)**

Press Enter to skip and keep cluster-internal Services (port-forward reaches them, add ingress later). With a domain, choose the hostname style: **subdomain** style (`agent-controller.xpander.<DOMAIN>`, needs wildcard DNS) or **dash** style (`agent-controller-xpander.<DOMAIN>`, individual A records).

**4. Control plane reachability**

Public internet (`https://deployment-manager.xpander.ai`), AWS PrivateLink (`https://deployment-manager-privatelink.xpander.ai`), or Staging. The installer auto-detects staging/local environments (a `.stg.` or local domain, or a kind/k3d/minikube/docker-desktop kube context) and leads with that option. Any of `--staging`, `--privatelink`, or `--dm-url` overrides detection.

**5. LLM providers (all optional)**

OpenAI, Anthropic, Google/Gemini, Amazon Bedrock, then optionally Fireworks, Nebius, Helicone and OpenRouter. All skippable: xpander starts without keys, and model calls work once you add a key in the UI or with a helm upgrade.

For **Bedrock** it asks how to authenticate: an API key / bearer token (`AWS_BEARER_TOKEN_BEDROCK`), or keyless IAM role / EKS pod identity. See [Amazon Bedrock: keyless IAM or API key](#amazon-bedrock-keyless-iam-or-api-key).

**6. Namespace, context confirmation, summary**

Namespace (default `xpander`, created if missing), an explicit confirmation of the kubectl context it is about to install into, and a masked configuration summary before anything changes.

### Install phases

The Hybrid install runs six phases, each announced as a stable `[N/6] name: ...` progress line (the same shape scripts can grep, see [Automation & CI](#automation--ci)):

```
[1/6] preflight       helm ≥ 3.13, kubectl, cluster reachable, Kubernetes ≥ 1.28
[2/6] configuration   context confirmation + masked summary
[3/6] chart-source    helm repo add xpander https://charts.xpander.ai (or --chart-path)
[4/6] values          secrets written to a mode-600 temp file, removed on exit
[5/6] install         helm upgrade --install xpander xpander/xpander
[6/6] verify          waits for the agent-controller rollout (5-minute timeout)
```

The preflight accepts Helm 3.13, but the documented upgrade command (`--reset-then-reuse-values`) needs Helm 3.14 or later.

Every failure prints `FIX:` (the repair) and `RERUN:` (your non-interactive one-liner, secrets placeholdered). The installer is idempotent.

### Amazon Bedrock: keyless IAM or API key

Bedrock calls take one of two paths, each with its own credentials:

| Path | Covers | Credentials |
| - | - | - |
| Chat and agent tasks | Chat, agent task executions and the summaries around them | `--bedrock-auth iam` (keyless, the pods' AWS identity) or `--bedrock-auth key` (a bearer token) |
| Claude Code on Bedrock | The CLI's model calls, straight to the in-cluster LLM Gateway, which gets the Bedrock credential from the Agent Controller | The controller's own AWS auth, or an organization Bedrock key under **Platform > AI governance > Models** |

**Chat and agent tasks.** `--bedrock-auth iam` signs calls with the pods' AWS identity (EKS Pod Identity or IRSA), in images built after 27 August 2026. The role needs the Bedrock permissions itself (`bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`, `bedrock:Converse`, `bedrock:ConverseStream` on `*`) and must be bound to both ServiceAccounts the LLM services run as: `default` and `agent-controller`. The chat and task paths do not assume a second role.

`--bedrock-auth key` (default when `--bedrock-key` is given) stores a bearer token. Either way, the account must be opted in to Bedrock's data retention mode for your models. The "IAM role" method under **Platform > AI governance > Models** covers only the Claude Code path below. If chat fails with a Bedrock entry set that way, remove it and restart the `agent-controller`, `agent-worker`, `agents`, `actions` and `functions` deployments.

**Claude Code on Bedrock.** The CLI calls the in-cluster LLM Gateway directly, and the gateway gets the Bedrock credential from the Agent Controller. The controller resolves AWS auth itself, in this order: `AWS_BEARER_TOKEN_BEDROCK`, then `BEDROCK_ASSUME_ROLE_ARN` (STS, self-refreshing), then static keys, then the pod's default credential chain (Pod Identity / IRSA). Set whichever you use as controller environment through the chart's env values, with `BEDROCK_REGION` or `AWS_REGION` next to it. An organization Bedrock key under **Platform > AI governance > Models** is preferred over the controller's credential.

> [!WARNING]
>
> **Fable- and Mythos-class Anthropic models on Bedrock need the AWS account's data-retention opt-in, per region.** With the default mode Bedrock answers `400 data retention mode 'default' is not available for this model`. There is no console switch: `PUT https://bedrock.<region>.amazonaws.com/data-retention` with `{"mode": "provider_data_share"}` (or `aws_review` where offered), SigV4-signed with `bedrock:PutAccountDataRetention`, in every region a `global.` inference profile can route to, on whichever account signs the call (the controller's, or the organization's own Bedrock key account). Details on [AI vendors](https://docs.xpander.ai/run/integrations/ai-vendors.md#anthropic-on-amazon-bedrock).

### 3. Verify the connection

The installer prints a summary. Within a minute or two the console shows **"Connection established"**.

```bash theme={"dark"}
kubectl get pods -n xpander
```

Expect one `Running`, `1/1`-ready pod per application service: `agent-controller`, `agent-worker`, `ai-gateway`, `api`, `aws-operator`, `chat`, `code-runner`, and `mcp`. `ai-gateway` is a registration-only compatibility component. Skill calls, including AWS ones, are served by `agent-controller` since chart 0.16.1. The LLM Gateway runs as two `llm-gw` pods (one on a single-node cluster installed with `install.sh`). You should also see the `postgres` and `redis` StatefulSets, and, at defaults, `docker-registry` and `metrics-server`.

Test locally without ingress:

```bash theme={"dark"}
kubectl -n xpander port-forward service/xpander-agent-controller 9016:9016 &
kubectl -n xpander port-forward service/xpander-ai-gateway 9018:9018 &
kubectl -n xpander port-forward service/xpander-mcp 8081:8081 &

curl http://localhost:9016/health
curl http://localhost:9018/health
curl http://localhost:8081/health
```

The installation then appears in the **ENVIRONMENTS** switcher and under **Platform > Agents > Runtime** (**Locations** tab), where **Components** shows heartbeats per service and **Configuration** holds the Chat, Streaming and Scheduler controls.

## PrivateLink

Your cluster must reach the xpander **deployment manager**. Two options:

| Option | URL | Security |
| - | - | - |
| **Public (TLS)** | `https://deployment-manager.xpander.ai` | Encrypted over public internet. No setup needed. The installer's default |
| **Private (PrivateLink)** | `https://deployment-manager-privatelink.xpander.ai` | Encrypted over AWS private backbone. This section |

Public needs only outbound HTTPS (via a NAT Gateway in private subnets). If traffic must stay in the AWS network, set up PrivateLink and pass `--privatelink` (or pick **AWS PrivateLink** in the wizard). `--privatelink`, `--staging` and `--dm-url` are mutually exclusive: pass one, or none for public.

### How PrivateLink works

xpander exposes the deployment manager as a **VPC Endpoint Service**. You create a **VPC Endpoint** in your VPC, which gets a private `10.x.x.x` IP. Pods connect to it over the AWS backbone, never the internet.

| Property | Value |
| - | - |
| Service Name | `com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197` |
| Service Region | `us-west-2` |
| Supported Regions | `us-west-2`, `us-west-1`, `eu-west-2`, `eu-central-1`, `ap-south-1` |
| Acceptance Required | No (auto-accepted for allowed accounts) |
| Private DNS Name | `deployment-manager-privatelink.xpander.ai` |

> [!WARNING]
>
> Contact xpander to have your AWS account ID added as an allowed principal before creating the VPC endpoint.

### Set it up

**1. Create the VPC endpoint**

> [!WARNING]
>
> The xpander endpoint service is in `us-west-2`. If your cluster is in a different region, you **must** include the `--service-region us-west-2` flag: this is the most common mistake (see [Troubleshooting](#troubleshooting)).

```bash theme={"dark"}
aws ec2 create-vpc-endpoint \
  --vpc-id <VPC_ID> \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  --subnet-ids <PRIVATE_SUBNET_A_ID> <PRIVATE_SUBNET_B_ID> \
  --no-private-dns-enabled \
  --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=xpander-privatelink}]' \
  --region <REGION> --profile <PROFILE>
```

Save from the output: `VpcEndpointId`, `Groups[0].GroupId` (the security group), `DnsEntries[0].DnsName`, and `DnsEntries[0].HostedZoneId`.

> [!TIP]
>
> If your cluster is in `us-west-2` (same region as the service), you can omit `--service-region`.

**2. Allow HTTPS on the endpoint security group**

The VPC endpoint is created with the VPC's default security group, which only allows inbound traffic from itself. Allow HTTPS from your VPC CIDR:

```bash theme={"dark"}
aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp \
  --port 443 \
  --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>
```

**3. Wait for the endpoint to become available**

The endpoint transitions from `pending` to `available` in 1-3 minutes (auto-accepted):

```bash theme={"dark"}
aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids <VPCE_ID> \
  --query 'VpcEndpoints[0].State' \
  --region <REGION> --profile <PROFILE>
```

**4. Create private DNS**

Create a private hosted zone so `deployment-manager-privatelink.xpander.ai` resolves to the endpoint ENI inside your VPC:

```bash theme={"dark"}
aws route53 create-hosted-zone \
  --name deployment-manager-privatelink.xpander.ai \
  --caller-reference "xpander-privatelink-$(date +%s)" \
  --vpc VPCRegion=<REGION>,VPCId=<VPC_ID> \
  --hosted-zone-config PrivateZone=true \
  --profile <PROFILE>
```

Save the `HostedZone.Id` from the output, then create the alias record:

```bash theme={"dark"}
cat <<EOF > /tmp/privatelink-dns.json
{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "deployment-manager-privatelink.xpander.ai.",
      "Type": "A",
      "AliasTarget": {
        "DNSName": "<VPCE_DNS_NAME>",
        "HostedZoneId": "<VPCE_HOSTED_ZONE_ID>",
        "EvaluateTargetHealth": true
      }
    }
  }]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <PRIVATE_ZONE_ID> \
  --change-batch file:///tmp/privatelink-dns.json \
  --profile <PROFILE>
```

**5. Verify connectivity**

```bash theme={"dark"}
kubectl run curl-test --restart=Never --image=curlimages/curl:latest -n default \
  --command -- sh -c "curl -sk -o /dev/null -w '%{http_code}' \
  --connect-timeout 10 https://deployment-manager-privatelink.xpander.ai/health; echo"

sleep 15 && kubectl logs curl-test
# Expected output: 200

kubectl delete pod curl-test
```

A timeout or `000`? See [Troubleshooting](#troubleshooting) below.

## External access & DNS

The xpander services communicate with each other internally via Kubernetes Services: no ingress is needed for that. You only need external access so that **users** can reach the chat UI, API, and other endpoints from their browser or client.

> [!NOTE]
>
> **Already have an ingress controller?** Give the installer a `--domain` (or set `ingress.enabled=true` in Helm values) and skip to [DNS records](#dns-records).

### Option A: VPN / corporate network access

If your organization has a VPN that routes into the VPC, users can access xpander services directly: no ingress controller, NLB, ACM certificate, or public DNS needed.

Point internal DNS records to the Kubernetes services:

```
chat.<DOMAIN>              → xpander-chat.xpander.svc.cluster.local
agent-controller.<DOMAIN>  → xpander-agent-controller.xpander.svc.cluster.local
ai-gateway.<DOMAIN>        → xpander-ai-gateway.xpander.svc.cluster.local
```

When installing, skip the domain prompt (ingress stays disabled, the default).

### Option B: Public access via ingress + NLB

For public or internet-facing access, set up an ACM certificate, an nginx ingress controller, and an NLB, using the three steps below: [SSL Certificate](#ssl-certificate), [Ingress Controller](#ingress-controller), and [DNS Records](#dns-records).

#### SSL certificate

Request an ACM certificate for your domain.

> [!WARNING]
>
> The certificate **must** include a wildcard for `*.chat.<DOMAIN>` because the xpander chat UI generates per-thread subdomains (e.g., `moccasin-prawn.chat.<DOMAIN>`). Standard wildcards only match one level, so `*.<DOMAIN>` does not cover these. If you omit `*.chat.<DOMAIN>`, the chat UI will show SSL certificate errors for thread URLs.

```bash theme={"dark"}
aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>
```

Add the DNS validation CNAME records to your Route 53 hosted zone. There will be two unique validation records, one for `<DOMAIN>` / `*.<DOMAIN>` (shared) and one for `*.chat.<DOMAIN>`:

```bash theme={"dark"}
aws acm describe-certificate --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE> \
  --query 'Certificate.DomainValidationOptions[0].ResourceRecord'
```

Wait for validation:

```bash theme={"dark"}
aws acm wait certificate-validated --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE>
```

#### Ingress controller

Install the nginx ingress controller with AWS NLB and ACM SSL termination:

```bash theme={"dark"}
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="<ACM_CERT_ARN>" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"="443" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"="tcp" \
  --set controller.service.targetPorts.https=http
```

Get the NLB hostname (you'll need it for the DNS records):

```bash theme={"dark"}
kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
```

### DNS records

Create wildcard CNAME records pointing `*.<DOMAIN>` and `*.chat.<DOMAIN>` to the NLB hostname. Both are required: the chat UI generates per-thread subdomains under `chat.<DOMAIN>`.

```bash theme={"dark"}
cat <<EOF > /tmp/dns-record.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.chat.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    }
  ]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <HOSTED_ZONE_ID> \
  --change-batch file:///tmp/dns-record.json \
  --profile <PROFILE>
```

This creates the following endpoints:

| Service | URL |
| - | - |
| API | `https://api.<DOMAIN>` |
| Chat | `https://chat.<DOMAIN>` |
| AI Gateway | `https://ai-gateway.<DOMAIN>` |
| Agent Controller | `https://agent-controller.<DOMAIN>` |
| Execution tier (`agent-worker`) | `https://agent-worker.<DOMAIN>` |
| MCP | `https://mcp.<DOMAIN>` |
| Code Runner | `https://code-runner.<DOMAIN>` |
| AWS Operator | `https://aws-operator.<DOMAIN>` |

Test it:

```bash theme={"dark"}
curl -sk https://api.<DOMAIN>/health
```

## LLM Gateway

The LLM Gateway (`llm-gw`) runs in your cluster, always on. Every agent model call goes through it, with your allowed models, budgets and usage reporting. Agents reach it at `http://xpander-llm-gw:4000`, the **Base URL** shown in **Platform > AI governance > Models** while the host is unpublished. Full details: [LLM Gateway on a self-hosted install](https://docs.xpander.ai/run/llm-gateway/self-hosted.md).

| What the chart adds | Default |
| - | - |
| Gateway pods | 2 (one on a single-node cluster installed with `install.sh`), HPA from 2 to 20 on CPU. Each requests 0.5 vCPU / 1 GiB and may use up to 1 vCPU / 4 GiB. No database of its own |
| Secrets | `xpander-llm-gw` (master key) and `xpander-llm-gw-content` (seals saved prompt and response text), generated once and kept on upgrade and on uninstall. Bring your own with `llm-gw.masterKey.existingSecret` / `llm-gw.contentKey.existingSecret` |
| Image | `xpanderaihub/llm-gw`, configured like every other image (`global.imageRegistry`, `images.llmGw`, the chart's pull secrets). Mirror it if you mirror images |
| Service account | `llm-gw-sa`. `llm-gw.serviceAccountAnnotations` carries workload identity (IRSA) for Bedrock or Vertex credentials |

> [!WARNING]
>
> Never delete or rotate `xpander-llm-gw-content`: saved prompt and response text cannot be read without it. With Argo CD, ignore `/data` on both gateway Secrets, or set both `existingSecret` values:
>
> ```yaml theme={"dark"}
> spec:
>   ignoreDifferences:
>     - kind: Secret
>       name: xpander-llm-gw
>       jsonPointers: ["/data"]
>     - kind: Secret
>       name: xpander-llm-gw-content
>       jsonPointers: ["/data"]
> ```

**Model keys.** Agents on a Hybrid install never use xpander's keys. Each provider needs a key in the organization (**Platform > AI governance > Models**) or on the install (`secrets.static`), or calls are refused with the missing key named. Set install keys as shown [below](#managing-llm-api-keys). Custom OpenAI-compatible endpoints keep working as configured.

**Health.** New agent runs wait for a healthy gateway. If its pods cannot start (for example, the image cannot be pulled), new runs are refused with "The xpander AI gateway is unavailable right now" until the pods are ready:

```bash theme={"dark"}
kubectl -n <namespace> get pods -l app=llm-gw
```

A healthy gateway shows two `llm-gw` pods (one on a single-node cluster installed with `install.sh`), each `1/1` and `Running`.

**Network.** With `networkPolicy.enabled`, the gateway gets its own policy: ingress on port 4000 only, egress to DNS, agent-controller, Redis and public model providers over HTTPS, never a private or link-local address. Add what else it must reach:

```yaml theme={"dark"}
llm-gw:
  networkPolicy:
    providerCidrs: []        # model endpoints on your own network, any port
    cloudEndpointCidrs: []   # private cloud endpoints such as Bedrock or Azure, port 443
    redisCidrs: []           # an external Redis
  env:                       # the gateway does not read global.env
    HTTPS_PROXY: http://<proxy-host>:<port>
    NO_PROXY: <hosts that bypass the proxy>
```

Your own firewall must let the gateway reach the same model providers agent-controller reaches today.

**Publishing `llm-gw.<domain>`.** Only for coding clients and scripts in your network (agents never need it): set `llm-gw.ingress.enabled: true` (it renders only with the chart-wide `ingress.enabled`). You then need a DNS record for the host and a certificate that covers it (the chart reuses `ingress.tls.secretName`). Any load balancer in front needs an idle timeout of at least 1500 seconds (ALB: `idle_timeout.timeout_seconds=1500`). `llm-gw.ingress.internal: true` serves the host on a private load balancer (`internalClassName`, `internalAnnotations`). While no gateway pod is ready, a small `xpander-llm-gw-unavailable` pod answers the published host with a JSON 503 that coding clients and SDKs retry. Set `llm-gw.unavailable.replicas: 2` to keep it through a node loss.

**It cannot be turned off.** From chart 0.22.5 the chart ignores `llm-gw.enabled` (0.22.0 to 0.22.4 still read it). Every `helm upgrade`, including `--reuse-values`, starts the gateway.

## Managing LLM API keys

Use `helm upgrade` with `--reset-then-reuse-values` (Helm 3.14 or later) to add or change API keys: it keeps the values you set and applies the chart's defaults for everything else:

```bash Single Key theme={"dark"}
helm upgrade xpander xpander/xpander \
  --namespace xpander --reset-then-reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --timeout 5m
```

```bash Multiple Keys theme={"dark"}
helm upgrade xpander xpander/xpander \
  --namespace xpander --reset-then-reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --set secrets.static.agentsOpenaiApiKey=<KEY> \
  --set secrets.static.fireworksApiKey=<KEY> \
  --timeout 5m
```

After the upgrade, pods that use the updated secrets will automatically restart. Verify with `kubectl get pods -n xpander`.

### Your own keys and credits

Keys can also live in the product: per provider under **Platform > AI governance > Models**, or as an agent-level credential (checked first). A call on a key you brought is **never charged credits**, but xpander still measures it, so budgets and usage stay accurate. Claude Code, Codex and OpenCode turns follow the same rule.

> **Secret not updating after helm upgrade?**
>
> The `xpander-static` secret has a Helm resource keep policy, so `helm upgrade` may not update it on subsequent installs. If your API key isn't being picked up after a helm upgrade, set it directly in the secret:
>
> ```bash theme={"dark"}
> # Set the key directly in the Kubernetes secret
> kubectl patch secret xpander-static -n xpander --type=merge \
>   -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"
>
> # Restart the worker to pick up the new key
> kubectl rollout restart deployment xpander-agent-worker -n xpander
> ```
>
> **Secret field name mapping:**
>
> | Helm Value | Secret Field |
> | - | - |
> | `secrets.static.anthropicApiKey` | `ANTHROPIC_API_KEY` |
> | `secrets.static.agentsOpenaiApiKey` | `AGENTS_OPENAI_API_KEY` |
> | `secrets.static.fireworksApiKey` | `FIREWORKS_API_KEY` |
> | `secrets.static.awsBedrockApiKey` | `AWS_BEARER_TOKEN_BEDROCK` |
> | `secrets.static.googleApiKey` | `GOOGLE_API_KEY` |
> | `secrets.static.nebiusApiKey` | `NEBIUS_API_KEY` |
> | `secrets.static.heliconeApiKey` | `HELICONE_API_KEY` |
> | `secrets.static.openrouterApiKey` | `OPENROUTER_API_KEY` |
> | `secrets.static.deploymentManagerApiKey` | `DEPLOYMENT_MANAGER_API_KEY` |

### Key names read from an environment variable

An agent, a custom model provider or the organization's default model can name its API key by an environment variable on agent-controller (`agent-controller.env` or `global.env`, for example `MY_VLLM_TOKEN`). Such a name never reads one of the install's own credentials. These names read as no key:

* names starting with `AWS_` (except `AWS_BEARER_TOKEN_BEDROCK`), `XPANDER_`, `FLEET_`, `LLM_GW`, `LLM_GATEWAY`, `LITELLM_`, `GITHUB_APP`, `SUPABASE`, `REDIS`, `OAUTH`, `JWT`, `LICENSE`, `MONGO`, `POSTGRES`, `PG_`, `ON_PREM_PG`, `SMTP`, `REGISTRY`, `SANDBOX_`, `HARNESS_` or `KUBE`
* names ending with `_SERVICE_API_KEY`, `_PASSWORD`, `_PRIVATE_KEY`, `_SECRET`, `_SECRET_KEY`, `_SESSION_TOKEN` or `_TOKEN_FILE`
* the platform's own service keys (`DEPLOYMENT_MANAGER_API_KEY`, `AGENT_CONTROLLER_API_KEY` and the like)

The standard provider names (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `AWS_BEARER_TOKEN_BEDROCK`, `GOOGLE_API_KEY`, `AZURE_OPENAI_API_KEY`, `OPENROUTER_API_KEY` and the other provider keys) keep working, and so does any other name, such as `VLLM_API_KEY`. Rename a name that matches the list (for example `VLLM_SECRET` to `VLLM_API_KEY`) in both places: the agent-controller env that carries the value and the setting that names it.

**Model endpoints on metadata addresses are refused.** This applies to a model endpoint set on an agent, a custom provider, the organization's default model or an Azure key. xpander refuses it when it resolves to a metadata or link-local address (`169.254.0.0/16`, `fe80::/10`, the cloud metadata hosts). The check runs when the endpoint is saved and again on every call, against the address actually dialled. Private addresses (a vLLM or model gateway on your own network) keep working.

### Using Kubernetes Secrets (recommended for production)

```bash theme={"dark"}
kubectl create secret generic ai-service-keys \
  --namespace xpander \
  --from-literal=openai-api-key=sk-your-openai-key \
  --from-literal=anthropic-api-key=sk-ant-your-anthropic-key
```

```yaml theme={"dark"}
# In your values file
agent-worker:
  envFromSecretKeys:
    AGENTS_OPENAI_API_KEY:
      secretName: "ai-service-keys"
      key: "openai-api-key"
    ANTHROPIC_API_KEY:
      secretName: "ai-service-keys"
      key: "anthropic-api-key"
```

## Upgrading

To upgrade, watch the [xpander releases feed](https://github.com/xpander-ai/releases) (Releases page / Atom feed), then run the `upgrade` subcommand:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade
# or, pinned:
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade --chart-version <X.Y.Z>
```

`install.sh upgrade`:

* finds the release (default `xpander` in namespace `xpander`, override with `--release` / `--namespace`),
* **infers its edition** from the installed chart: you never pass `--edition` to an upgrade,
* **reuses the values the release was installed with** (`helm get values`), plus anything extra you pass (`--set`, `--values`),
* shows current → target version with a link to the release notes, and asks before changing anything,
* always runs with `--atomic`: a failed upgrade (including a failed hook) rolls back automatically and the previous revision keeps serving.

Manual-Helm equivalent (Helm 3.14 or later):

```bash theme={"dark"}
helm repo update
helm upgrade xpander xpander/xpander --namespace xpander --reset-then-reuse-values
```

`--reset-then-reuse-values` keeps the values you set and applies the new chart's defaults for everything else. Plain `--reuse-values` replays the previous release's values, including the old chart's defaults, so it misses defaults that changed (see below).

### Upgrading to the current chart

The current Hybrid chart is `xpander` 0.22.5 (release v2026.10.22). Database migrations run automatically. Upgrade with Helm 3.14 or later:

```bash theme={"dark"}
helm repo add xpander https://charts.xpander.ai && helm repo update xpander
helm upgrade xpander xpander/xpander --version 0.22.5 -n <namespace> --reset-then-reuse-values --wait --timeout 30m
```

**Before you upgrade from a chart between 0.22.0 and 0.22.4:** the changes below arrived across charts 0.22.1 to 0.22.5. Each one happens on the first upgrade to the chart that brings it, so skip the ones your current chart already has. On 0.22.3, for example, only the last two rows still apply.

| Change | Chart that brings it |
| - | - |
| In-cluster Postgres on the pgvector build, with the pre-upgrade check | 0.22.1 |
| `BRAIN_INGEST_API_KEY` in the generated Secret | 0.22.3 |
| `envFromSecret` and `envFromConfigMap` render | 0.22.4 |
| The LLM Gateway is always on | 0.22.5 |

* **In-cluster Postgres moves to the pgvector build (0.22.1).** The `postgres` StatefulSet runs `pgvector/pgvector:0.8.7-pg15-trixie` instead of `postgres:15`: PostgreSQL 15.19, the same major version, on the same volume. If you use `global.imageRegistry`, mirror `docker.io/pgvector/pgvector:0.8.7-pg15-trixie` first. Take a volume snapshot of `postgres-data-xpander-postgres-0`, or set `postgres.upgradeGuard.backup.pvcName` to an existing PVC with room for a full dump. Postgres restarts once. If you already run 0.22.1 or later, Postgres is on pgvector and this step is done.
* **A pinned Postgres image stays on pgvector (0.22.1).** If your values set `images.postgres`, keep it on a pgvector build of PostgreSQL 15. A pre-upgrade check refuses the upgrade when the major version would change.
* **A new key in the generated Secret (0.22.3).** `BRAIN_INGEST_API_KEY` is added automatically and kept from then on. No action needed.
* **Environment from your own Secrets and ConfigMaps renders (0.22.4).** `global.envFromSecret`, `global.envFromConfigMap` and the per-component `envFromSecret` and `envFromConfigMap` lists now render as the container's `envFrom`. Earlier charts failed the render when they were set. Installs that never set them render the same as before.
* **The LLM Gateway is always on (0.22.5).** The chart ignores `llm-gw.enabled`. If your install had `llm-gw.enabled: false`, mirror the `llm-gw` image if you mirror images, make sure every model provider your agents use has a key, and open the gateway's network paths ([LLM Gateway](#llm-gateway)). The upgrade starts the gateway pods and restarts agent-controller and api once. Then check `kubectl -n <namespace> get pods -l app=llm-gw`. If the gateway was already on, nothing changes.
* **Per-agent MCP URLs move to the MCP host (release change, no chart change).** MCP clients that used a per-agent MCP URL connect to `https://mcp.<DOMAIN>` instead. Per-agent MCP URLs are no longer served. The API's own MCP endpoint, `https://api.<DOMAIN>/mcp/` ([Model Context Protocol](https://docs.xpander.ai/api-reference/mcp.md)), does not change, so clients set up on it keep working. The chart has served `mcp.<DOMAIN>` from its ingress since 0.22.0. There is no value to set.

**The Postgres upgrade, step by step.** A pre-upgrade Job, `xpander-postgres-upgrade-guard`, refuses the upgrade when the running major version differs from the image's, and lists every database. With `postgres.upgradeGuard.backup.pvcName` set, it first writes a `pg_dumpall` to that PVC, and refuses the upgrade if the dump cannot be taken in full. The dump must finish within `postgres.upgradeGuard.timeoutSeconds` (default 1500) and within the `--timeout` of the upgrade, so raise both for a large database. After the upgrade, the `xpander-postgres-pgvector` Job adds the `vector` extension to every database. It never fails the release, and its log ends with one status line:

```bash theme={"dark"}
kubectl -n <namespace> logs job/xpander-postgres-pgvector
```

To go back, `helm rollback <release> <previous revision>` returns the pod to `postgres:15` on the same volume with all its data.

**Coming from a chart before 0.22.0**, these changes also apply:

* **The LLM Gateway runs.** Before you upgrade, mirror the `llm-gw` image if you mirror images, make sure every provider your agents use has a key, and open any network path the gateway needs ([LLM Gateway](#llm-gateway)). agent-controller and api restart once to pick up the gateway's address. After the upgrade, check `kubectl -n <namespace> get pods -l app=llm-gw` ([healthy result](#llm-gateway)).

* **Argo CD.** Add `ignoreDifferences` on `/data` for `xpander-llm-gw` and `xpander-llm-gw-content`, or set `llm-gw.masterKey.existingSecret` and `llm-gw.contentKey.existingSecret`.

* **Harness agents run only on the fleet executors.** There is no pod per agent. When no executor is up, a turn waits in the queue. The chart always keeps at least one executor: `fleet.replicas: 0` (or blank) renders one, and with `fleet.keda.enabled`, `fleet.keda.maxReplicas: 0` holds one. `agentContainers.harness` (`enabled`, `resources`, `storage`, `shm`) is ignored and can be removed from your values. `images.agentHarness` stays. It is the image every executor runs.

* **The release Role reads pods but no longer writes them.** agent-controller keeps get, list and watch on pods and still manages the Deployments, Services, volumes, Secrets and Jobs of agent workspaces. If your own automation used the release ServiceAccount to create, update or delete pods, add the rule back:

  ```yaml theme={"dark"}
  rbac:
    additionalRules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["create", "update", "patch", "delete"]
  ```

* **Fleet agents reach private networks like any other pod.** `fleet.networkPolicy.exceptCidrs` now defaults to `["169.254.0.0/16"]` and `exceptCidrsV6` to `["fe80::/10"]`, so agents and the MCP endpoints they call reach `10/8`, `172.16/12`, `192.168/16` and in-cluster services. Cloud metadata stays blocked. `--reset-then-reuse-values` picks this up. With `--reuse-values`, pass `--set-json 'fleet.networkPolicy.exceptCidrs=["169.254.0.0/16"]' --set-json 'fleet.networkPolicy.exceptCidrsV6=["fe80::/10"]'`. An install that set `exceptCidrs` itself keeps its own list.

* **Executor nodes need Landlock, or `fleet.landlock.enabled: false`.** See [Node prerequisites](https://docs.xpander.ai/self-hosted/sizing.md#node-prerequisites).

* **Model key names and endpoints.** Environment-variable key names that match the install's own credentials read as no key, and model endpoints on metadata addresses are refused ([details](#key-names-read-from-an-environment-variable)). Check the key names on your agents, custom providers and the organization's default model before you upgrade.

* **Custom-container agents** move to the standard agent runtime during the upgrade.

`--reuse-values` with the new defaults:

```bash theme={"dark"}
helm upgrade xpander xpander/xpander --namespace xpander --reuse-values \
  --set-json 'fleet.networkPolicy.exceptCidrs=["169.254.0.0/16"]' \
  --set-json 'fleet.networkPolicy.exceptCidrsV6=["fe80::/10"]'
```

What else to expect from the current chart line:

* **PostgreSQL no longer restarts on every chart release.** The config checksum hashes the ConfigMap data only, so a release whose Postgres configuration is byte-identical leaves the StatefulSet alone. A real `postgres.*` change still rolls it.
* **The fleet is always rendered, and `--atomic` waits for it.** A running turn holds the roll for up to the drain window (`fleet.executor.drainSeconds`, 3600 s). The idle guard holds the executor at zero replicas on a cluster without its StorageClass, so the upgrade completes anyway. If you drive `helm upgrade --wait` by hand, give `--timeout` at least the drain window or upgrade in a quiet period. Details on [Sizing and requirements](https://docs.xpander.ai/self-hosted/sizing.md#safe-rollout-drain-and-upgrades).

## Installer flags reference

```bash theme={"dark"}
bash install.sh [flags]            # install (the wizard, then helm upgrade --install)
bash install.sh upgrade [flags]    # upgrade an existing release in place
bash install.sh status [flags]     # read-only health report (--json for one object)
```

**Identity** (Hybrid identifies with the three values shown in xpander Chat. Required non-interactively.):

| Flag | Description |
| - | - |
| `--edition hybrid\|airgap` | Edition. `hybrid` (default): execution tier here, control plane in xpander cloud. `airgap`: every component here, no connection to xpander cloud at run time |
| `--org UUID` | Organization ID |
| `--env UUID` | Environment ID |
| `--dm-api-key KEY` | Deployment manager API key |

**LLM providers** (all optional, the wizard asks for these):

| Flag | Description |
| - | - |
| `--openai-key KEY` | OpenAI |
| `--anthropic-key KEY` | Anthropic |
| `--google-key KEY` | Google / Gemini |
| `--bedrock-key KEY` | Bedrock bearer token (`AWS_BEARER_TOKEN_BEDROCK`) |
| `--bedrock-auth key\|iam` | `key` (default when `--bedrock-key` is given) or keyless IAM role / EKS pod identity, see [Amazon Bedrock: keyless IAM or API key](#amazon-bedrock-keyless-iam-or-api-key) |
| `--fireworks-key KEY` | Fireworks AI |
| `--nebius-key KEY` | Nebius |
| `--helicone-key KEY` | Helicone observability |
| `--openrouter-key KEY` | OpenRouter |

**Hybrid connectivity & ingress:**

| Flag | Description |
| - | - |
| `--domain DOMAIN` | Enable ingress for this domain |
| `--dash-domains` | Dash-separated hostnames (`agent-controller-xpander.<DOMAIN>`) |
| `--privatelink` | Reach the deployment manager over [AWS PrivateLink](#privatelink) |
| `--staging` | Point at the xpander staging control plane (auto-detected for local/staging domains and local kube contexts) |
| `--dm-url URL` | Custom deployment manager URL. `--privatelink` / `--staging` / `--dm-url` are mutually exclusive |

**Chart and release:**

| Flag | Description |
| - | - |
| `--chart-path PATH` | Install from a local chart directory or `.tgz` |
| `--chart-version VERSION` | Pin the chart version |
| `--set KEY=VALUE` | Extra helm `--set` (repeatable) |
| `--values FILE` | Extra helm `-f` values file (repeatable) |
| `--namespace NAME` | Default: `xpander` |
| `--release NAME` | Default: `xpander` |
| `--kube-context NAME` | kubectl/helm context |

**Other:**

| Flag | Description |
| - | - |
| `--manual` | Print the whole install as copy-pasteable commands, personalized from the flags you passed, and do nothing. Both self-hosted deployments. Secrets stay placeholders. See [Advanced](#advanced-review-first--manual-install) |
| `--values-out PATH` | Write the generated values file to PATH and exit: no cluster needed. Useful for GitOps: review the file, then `helm upgrade --install` it yourself |
| `--dry-run` | Render everything through helm, install nothing |
| `-y`, `--yes` | Accept every confirmation |
| `--non-interactive` | Never read the terminal, even if one exists. Implies `--yes`. Missing required flags exit 2 instead of prompting |
| `-q`, `--quiet` | Phase lines and the outcome only |
| `--json` | `status` only: emit one JSON object |
| `--help` / `--version` | Usage text / installer version |

Air-gap-only flags (license, mirroring, image tags, data tier, base domain, code execution): [Air-gapped deployment](https://docs.xpander.ai/self-hosted/airgap.md).

## Automation & CI

The installer is built to be driven by scripts, CI pipelines, and coding agents:

* **Every prompt has a flag.** With `--non-interactive` nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies `--yes`.
* **Stable progress output.** Progress is one line per transition in a grep-friendly shape, `[3/6] chart-source: ok`, and `--quiet` reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
* **`status` is the poll target.** A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. `--json` emits one JSON object.

**Exit codes:**

| Code | Meaning |
| - | - |
| `0` | Success |
| `1` | Install or upgrade failure (`status`: degraded) |
| `2` | Usage error, or a required flag missing with no terminal |
| `3` | `status`: no such release installed |
| `4` | Preflight failure (missing helm/kubectl, unreachable cluster, version too old) |
| `5` | External data-tier probe failed and `--on-probe-failure` was `abort` (air-gap external data tier) |

**Hybrid, end to end, no terminal:**

```bash theme={"dark"}
bash install.sh --non-interactive --quiet \
  --edition hybrid --org <uuid> --env <uuid> --dm-api-key <key> \
  --domain xpander.example.com --openai-key <key>
```

**Poll until healthy:**

```bash theme={"dark"}
until bash install.sh status --quiet; do sleep 30; done
```

When a scripted run fails, the exit code plus the last `FIX:`/`RERUN:` lines tell you what to repair. See [Troubleshooting](#troubleshooting).

## Advanced: Review-first / manual install

### Read the installer before running it

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | less
curl -fsSL https://charts.xpander.ai/install-v<version>.sh | less   # this exact version
```

* The **versioned URL is immutable**: `install-v<version>.sh` is published once per release and never rewritten in place, so pin it in runbooks and CI (`bash install.sh --version` prints the version you're holding).
* The script is **shellcheck-clean**, enforced by CI on every push.
* **Secrets never touch the helm command line**: they are written to a temporary values file (mode 600, removed on exit), so they don't land in shell history or process listings.
* **Nothing runs against a cluster before the kube-context confirmation**: the installer names the context and cluster it is about to install into and waits for your yes.

### Do it yourself

| Path | What you get |
| - | - |
| `--manual` | The whole install as copy-pasteable commands (repo setup, secrets, a complete values file, the helm command, verification) personalized from the flags you passed, **executing nothing**. Secrets stay placeholders |
| `--values-out PATH` | Just the generated values file (mode 600), plus the printed `helm upgrade --install` command to run yourself. No cluster needed |
| `--dry-run` | The full flow rendered through `helm --dry-run`, installing nothing |

```bash theme={"dark"}
bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> --manual
```

> **Manual Helm install (no installer at all)**
>
> The installer is a wrapper around `helm upgrade --install`. Everything it does can be done by hand. The cleanest middle path is `--values-out`:
>
> ```bash theme={"dark"}
> bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> \
>   --domain <DOMAIN> --values-out xpander-values.yaml
> helm upgrade --install xpander xpander/xpander \
>   --namespace xpander --create-namespace -f xpander-values.yaml
> ```
>
> Or fully by hand. Add the repository:
>
> ```bash theme={"dark"}
> helm repo add xpander https://charts.xpander.ai
> helm repo update
> ```
>
> Set the deployment manager URL based on your connectivity choice (public `https://deployment-manager.xpander.ai`, or `https://deployment-manager-privatelink.xpander.ai` with [PrivateLink](#privatelink)) and install:
>
> ```bash theme={"dark"}
> helm upgrade --install xpander xpander/xpander \
>   --namespace xpander --create-namespace \
>   --set ingress.enabled=true \
>   --set domain=<DOMAIN> \
>   --set global.organizationId=<ORGANIZATION_ID> \
>   --set global.environmentId=<ENVIRONMENT_ID> \
>   --set secrets.static.deploymentManagerApiKey=<DEPLOYMENT_MANAGER_API_KEY> \
>   --set urls.deploymentManager=<DEPLOYMENT_MANAGER_URL> \
>   --timeout 10m
> ```
>
> LLM provider keys are `secrets.static.*` values. The full list is in the [key mapping table](#managing-llm-api-keys). Common optional parameters:
>
> | Parameter | Default | Description |
> | - | - | - |
> | `domain` | `""` (ingress disabled) | Base domain for ingress hostnames |
> | `ingress.enabled` | `false` | Expose services via ingress |
> | `ingress.tls.enabled` | `false` | Enable TLS on ingress |
> | `ingress.tls.source` | `self-signed` | `self-signed`, `cert-manager`, or `external` |
> | `redis.storage.size` | `8Gi` | Redis PVC size |
> | `redis.storage.storageClass` | cluster default | Storage class for Redis PVC |
> | `dockerRegistry.enabled` | `true` | In-cluster registry for agent container images. Disable it if your cluster provides one |
> | `metricsServer.enabled` | `true` | metrics-server for autoscaling. Disable it if your cluster runs one |
>
> For production, use a values file instead of `--set` flags:
>
> ```yaml xpander-values.yaml theme={"dark"}
> domain: "xpander.production.com"
>
> global:
>   organizationId: "your-org-id"
>   environmentId: "your-env-id"
>   env:
>     LOG_LEVEL: "info"
>     ENVIRONMENT: "production"
>
> urls:
>   deploymentManager: "https://deployment-manager.xpander.ai"
>
> secrets:
>   static:
>     deploymentManagerApiKey: "your-key"
>     agentsOpenaiApiKey: "sk-your-openai-key"
>     anthropicApiKey: "sk-ant-your-anthropic-key"
>
> ingress:
>   enabled: true
>   tls:
>     enabled: true
>     source: "cert-manager"
>   annotations:
>     cert-manager.io/cluster-issuer: "letsencrypt-prod"
>     nginx.ingress.kubernetes.io/ssl-redirect: "true"
>
> resources:
>   agentController:
>     requests:
>       cpu: "1000m"
>       memory: "2Gi"
>     limits:
>       cpu: "3000m"
>       memory: "4Gi"
>
> redis:
>   storage:
>     size: "32Gi"
>     storageClass: "fast-ssd"
> ```
>
> ```bash theme={"dark"}
> helm upgrade --install xpander xpander/xpander \
>   --namespace xpander --create-namespace \
>   --values xpander-values.yaml
> ```
>
> See [Sizing and requirements](https://docs.xpander.ai/self-hosted/sizing.md) for recommended resource values per service.

## Troubleshooting

Every installer failure prints two lines: **`FIX:`**, the action that repairs the condition, and **`RERUN:`**, your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under `~/.xpander/logs/` (`latest.log` points at the most recent run). For scripted runs, the exit code names the failure class ([Automation & CI](#automation--ci)), and `bash install.sh status` gives a read-only health verdict.

### Installer

> **Release stuck in pending-install / pending-upgrade / failed**
>
> `install.sh upgrade` requires the release to be in `deployed` status. Operating on a stuck release (`pending-install`, `pending-upgrade`, `pending-rollback`, `failed`, `uninstalling`) would strand it. Recover first:
>
> ```bash theme={"dark"}
> helm history xpander -n xpander
> helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
> helm uninstall xpander -n xpander                        # first install that never converged
> ```
>
> Then paste your `RERUN:` line. A `pending-install` status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out: `helm uninstall` is the right recovery for it.

> **Release already runs the other deployment's chart**
>
> The two self-hosted deployments share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (`--release` / `--namespace`), or remove the existing release first (`helm uninstall xpander -n xpander`).

> **Piped run exits with 'these required flags are missing'**
>
> Piping the installer into `bash` leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as `--non-interactive`). Hybrid requires `--org`, `--env`, and `--dm-api-key`. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console, it carries the identity flags, or download the script and run `bash install.sh` for the interactive wizard.

### Cluster and pods

> **Pods stuck in Init**
>
> The init containers wait for `agent-controller` to be ready. Check its logs:
>
> ```bash theme={"dark"}
> kubectl logs -n xpander deployment/xpander-agent-controller
> ```
>
> **Common cause:** Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly: see [PrivateLink](#privatelink).

> **Pods stuck in Pending**
>
> **Cause:** Storage class or PVC issues.
>
> ```bash theme={"dark"}
> kubectl -n xpander get pvc
> kubectl -n xpander describe pvc
> ```
>
> Check that a default StorageClass exists and the EBS CSI driver is running:
>
> ```bash theme={"dark"}
> kubectl get storageclass  # Should show a default (gp3)
> kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver  # Should show Running
> ```
>
> If using EKS, ensure the node role has `AmazonEBSCSIDriverPolicy` attached.

> **exec format error**
>
> You are running on ARM/Graviton nodes. xpander images are **amd64 only**. Switch to x86 instance types (`t3`, `m5`, `c5`, etc.).

> **Insufficient CPU**
>
> The `agent-worker` pod requests 2 CPU by default. Two options:
>
> * Add more nodes, or use larger instances.
> * For non-production environments only, lower the CPU request directly on the `xpander-agent-worker` Deployment in the `xpander` namespace:
>
> ```bash theme={"dark"}
> kubectl patch deployment xpander-agent-worker -n xpander \
>   --type='json' \
>   -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]'
> ```

> **Health checks failing**
>
> Check application logs (skill calls, including AWS ones, log in `agent-controller`):
>
> ```bash theme={"dark"}
> kubectl -n xpander logs deployment/xpander-agent-controller
> kubectl -n xpander logs deployment/xpander-agent-worker
> kubectl -n xpander logs deployment/xpander-mcp
> ```

### PrivateLink

> **PrivateLink InvalidServiceName (cross-region)**
>
> When creating a VPC endpoint to the xpander service from any region other than `us-west-2`, you **must** include `--service-region us-west-2`:
>
> ```bash theme={"dark"}
> aws ec2 create-vpc-endpoint \
>   --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
>   --service-region us-west-2 \
>   ...
> ```
>
> Without `--service-region`, AWS looks for the service in your local region and fails with `InvalidServiceName`.

> **PrivateLink connection timeout (HTTP 000)**
>
> Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR:
>
> ```bash theme={"dark"}
> aws ec2 authorize-security-group-ingress \
>   --group-id <ENDPOINT_SG_ID> \
>   --protocol tcp --port 443 --cidr <VPC_CIDR> \
>   --region <REGION> --profile <PROFILE>
> ```
>
> Also verify the private DNS hosted zone and alias record were created correctly: see [PrivateLink → Create private DNS](#privatelink).

### Ingress and networking

> **Ingress not accessible**
>
> Verify ingress configuration:
>
> ```bash theme={"dark"}
> kubectl -n xpander describe ingress
> kubectl -n xpander get ingress
> ```
>
> Check that the NLB was provisioned and DNS CNAME records point to it:
>
> ```bash theme={"dark"}
> kubectl get svc -n ingress-nginx ingress-nginx-controller \
>   -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
> ```

> **Load Balancer not provisioning**
>
> **Symptom:** the ingress controller's Service stays `<pending>`. Check its events (`kubectl -n ingress-nginx describe svc ingress-nginx-controller`):
>
> * `could not find any suitable subnets for creating the ELB`: the VPC's subnets are missing the ELB role tags. Tag public subnets `kubernetes.io/role/elb=1` for an internet-facing NLB. For an internal one, tag the private subnets `kubernetes.io/role/internal-elb=1` and annotate the Service `service.beta.kubernetes.io/aws-load-balancer-internal: "true"`.
> * On EKS Auto Mode, ensure the cluster role trust policy includes `sts:TagSession`:
>
> ```json theme={"dark"}
> {
>   "Effect": "Allow",
>   "Principal": { "Service": "eks.amazonaws.com" },
>   "Action": ["sts:AssumeRole", "sts:TagSession"]
> }
> ```

> **SSL certificate errors on chat URLs**
>
> The ACM certificate must include `*.chat.<DOMAIN>` as a subject alternative name. The chat UI generates per-thread subdomains (e.g., `moccasin-prawn.chat.<DOMAIN>`) that are not covered by `*.<DOMAIN>`.
>
> Request a new certificate with:
>
> ```bash theme={"dark"}
> aws acm request-certificate \
>   --domain-name "<DOMAIN>" \
>   --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
>   --validation-method DNS \
>   --region <REGION> --profile <PROFILE>
> ```

### API keys

> **API key not being picked up after helm upgrade**
>
> The `xpander-static` secret has a Helm resource keep policy, so a `helm upgrade` does not update it on its own. Patch the key directly on the `xpander-static` Kubernetes secret in the `xpander` namespace:
>
> ```bash theme={"dark"}
> kubectl patch secret xpander-static -n xpander --type=merge \
>   -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"
>
> kubectl rollout restart deployment xpander-agent-worker -n xpander
> ```
>
> See the [secret field name mapping](#managing-llm-api-keys) for all key names.

> **New runs refused: The xpander AI gateway is unavailable right now**
>
> New agent runs wait for a healthy LLM Gateway. Check its pods and the reason they are not ready (an image the cluster cannot pull is the usual one):
>
> ```bash theme={"dark"}
> kubectl -n <namespace> get pods -l app=llm-gw
> kubectl -n <namespace> describe pods -l app=llm-gw
> ```
>
> See [LLM Gateway](#llm-gateway) for the healthy result.
>
> If you mirror images, mirror `xpanderaihub/llm-gw` and point `images.llmGw` or `global.imageRegistry` at it. The gateway cannot be turned off, so new runs wait until its pods are ready.

> **LLM calls fail with a missing API key**
>
> A call to a provider with no key, neither in the organization (**Platform > AI governance > Models**) nor on the install (`secrets.static`), is refused with a message that names the missing key. Add the key in either place. To check the install's keys, verify they reach the `agent-worker` pod:
>
> ```bash theme={"dark"}
> kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY
> ```
>
> If missing, re-run the upgrade with the key set (`--set secrets.static.<key>=...`), or check that your secret is correctly mounted through `envFromSecretKeys`: see [Managing LLM API Keys](#managing-llm-api-keys).

### Debug commands

```bash theme={"dark"}
# Get all resources
kubectl -n xpander get all

# Check events
kubectl -n xpander get events --sort-by=.metadata.creationTimestamp

# Describe problematic pods
kubectl -n xpander describe pod <pod-name>

# Check service endpoints
kubectl -n xpander get endpoints
```

## Uninstalling

```bash theme={"dark"}
# Remove the Helm release
helm uninstall xpander --namespace xpander

# Clean up the namespace (optional)
kubectl delete namespace xpander
```

## Next steps

> **[Review the Hybrid network boundary](https://docs.xpander.ai/resources/security-compliance/security-architecture-hybrid.md)**
>
> Network boundary, PrivateLink, and what leaves your VPC

> **[Compare Hybrid and Air-Gapped](https://docs.xpander.ai/self-hosted/index.md)**
>
> Hybrid vs Air-Gapped comparison

> **[Run fully air-gapped instead](https://docs.xpander.ai/self-hosted/airgap.md)**
>
> Every component in your environment, no connection to xpander cloud at run time

> **[Set up identity and access](https://docs.xpander.ai/resources/security-compliance/access-control.md)**
>
> Identity by deployment, SSO, API keys, audit logs


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.