> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Air-gapped deployment

> Run every xpander component in your environment, with no connection to xpander cloud at run time.

An Air-Gapped install runs every xpander component in your environment, and nothing connects to xpander cloud at run time. It installs in a private cloud VPC (no internet or NAT gateways) or any disconnected environment.

![Diagram](https://docs.xpander.ai/images/diagrams/self-hosted-airgap-61d70e52.svg)

## What runs where

* **In your environment:** the web UI, control plane, a self-hosted Supabase stack (auth, storage, edge functions) and the data tier (PostgreSQL, Redis Stack, MongoDB), as one Helm release in one namespace. Agent skill calls (including MCP) and agent workspaces run there too.
* **Sent to xpander:** nothing. No heartbeats, no catalog sync, no telemetry.
* **Your choice:** your LLM endpoint, cloud services such as Bedrock, S3 or SES, and the registry you mirror from at install time.
* **MCP clients** (Claude, ChatGPT, Cursor) reach Omni once you set `global.mcp.publicUrl`. Until then, with `ui.env.VITE_OMNI_MCP_URL` empty too, **Platform > Agents > Channels** shows **Not offered on this install**, and the License page's **Capabilities** card shows the value to add.
* **Private mode** needs `global.network.internetEgress: false`. With `true`, built-in skills that use public APIs (for example arXiv and HackerNews) call them from the cluster. With `false`, those skills do not work.

> [card] [Air-Gapped Security Architecture](https://docs.xpander.ai/resources/security-compliance/air-gapped-deployments.md)
>
> Zero egress, the signed Ed25519 license, local sign-in, and the full install-time / run-time / never egress table.

## Three things define an air-gapped install

1. **The license is the identity.** Two sealed files (`install.json` and `activation.json`, format `xpander-license/v2`) plus the organization id you pass with `--org`. **No console IDs and no API key.** Older single-file `license.json` (v1) installs carry the org id inside. The installer generates the environment id and the first install creates your organization in-cluster. Single-tenant: one license, one organization.
2. **Images come from your own registry mirror.** xpander delivers them through the gated `xpanderaihub` Docker Hub org with a per-customer read token. The installer mirrors them, or you mirror them yourself on a sealed network.
3. **Image tags are pinned per release.** The chart refuses unpinned images, and an upgrade needs the new bundle mirrored first.

The interactive installer wizard runs with `--edition airgap`: a checklist, license validation, mirroring when needed, and a live watch over the install.

> [!NOTE]
>
> "Air-gapped" means **from xpander's cloud**, not necessarily from the internet. Internet-backed skills (web search, docs search, TTS) can stay on or be sealed off. Ask your xpander contact about the sealed hardening profile.

## What talks to what

**Outbound.** Nothing to xpander at run time. These are the connections you choose (full table: [Air-Gapped Security Architecture](https://docs.xpander.ai/resources/security-compliance/air-gapped-deployments.md#what-leaves-the-environment)):

| From your cluster | To | Purpose |
| - | - | - |
| LLM-serving components | Your LLM endpoint: in-network vLLM/Ollama/NIM, or Amazon Bedrock over a PrivateLink VPC endpoint, or a hosted provider if you allow it | Model calls |
| Skills (including MCP) | Only the external systems you choose to allow | Egress is your policy. The chart's internet-posture setting shapes which internet-backed skills are even offered |
| (install/upgrade time only) | xpander's registry, only when you use the [in-wizard mirroring](#when-the-mirror-is-empty-the-wizard-offers-to-fill-it) from a connected machine | Image delivery. A sealed network uses the [two-hop archive flow](#sealed--manual-mirroring) instead |

**Inbound.** All land on your ingress controller, over your own path (internal DNS, VPN, private load balancer). The host names are the chart's defaults and each can be renamed ([Domains, Ingress & DNS](#domains-ingress--dns)):

| From | To | Purpose |
| - | - | - |
| Users' browsers | `chat.<domain>`, `supabase.<domain>` | Using the web UI: invoking agents, chat, sign-in and file storage |
| SDK and API clients, CI pipelines | `api.<domain>`, `agent-controller.<domain>` | Invoking agents programmatically (REST, streaming, SDK) |
| In-network systems that trigger agents | `webhook.<domain>` | Webhook-triggered agent runs |
| OAuth redirect flows (browser) | `client-auth.<domain>` | Authorizing skills, [per-person MCP connector sign-in](https://docs.xpander.ai/run/integrations/mcp-per-person-sign-in.md) and the SSO login callback |
| Users' browsers (optional) | The short-links origin (`global.shortLinks.publicUrl`) | Short links to signed attachments, served by the `functions` Service |

***

## Prerequisites

| Requirement | Value |
| - | - |
| Nodes | 3 × 8 vCPU / 32 GiB (e.g. `m5.2xlarge`), amd64. Larger installs: see [Sizing](https://docs.xpander.ai/self-hosted/sizing.md#air-gapped) |
| Kubernetes | ≥ 1.28, with a CNI, cluster DNS (CoreDNS) and a **CSI driver with a default RWO StorageClass**. On EKS: the `vpc-cni`, `coredns`, `kube-proxy` and `aws-ebs-csi-driver` add-ons ([EKS Cluster Setup](https://docs.xpander.ai/self-hosted/eks-setup.md#eks-add-ons)), or the built-in `ebs.csi.eks.amazonaws.com` provisioner on [EKS Auto Mode](https://docs.xpander.ai/self-hosted/eks-setup.md#eks-auto-mode). Metrics-server is not required |
| Storage | \~700 GiB on an RWO StorageClass, **encrypted at rest**. On EKS, set `encrypted: "true"` in the [StorageClass](https://docs.xpander.ai/self-hosted/eks-setup.md#create-default-storageclass) |
| Network | An LLM endpoint reachable in-network. No connection to xpander cloud ([egress table](https://docs.xpander.ai/resources/security-compliance/air-gapped-deployments.md#what-leaves-the-environment)) |
| License files | `install.json` and `activation.json` plus your organization id, issued by the xpander.ai team and delivered over email ([Two-File Licensing](https://docs.xpander.ai/self-hosted/licensing.md)) |
| Container registry | Reachable by the cluster. The installer can fill it for you |
| Optional | One GPU node (L4/A10 class) for voice / local inference |
| Tooling | `helm` ≥ 3.13 and `kubectl` where you install. `skopeo` + `python3` where you mirror (Docker alone also works). `crane` or `skopeo` for the installer's mirror check |

Published requirements: [requirements.json](https://charts.xpander.ai/requirements.json).

* **The executor fleet** (where Claude Code, Codex and OpenCode turns run) adds to these numbers and needs a namespace that admits PodSecurity `baseline`, Landlock on its nodes (or `global.fleet.landlock.enabled: false`) and XFS with `prjquota`. No privileged container or node preparation. See [Node prerequisites](https://docs.xpander.ai/self-hosted/sizing.md#node-prerequisites).
* **The LLM Gateway** is always on, and agents' model calls go through it. It adds its pods (0.5 vCPU / 1 GiB requested each, up to 1 vCPU / 4 GiB) and the `llm-gw` image. See [LLM Gateway on a self-hosted install](https://docs.xpander.ai/run/llm-gateway/self-hosted.md).

The fleet starts warm with one executor and needs a StorageClass that mounts XFS with `prjquota` for its 500 GiB per-executor volume:

| Your cluster | What to set |
| - | - |
| You bring the class | Create it under the name in `global.fleet.storage.className` (`fleet-xfs`). `createClass: false` is the Air-Gapped default. |
| EKS | Set `global.fleet.storage.createClass: true` to let the chart render the class for the EBS CSI driver (`ebs.csi.aws.com`, gp3, encrypted). |
| EKS Auto Mode | Also set `global.fleet.storage.provisioner: ebs.csi.eks.amazonaws.com`. The classic driver object exists there but has no controller behind it, so a class on `ebs.csi.aws.com` never binds. |

Without that class or driver, the chart renders the executor idle at `replicas: 0`, annotated `xpander.ai/fleet-idle-reason`, so an `--atomic` install never waits on a volume that cannot bind. Create the class and the next upgrade starts it. Sizing tiers: [Sizing and requirements](https://docs.xpander.ai/self-hosted/sizing.md#air-gapped).

***

## Install in three steps

**1. Receive the license files and pull token by email**

The xpander.ai team emails you:

| Item | What it is |
| - | - |
| **Two sealed license files** | `install.json` (identity: org, edition, no expiry) and `activation.json` (entitlement: term, caps, features) |
| Your **organization id** | The value for `--org` |
| The **license public key** | Used to verify the license signature |
| **A registry pull token** | For the gated `xpanderaihub` Docker Hub org: read-only, scoped to your account, revocable. It gates *delivery* only. The license is the enforcement. |

Details: [Two-File Licensing](https://docs.xpander.ai/self-hosted/licensing.md).

**2. Run the one-liner**

From a terminal with `kubectl` access to the cluster:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition airgap --org <ORG_ID> \
  --install-license-file ./install.json \
  --activation-license-file ./activation.json
```

A legacy v1 install passes `--license-file ./license.json` instead of the three flags.

The wizard runs preflight checks, installs with Helm and seeds an admin login. It prompts for:

* Registry
* Chart source (OCI pull or local path)
* Data tier
* Admin email
* Organization display name
* StorageClass
* Agent code execution (`--code-execution` / `--no-code-execution`). The runner embeds dockerd, so its pod is privileged, and a restricted PodSecurity policy refuses it
* A [base domain](#domains-ingress--dns) for the public hosts
* LLM provider keys, for any hosted providers your egress policy allows

Image tags come from the [release](#releases).

**If your registry has no xpander images yet, the wizard offers to [mirror them right there](#when-the-mirror-is-empty-the-wizard-offers-to-fill-it)** with the pull token from step 1, and creates ECR repositories itself. On a sealed network, use [Sealed / manual mirroring](#sealed--manual-mirroring).

**3. Sign in as the seeded admin**

The first install creates your organization and a bootstrap admin in your in-cluster Supabase. The closing summary prints how to sign in: see [Signing In](#signing-in).

Non-interactive (image tags resolve from the mirror unless pinned):

```bash theme={"dark"}
bash install.sh --edition airgap \
  --org <ORG_ID> --install-license-file ./install.json --activation-license-file ./activation.json \
  --registry <your-registry>/xpander \
  --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --domain xpander.acme.internal \
  --admin-email <you@company.example> --org-name "<Your Company>" --yes
```

To review every command first, `bash install.sh --edition airgap --manual` prints a personalized runbook and runs nothing: see [Advanced: review-first / manual install](https://docs.xpander.ai/self-hosted/hybrid.md#advanced-review-first--manual-install).

***

## What happens when you run the installer

Steps 1–6 take a few minutes. Step 7 takes **10–20 minutes**. Each phase prints a `[N/8] name: ...` line for scripts to grep ([Automation & CI](#automation--ci)).

> [!NOTE]
>
> Every hook failure **names the Job**, prints its **last 30 log lines**, and ends with a `FIX:` line and a `RERUN:` line that resumes the install.

**1. Resolve: load the latest release**

One release manifest lists the version and SHA-256 digest of every xpander image, including `llm-gw` and `tool-repo-seed`. The installer passes those pins to Helm. To repeat a release or install offline, pass `--release-manifest ./release-manifest.json`. A missing or invalid manifest stops the run. [How releases work](#releases).

**2. Checklist: preconditions verified before anything changes**

The registry answers `/v2/` (401 counts as up), `agent-controller` and `deployment-manager` exist **at their pinned tags**, an RWO StorageClass exists, the image pins are in place, `--org-name` and `--admin-email` are set, and a chart source exists. Any failure stops here with nothing to clean up.

**3. License: validated locally, applied as a Secret**

Two `xpander-license/v2` envelopes, one `kind: install` and one `kind: activation`, bound to `--org` (a legacy v1 file: envelope, expiry and org id). They become the `xpander-license` Secret. The **Ed25519 signature** is verified in the preflight hook and again at deployment-manager boot.

**4. Chart: pulled over OCI**

The `xpander-airgap` chart comes from the gated OCI registry, your own mirror, or a `.tgz` via `--chart-path`.

**5. Values: secrets never touch the command line**

Configuration goes to a temporary values file (mode 600, removed on exit), never to the helm command line, shell history or process list.

**6. Install: helm, without --atomic on purpose**

The **first** install runs without `--atomic`, because `--wait` would hold the seed hooks that the api and agent-controller pods are waiting on. The smoke hook is the readiness gate. **Upgrades always run with `--atomic`**, so a failed upgrade rolls back and the previous revision keeps serving.

**7. Pipeline: the cluster comes up, hook by hook (10–20 min)**

The installer watches the hook pipeline live:

```
preflight → keygen → data-checks → workloads → migrations → org-seed → catalog-seed → smoke
```

| Stage | What it does |
| - | - |
| **preflight** | StorageClass, node architecture, and the license's shape, expiry and **Ed25519 signature**. A bad license fails here, once, readably |
| **keygen** | Every key is **minted in your cluster** (15+ inter-service keys, Supabase JWTs, the admin password) into `xpander-generated-keys`. **Lookup-preserve**: existing keys are never regenerated |
| **data-checks** | External data tier only: `pg_isready` and `redis-cli ping` **from inside the cluster**, so network problems surface by name before any workload starts |
| **workloads** | Data tier, Supabase, control plane and UI roll out |
| **migrations** | The **full schema is replayed** into your Supabase, one versioned Job per revision. Forward-only: rollback means restore-from-backup |
| **org-seed** | **Your organization is created in your cluster**: the licensed org, the admin user and the vault. Idempotent |
| **catalog-seed** | The skills catalog loads from the mirrored artifacts image |
| **smoke** | Every enabled component is probed through its Service. Success only when all answer |

**8. Summary: URLs, password, logs**

The closing box prints the **derived URLs** (with a [base domain](#domains-ingress--dns): `https://chat.<domain>`, `https://api.<domain>`, `https://supabase.<domain>`), the **bootstrap admin password**, and how to read it later:

```bash theme={"dark"}
kubectl -n xpander get secret xpander-generated-keys \
  -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
```

The secret-scrubbed session log (mode 600) is under `~/.xpander/logs/`, with `latest.log` pointing at the last run.

***

## Releases

The release manifest supplies xpander image versions and digests to Helm. Helm still deploys. By default, the installer and mirror script use the [latest release](https://github.com/xpander-ai/releases/releases/tag/latest). A manifest names every xpander image: the services (including `billing`, `openapi-spec-generator` and `llm-gw`), the agent runtime images, and the artifacts (UI, migrations, edge functions, skills, catalog specs and `tool-repo-seed`). Logos ship in the UI image.

* **Latest release:** omit `--release-manifest` on a connected machine.
* **Repeat a release:** pass the same `--release-manifest FILE` to both scripts, so a new release between mirroring and installing changes nothing.
* **Invalid release:** stops the run. It never falls back to `latest` or guesses tags.
* **Consistency:** tag flags must match the manifest, and its pins override conflicting image values. Registry, infrastructure and secrets stay yours.

Third-party image versions come from the chart, so keep the matching chart package with each release.

**Keep mirror tags longer than the releases you may re-install.** A "keep the last twenty images" rule spans only days, and a re-install then fails on an expired image. Keep hundreds of tags, or exempt pinned ones. If repositories must exist before a push, create one per release image, including `agent-harness`, `openapi-spec-generator`, the third-party data-tier and Supabase images, and `charts/xpander-airgap`. `image_sync.sh` creates missing ECR repositories itself.

**Tracking pre-release builds** (staging only): set `XPANDER_IMAGE_STREAM=dev` for the installer and pass `--stream dev` to `image_sync.sh`. Otherwise an air-gapped install refuses development overrides.

Per-artifact tag flags:

| Artifact | Mirror script (`image_sync.sh`) | Installer |
| - | - | - |
| UI | `--ui-tag` | not available |
| Skills | `--skills-tag` | `--skills-tag` |
| Catalog specs | `--catalog-specs-tag` | `--catalog-tag` |
| Skills repository seed (`tool-repo-seed`) | `--tool-repo-seed-tag` | read from the release manifest |
| Migrations | `--migrations-tag` | `--migrations-tag` |
| Edge functions | `--edge-functions-tag` | `--edge-functions-tag` |

Other flags: `--manifest-out <file>` writes the copied tags as a values file (pass it last with `--values`). `--release-candidate` selects a candidate. Credentials: `image_sync.sh --source-user xpanderaihub --source-token <token>` (and `--dest-user` / `--dest-token`), installer `--xpander-registry-token <token>`. `--icon-pack-tag` no longer exists, and `image_sync.sh` refuses unknown flags.

> [!WARNING]
>
> Installer 0.17.1 cannot parse the JSON release manifest, so `--release-manifest` alone fails. Until the fix ships, mirror with `--manifest-out mirror-pins.yaml` and pass `--values mirror-pins.yaml` as the last values file, alongside `--release-manifest`.

**Publishing a release does not upgrade clusters.** Your Helm, GitOps or upgrade schedule decides when.

### Existing Helm and GitOps pipelines

| Deployment flow | Release handling |
| - | - |
| Installer invokes Helm | Refresh the installer and mirror scripts together. |
| Direct `helm upgrade`, Argo CD, or Flux | Supply the generated release values with your configuration, and update chart and image values together. Helm does not read the JSON manifest. |
| Fully offline installation | Transfer the image bundle, manifest, scripts, validator and chart package before upgrading. |

Example pipeline: [CI/CD & GitOps](#cicd--gitops).

### When the mirror is empty, the wizard offers to fill it

Choose **Mirror them now**, give the xpander pull credentials and any destination credentials, and let the copy finish. ECR repositories are created when your AWS credentials allow. Non-interactively: `--mirror now|guide|skip` (`now` needs the credentials). Sealed networks use the archive flow below.

## Sealed / manual mirroring

Mirror with the current `image_sync.sh`: it includes `llm-gw` and `tool-repo-seed`, and stops on a release without the gateway image. It needs **skopeo and Python 3**. The Docker pull/tag/push fallback does not preserve release digests.

On a connected workstation, download the manifest once and use that file for both the copy and the upgrade:

```bash theme={"dark"}
curl -fsSLO https://charts.xpander.ai/image_sync.sh
curl -fsSLO https://charts.xpander.ai/install.sh
curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json \
  -o release-manifest.json
bash image_sync.sh --source docker.io/xpanderaihub \
  --dest <your-registry>/xpander \
  --release-manifest ./release-manifest.json
```

Pass registry credentials with the source/destination flags or registry auth. `--dry-run` prints the plan. Keep the JSON manifest: a single `--tag` or a legacy service-only manifest is not a complete release.

**Fully sealed network?** Archive outside, carry the whole directory in, import inside:

```bash theme={"dark"}
# On a connected workstation:
bash image_sync.sh --source docker.io/xpanderaihub \
  --release-manifest ./release-manifest.json --archive-dir ./xpander-images

# After transferring the complete directory, on the inside:
bash image_sync.sh --from-archive ./xpander-images --dest <your-registry>/xpander
```

Carry `install.sh`, `image_sync.sh` and their `release_manifest.py` with the bundle. Get the validator from the immutable `release_manifest-<sha256>.py` URL in the scripts and save it as `release_manifest.py` beside them. The scripts verify its checksum. Import validates the full image set and digests first. Complete an incomplete archive on the connected side. Legacy archives need the previous importer.

The chart is an OCI artifact. Mirror it with the images (`--chart-version <X.Y.Z> --chart-source registry-1.docker.io/xpanderaihub/xpander-airgap`, landing at `oci://<your-registry>/xpander/charts/xpander-airgap`), or hand-carry the `.tgz`:

```bash theme={"dark"}
helm registry login registry-1.docker.io --username <user> --password-stdin
helm pull oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version <X.Y.Z>
# copy xpander-airgap-<X.Y.Z>.tgz onto the sealed-network host, then --chart-path it
```

> [accordion] Using Nexus, Artifactory, or Harbor
>
> **Hosted repository: mirror into it.** Use it as `--registry`, and fill it with the wizard or `image_sync.sh`. `--dest-user` / `--dest-token` handle Basic auth:
>
> ```bash theme={"dark"}
> bash image_sync.sh --source docker.io/xpanderaihub \
>   --dest nexus.corp.example:5000/xpander \
>   --dest-user <user> --dest-token <token>
> ```
>
> Then install with `--registry nexus.corp.example:5000/xpander`.
>
> **Pull-through proxy, no mirror step.** Create a Docker *proxy* repository upstream of Docker Hub (`https://registry-1.docker.io`) with the xpander read token as its **upstream** credentials, and keep the org path in the registry value:
>
> ```bash theme={"dark"}
> --registry <nexus-host>/<proxy-repo>/xpanderaihub
> ```
>
> Images cache on first pull. Keep the `xpanderaihub` segment: the chart builds `<registry>/<name>`, so `<proxy-repo>/xpanderaihub/agent-controller` maps to `xpanderaihub/agent-controller` upstream. A **fully sealed** network cannot use a proxy: use the archive flow into a hosted repository.

> [!TIP]
>
> `bash install.sh --edition airgap --guide` prints this runbook personalized with your registry and namespace (also printed when no cluster is reachable). `--manual` prints the [whole install as commands](https://docs.xpander.ai/self-hosted/hybrid.md#advanced-review-first--manual-install) and runs nothing.

***

## Data tier: in-cluster or external

Choose carefully: moving from chart-managed StatefulSets to a managed service later is a migration. The wizard asks per store. Both default to in-cluster.

> [tab] In-cluster (default)
>
> Chart-managed StatefulSets on your StorageClass:
>
> * **PostgreSQL 16**: history, memory and per-agent databases
> * **Redis Stack** (`redis-stack-server`): queues, cache and knowledge-base vectors. Not plain Redis: knowledge bases use RediSearch (`FT.*`)
> * **MongoDB**: catalog and monitoring
> * Supabase ships its **own** pinned PostgreSQL 15 (it needs extensions managed services lack)
>
> No flags needed. `--storage-class <name>` picks a non-default class.

> [tab] External (RDS / MemoryDB)
>
> Bring your own PostgreSQL and/or a Valkey / MemoryDB cache with the search module. Interactively:
>
> ```bash theme={"dark"}
> bash install.sh --edition airgap ... \
>   --pg-mode external --pg-dsn 'postgresql://xpander:password@db.internal:5432/xpander' \
>   --redis-mode external --redis-endpoint <clustercfg endpoint, no port> \
>   --redis-tls --redis-cluster --redis-username <acl-user> --redis-password <pass>
> ```
>
> In a pipeline, put it in values and pass passwords with `--set`, never in Git:
>
> ```yaml theme={"dark"}
> postgres:
>   mode: external
>   external:
>     host: postgres.internal.acme.example
>     port: 5432
> redis:
>   mode: external
> global:
>   redis:
>     endpoint: <clustercfg endpoint, no port>
>     clusterMode: true
>     auth:
>       username: <acl-user>
> ```
>
> ```bash theme={"dark"}
> bash install.sh --edition airgap ... --values values.yaml \
>   --set keygen.overrides.POSTGRES_PASSWORD=<postgres password> \
>   --set global.redis.auth.password=<redis password>
> ```
>
> * **PostgreSQL**: create the role and database first. The role needs **`CREATEDB`** for per-agent databases:
>
>   ```sql theme={"dark"}
>   CREATE ROLE xpander LOGIN PASSWORD '<password>' CREATEDB;
>   CREATE DATABASE xpander OWNER xpander;
>   ```
>
>   `keygen.overrides.POSTGRES_PASSWORD` overrides every generated value.
> * **On RDS**, name the master user and initial database `xpander`: the master role already has `CREATEDB`, so no SQL is needed.
> * **Valkey / MemoryDB** must serve **RediSearch (`FT.*`) commands**, or knowledge bases accept writes and return nothing.
>   * On MemoryDB: **Valkey 7.3 on parameter group `default.memorydb-valkey7.search`**, chosen at creation (it cannot be added later).
>   * An ACL user with `on ~* &* +@all`.
>   * TLS (`--redis-tls`) and cluster mode (`--redis-cluster`). Encrypted MemoryDB and ElastiCache answer TLS only.
>   * **One cluster per environment**: ACLs cannot split `FT.*` namespaces. Recreating it means re-ingesting knowledge-base vectors.
>   * `--redis-username` / `--redis-password` for ACL auth, `--redis-port` if not 6379.
>   * The chart adds a `redis` ExternalName Service, so service DNS stays the same.
>   * Probe errors echo the endpoint's replies (`NOAUTH`, `WRONGPASS`).
> * **MongoDB has no external mode** and Supabase's Postgres is **never external** (its migrations need extensions RDS lacks).
> * **Blob storage**: Supabase Storage keeps uploads, attachments and live-surface assets on its own volume. No S3 backend. See [File storage](https://docs.xpander.ai/run/integrations/file-storage.md#object-storage).
>
> **Live connectivity probes.** Before installing, a probe pod runs `pg_isready` and `redis-cli ping` from your namespace. On failure, retry, continue (the pre-install hook re-checks) or abort with nothing changed. `--on-probe-failure retry|continue|abort` preselects. Non-interactive runs default to `abort` and exit with code 5.

***

## Local LLM inference

xpander speaks the OpenAI-compatible API, so any local runtime that exposes it plugs in:

| Runtime | Models | Notes |
| - | - | - |
| vLLM | Llama, Mistral, Qwen | High-throughput inference. OpenAI-compatible API |
| Ollama | Llama, Mistral, Phi, Qwen | Easier to operate. Good for smaller deployments |
| NVIDIA NIM | Llama, Mistral, others | Optimized GPU runtime. Enterprise support |
| AWS Bedrock (private) | Anthropic, Mistral, Llama | Only for VPC deployments with PrivateLink to Bedrock: *not* fully air-gapped |

Size the inference pool separately: it is your infrastructure, not part of the chart. Hosted providers also work if your egress allows, with the same [LLM provider keys](https://docs.xpander.ai/self-hosted/hybrid.md#managing-llm-api-keys).

Two models are declared at install: the agents' default and the **gateway model** that filters requests first (`global.llm.gateway.{provider,model,apiBase}`). Each install or upgrade converges the organization's defaults to the chart's declaration only when that declaration changed, so defaults an admin edited in the UI survive upgrades.

> [!NOTE]
>
> **Amazon Bedrock without a key.** In images built after 27 August 2026, chat, tasks and their summaries use the pods' own AWS identity (EKS Pod Identity or IRSA): leave `global.llm.bedrock.bearerTokenSecret` empty and set `global.llm.bedrock.region`.
>
> * The pod role needs `bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`, `bedrock:Converse` and `bedrock:ConverseStream` on `*` (the `global.*` profiles route across regions), bound to both `default` and `agent-controller` ServiceAccounts. No second role is assumed.
> * The account must be opted in to Bedrock data retention for those models in every region a `global.` profile spans. A sealed network needs VPC endpoints for `bedrock-runtime`, `sts` and `eks-auth`.
> * A Bedrock API key in `bearerTokenSecret` is the alternative.
> * The "IAM role" method under **Platform > AI governance > Models** applies to the Claude Code leg only. If such an entry exists and chat fails, remove it and restart `agent-controller`, `agent-worker`, `agents`, `actions` and `functions`.

A working keyless declaration, from a customer install on EKS:

```yaml theme={"dark"}
global:
  llm:
    defaults:
      provider: amazon_bedrock
      model: global.anthropic.claude-sonnet-5
    gateway:
      provider: amazon_bedrock
      model: global.anthropic.claude-sonnet-4-6
    bedrock:
      region: <your Bedrock region>
```

Leave `bearerTokenSecret` and `assumeRoleArn` unset, put the Bedrock permissions on the role the pods run as (`xpander-sa-role` there), bind it to `default` and `agent-controller`, and upgrade. No separate Bedrock role is needed.

**Claude Code on Bedrock**: the CLI calls the in-cluster LLM Gateway directly, and the gateway gets the Bedrock credential from the Agent Controller, the only place assume-role applies: `global.llm.bedrock.assumeRoleArn` (plus `assumeRoleExternalId`), or with no token and no role, the pod's own credentials. `global.llm.bedrock.baseUrl` points it at your VPC endpoint. Fable- and Mythos-class models also need the per-region data-retention opt-in. See [AI vendors](https://docs.xpander.ai/run/integrations/ai-vendors.md#anthropic-on-amazon-bedrock).

Summaries around agent work (thread titles, context compaction, evaluation judges) use the org default model, backed by the install's other providers. None dials a public endpoint on a sealed install.

### A worked vLLM setup on EKS Auto Mode

Checked end to end on an Air-Gapped install: vLLM serving a small instruct model on a T4, registered as a custom provider. The cluster side is yours. xpander needs only the address.

**GPU capacity.** A `NodePool` limited to `g4dn` with a `nvidia.com/gpu` taint gets a T4 node in about two minutes, exposing `nvidia.com/gpu: 1` with no device plugin. It needs `nodeClassRef` `group: eks.amazonaws.com`, `kind: NodeClass`, `name: default`. Keep weights on a PersistentVolumeClaim mounted as `HF_HOME` (pre-loaded on a sealed network).

**Five things that stop vLLM:**

1. A Service named `vllm` injects `VLLM_PORT=tcp://...` and vLLM refuses to start (`VLLM_PORT ... appears to be a URI`). Set `enableServiceLinks: false` or rename the Service.
2. A T4 needs `--dtype=half`. vLLM 0.11 crash-loops on a T4. `vllm/vllm-openai:v0.9.2` works.
3. Pass the key as `VLLM_API_KEY`, not `--api-key` (which logs it). It guards only `/v1`. `/health`, `/metrics` and `/tokenize` answer without it, so keep port 8000 on a `ClusterIP` Service, never an ingress or load balancer.
4. Start with `--enable-auto-tool-choice --tool-call-parser=hermes`, or **Add provider** marks the model as unable to run agents.
5. Leave room above the composed prompt. `--max-model-len=8192` failed with `400 This model's maximum context length is 8192 tokens. However, you requested 8298 tokens`, shown as **Unknown model error** (controller log: `Non-retryable model provider error: Unknown model error` next to `[prompt-budget] system_message_tok=...`). `--max-model-len=32768` ran. Custom models also log `unknown model.id ... defaulting to 128,000 window`.

The manifests, with account-specific values replaced by placeholders:

```yaml theme={"dark"}
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        workload: gpu
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: eks.amazonaws.com/instance-family
          operator: In
          values: ["g4dn"]
        - key: eks.amazonaws.com/instance-size
          operator: In
          values: ["xlarge", "2xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      taints:
        - key: nvidia.com/gpu
          value: "present"
          effect: NoSchedule
      terminationGracePeriod: 10m
  limits:
    cpu: "16"
    nvidia.com/gpu: "2"
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 10m
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-models
  namespace: <namespace>
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: <storage class>
  resources:
    requests:
      storage: 40Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: <namespace>
  labels: { app: vllm }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector: { matchLabels: { app: vllm } }
  template:
    metadata:
      labels: { app: vllm }
    spec:
      enableServiceLinks: false   # a Service named vllm would inject VLLM_PORT=tcp://... and stop vLLM
      nodeSelector:
        workload: gpu
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: docker.io/vllm/vllm-openai:v0.9.2   # from your mirror; 0.11 does not run on a T4
          args:
            - --model=Qwen/Qwen2.5-1.5B-Instruct
            - --served-model-name=qwen2.5-1.5b-instruct
            - --dtype=half
            - --max-model-len=32768          # 8192 fails the first turn: the composed prompt is larger
            - --gpu-memory-utilization=0.90
            - --host=0.0.0.0
            - --port=8000
            - --enable-auto-tool-choice     # xpander's Add provider dialog probes for this
            - --tool-call-parser=hermes
          env:
            - name: HF_HOME
              value: /models
            - name: VLLM_API_KEY             # the variable, not --api-key: the flag logs the key
              valueFrom:
                secretKeyRef: { name: vllm-api-key, key: key }
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 12Gi
            requests:
              cpu: "2"
              memory: 8Gi
          volumeMounts:
            - { name: models, mountPath: /models }
            - { name: shm, mountPath: /dev/shm }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 60
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: vllm-models }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 2Gi }
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: <namespace>
spec:
  selector: { app: vllm }
  ports:
    - port: 8000
      targetPort: 8000
```

### Register the endpoint in xpander

1. **Platform > AI governance > Models**, **Add provider**.
2. **Name** is free text, and **ID (used in API calls)** derives from it and is fixed. **Protocol**: **OpenAI-compatible**. **Base URL**: the address the pods reach, for example `http://vllm.<namespace>.svc.cluster.local:8000/v1`. In-cluster addresses work on Hybrid and Air-Gapped. xpander cloud accepts only a public `https` address.
3. **API key**: **No key (local or unauthenticated server)**, **Use an existing key**, or **Create a new key** with a **Key name** and the **API key** (the `VLLM_API_KEY` value).
4. **Test connection** shows **Reachable** and the served models. A `tools` badge marks models that accept function calls. **Add** the model, with an optional **Display name**.
5. Click **Add provider**.
6. In the provider's **Keys and models…**, tick the models the organization may use. Prices (USD per 1M tokens, via **Edit model**) track spend and are never charged. **Who may use it…** sets access. New models wait under **New models** unless **Approve new models automatically** is on. **Agent readiness** shows, from the last **Test model**, which harnesses work with each model.

![Models page with the xpander.ai vllm provider panel open, showing Keys, LLM gateway, New models with Approve new models automatically, a Models list with muse-glimmer ticked and its prices](https://mintcdn.com/xpanderai-099931d1/wqIIXSsh0kqA2hZ_/images/verify/lab-llm-model-cost.png?fit=max\&auto=format\&n=wqIIXSsh0kqA2hZ_\&q=85\&s=8d73c56a36da4da694142831c00a1983)

Keys and models on a custom provider: the key in use, New models waiting for approval, the models the organization may use with their prices. Shown with sample data and a different model than the one deployed above.

`POST /v1/agents` accepts only built-in providers in `model_provider` (a custom ID gets `422`). Pick a custom provider per conversation in **Harness and model**. The header then reads, for example, **OpenCode · Qwen2.5 1.5B (lab T4)**, the controller logs `gateway turn start` with `model=<host>:<port>/<model id>`, and vLLM logs `POST /v1/chat/completions 200`.

***

## The executor fleet on Air-Gapped

Agent turns on Claude Code, Codex and OpenCode run on the executor fleet. Sizing: [Sizing and requirements](https://docs.xpander.ai/self-hosted/sizing.md). On Air-Gapped, also set:

| Concern | What to set |
| - | - |
| **An LLM route** | CLIs never call a provider directly: calls go through the Agent Controller and the LLM Gateway. Preflight needs one route: an org default provider (`global.llm.defaults` or `global.llm.bedrock`), a gateway provider (`global.llm.gateway`), an in-cluster endpoint (`LLM_PROXY_<PROVIDER>_BASE_URL`, for example `LLM_PROXY_ANTHROPIC_BASE_URL`), or `global.fleet.llmProxyUrl`. With `global.network.internetEgress: false`, a leg with no override is refused |
| **Which CLI, provider and model a new agent gets** | Explicit choices win. Otherwise the **organization's default LLM** (Anthropic: Claude Code, Bedrock with an Anthropic model: Claude Code on Bedrock, OpenAI: Codex, others: OpenCode), then `global.harness.defaultModels.{claudeCode,codex,opencode,xpander}` and `global.harness.defaultProviders.claudeCode` (chart 0.12.10+), then baked defaults. With neither, a Claude Code agent runs **Fable 5.1 over Anthropic direct** and needs an Anthropic key, or Bedrock declared instead |
| **Bedrock auth** | `global.llm.bedrock.region`, then `bearerTokenSecret`, `assumeRoleArn` (+ `assumeRoleExternalId`), or nothing for Pod Identity / IRSA. `baseUrl` for the VPC endpoint. Fable- and Mythos-class models need the per-region data-retention opt-in on the signing account, including an organization's own Bedrock key account |
| **The license** | The activation file must carry the fleet entitlement (`harness / containers`), or agent creation is refused |
| **CLI versions with no registry** | Seal the CLIs on a connected machine (`python -m xpander_harness.cli_updates --root /tmp/cli bundle --out cli-bundle.tar.gz`), put the tarball on a PVC and set `global.fleet.cliBundle.enabled: true` with `global.fleet.cliBundle.pvc`. A post-install Job imports it |
| **Capacity** | Fixed `global.fleet.replicas` for peak (sealed networks), or KEDA with `global.fleet.keda.enabled` after `image_sync.sh --with-keda` |
| **Claude Code on Bedrock** | `global.harness.defaultProviders.claudeCode: amazon_bedrock` and `global.harness.defaultModels.claudeCode: <Bedrock model id>`, or the first turn fails with `503 no anthropic credential configured for harness agents` |
| **Fleet volumes on on-demand storage** | `global.fleet.storage.createClass: true` with `global.fleet.storage.provisioner` set to your CSI driver (`ebs.csi.eks.amazonaws.com` on EKS Auto Mode) |

***

## AI skill generation

Upload an OpenAPI spec (`Connectors` → `Create connector`) and the `openapi-spec-generator` service turns it into a full skill in the background: grouping, naming, descriptions, summaries and validation. You can delete generated skills or upload a new spec version over one. The air-gap license carries the entitlement (`Custom connectors`), and the image is in the standard mirrored set.

The generator uses **exactly one LLM provider**, chosen at install: Bedrock if configured, else OpenAI if a key was given. It refuses runs without that provider's credentials. **With neither, it is not installed** and skill creation is **spec-only**, with no enrichment (`VITE_CONNECTOR_AI_ENRICHMENT` is set for you).

It writes specs to an object store bucket, set on **both** `catalog` and `openapi-spec-generator`, with **versioning enabled** (it lists object versions). Without the bucket, upload fails.

```yaml theme={"dark"}
catalog:
  env:
    OAS_GEN_BUCKET_NAME: <bucket>
    AWS_DEFAULT_REGION: <region>
openapi-spec-generator:
  env:
    OAS_GEN_BUCKET_NAME: <bucket>
    AWS_DEFAULT_REGION: <region>
```

> [!WARNING]
>
> On Bedrock, the generator follows `global.llm.bedrock.region`, falling back to **us-west-2**. Its default models are US cross-region profiles, so that region must be in the US, and IAM and PrivateLink must cover Bedrock there.

***

## Custom actions and code execution

Custom actions and code run in short-lived jobs from the `actions` service. To let them reach internal systems, give them the cluster's resolver:

```yaml theme={"dark"}
actions:
  env:
    XPANDER_SANDBOX_DNS: <ClusterIP of the CoreDNS Service>   # 172.20.0.10 on EKS; kubectl -n kube-system get svc kube-dns
```

The job image is xpander's sandbox image from your mirror. `api.env.SANDBOX_IMAGE` overrides it for a development stream.

***

## The license

New installs use **two sealed files** (`install.json` + `activation.json`, `xpander-license/v2`): see **[Two-File Licensing](https://docs.xpander.ai/self-hosted/licensing.md)** for caps, expiry and renewal. The installer validates both and creates the `xpander-license` Secret:

```bash theme={"dark"}
kubectl -n xpander create secret generic xpander-license \
  --from-file=install.json=./install.json \
  --from-file=activation.json=./activation.json \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -
```

The preflight hook verifies the **Ed25519 signature** against `global.license.publicKey`, and deployment-manager refuses to start on a bad one.

* **Your own `--values` file** displaces the installer's license block: also pass `--set global.license.installKey=install.json --set global.license.activationKey=activation.json --set global.license.publicKey=<license public key>`.
* **Renewing** replaces only `activation.json` and redeploys. See [Renewing: re-activate](https://docs.xpander.ai/self-hosted/licensing.md#renewing-re-activate).
* **Expired activation**: the installer warns and preflight refuses a fresh install. A running install degrades in stages. `GET /v1/license/status` returns `valid`, `expiring`, `grace` or `expired`. Alert on anything past `expiring`.
* **Single-tenant**: one license, one organization.
* **Legacy single file** (`xpander-license/v1`): `--license-file ./license.json` creates a Secret with one `license.json` key, wired through `global.license.key`. Renew by replacing that file and redeploying.

***

## Domains, ingress & DNS

One `--domain` (for example `xpander.acme.internal`) names every public host:

| Host | Serves |
| - | - |
| `chat.<domain>` | The web UI: the address your users open |
| `api.<domain>` | The REST API |
| `agent-controller.<domain>` | API calls and the UI's chat backend |
| `supabase.<domain>` | Auth, storage, and edge functions (the `supabase-kong` gateway) |
| `webhook.<domain>` | Webhook triggers |
| `client-auth.<domain>` | OAuth flows for skills, SSO login |
| `actions.<domain>` | The actions service |

A reference customer renames most of them: `app.` (UI), `auth.` (Supabase), `api.`, `chat.` (agent-controller), `hooks.` (webhook), `links.` (short links, the `functions` Service), `actions.`, `client-auth.`, and `dm.internal.` for deployment-manager on a second, internal ingress class (`deployment-manager.ingress.className`), reached over a backchannel.

Set each with `<name>.ingress.host` (the UI also with `--ui-host`), and move the matching knobs in the same change:

| Host you rename | Knobs that must follow it |
| - | - |
| UI (`chat.` / `app.`) | `ui.ingress.host` or `--ui-host`, `supabase.auth.siteUrl`, `supabase.edgeRuntime.functionsEnv.APP_URL`, `client-auth.env.APP_URL`, `ui.env.VITE_PLATFORM_URL` |
| Supabase gateway (`supabase.` / `auth.`) | `supabase.ingress.host`, `supabase.auth.apiExternalUrl`, `supabase.storage.publicUrl`, `supabase.edgeRuntime.publicUrl`, `global.assetStore.publicUrl`, `ui.env.VITE_SUPABASE_URL` |
| API (`api.`) | `api.ingress.host`, `api.env.API_URL`, `ui.env.VITE_AGENTS_API_URL` |
| agent-controller (`agent-controller.` / `chat.`) | `agent-controller.ingress.host`, `ui.env.VITE_CHAT_BACKEND_URL` (`https://<host>/chat-backend/{agent_id}`) |
| Webhook (`webhook.` / `hooks.`) | `webhook.ingress.host`, `supabase.edgeRuntime.functionsEnv.INBOUND_GW_URL`, `ui.env.VITE_WEBHOOK_URL` |
| Short links (`links.`, required when agents share links) | `functions.ingress.host`, `global.shortLinks.publicUrl` (its `/s/*` path must land on the `functions` Service) |
| actions, client-auth | `actions.ingress.host`, `ui.env.VITE_ACTIONS_SERVICE_URL`, `client-auth.ingress.host`, `ui.env.VITE_CLIENT_AUTH_URL` |
| deployment-manager (`dm.internal.`, required when a remote location connects) | `deployment-manager.ingress.host` and `className` (an internal class). `supabase.edgeRuntime.functionsEnv.DEPLOYMENT_MANAGER_PUBLIC_URL` |
| Every install that sets hosts by hand | `supabase.edgeRuntime.functionsEnv.AGENT_CONTAINER_INGRESS_DOMAIN` (agent workspaces), `ui.env.VITE_LICENSE_MODE: "true"`, `ui.env.VITE_EMAIL_DOMAIN` when agents have mailboxes, and `ui.extraEnv` with `VITE_SUPABASE_ANON_KEY` read from the `xpander-generated-keys` Secret |

Setting the domain **publishes** these Ingresses. `<name>.ingress.enabled: false` keeps one private, and `global.ingress.enabled: false` suppresses all. The domain also sets the UI's runtime origins: without it, or `ui.env` overrides, browsers on a sealed network get unreachable origins.

* `--ingress-scheme https|http` (default `https`) feeds every derived URL. Use `http` only without certificates.
* No ingress controller is assumed: set `<name>.ingress.className` / `annotations`, or `global.ingress.className` / `global.ingress.annotations`.

**TLS at your controller:** the Ingress `tls:` block, per component (`<name>.ingress.tls`) or via `global.ingress.tls`. A wildcard `*.xpander.acme.internal` covers the derived set. `dm.internal.<domain>` needs its own wildcard:

```yaml theme={"dark"}
global:
  ingress:
    className: traefik
    tls:
      - hosts: ["*.xpander.acme.internal"]
        secretName: xpander-wildcard-tls
```

**TLS at a cloud load balancer** (ALB, Azure Application Gateway, Google): a managed certificate covers every host, and the Ingresses carry annotations and no `tls:`. A wildcard covers one level only: `*.xpander.acme.example` does not cover `*.staging.xpander.acme.example`. On EKS Auto Mode, put the group in an `IngressClassParams` (Auto Mode ignores `alb.ingress.kubernetes.io/group.name`). The load balancer appears after the first Ingress, so DNS aliases are a second pass.

### Making the hosts resolve

> [tab] Internal DNS zone
>
> A wildcard in the zone your users resolve:
>
> ```
> *.xpander.acme.internal.  CNAME  <your-ingress-controller's address>
> ```
>
> Or one record per host when wildcards are not allowed.

> [tab] VPN + internal load balancer
>
> Expose the ingress controller on an **internal** load balancer reachable over the VPN. In a sealed VPC, tag private subnets `kubernetes.io/role/internal-elb=1` and annotate the Service `aws-load-balancer-internal`: commands in [Troubleshooting](#troubleshooting).

> [tab] Local test: port-forward + /etc/hosts
>
> One port-forward to the ingress controller serves every host (it routes by `Host` header). **Forward to the real scheme port** (443 for https, 80 for http): xpander Chat's origins carry no port, so another port breaks every call after login. Binding 443/80 needs admin/root:
>
> ```bash theme={"dark"}
> sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 443:443
> # installed with --ingress-scheme http, forward port 80 instead:
> sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 80:80
> ```
>
> ```
> # /etc/hosts
> 127.0.0.1 chat.xpander.acme.internal api.xpander.acme.internal agent-controller.xpander.acme.internal supabase.xpander.acme.internal webhook.xpander.acme.internal client-auth.xpander.acme.internal actions.xpander.acme.internal
> ```
>
> Then open `https://chat.xpander.acme.internal` (or `http://…` for an `--ingress-scheme http` install), with no port.

> [!NOTE]
>
> **Sealed networks:** resolution must happen where your users are (internal zone or VPN DNS). The domain never needs to exist publicly.

***

## Signing in

The summary box prints the URLs **and the bootstrap admin password**. `helm get notes xpander -n xpander` repeats the commands.

1. **Bootstrap password** (default): open `https://chat.<domain>/login-password` with your `--admin-email`. Read the password again with:

   ```bash theme={"dark"}
   kubectl -n xpander get secret xpander-generated-keys \
     -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
   ```

2. **Email OTP without SMTP**: codes are not sent. Get one with `scripts/get_login_code.sh <email> <namespace>`. With `global.smtp.*` set, codes arrive by email. On Amazon SES in the sandbox, invitations reach only verified identities, and the SMTP IAM policy must list each verified recipient identity ARN: request production access first ([Email](https://docs.xpander.ai/run/integrations/email.md#transactional-email)). Agent mailbox replies use the same relay: `INBOUND_EMAIL_REPLY_DOMAIN` sets the From domain, and the SMTP identity must be allowed to send as the agents' mailbox subdomain.

3. **SSO (OIDC)**: configured in xpander Chat under **Platform > Organization > Access** and served by `client-auth`. See [SSO with your identity provider](https://docs.xpander.ai/self-hosted/sso.md). Google and GitHub buttons stay GoTrue providers (`supabase.auth.extraEnv`).

No domain yet? Port-forward to the UI:

```bash theme={"dark"}
kubectl -n xpander port-forward svc/ui 8080:80   # then http://localhost:8080
```

**Access tokens live one day** (`supabase.db.jwtExpiry`, default `86400`), renewed through the rotating refresh token. The one value feeds GoTrue, PostgREST and the database. Older tokens keep their expiry until refresh, and the migrations hook re-applies it on every upgrade, so installs that started on the one-hour default converge.

***

## Upgrading an Air-Gapped install

### Slack handle management compatibility

For releases with administrator-authorized [Slack agent handles](https://docs.xpander.ai/run/integrations/slack.md#authorize-handle-management), mirror the matching Supabase migrations and bots images before upgrading the chart. The migrations image must include `20260910231402` and `20260911192635`, and the chart's post-migration check requires `get_slack_handle_authorization`, `save_slack_handle_authorization` and `complete_slack_handle_authorization`. Never upgrade the chart alone before those images are in your mirror.

Add `usergroups:write` to your Slack app's **User Token Scopes**, then complete administrator consent in **Platform > Agents > Channels**, on the **Slack** row. The token is encrypted in the local Supabase Vault. Slack needs explicitly allowed connectivity, and this upgrade does not turn it on.

### Release v2026.10.22 (chart `xpander-airgap` 0.17.2)

Carries the changes of Hybrid chart 0.22.5 ([Hybrid: Upgrading to the current chart](https://docs.xpander.ai/self-hosted/hybrid.md#upgrading-to-the-current-chart)). Before you upgrade:

* **The LLM Gateway is always on** and `llm-gw.enabled` is ignored. If you had it off, mirror `llm-gw`, give every provider your agents use a key, and open its network paths ([LLM Gateway on a self-hosted install](https://docs.xpander.ai/run/llm-gateway/self-hosted.md)).
* **In-cluster Postgres moves to the pgvector build** of PostgreSQL 16.15. Mirror `pgvector/pgvector`, and snapshot the Postgres volume or set `postgres.upgradeGuard.backup.pvcName`. Postgres restarts once. A pinned `postgres.image` must stay on a pgvector build of the same major version.
* **Per-agent MCP URLs are gone.** Use `https://api.<your domain>/mcp/`. There is no per-person library endpoint on air-gap, and Omni over MCP needs `global.mcp.publicUrl`.
* **`BRAIN_INGEST_API_KEY`** is added to the generated secret automatically.

Mirror and upgrade (replace `<namespace>`, `<your-registry>` and `<xpander-token>`, Helm 3.14+):

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/image_sync.sh -o image_sync.sh
curl -fsSL https://github.com/xpander-ai/releases/releases/download/v2026.10.22/release-manifest.json -o release-manifest.json

bash image_sync.sh --source-user xpanderaihub --source-token <xpander-token> \
  --dest <your-registry> --release-manifest release-manifest.json --manifest-out pins.yaml

helm registry login registry-1.docker.io -u xpanderaihub -p <xpander-token>
helm upgrade xpander oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version 0.17.2 \
  -n <namespace> --reset-then-reuse-values -f pins.yaml --wait --timeout 30m
```

Then run the [verification guide](https://pages.xpander.ai/airgap-verification-guide).

### Release v2026.10.15 (chart `xpander-airgap` 0.16.2)

Carries the changes of Hybrid chart 0.22.0 ([Hybrid: Upgrading to the current chart](https://docs.xpander.ai/self-hosted/hybrid.md#upgrading-to-the-current-chart)):

* **Mirror with the current `image_sync.sh`** (adds `llm-gw` and `tool-repo-seed`, drops `--icon-pack-tag`).
* **The LLM Gateway is on by default.** Its generated keys are kept on every upgrade. Never rotate the content key. Every provider your agents use needs a key, or calls are refused naming the missing key. Check `kubectl -n <namespace> get pods -l app=llm-gw`: two `llm-gw` pods (one on a single-node `install.sh` cluster), `1/1` `Running`. See [LLM Gateway on a self-hosted install](https://docs.xpander.ai/run/llm-gateway/self-hosted.md).
* **Key names and endpoints:** env-var key names matching the install's own credentials read as no key, and metadata-address endpoints are refused ([details](https://docs.xpander.ai/self-hosted/hybrid.md#key-names-read-from-an-environment-variable)).
* **Helm 3.14+** for `--reset-then-reuse-values`. Migrations run automatically.

### Upgrade procedure

Check the [latest release](https://github.com/xpander-ai/releases/releases/tag/latest). Two steps, **in this order**:

**1. Mirror the new bundle FIRST**

Pinned digests mean upgrading before mirroring leaves pods in `ImagePullBackOff`. Re-run `image_sync.sh` (`curl -fsSLO https://charts.xpander.ai/image_sync.sh`) for the target release as in [Sealed / manual mirroring](#sealed--manual-mirroring), and carry the new chart `.tgz` if needed.

**2. Run the upgrade**

```bash theme={"dark"}
bash install.sh upgrade --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --release-manifest ./release-manifest.json
```

It reuses the installed values (`helm get values`), loads the release pins over them, and stops if the manifest is invalid. It runs `--atomic`. Migrations stay forward-only. Pipelines should use the [declarative re-install](#declarative-re-install-for-ci) instead.

### Declarative re-install for CI

A pipeline should inherit nothing from the release it upgrades. Run `bash install.sh --edition airgap ...` on every deploy with all values files, the release manifest and mirror pins, the license flags and `--org`, as on day one. Against a deployed release the installer adds `--atomic` itself. (`install.sh upgrade` once let a stored digest ride along and ran a stale image on staging for four days.)

* **`--helm-timeout 20m`** (default `30m`). The `--atomic` rollback waits the same time again, so budget twice the value plus hooks within your job's credential lifetime. A one-hour OIDC session cannot host a 30-minute timeout.

* **Heal first.** The installer refuses a release in `pending-install`, `pending-upgrade` or `failed`. Check `helm history xpander -n xpander`, then `helm rollback xpander <last deployed revision> -n xpander`, or `helm uninstall xpander -n xpander` when none deployed (`xpander-generated-keys` survives). Delete Failed pods and failed hook Jobs first.

* **Pin the environment id.** After the first install, pass `--env <existing default environment id>`: the default location's runtime environment cannot move to a new id. Read it first:

  ```bash theme={"dark"}
  kubectl -n xpander exec supabase-db-0 -c postgres -- psql -U supabase_admin -d postgres -tAc \
    "SELECT id FROM public.organization_environments WHERE is_default LIMIT 1"
  ```

* **Wait for Helm's verdict.** The stage watcher exits on the first failed hook pod, even an evicted one whose Job then succeeds. If `install.sh` exits non-zero while `helm upgrade --install xpander` still runs, do not kill Helm (it strands the release in `pending-upgrade`). Wait and read the release status.

* **`global.upgradeStrategy: replace`** on clusters that provision nodes on demand, so a surge upgrade does not trigger provisioning waves that outlive the timeout.

**The fleet and `--atomic`.** A **running harness turn holds the roll**: on `SIGTERM` an executor drains for up to `global.fleet.executor.drainSeconds` (3600 s). The installer's timeout covers it. On a hand-driven `helm upgrade`, set `--timeout` to at least the drain window, or upgrade when quiet. A turn still running at the deadline stops with "The executor is restarting; send the message again to continue where it left off." and resumes when the user sends again. An **idle executor** (zero replicas, `xpander.ai/fleet-idle-reason`) is skipped and never stalls an upgrade.

What survives what:

* **Generated keys survive every upgrade and rollback** (lookup-preserve, and the Secret is Job-created, never Helm-rendered).
* **`helm rollback xpander <N>` restores any prior revision**, **except the database schema**: migrations are forward-only, so a schema rollback means restore-from-backup.
* **The one-day token lifetime** (`supabase.db.jwtExpiry`) is re-applied by the migrations hook on every upgrade.

> [!WARNING]
>
> **Avoid `--reuse-values` on hand-driven air-gap upgrades.** It replays the old computed values, keeping changed defaults (from v2026.10.15 the LLM Gateway would stay off) and resurrecting removed settings, such as a stale edge-runtime image pin that crash-loops and fails every `--atomic` upgrade. Re-supply your values files, or clear it once with `--set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null`.

***

## Automation & CI

* **Every prompt has a flag.** `--non-interactive` reads nothing from the terminal, exits 2 with the usage and a rerun line when a value is missing, and implies `--yes` and `--on-probe-failure abort`.
* **Stable progress lines** such as `[3/8] license: ok`. `--quiet` prints phases and the outcome only. Color only on a terminal.
* **`status`** is a read-only health report (edition, chart, release, hooks, readiness, URLs). `--json` emits one object.

**Exit codes:**

| Code | Meaning |
| - | - |
| `0` | Success |
| `1` | Install or upgrade failure (`status`: degraded) |
| `2` | Usage error, or a required flag missing with no terminal |
| `3` | `status`: no such release installed |
| `4` | Preflight failure (missing helm/kubectl, unreachable cluster, version too old) |
| `5` | External data-tier probe failed and `--on-probe-failure` was `abort` |

**Air-gap, end to end, no terminal** (release already mirrored):

```bash theme={"dark"}
bash install.sh --non-interactive --quiet \
  --edition airgap --release-manifest ./release-manifest.json \
  --org <ORG_ID> --install-license-file ./install.json --activation-license-file ./activation.json \
  --registry registry.example:5000/xpander \
  --org-name "Example Inc" --admin-email ops@example.com \
  --domain xpander.example.internal --storage-class gp3 \
  --on-probe-failure abort
```

If the mirror may be empty, add `--mirror now --xpander-registry-token <token>`. Without a `--mirror` answer, an empty mirror exits 2.

**Poll until healthy:**

```bash theme={"dark"}
until bash install.sh status --quiet; do sleep 30; done
```

### Reference values file

A complete production values file. Hosts, buckets and endpoints are placeholders from your infrastructure outputs. Pass passwords with `--set`, and the mirror manifest last for image pins.

```yaml theme={"dark"}
global:
  upgradeStrategy: replace                  # in-place recreation; surge upgrades cascade into node waves on autoscaled clusters
  imageRegistry: <registry>/<mirror path>   # every image is pulled from the mirror
  license:
    secretName: xpander-license
  smtp:                                     # one block feeds sign-in mail, invitations and xpander's notifications
    host: email-smtp.<region>.amazonaws.com
    port: 587
    starttls: true
    user: <smtp user>
    passwordSecret: { name: xpander-smtp, key: password }
    adminEmail: admin@<domain>
    senderName: xpander
  assetStore: { publicUrl: https://auth.<domain> }
  shortLinks: { publicUrl: https://links.<domain> }
  # mcp.publicUrl unset: Platform > Agents > Channels shows the MCP row as "Not offered on this install" until you set it
  llm:
    defaults: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-5, apiBase: "", credentialsKey: "" }
    bedrock: { region: us-west-2 }          # may differ from the cluster's region
    gateway: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-4-6 }
  harness:
    defaultProviders: { claudeCode: amazon_bedrock }
    defaultModels: { claudeCode: global.anthropic.claude-sonnet-5 }
  fleet:
    replicas: 1
    storage: { createClass: true, provisioner: ebs.csi.eks.amazonaws.com }
  network:
    internalFetchAllowlist: [".internal.<domain>"]   # intranet hosts agents may fetch for knowledge ingest
  redis: { endpoint: <clustercfg endpoint>, clusterMode: true, auth: { username: <acl user> } }
redis: { mode: external }
postgres: { mode: external, external: { host: <rds host>, port: 5432 } }
supabase:
  ingress: { enabled: true, className: alb, host: auth.<domain> }
  auth: { apiExternalUrl: https://auth.<domain>, siteUrl: https://app.<domain> }
  storage:
    persistence: { size: 100Gi }
    publicUrl: https://auth.<domain>
  edgeRuntime:
    publicUrl: https://auth.<domain>
    functionsEnv:
      APP_URL: https://app.<domain>
      INBOUND_GW_URL: https://hooks.<domain>
      AGENT_CONTAINER_INGRESS_DOMAIN: <domain>
      DEPLOYMENT_MANAGER_PUBLIC_URL: https://dm.internal.<domain>
ui:
  ingress: { host: app.<domain>, className: alb }
  authProviders: [sso]
  env:
    VITE_WEBHOOK_URL: https://hooks.<domain>     # the app's baked default is the cloud host
    VITE_EMAIL_DOMAIN: agents.<domain>
    VITE_PLATFORM_URL: https://app.<domain>
    VITE_SUPABASE_URL: https://auth.<domain>
    VITE_AGENTS_API_URL: https://api.<domain>
    VITE_CHAT_BACKEND_URL: https://chat.<domain>/chat-backend/{agent_id}
    VITE_ACTIONS_SERVICE_URL: https://actions.<domain>
    VITE_CLIENT_AUTH_URL: https://client-auth.<domain>
    VITE_LICENSE_MODE: "true"
    VITE_OMNI_MCP_URL: ""
  extraEnv:
    - name: VITE_SUPABASE_ANON_KEY
      valueFrom: { secretKeyRef: { name: xpander-generated-keys, key: SUPABASE_ANON_KEY } }
client-auth: { ingress: { host: client-auth.<domain> }, env: { APP_URL: https://app.<domain>, CLIENT_AUTH_EXTERNAL_URL: https://client-auth.<domain> } }   # the second turns on MCP OAuth
api:
  ingress: { host: api.<domain> }
  env: { API_URL: https://api.<domain>, INBOUND_EMAIL_REPLY_DOMAIN: agents.<domain>, SMTP_HOST: <smtp host>, SMTP_PORT: "587", SMTP_STARTTLS: "true", SMTP_USERNAME: <smtp user> }
  extraEnv:
    - { name: SMTP_PASSWORD, valueFrom: { secretKeyRef: { name: xpander-smtp, key: password } } }
    - { name: INBOUND_EMAIL_TOKEN, valueFrom: { secretKeyRef: { name: xpander-inbound-mail, key: token } } }
    - { name: MAILGUN_SIGNING_KEY, valueFrom: { secretKeyRef: { name: xpander-inbound-mail, key: signing-key } } }
agent-controller: { ingress: { host: chat.<domain> } }   # API clients on this host add /<organization id> to the base URL
webhook: { ingress: { host: hooks.<domain> } }
functions: { ingress: { host: links.<domain> } }
deployment-manager: { ingress: { host: dm.internal.<domain>, className: alb-internal } }
actions:
  ingress: { host: actions.<domain> }
  env: { XPANDER_SANDBOX_DNS: <CoreDNS ClusterIP> }
catalog: { env: { OAS_GEN_BUCKET_NAME: <versioned bucket>, AWS_DEFAULT_REGION: <region> } }
openapi-spec-generator: { env: { OAS_GEN_BUCKET_NAME: <versioned bucket>, AWS_DEFAULT_REGION: <region> } }
```

Never repeat the installer's own keys (`orgSeed`, generated passwords, image pins): extra values files merge last and override them.

### CI/CD & GitOps

> [accordion] Jenkins (or any CI): watch the releases feed, mirror, upgrade
>
> Fetch the release manifest once, mirror it with CI-stored credentials, re-run the install with that file, your values and the license flags ([declarative re-install](#declarative-re-install-for-ci)), then gate on `status`. The agent needs cluster access, Helm 3.13+, skopeo and Python 3.
>
> ```groovy theme={"dark"}
> pipeline {
>   agent { label 'xpander-admin' }        // has kubectl access to the cluster
>   triggers { cron('H 6 * * 1') }         // or trigger from releases.atom
>   environment {
>     XPANDER_PULL = credentials('xpander-pull-token')  // username: xpanderaihub, password: the read token
>     XPANDER_ORG  = credentials('xpander-org-id')      // the organization id from the license handoff
>   }
>   stages {
>     stage('Mirror the release') {
>       steps {
>         sh '''
>           curl -fsSLO https://charts.xpander.ai/image_sync.sh
>           curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json -o release-manifest.json
>           bash image_sync.sh --source docker.io/xpanderaihub \
>             --dest registry.corp.example:5000/xpander \
>             --release-manifest ./release-manifest.json \
>             --source-user "$XPANDER_PULL_USR" --source-token "$XPANDER_PULL_PSW"
>         '''
>       }
>     }
>     stage('Upgrade and verify') {
>       steps {
>         sh '''
>           curl -fsSL https://charts.xpander.ai/install.sh -o install.sh
>           bash install.sh --non-interactive --quiet --edition airgap \
>             --org "$XPANDER_ORG" --install-license-file install.json --activation-license-file activation.json \
>             --registry registry.corp.example:5000/xpander --mirror skip \
>             --release-manifest ./release-manifest.json --values values.yaml \
>             --helm-timeout 20m
>           bash install.sh status --quiet
>         '''
>       }
>     }
>   }
> }
> ```
>
> The install step fails on `1` (upgrade failed, already rolled back by `--atomic`), `2` (usage) and `4` (preflight). `status` fails on `1` (degraded, with reasons) and `3` (not installed). `status --json` emits one object if you prefer to parse.

> [accordion] ArgoCD: the chart as Helm OCI from your registry
>
> [Mirror the chart with the images](#sealed--manual-mirroring) and keep configuration and generated release values in Git. A newer chart alone does not fetch the release manifest. Generate values with your install arguments plus `--release-manifest ./release-manifest.json --values-out xpander/values.yaml`. It writes `xpander/values.yaml` and `xpander/values.yaml.release.json` (Helm values, not the manifest). Keep the manifest for your record and pass both files, release values last:
>
> ```bash theme={"dark"}
> helm upgrade --install xpander ./xpander-airgap-<X.Y.Z>.tgz \
>   --namespace xpander --create-namespace \
>   -f xpander/values.yaml -f xpander/values.yaml.release.json
> ```
>
> For Argo CD:
>
> ```yaml theme={"dark"}
> apiVersion: argoproj.io/v1alpha1
> kind: Application
> metadata:
>   name: xpander
>   namespace: argocd
> spec:
>   project: default
>   destination:
>     server: https://kubernetes.default.svc
>     namespace: xpander
>   sources:
>     - repoURL: registry.corp.example:5000/xpander/charts   # OCI repo credential with "Enable OCI"
>       chart: xpander-airgap
>       targetRevision: <the chart version you mirrored>     # keep it in step with the release values
>       helm:
>         valueFiles:
>           - $values/xpander/values.yaml           # generated once: install.sh --values-out
>           - $values/xpander/values.yaml.release.json  # generated release image pins, applied last
>     - repoURL: https://git.corp.example/platform/xpander-config.git
>       targetRevision: main
>       ref: values
>   syncPolicy:
>     syncOptions:
>       - CreateNamespace=true
> ```
>
> * **Values**: regenerate per release, review with the chart version, and add no image overrides after the release values. Direct Helm does not run the installer's checks. Strip secrets (LLM keys, DSNs) into your secret manager before committing.
> * **Bootstrap secrets**: create `xpander-license` (and any pull secret) once out-of-band ([The License](#the-license)), or with SealedSecrets / External Secrets Operator.
> * **Hooks**: Argo CD maps the chart's hook pipeline onto sync phases and waves. The chart is not certified under Argo CD: **verify the hook ordering on your Argo version**. `install.sh` is the certified path.

***

## Skill credentials on Air-Gapped

Built-in skills that call a third-party service need a credential, or they fail with a 401:

* **RapidAPI skills** (for example Google News, Company Insights and Social Media): set `global.network.toolKeys.RAPID_API_KEY`, or calls return `401 Invalid API key`.
* **Skills with their own sign-in** (for example BigQuery and Figma): create the connection with valid credentials.
* **AWS skills** (Redshift, S3, Athena, EKS): the IAM role and its permissions on [AWS Operator Setup](https://docs.xpander.ai/self-hosted/aws-operator.md#air-gapped-installs).
* **Public skills** need internet egress, and do not work with `global.network.internetEgress: false`.

***

## Backups

The chart creates no backup job. Back up these volumes yourself, for example with AWS Backup or EBS snapshots:

| Volume | What it holds | Size on a reference install |
| - | - | - |
| `data-supabase-db-0` | Supabase PostgreSQL: organizations, users, agents, schema | 50Gi |
| `data-mongodb-0` | MongoDB data | 20Gi |
| `supabase-storage-data` | Supabase storage files | 50Gi |
| `fleet-volume-fleet-executor-<n>` | Executor fleet workspaces | 500Gi |
| `ag-<agent-id>-data` | The workspace of each agent that uses one (harness agents keep their files on the executor fleet volume) | 5Gi |
| `data-postgres-0` | Platform PostgreSQL, when `postgres.mode: internal` (the default) | 50Gi |
| `data-redis-0` | Redis Stack, when `redis.mode: internal` (the default) | 8Gi |

External PostgreSQL or Redis use the managed service's backups. Migrations are forward-only, so a schema rollback needs a restore.

***

## Monitoring

No metrics stack or `ServiceMonitor` ships. The `xpander-smoke` Job logs `smoke passed: every required component serves` when all components answer, and `bash install.sh status` gives a read-only verdict. Scrape the services with your own stack for metrics.

***

## Troubleshooting

Every failure prints **`FIX:`** (the repair) and **`RERUN:`** (your non-interactive one-liner, secrets as placeholders). The installer is idempotent: fix, paste the rerun line, and it resumes. The checklist stops most problems before anything changes. Scripts read the exit code ([Automation & CI](#automation--ci)), and `bash install.sh status` gives a read-only verdict.

### Mirroring and registries

> [accordion] Mirror push fails: name unknown, 'The repository ... does not exist' (ECR)
>
> ```
> writing blob: initiating layer upload to /v2/xpander/ai-gateway/blobs/uploads/ in
> 123456789012.dkr.ecr.eu-west-2.amazonaws.com: name unknown: The repository with
> name 'xpander/ai-gateway' does not exist in the registry with id '123456789012'
> ```
>
> ECR refuses pushes to repositories that do not exist. The current `image_sync.sh` creates them (with AWS CLI credentials). Refresh your copy:
>
> ```bash theme={"dark"}
> curl -fsSLO https://charts.xpander.ai/image_sync.sh
> ```
>
> On an older copy, pre-create the repository and re-run (copied images are skipped):
>
> ```bash theme={"dark"}
> aws ecr create-repository --repository-name xpander/ai-gateway --region <region>
> ```

> [accordion] Mirror push fails: authentication required at the destination
>
> `authentication required` even though `docker login` works: `skopeo` cannot read Docker Desktop's credential store. The current `image_sync.sh` logs in to ECR itself (`aws ecr get-login-password | skopeo login --username AWS --password-stdin`). For other registries, pass credentials:
>
> ```bash theme={"dark"}
> bash image_sync.sh --source docker.io/xpanderaihub --dest <registry>/<path> \
>   --dest-user <user> --dest-token <token>
> ```

> [accordion] 401 from Docker Hub when resolving or mirroring
>
> `401 from auth.docker.io (incorrect username or password)`: the credential is the **organization username** `xpanderaihub` plus the token from your email, not a personal username. Re-run with `--xpander-registry-user xpanderaihub --xpander-registry-token <token>`. Still refused? Ask xpander to re-issue the token with repository-read and image-pull access.

> [accordion] Pods can't pull: manifest unknown at your pinned tag
>
> Pod events show `manifest unknown`, or the checklist reports:
>
> ```
> ✗ deployment-manager:0.0.417 not found in registry.example:5000/xpander
> ```
>
> The mirror lacks that repository at the pinned tag. Tags differ **per image**, so one stale repo is enough. Re-run the mirroring ([in-wizard](#when-the-mirror-is-empty-the-wizard-offers-to-fill-it) or `image_sync.sh`), then check the exact coordinate:
>
> ```bash theme={"dark"}
> crane digest registry.example:5000/xpander/deployment-manager:0.0.417
> ```

### Cluster and network

> [accordion] Nodes time out pulling from docker.io (dial tcp ... i/o timeout)
>
> ```
> failed to pull image "docker.io/...": dial tcp 44.208.254.194:443: i/o timeout
> ```
>
> An image reference still points at `docker.io` instead of your mirror. Install with `--registry <your-mirror>/<path>`, and make sure the **nodes** reach the mirror (on an ECR-only sealed VPC: `ecr.api` + `ecr.dkr` endpoints, the S3 gateway endpoint, ECR read on the node role). Find the reference:
>
> ```bash theme={"dark"}
> kubectl -n xpander get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | grep docker.io
> ```

> [accordion] Preflight: no node with a supported architecture (Graviton/ARM)
>
> ```
> ✗ no node with a supported architecture: cluster has [arm64], the images need [amd64]
> ```
>
> (On a mixed cluster: `exec format error`.) xpander images are amd64-only, so ARM/Graviton (`t4g`, `m7g`, `c7g`, ...) cannot run them. Add an amd64 node group. Only for genuinely arm64 images: `--set 'preflight.allowedArchitectures[0]=arm64'`.

> [accordion] LoadBalancer Service stuck \<pending>: could not find any suitable subnets
>
> The ingress Service's events show:
>
> ```
> could not find any suitable subnets for creating the ELB
> ```
>
> A sealed VPC has no public subnets and the private ones lack ELB role tags. Tag them and annotate the Service:
>
> ```bash theme={"dark"}
> aws ec2 create-tags --resources <subnet-ids> \
>   --tags Key=kubernetes.io/role/internal-elb,Value=1
> ```
>
> ```yaml theme={"dark"}
> metadata:
>   annotations:
>     service.beta.kubernetes.io/aws-load-balancer-internal: "true"
> ```
>
> Or skip the load balancer: use the VPN, or `kubectl -n xpander port-forward svc/ui 8080:80`.

> [accordion] External data-tier probe fails: namespaces 'xpander' not found
>
> ```
> Error from server (NotFound): namespaces "xpander" not found
> ```
>
> The probe ran before the namespace existed (fixed in current installers). Refresh (`curl -fsSL https://charts.xpander.ai/install.sh`) or create the namespace and re-run:
>
> ```bash theme={"dark"}
> kubectl create namespace xpander
> ```

> [accordion] Pods stuck in Pending
>
> Unbound PVCs (`kubectl -n xpander describe pvc`) mean no usable RWO StorageClass or too little capacity for the [footprint](#prerequisites). Check `kubectl get storageclass`, pass `--storage-class <name>`, and look for `FailedScheduling` in `kubectl -n xpander get events --sort-by=.lastTimestamp`.

> [accordion] UI loads blank: Missing or invalid environment variables
>
> The browser console shows:
>
> ```
> Missing or invalid environment variables
> ```
>
> Charts before **0.6.0** did not derive the UI's origins from the domain. Upgrade with `bash install.sh upgrade`, or set `ui.env` (`VITE_SUPABASE_URL`, `VITE_CHAT_BACKEND_URL`, ...) yourself.

> [accordion] Only the UI host resolves. Other subdomains 404 or NXDOMAIN
>
> `chat.<domain>` opens but sign-in or API calls fail: only one of the [published hosts](#domains-ingress--dns) has a record. Add a wildcard `*.<domain>`, or records for `chat`, `api`, `agent-controller`, `supabase`, `webhook`, `client-auth` and `actions` (or your `<name>.ingress.host` values). Check what is published:
>
> ```bash theme={"dark"}
> kubectl -n xpander get ingress
> ```

### Install and upgrade

> [accordion] Release stuck in pending-install / pending-upgrade / failed
>
> The installer refuses a stuck release, and `install.sh upgrade` needs `deployed`. Recover first:
>
> ```bash theme={"dark"}
> helm history xpander -n xpander
> helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
> helm uninstall xpander -n xpander                        # first install that never converged
> ```
>
> Then paste your `RERUN:` line. `pending-install` (an interrupted first install) is fixed with `helm uninstall`. `xpander-generated-keys` survives it, so the reinstall keeps the same keys.

> [accordion] 'images are unpinned' / chart refuses to render
>
> Load a complete [latest release](#releases), mirror its images, and retry with that manifest. For direct Helm, pass both generated values files, release values last. Never guess a shared image tag.

> [accordion] License expired or rejected
>
> The installer checks the two `xpander-license/v2` files (or the legacy v1 file) and the `--org` binding. Preflight repeats it with the Ed25519 signature, and deployment-manager repeats it at boot. `GET /v1/license/status` answers `valid`, `expiring`, `grace` or `expired`.
>
> **Expired:** the installer warns `License expired on <date> - preflight will refuse the install`. Replace only `activation.json` in the Secret with the renewed file and redeploy:
>
> ```bash theme={"dark"}
> kubectl -n xpander create secret generic xpander-license \
>   --from-file=install.json=./install.json \
>   --from-file=activation.json=./activation-new.json \
>   --dry-run=client -o yaml | kubectl -n xpander apply -f -
> ```
>
> Then run `install.sh upgrade` or your pipeline's re-install (the Secret checksum rolls the services). Keep the `--set global.license.installKey/activationKey/publicKey` flags if you pass your own `--values` ([The License](#the-license)). A legacy v1 install swaps its `license.json` the same way.

> [accordion] An agent answers 500 Failed to build agent details
>
> Usually the agent holds a connection to an API connector that never finished (status `faulty`, no spec). The `agents` pod log shows a pydantic error on `ConnectedConnector.open_api_spec_id`. Recover from the in-cluster Supabase database:
>
> ```sql theme={"dark"}
> -- the connections of the faulty connector
> update public.connector_organization set deleted_at = now()
>   where connector_id = '<connector id>' and deleted_at is null;
> -- take the connection id out of the agent's skill list
> update public.ai_agents
>   set attached_tools = (select coalesce(jsonb_agg(t), '[]'::jsonb)
>     from jsonb_array_elements(attached_tools::jsonb) t
>     where t->>'id' <> '<connection id>')::json
>   where id = '<agent id>';
> ```
>
> Reload the agent in every open tab before saving it again: a stale tab writes the dead connection back.

> [accordion] A pipeline stage failed (preflight / keygen / migrations / org-seed / smoke)
>
> The installer printed the Job's last 30 lines. Dig deeper:
>
> ```bash theme={"dark"}
> kubectl -n xpander logs job/<failed-job> --tail=100
> kubectl -n xpander describe job/<failed-job>
> kubectl -n xpander get pods,jobs
> kubectl -n xpander get events --sort-by=.lastTimestamp
> ```
>
> org-seed needs `--org-name` and `--admin-email` whenever Supabase is on. A hand-driven `helm upgrade` without them fails at render time.

> [accordion] Services fail auth after an upgrade with a custom values file
>
> A username with `password: ""` beside it **overrides the generated password with an empty string**. Remove the empty `password:` key: an absent key is preserved, `""` wins.

> [accordion] Creating a skill from an OpenAPI spec stops at "Generating the enriched spec", but the skill appears anyway
>
> On releases before the single-provider generator, a missing OpenAI key made the dialog show *"The AI enrichment pipeline reported a failure"* although the skill finished seconds later. Check the skills list before retrying. Upgrading fixes it.

> [accordion] Pods in ImagePullBackOff after an upgrade
>
> You upgraded before mirroring (typically [`manifest unknown`](#troubleshooting)). Mirror the target release and re-run the upgrade. `--atomic` already rolled back the failed attempt.

> [accordion] Edge-runtime crash-looping after upgrading with --reuse-values
>
> `--reuse-values` resurrected a stale edge-runtime image pin. Upgrade with your real values files (as `install.sh upgrade` does), or clear it once:
>
> ```bash theme={"dark"}
> --set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null
> ```
>
> Better: pin the mirrored edge-functions build (`--edge-functions-tag`).

> [accordion] Release already runs the other deployment's chart
>
> Hybrid and Air-Gapped cannot be upgraded into each other. Use another `--release` / `--namespace`, or `helm uninstall xpander -n xpander` first.

> [accordion] Piped run exits with 'these required flags are missing'
>
> Piped into `bash`, the run is non-interactive and needs `--org`, `--install-license-file` and `--activation-license-file` (or `--license-file` for v1). It exits with code 2 and a rerun line. For the wizard, download the script and run `bash install.sh --edition airgap`.

***

## Next steps

> [card] [Review the security architecture](https://docs.xpander.ai/resources/security-compliance/air-gapped-deployments.md)
>
> Zero egress, signed license, local sign-in.

> [card] [Review the install before you run it](https://docs.xpander.ai/self-hosted/hybrid.md#advanced-review-first--manual-install)
>
> `--manual`, `--values-out` and `--dry-run`.

The chart README (in the `.tgz`) covers bring-your-own ingress, private CA and proxy, SMTP, internet posture, builtin-skills seeding and Bedrock region pinning.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.