Sizing and requirements

How much cluster a self-hosted xpander needs, and what the nodes must support.

A self-hosted install has three footprints:

Footprint	What it is	Sized by	Section
base services	The web UI, control plane and data tier	The deployment (Hybrid or Air-Gapped)	Base footprint
agent workspace pod	A pod created on demand for an agent whose harness is xpander and that uses a workspace	Concurrently active workspaces	Agent workspaces
executor fleet	A shared pool of warm pods where every Claude Code, Codex and OpenCode turn runs	Concurrent turns	The executor fleet

The turns of every Claude Code, Codex and OpenCode agent land on the fleet, however the agent was created. Two kinds of agent do not. An agent whose harness is xpander runs on the execution tier (agent-worker), and on a workspace pod when it uses one. An SDK agent runs its loop in your own process.

Base footprint

Hybrid

The installer shows the same minimums, published at charts.xpander.ai/requirements.json.

	Hybrid minimum
Nodes	3 × 4 vCPU / 16 GiB (e.g. m5.xlarge)
Storage	~550 GiB: a 500 GiB agent volume plus ~30 GiB for the databases. For a small install, lower the agent volume with fleet.storage.size
Kubernetes	≥ 1.28
Architecture	amd64
Network	Egress to deployment-manager.xpander.ai (or PrivateLink) and to your LLM providers
Extras	Ingress controller + wildcard DNS optional

Size for tasks running concurrently at peak, whatever triggers them. In xpander cloud production only ~5% of executions are interactive chat. The rest come from schedules, webhooks, the API and other agents.

Concurrent tasks	Agents	Cluster capacity	Example nodes	Persistent storage
3	10	11 vCPU / 22 GiB	3 × m5.xlarge	~110 GiB
10	25	21 vCPU / 37 GiB	3 × m5.2xlarge	~180 GiB
20	50	31 vCPU / 52 GiB	4 × m5.2xlarge	~310 GiB
50	100	67 vCPU / 108 GiB	9 × m5.2xlarge, enable autoscaling	~570 GiB

Capacity includes ~50% headroom over pod requests for bursts, system pods and node upgrades. Capacity and storage cover the base services and workspaces. Add the executor fleet on top: each executor requests 2 vCPU / 8 GiB and has a 500 GiB volume (see Capacity Planning).

Sizing calculator

Enter peak concurrent tasks, agents, and how many use a workspace.

How the estimate works

The calculator applies the same model we use to size xpander cloud:

1. Concurrent tasks is the primary driver. A task is any agent execution, from any source: chat, API calls, schedules, webhooks, MCP clients, Slack, or another agent. If you don’t know your peak concurrency yet, assume one concurrent task per agent - that covers most production organizations we measured. In practice most teams peak at just a few concurrent tasks (typically under 5), and even the busiest organization peaks around 30.
2. Execution replicas (agent-worker) = concurrent tasks ÷ 4 (each replica processes 4 tasks by default, configurable with MAX_CONCURRENT_EXECUTIONS). Each replica requests 2 vCPU / 2.5 GiB.
3. Agent Controllers: 1 replica up to ~8 concurrent tasks, 2 up to ~20, 3 beyond. Each requests 1 vCPU / 2 GiB.
4. Active workspaces = concurrent tasks × workspace share, capped at the agent count. Budget ~0.5 vCPU / 1 GiB per active workspace. That is the measured typical load, and each can burst to its 2 vCPU / 3 GiB limit.
5. Fixed base (~3 vCPU / 8 GiB): AI Gateway, MCP, Chat, Code Runner, AWS Operator, API, Redis, PostgreSQL, container registry and metrics-server at one replica each. Add the LLM Gateway’s two pods (0.5 vCPU / 1 GiB requested each).
6. Cluster capacity = requests × 1.5 for burst, DaemonSets, and safe node drains. Storage = 48 GiB of chart volumes + 5 GiB per agent (an upper bound for workspace volumes; only an agent whose harness is xpander and that uses a workspace keeps one) + image cache per node.
7. Executor fleet, where Claude Code, Codex and OpenCode turns run, is not in the estimate. Each executor requests 2 vCPU / 8 GiB and has a 500 GiB volume. Add it from Capacity Planning.

Per-service requirements

Per-replica values. Defaults are what the Helm chart applies out of the box. The recommended values raise the ones we found low against measured production usage.

Service	CPU request	Memory request	CPU limit	Memory limit	Notes
Agent Controller	1000m	2 Gi	3000m	4 Gi	Orchestrator; observed steady-state memory is ~1.5-2 GiB, and the limit carries headroom for peaks
Execution tier (agent-worker)	2000m	2.25 Gi → 2.5 Gi	4000m	3 Gi	Scale replicas for task concurrency, not size
API	100m → 250m	256 Mi → 1 Gi	- → 2000m	- → 1.5 Gi	Chart sets no limits; observed usage ~0.9-1.4 GiB
MCP	200m	750 Mi	400m	1 Gi
Chat	300m	1 Gi	600m	1.5 Gi
AI Gateway	200m	400 Mi	400m	600 Mi
Code Runner	400m	800 Mi	800m	1229 Mi	Runs privileged (Docker-in-Docker); each concurrent code execution adds up to 1 vCPU / 2 GiB
AWS Operator	200m	400 Mi	400m	600 Mi	Optional component
LLM Gateway (llm-gw)	500m	1 Gi	1000m	4 Gi	Always on. 2 pods (1 on a single-node cluster installed with install.sh), HPA 2 to 20 on CPU (llm-gw.autoscaling); scale out, never up. No database of its own

Override any of these in your Helm values (keys are camelCase):

resources:
  agentController:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "3000m"
      memory: "4Gi"

Horizontal Pod Autoscaling is available for Agent Controller, AI Gateway, the execution tier (agent-worker), and MCP via the chart’s autoscaling values. The LLM Gateway’s HPA is on by default. The chart installs metrics-server by default to support it.

Air-Gapped

	Air-Gapped minimum
Nodes	3 × 8 vCPU / 32 GiB (e.g. m5.2xlarge), amd64
Storage	~700 GiB on an RWO StorageClass, encrypted at rest: a 500 GiB agent volume plus the databases. For a small install, lower the agent volume with global.fleet.storage.size
Kubernetes	≥ 1.28, with a CSI driver and a default RWO StorageClass

The full prerequisite list (network, license files, registry, tooling) is on Air-Gapped: Prerequisites.

Sizing tiers

Air-Gapped runs everything in-cluster: the execution tier (agent-worker) plus the control plane, the web UI, the Supabase stack (its own PostgreSQL 15, auth, storage, edge functions) and MongoDB.

Tier	Nodes	Fits	Storage (PVC)
Initial deployment	3 × (8 vCPU / 32 GiB)	~10 users, 5–10 concurrent agent runs	~700 GiB
Production	5 × (16 vCPU / 64 GiB)	~50 users, 30–50 concurrent agent runs, headroom	~1 TiB
+ GPU (only if needed)	+1 GPU node (L4/A10 class, 8 vCPU / 32 GiB)	Local voice STT; in-cluster LLM inference is your own pool, sized separately by model	+ model weights

Storage breakdown at the production tier:

Volume	Size
Agent volume	500 GiB per executor
Supabase PostgreSQL	50 GiB
xpander’s PostgreSQL	50 GiB
Supabase Storage	50 GiB+
MongoDB	20 GiB
Redis	8 GiB
Registry	~50 GiB
Mirror caches	50–100 GiB

All are RWO, on one StorageClass. The chart’s preflight verifies it exists.

External data tier. You can bring managed stores instead (Data Tier). A reference customer install uses:

• RDS PostgreSQL 16 on db.t4g.small with 50 GiB gp3
• MemoryDB on db.t4g.small with one shard and one replica
• S3 as the Supabase Storage backend for blobs

With external stores, xpander’s PostgreSQL, Redis and storage PVCs above fall away.

Most services run one replica. deployment-manager, actions and the Supabase edge runtime ship at ×2 with soft anti-affinity. For production, also run agent-controller, agent-worker, api, agents and functions at ×2.

Scaling with usage

The chart ships fixed replica counts and no autoscalers. Scaling policy is yours:

What grows	Grows with	How to scale it
agent-worker: executes agent tasks	Concurrent agent runs. Each replica runs up to 4 tasks at once (MAX_CONCURRENT_EXECUTIONS); beyond replicas × 4, tasks queue	Raise agent-worker.replicas. It is stateless, so adding replicas is always safe. Don’t autoscale it on CPU: a replica waiting on a model looks idle when it is full. Scale on runs waiting to start (a queue in xpander’s Redis), for example with KEDA
Agent workspace pods (sb-*): the agents’ computers	Number of agents with an active workspace: one pod + PVC each, created on demand by agent-controller and reaped when idle	Not a Deployment you own, so not HPA territory: this is node capacity. Size headroom for your expected concurrent workspaces, or run a node autoscaler
Executor fleet (the fleet-executor StatefulSet), where agent turns run	Peak concurrent turns. One executor per node, 12 turn slots (8 GiB / 4 vCPU ceiling each; the chart’s global.fleet.turnsMax and global.fleet.turn values, which override the image’s own default of 6 turns and 16 GiB per turn), 500 GiB XFS volume with a per-agent quota; idle agents cost only disk	global.fleet.replicas (fixed capacity, the sealed-network posture), or KEDA with global.fleet.keda.enabled (mirror KEDA with image_sync.sh --with-keda). Prerequisites and capacity tables: the executor fleet
Code-execution Jobs: spawned per run by the actions service	Bursts of agent code execution	Node capacity, same as workspaces
Request path: api, agent-controller, agents, functions	Users and API traffic; functions also runs CPU-heavy local embedding during knowledge-base ingest	Raise .replicas; standard CPU-based HPA works here if you install metrics-server (not required otherwise)

Node capacity is the real ceiling, because workspaces and code-execution Jobs create pods on demand. Two postures:

• Static capacity: size the node group for peak and leave it fixed. Simplest, fully offline, recommended for sealed networks.
• Cluster Autoscaler: adds nodes when pods can’t schedule. It works in a no-internet VPC if your cloud’s scaling API is reachable (on AWS, the autoscaling and ec2 VPC endpoints).

Karpenter does not fit sealed networks: it fetches instance pricing from the internet.

Agent workspaces

Each agent whose harness is xpander and that uses a workspace gets its own pod, created on demand and reclaimed when idle. Size for concurrently active workspaces, not total agents.

Resource	Value
CPU	250m request, 2000m limit
Memory	256 Mi request, 3 Gi limit
Storage	5 GiB persistent volume per agent
Idle shutdown	5 minutes without activity (configurable via SANDBOX_IDLE_TIMEOUT on the Agent Controller)
Maximum lifetime	3 hours, then the pod is recycled

Idle workspaces use ~50 Mi. An active one uses ~0.5-1 vCPU and up to 1 GiB, bursting to its limits. The volume persists after the pod is reclaimed, so plan 5 GiB per workspace-using agent, not per pod.

The workspace image is ~1 GiB and is pulled on the first start on each node. Budget ~2 GiB of node disk for the image cache.

The fleet is always rendered by both charts (there is no enable flag). It starts warm with one executor (fleet.replicas: 1 on Hybrid, global.fleet.replicas: 1 on Air-Gapped). Harness agents (Claude Code, Codex, OpenCode) run only on the executors, never as a pod per agent. So the chart always keeps at least one: replicas: 0 (or a blank value) renders one executor. The one case that renders zero is a live cluster whose executor volume can never bind (see Idle guard). Agent turns then wait in the queue until it is fixed.

The executor fleet

The fleet is a fleet-executor StatefulSet. Each replica is one PodSecurity baseline pod over one XFS volume: no privileged container, no host namespaces, no user namespaces and no node preparation. The pod holds the executor container, which claims turns, plus one slot-<i> container per turn slot (fleet.turnsMax). Each turn runs in its own slot container, as the agent’s own uid. It has its own processes, /proc, /tmp and memory and CPU limit, behind a seccomp profile, no_new_privs and Landlock. An agent’s directory lives on exactly one executor’s volume, and its turns always run there.

	Executor fleet
What an agent is	A directory (its home, CLI state, working tree) on the executor’s XFS volume, under a 20 GiB quota
What runs a turn	A free slot container on the executor that holds the agent’s directory, as the agent’s own uid, limited by Landlock (unless fleet.landlock.enabled: false) and the seccomp profile
Cost of an idle agent	Disk only; no pod, no requests
Cost of a new agent	mkdir on the first turn, seconds
Scaling axis	Concurrent turns, never the number of agents
Model credentials	None in the pod. Each turn calls the in-cluster LLM Gateway directly, with the run’s own short-lived token

Per executor

Resource	Self-hosted chart default
Executor CPU request / limit	2 / 8 vCPU (fleet.resources)
Executor memory request / limit	8 / 28 GiB
Slot containers	One per slot (fleet.turnsMax, default 12); each requests 10m CPU / 64 Mi and is limited to the turn ceiling (fleet.turn, 4 vCPU / 8 GiB). Changing turnsMax rolls the executor pods
Volume	500 GiB, XFS, prjquota, gp3 encrypted, RWO
Per-agent disk quota (quotaPerAgent)	20 GiB (XFS project quota, hard; a ceiling, not a reservation)
/dev/shm (memory-backed, for Chromium)	4 GiB
Package cache (emptyDir, node disk)	20 GiB
Termination grace	7500 s (fleet.executor.terminationGraceSeconds), a ceiling: the executor first drains running turns for up to 3600 s (fleet.executor.drainSeconds)
Probes	readiness 10 s x 6, startup 5 s x 36 (3 min)
Placement	One executor per node (required anti-affinity); fleet.nodeSelector / fleet.tolerations

Size the executor’s memory against turnsMax. Each slot container is limited to the turn ceiling (8 GiB by default), and the node must back them. If the turns together use more memory than the node has, the kernel OOM-kills one. The chart default (12 slots, executor limit 28 GiB) is a conservative starting footprint. It assumes typical turns stay well below their ceiling. For a production fleet, either use memory-rich nodes sized toward turnsMax x 8 GiB or lower turnsMax.

# Hybrid (xpander chart); on Air-Gapped the same keys live under global.fleet
fleet:
  replicas: 1                       # 0 or blank also renders 1
  turnsMax: 12
  turn: { memory: "8Gi", cpu: "4", pids: 1024 }
  landlock:
    enabled: true                   # false on nodes without Landlock
  executor:
    drainSeconds: 3600              # a roll waits this long for running turns
    terminationGraceSeconds: 7500   # pod grace; keep drainSeconds at least 120 s below it
  resources:
    requests: { cpu: "2", memory: "8Gi" }
    limits:   { cpu: "8", memory: "28Gi" }
  storage:
    className: fleet-xfs
    createClass: true          # false on Air-Gapped: bring your own XFS + prjquota class
    provisioner: ebs.csi.aws.com
    size: 500Gi
    quotaPerAgent: 20Gi
  networkPolicy:
    exceptCidrs: ["169.254.0.0/16"]
    exceptCidrsV6: ["fe80::/10"]
  shm:   { size: 4Gi }
  cache: { size: 20Gi }

Per turn

Ceilings per turn (fleet.turn). An agent can lower its own, never raise it.

Runner	Memory	CPU	Processes	Notes
Agent turn (Claude Code / Codex / OpenCode)	8 GiB	4 vCPU	1024 pids	Sized for real node / python builds inside the turn
Disk per agent	20 GiB			XFS project quota, hard limit; a du-based sweeper enforces it softly where prjquota is missing

Concurrency caps

Cap	Where	Default
Turns one executor accepts	fleet.turnsMax (XPANDER_FLEET_TURNS_MAX)	12
Turns above capacity	Queue in Redis and start as slots free	Nothing is dropped, only delayed

There is no per-organization cap. fleet.orgMaxTurns is removed, and a license’s agent cap does not limit concurrent turns. A values file that still sets it upgrades cleanly, and the key renders nothing.

Reading the runtime logs

One turn shows in two logs, the executor and the Agent Controller:

kubectl -n xpander logs -f fleet-executor-0 -c executor
kubectl -n xpander logs -f deploy/agent-controller

Executor (fleet-executor-<n>, container executor), in order:

Event	Carries
harness.fleet.turn_start	The turn was claimed
harness.cli.selected	The CLI and where the choice came from, for example cli=claude-code source=task
execution.runtime_selected
harness.sandbox.open	The sandbox’s uid and cgroup
harness.skills.sync	synced= and exposed=, the skills placed in the sandbox
harness.mcp_remote.resolved	servers= and tools=, the MCP endpoints resolved for the turn and how many actions they expose
harness.run.start, harness.cli.spawn, harness.cli.init	The CLI process is up
harness.cli.tool_call, harness.mcp.call, harness.mcp.result, harness.cli.tool_result	One group per skill call the agent makes
harness.cli.exit	The CLI finished
harness.sandbox.close	cpu_usec, the turn’s CPU time
harness.run.end, harness.fleet.turn_end	The outcome

Agent Controller (deploy/agent-controller). The same turn shows as:

• direct turn conversation=<id> task=<id> status=... received_to_enqueued_ms=... enqueued_to_claimed_ms=...: the message became a task and was picked up by the fleet. enqueued_to_claimed_ms is where a cold executor shows. The turn runs on the model the conversation picked. The agent record’s model is only the default for new conversations.
• event=fleet.claimed executor=fleet-executor-0: which executor took it.
• event=compute_usage.emitted cli=claude-code cpu_seconds=...: the compute record written when the turn ended.

LLM Gateway (llm-gw, kubectl -n xpander logs -f -l app=llm-gw). Each model call the turn made is one event=llm_gw.call llm_route=gateway_direct status=... org=<id> agent=<id> task=<id> client=... provider=... request_id=<id> line. To see the model a call used, paste its request_id into Request or trace ID and press Enter in Platform > AI governance > Logs, on your install’s own address.

Node prerequisites

All required: a cluster that fails one runs no agent turns. install.sh and the Air-Gapped preflight check each and name the fix. There is no Kubernetes 1.33, kernel 6.3, containerd 2.0 or user.max_user_namespaces minimum, and no node configuration.

Requirement	Why	How to check
amd64 / x86_64 nodes	The agent-harness image is amd64 only	Same rule as the rest of xpander
Release namespace admits PodSecurity baseline	The executor and slot containers are baseline pods (root with the runtime’s default capabilities, RuntimeDefault seccomp)	Unlabelled is fine; restricted is refused
Landlock on the executor nodes, or fleet.landlock.enabled: false	Landlock limits each turn to its own files and ports	grep -o landlock /sys/kernel/security/lsm prints landlock; the executor’s /health reports it too. See Landlock
A StorageClass that formats XFS and mounts it with prjquota	Hard per-agent quotas	Hybrid: fleet.storage.createClass: true (default) renders fleet-xfs for the EBS CSI driver (ebs.csi.aws.com, gp3, encrypted, XFS, prjquota; on EKS Auto Mode set provisioner: ebs.csi.eks.amazonaws.com, see EKS Auto Mode; on GKE pd.csi.storage.gke.io, on AKS disk.csi.azure.com). Air-Gapped: global.fleet.storage.createClass: false by default, so bring a class that mounts XFS with prjquota, or set it to true on EKS. Without prjquota the sweeper enforces quotas softly
RWO block storage, 500 GiB per executor	One volume per executor, via volumeClaimTemplates	The class exists and provisions
Cloud metadata blocked from the pod	The executor refuses to register while 169.254.169.254 answers from the pod	The fleet NetworkPolicy blocks it; with fleet.networkPolicy.enabled: false, block it yourself (your own policy, or IMDS hop limit 1 on EKS)
KEDA (optional)	Holds the executor count with a ScaledObject	Only with fleet.keda.enabled; KEDA is not bundled. Air-Gapped mirrors its images with image_sync.sh --with-keda and installs it first
Pinned image	The executor and slot containers run agent-harness	Air-Gapped render validation refuses an unpinned global.images.agent-harness tag or digest
Memory-rich nodes	Turn memory dominates; CPU is bursty	r-class (1:8 vCPU:GiB) fits the profile

Landlock on the executor nodes

Landlock limits each turn to the agent’s own files. It is on by default (fleet.landlock.enabled: true, or global.fleet.landlock.enabled on Air-Gapped). On a node without it, the executor takes no turn: /health answers 503, and its refusal field and the log say the kernel has no Landlock.

Platform	Node image	Agent executors
Amazon EKS (incl. Auto Mode)	Amazon Linux 2023, Bottlerocket	Supported, nothing to set
Google GKE Standard	Ubuntu (UBUNTU_CONTAINERD)	Supported, nothing to set (recommended on GKE)
Google GKE Standard	Container-Optimized OS	Supported with fleet.landlock.enabled: false
Google GKE Autopilot	Container-Optimized OS	Supported with fleet.landlock.enabled: false
Azure AKS	Ubuntu, Azure Linux	Supported when the check above prints landlock, else with fleet.landlock.enabled: false

On GKE, keep the cluster on Container-Optimized OS and add one Ubuntu node pool for the executors:

gcloud container node-pools create xpander-agents --cluster <cluster> --image-type=UBUNTU_CONTAINERD \
  --machine-type <type> --num-nodes <n> --node-taints xpander.ai/agents=true:NoSchedule

fleet:
  nodeSelector:
    cloud.google.com/gke-nodepool: xpander-agents
  tolerations:
    - {key: xpander.ai/agents, operator: Equal, value: "true", effect: NoSchedule}

The taint is optional. If you cannot add a node pool, set fleet.landlock.enabled: false, and Landlock is off on every node.

Still in place without Landlock	Lost without Landlock
Each agent still runs as its own user. Its files and turn secrets are readable by that user only. Each turn runs in its own slot container behind the seccomp profile and no_new_privs. The slot is cleared between agents. Agents never hold platform keys.	Landlock’s file rules (a turn can read what ordinary file permissions allow) and its port rules

/health then shows landlock_enabled: false.

Fleet egress

The executors’ NetworkPolicy (fleet.networkPolicy.enabled, default true) gives agents the same reachability as any other pod, private ranges and in-cluster services included, minus link-local. fleet.networkPolicy.exceptCidrs defaults to ["169.254.0.0/16"] and exceptCidrsV6 to ["fe80::/10"]. Setting either replaces the whole list, so keep the link-local range in it. To close more ranges, use extraExceptCidrs. extraEgress appends raw egress rules. fleet.egress.allowHosts is still accepted for older agent-harness images and is no longer needed.

Autoscaling

An agent runs only on the executor that holds its home, so scaling in would strand agents. With fleet.keda.enabled: true the chart renders a KEDA ScaledObject that keeps every executor running:

Knob	Chart default
fleet.keda.maxReplicas	12; the ScaledObject sets both minReplicaCount and maxReplicaCount to it (0 holds one). Set it to your fleet node count
fleet.keda.minReplicas	Removed; ignored if still set
Trigger	CPU utilization 70 % (fleet.keda.cpuUtilization)
Cooldown	300 s (fleet.keda.cooldownSeconds)

One executor per node, so grow the node group and maxReplicas together. Without KEDA (the default), set fleet.replicas. On a sealed network, a fixed count sized for peak is recommended.

Safe rollout, drain, and upgrades

A roll never kills a running turn

A rolling update, a KEDA scale-in and a pod delete all send SIGTERM, which the executor treats as a drain, not a kill:

1. The executor stops claiming work, and its heartbeat says draining. The Agent Controller routes around it: nothing new lands on it, and whatever was queued for it goes back to the shared queue. It still counts as fleet presence, so new turns wait for the replacement instead of starting anywhere else.
2. The turns in flight keep running for up to fleet.executor.drainSeconds (3600 s by default). A turn still running when that window ends is stopped the way a user stop is. The CLI is interrupted, its session is kept, and the task ends stopped with “The executor is restarting; send the message again to continue where it left off.” Sending the message again resumes the session on the new pod. No task is left hanging as executing.
3. Each slot container holds its turn until the executor’s health endpoint reports draining with no turn in flight. The pod grace (fleet.executor.terminationGraceSeconds, 7500 s: the CLI run budget plus five minutes) is a ceiling, never a wait. The pod leaves as soon as its containers exit.

A roll replaces one pod at a time, so a roll during a long turn takes as long as that turn, up to the drain window. So:

• If your turns run long, raise drainSeconds toward the run budget. It must stay at least 120 s under the pod grace. The Air-Gapped chart refuses to render otherwise.
• Never shorten a termination with --grace-period=0 or --force on a pod that still has turns. That is the one way to kill a turn.
• Plan image rolls for quiet hours when you can. They are safe at any time, only slower under load.
• The drain needs an agent-harness image that carries it. An older executor ignores the setting and exits as before.

Idle guard when the StorageClass is missing

The executor is Ready only when its volume binds on an XFS + prjquota StorageClass. So an install never waits on one that cannot bind, both charts check the live cluster:

Situation	What the chart renders
fleet.storage.createClass: false and no StorageClass named fleet.storage.className exists	the executor StatefulSet with replicas: 0, annotated xpander.ai/fleet-idle-reason
fleet.storage.createClass: true but no CSIDriver named fleet.storage.provisioner is installed (the rendered class could never provision, the non-AWS cluster case)	same: idle, with the reason on the annotation
the class exists (or the driver is present for the rendered class)	the configured replicas

helm template and dry-runs always render the configured replicas (at least one). fleet.idleGuard: "off" removes the lookup for Argo CD, Flux and policy gates, and the executor stays Pending until its class exists. Create the class (or point provisioner at your CSI driver) and the next upgrade brings it up. Meanwhile turns queue.

--wait, --atomic and the installer

helm upgrade --wait (and --atomic) waits for the executor StatefulSet:

• A running turn holds the roll. With --wait, an upgrade during a long turn can take up to the drain window plus pod start. Give --timeout at least the drain window (3600 s by default) or upgrade in a quiet period.
• An idle executor never stalls an upgrade. With the idle guard above, a cluster without the storage prerequisites renders the executor at zero replicas and the upgrade completes.

install.sh handles both for you:

Run	What install.sh does
A first install	Runs without --wait or --atomic (the post-install smoke hook is the readiness gate), and prints the xpander.ai/fleet-idle-reason annotation when the executor is at zero
upgrades	Run helm upgrade --atomic with the installer’s timeout, then verify workloads, skipping zero-replica ones

Capacity planning

Plan in memory-backed executors: 10 slots on a 16 vCPU / 128 GiB node (for example r6i.4xlarge), with executor memory raised so each turn’s 8 GiB is backed. The rest covers the kubelet, DaemonSets, the 4 GiB /dev/shm and the 20 GiB package cache.

Peak concurrent turns	Executors (10 slots each)	Nodes (16 vCPU / 128 GiB each)	Fleet volumes	Agents the volumes hold at full 20 GiB quota	Also raise
10	1	1	1 x 500 GiB	~25	nothing
50	5	5	5 x 500 GiB = 2.5 TiB	~125	fleet.keda.maxReplicas: 5 (or replicas: 5)
200	20	20	20 x 500 GiB = 10 TiB	~500	maxReplicas: 20 (or replicas: 20) and a fixed 20-node group

• Slots, not agents. One executor serves many idle agents but runs at most turnsMax turns at once. Extra turns queue.
• Agents per volume is a ceiling. Most directories stay far below the quota, and clones share one object store per repository, so a 500 GiB volume holds far more than 25 agents in practice.
• Memory headroom. Keep the executor’s limit at or below ~80 % of the node, and turnsMax x 8 GiB within what the node can back.
• Smaller executors are fine. On the defaults (8 vCPU / 28 GiB, 12 slots) one fits on a 16 vCPU / 32 GiB node, with the risk in the warning above. Lower turnsMax to 3 or 4 for heavy builds.
• Workspace pods are separate: 0.25 vCPU / 256 Mi request, up to 2 vCPU / 3 GiB while active. See Agent workspaces.

Data services and storage

The chart deploys single-replica Redis and PostgreSQL StatefulSets and a container registry alongside xpander:

Component	Persistent volume (default)	Guidance
Redis	8 GiB	Queues, caches, and runtime state. 2-4 GiB of memory is sufficient for a single organization; give the pod a 512 Mi-1 GiB memory request
PostgreSQL	20 GiB	xpander state; grows slowly
Container registry	20 GiB	Stores agent container images built in your cluster

Baseline storage: 48 GiB of persistent volumes plus the 500 GiB agent volume per executor, plus the workspace volume of each workspace-using agent (see above). All volumes use your cluster’s default StorageClass unless overridden.

Production checklist:

• Node types: x86_64 only. General-purpose instances with a 1:4 vCPU:GiB ratio (m5, m6i) fit the workload profile. t3 burstable instances are fine for evaluation, not for production.
• Raise the low defaults: API memory (table above) before real load.
• Enable autoscaling for the execution tier (agent-worker) once concurrency grows beyond a couple of replicas.
• Keep headroom: pod requests should stay under ~65% of cluster capacity so node upgrades and bursts don’t evict workloads.
• Storage class: SSD-backed volumes for Redis and PostgreSQL.

Image and registry

Item	Value
Image	agent-harness (Docker Hub xpanderaihub/agent-harness, mirrored into your registry on Air-Gapped)
Compressed size	~1.5 GB (amd64)
On node disk	Plan ~4 GiB for the extracted image
Contents	Claude Code, Codex, OpenCode, Playwright Chromium, Node 22, Python 3.12, git, plus kubectl, helm, aws, terraform, gh, docker client and other developer utilities
Pull policy	Executors pull on every roll (Always); pin the tag or digest so a roll is deliberate

The executor and every slot use this image. Budget the ~1.5 GB pull per executor node. On Air-Gapped, mirror it at the pinned tag before the first executor starts.

Network egress

Executors hold no LLM provider key. A turn’s model calls go straight to the in-cluster LLM Gateway, which is always on, with the run’s own token. The gateway is the component that egresses to the model provider.

From	To	When
Executor	Agent Controller (registration, heartbeat, claim, run tokens) and the LLM Gateway Service on port 4000 (model calls)	Always, in-cluster (fleet.controllerUrl, default the in-cluster Service)
LLM Gateway	api.anthropic.com, bedrock-runtime.<region>.amazonaws.com, or your OpenAI-compatible endpoint	Every model call, on behalf of the CLIs
Executor	registry.npmjs.org (or your XPANDER_NPM_REGISTRY mirror)	Only when CLI auto-update is on (cli_auto_update, default on). The preinstalled CLIs need no egress; Air-Gapped uses a sealed CLI bundle instead
A turn’s own commands	npm / PyPI / apt / Homebrew, or their mirrors	Only when an agent’s runtime environment or the agent itself installs packages; follows the install’s package-mirror contract
Executor (NetworkPolicy)	DNS, the Agent Controller, the LLM Gateway, and every other address except link-local (fleet.networkPolicy.exceptCidrs), private ranges included	The chart renders this policy (Fleet egress); on Air-Gapped the internet leg is dropped with global.network.internetEgress: false

The executor refuses to register while the instance metadata endpoint (169.254.169.254) answers from the pod, so a turn can never read node credentials.

Model providers

Keys are saved in the organization (Platform > AI governance > Models) or on the install (secrets.static), and the LLM Gateway adds them to each call. Per-provider keys, Bedrock authentication and default harnesses: AI vendors.

Air-Gapped vs Hybrid

	Hybrid	Air-Gapped
Fleet values	fleet.*, image images.agentHarness	global.fleet.*, image global.images.agent-harness from your mirror, tag or digest pinned
Starting executors	fleet.replicas: 1	global.fleet.replicas: 1
Fleet token	Generated into the chart’s Secret, or fleet.tokenSecret.existing	One more key in xpander-generated-keys
StorageClass	createClass: true renders the EBS gp3 XFS class (ebs.csi.aws.com; ebs.csi.eks.amazonaws.com on EKS Auto Mode)	createClass: false; bring a class that mounts XFS with prjquota, or set it to true on EKS
KEDA	Install kedacore/keda yourself, then fleet.keda.enabled	Mirror with image_sync.sh --with-keda, install, then global.fleet.keda.enabled
CLI updates	Executors update from the npm registry or your XPANDER_NPM_REGISTRY mirror	Sealed CLI bundle imported from a PVC (global.fleet.cliBundle)
LLM	In-cluster LLM Gateway (always on) egresses to Anthropic / Bedrock / your endpoint on keys saved in the organization or set on the install, never xpander’s keys; defaulted agents follow the org default LLM, else Anthropic direct	In-network endpoint only; VPC Bedrock via global.llm.bedrock.baseUrl (Pod Identity, assume role or bearer token); defaulted agents follow the org default LLM, then global.harness.defaultModels / defaultProviders
Upgrades	install.sh upgrade runs --atomic; a running turn holds the roll up to fleet.executor.drainSeconds; the idle guard covers a missing class	same, under global.fleet.executor; the first install runs without --wait
Concurrency cap	fleet.turnsMax per executor; no per-organization cap	global.fleet.turnsMax per executor
Landlock off switch	fleet.landlock.enabled: false	global.fleet.landlock.enabled: false
Recommended posture	fleet.replicas sized for peak, or KEDA with maxReplicas set to the fleet node count	Fixed replicas sized for peak (static capacity, no autoscaler), KEDA optional

Next steps

Install Hybrid

The Hybrid installer.

Install Air-Gapped

The Air-Gapped installer.

Set up an EKS cluster

Nodes and StorageClasses.

Understand two-file licensing

Seat and agent caps.