Licensed to be used in conjunction with basebox, only.
// installation
Multi-node
Applies to
Product: Server · Audience: Platform Operator
Distributing the basebox platform, service models and inference across several nodes: what becomes possible (capacity, security zones, resilience) and what gets harder (state, shared storage, model caches). Part of this is documented and proven – separating application and inference hosts – while high availability is an architecture of its own with its own effort.
The three topologies
From Deployment topologies:
| Topology | When | Status |
|---|---|---|
| One server, one cluster | Evaluation, compact appliance – all reference configurations | Validated |
| One cluster with CPU and GPU nodes | Kubernetes already separates the nodes; platform on CPU nodes, inference and service models on GPU nodes | Supported |
| Separate application and inference servers | GPU system is managed separately or sits in another security zone; application server CPU-only | Supported – documented and proven with basebox 1.7.1 / vLLM 0.15.0 |
What stays the same in all three: users talk to basebox, never to the inference server; AISRV reaches inference via an authenticated OpenAI-compatible API.
What belongs on which node
| Layer | Components | Node type | What to watch |
|---|---|---|---|
| Platform | frontend, AISRV, storesrv, Keycloak, PostgreSQL clusters, MCP connectors | CPU nodes | State lives in the databases and the media volume |
| Service models | ragsrv, ragsrv-support, OCR/STT endpoints | GPU nodes (MIG or own GPU) or CPU mode | Shared temp volume /tmp/ragsrv needs ReadWriteMany when ragsrv and ragsrv-support run on different nodes |
| Inference | vLLM or compatible runtime | GPU nodes | Model cache per node; cold start several minutes |
Placement via nodeSelector/tolerations – the chart already ships nvidia.com/gpu: "true" and the matching toleration for GPU workloads (Multi-GPU → Configuration). For CPU nodes, conversely, make sure platform services do not land on GPU nodes and block resources there.
Separate application and inference servers
The documented path for organizations with an existing GPU system:
- Prepare the application server CPU-only: Kubernetes, ingress, storage, CloudNativePG – without the NVIDIA stack. ragsrv/ragsrv-support in CPU mode or on their own GPU host.
- Inference server with the runtime of your choice; the endpoint requires an API key, TLS, reachable only by AISRV.
- Connect per Connect inference:
inference.enabled: false,AISRV_LLM_URL, provider, model, context, key from secret; import the private CA into the basebox workload if needed. - Firewall: exactly one path AISRV → inference endpoint; no exposure to users.
Both hosts sit inside the same approved data processing boundary – prompts, retrieved context and document contents travel between them. What that means for security reviewers: Security → Data flows.
What multi-node makes harder
- State. The four PostgreSQL clusters hold the state; platform pods are interchangeable, the databases are not. CloudNativePG can replicate across nodes with
instances: 3– then you need anti-affinity and a storage class that provisions on every node. - Shared storage. The media volume (
AISRV_MEDIA_ROOT) and the RAG temp volume must be reachable by the pods that need them – with several replicas or separate nodes that meansReadWriteMany(NFS or comparable), see Storage. - Model caches. Every inference and service node keeps its own cache; in air-gapped operation the mirror must feed every node.
- Network. More hops, more TLS, more firewall rules – exactly one rule per path (Networking).
- Operations. Updates, backups and monitoring span several hosts; the manifest becomes more important, not less.
High availability
Multi-node enables resilience but does not deliver it by itself. Building blocks of an HA architecture:
| Block | Means | Note |
|---|---|---|
| Platform services | Several replicas of frontend, AISRV, storesrv, Keycloak on different nodes | Stateless; mind Keycloak clustering |
| Databases | CloudNativePG instances: 3, automatic failover on the -rw service |
Backups remain mandatory (Backup & restore) |
| Ingress | Redundant ingress controller, virtual IP or external load balancer | Certificate on all instances |
| Storage | Replicated or external storage with ReadWriteMany |
Local storage (local-path) is not HA |
| Inference | Second instance on its own GPU node (Multiple inference instances) | Doubles GPU demand; without a second instance, inference remains the single point of failure |
Status: Custom. basebox runs individual building blocks but has not measured a complete HA configuration as a reference. Whoever builds one documents it following the Custom hardware template and aligns it with basebox.
Verify
kubectl get nodes -o wide # roles, readiness
kubectl get pods -n basebox -o wide # placement per node
kubectl -n basebox describe node <gpu-node> | grep nvidia # GPU resources only on GPU nodes
kubectl get cluster -n basebox # CNPG instances and primary
Then: chat, upload, knowledge base, audio – the acceptance from Validate the installation. For HA, additionally drain one node and watch whether the application stays reachable.
Common failure patterns
| Symptom | Cause | Solution |
|---|---|---|
Platform pod Pending on GPU node |
Toleration set but no CPU node free | Check resources/selector |
| ragsrv-support cannot find files | Temp volume not shared (RWO on another node) | ReadWriteMany volume or both services on the same node |
| Inference unreachable after failover | DNS/firewall points at the old host | Stable address (Service, VIP) for the endpoint |
| Database read-only after node failure | CNPG primary gone, no replica | instances: 3 and storage on several nodes |
| TLS errors between hosts | Private CA not imported | Add the CA to the basebox workload |
Next step: Operations