Licensed to be used in conjunction with basebox, only.
// installation
Scaling
Applies to
Product: Server · Audience: Platform Operator
Scaling each layer independently: more users, more documents, larger models. When to add GPUs, when nodes – and when none of that is needed but a setting. This page is the operations view; the architecture behind it is under Resources & scaling.
First: what is actually scarce?
Scale on measurements. The most common symptoms and their cause:
| Symptom | Probably scarce | Visible in |
|---|---|---|
| Chat answers slow down when many work simultaneously | Inference: KV cache / compute | Inference queue, latency p95, GPU utilisation near 100 % |
| Long requests fail, OOM in inference logs | Inference: VRAM | GPU memory > 90 % |
| Uploads stay "processing" for a long time, chat stays fast | Service models | ragsrv/ragsrv-support utilisation, processing time per document |
| Chat slows down while documents are processed | Service models share the inference GPU | Inference GPU memory fluctuates with uploads |
| Interface sluggish, login slow, many users | Platform: AISRV, Keycloak, database | Pod CPU/RAM, DB connections |
| Uploads fail, model downloads abort | Storage | PVC fill level |
What to measure and with what: Monitoring.
Scaling inference
| Goal | Knob | Note |
|---|---|---|
| More concurrent users, same model | Lower context (AISRV_LLM_CONTEXT_SIZE and runtime context); enable KV-cache quantisation |
KV cache = context × users × cache/token; quantisation roughly doubles capacity |
| More concurrent users, keep context | Second inference instance on another GPU | Multiple inference instances |
| Larger model or longer context | Stronger quantisation (FP8/AWQ/MXFP4); tensor parallelism across two GPUs (NUM_GPUS: "2", nvidia.com/gpu: 2) |
TP must divide the attention heads; pair with the fastest peer path – Multi-GPU |
| Better latency | Less batching headroom, more hardware per user; faster model (MoE such as GPT-OSS) | Recommended models |
| Separate inference from the application server | Separate GPU host, AISRV via TLS/API key | Deployment topologies |
Guidelines: Llama 3.3 70B FP8 on 4 × H100 – 65k context ≈ 10–12 users, 32k ≈ 20–25, 16k ≈ 40–50 (LLM recommendations).
Scaling service models
| Goal | Knob |
|---|---|
| Faster document processing | From CPU to GPU mode (COMPUTE: gpu); dedicated GPU or MIG slices instead of a shared GPU |
| More parallel document volume | Increase ragsrv.replicaCount; ragsrv-support replicas; RWX volume for /tmp/ragsrv if pods run on different nodes |
| More OCR/audio | Larger MIG instance or its own GPU for ragsrv-support / Whisper |
| Larger knowledge bases | Increase ragsrv-db.cluster.storage.size; fast storage class; watch pgvector indexes |
Never put service models on the inference GPU to "save space" – that produces exactly the latency spikes users report. Pattern: Dedicated service GPU.
Scaling the platform
| Goal | Knob |
|---|---|
| More concurrent users in interface/API | aisrv.replicaCount or autoscaling.enabled: true (target CPU 70 %); frontend.replicaCount; storesrv if needed |
| Login load | idp.replicaCount / autoscaling; mind the Infinispan cache with several replicas |
| Database load and resilience | *-db.cluster.instances: 3 (read replicas); fast storage class; increase storage.size; regular VACUUM/ANALYZE |
| More organisations | Limits per organisation in the administration; platform as above |
Example:
aisrv:
replicaCount: 3
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
aisrv-db:
cluster:
instances: 3
storage:
size: 50Gi
storageClass: fast-ssd
Scaling storage
Enlarge PVCs (the storage class must allow expansion), consider media volume and model caches separately, let backups grow along – Storage.
When GPUs, when nodes
| Situation | Measure |
|---|---|
| Inference needs more VRAM, server has free GPU slots | Add a GPU; TP or second instance |
| Service models compete with inference | Dedicated service GPU or MIG on a large GPU |
| Server is full (GPU slots, power, cooling) | Second node: inference on its own GPU host, platform stays – Deployment topologies |
| High availability required | Several nodes, replicated databases, anti-affinity – separate architecture, Multi-node |
| Only the interface is sluggish | No new node – replicas and database instances |
A single node covers the reference configurations; multi-node is the step when capacity or availability demand it – not the first reflex.
Procedure
- Measure (one week, including peaks).
- Change one layer, not three.
- Render, apply in the maintenance window, check mixed load.
- Update the manifest.
Next step: Troubleshooting