Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Resources & scaling

Applies to

Product: Server · Audience: Platform Operator

How resource demand is distributed across the three layers, what grows with users, documents or model size, and which knobs exist to scale each layer independently. The numbers on this page are guidelines from the chart and the reference configurations – exact supported minimum and recommended values per layer are pending DevOps confirmation and will be stated on the reference configuration pages once confirmed.

What grows with what

Layer Grows with Resources Knobs
basebox platform Number of users and concurrent sessions, API traffic, number of organisations CPU, RAM, database IOPS, storage for media Replicas and autoscaling for AISRV/frontend/storesrv, CNPG instances, storage class, database size
Service models Document volume (uploads, knowledge bases), share of OCR, audio minutes GPU memory and compute; in CPU mode CPU and RAM Dedicated GPU or MIG slices, COMPUTE: gpu/cpu, ragsrv replicas, size of model cache volumes
Inference Model size, context length, concurrent requests, latency target GPU memory (weights + KV cache), GPU compute, interconnect Model and quantisation, number of GPUs / tensor parallelism, AISRV_LLM_CONTEXT_SIZE, KV-cache quantisation

The three layers scale independently: more users does not mean more GPU for inference as long as concurrency does not rise; more documents means storage and service GPU, not inference.

Platform

Chart guidelines for a production deployment: minimum 5+ cores and 12 GB+ RAM, recommended 10+ cores and 32 GB+ RAM – for the whole stack excluding GPU workloads. Example requests from the service pages:

# AISRV / STORESRV
resources:
  requests: {cpu: 1000m, memory: 2Gi}
  limits:   {cpu: 2000m, memory: 4Gi}

Scaling:

  • AISRV horizontally (replicaCount, autoscaling.enabled with target CPU) – stateless, database via CloudNativePG.
  • Keycloak horizontally as well; mind the Infinispan cache with several replicas.
  • Databases: *-db.cluster.instances: 3 for read replicas and resilience; fast storage class (SSD/NVMe); storage.size from the 5–10 Gi defaults to 20–50 Gi per service in production.
  • Ingress is set up for long connections (24 h) and a 20 MB body; file uploads into knowledge bases may require larger limits.

Service models

They need a stable, predictable footprint. Typically useful: around 48 GB GPU memory, better about 96 GB for larger usage scenarios – as architectural orientation, not a minimum requirement. The reference configurations show two ways: a dedicated GPU (RTX PRO 6000) or MIG slices (4 × 1g.35gb on one H200; 2 × 3g.40gb on each of two H100s).

# ragsrv / ragsrv-support in GPU mode (example)
resources:
  requests: {cpu: 2000m, memory: 8Gi,  nvidia.com/gpu: 1}
  limits:   {cpu: 4000m, memory: 16Gi, nvidia.com/gpu: 1}
# CPU mode
resources:
  requests: {cpu: 4000m, memory: 16Gi}
  limits:   {cpu: 8000m, memory: 32Gi}

Scaling: for high document volume more ragsrv replicas and a larger ragsrv-db; for lots of OCR or audio more GPU memory for the service GPU; in CPU mode more cores and RAM – functional but considerably slower. A measured data point from the 2 × H200 reference configuration: 120-page PDF extracted in 28.47 s; 60 minutes of audio transcribed in 31.31 s.

Inference

This is where the large GPU demand lives. Rule of thumb:

Available VRAM = model weights + (context size × concurrent users × KV cache per token)

Example Llama 3.3 70B FP8 on 4 × H100 (320 GB): 65k context → 10–12 concurrent users; 32k → 20–25; 16k → 40–50. Context and concurrency are a trade-off; KV-cache quantisation can roughly double capacity. Tables per hardware class and model: LLM recommendations.

Scaling:

  • Larger model or longer context → more VRAM: stronger quantisation, a second GPU with tensor parallelism (TP must divide the attention heads: GPT-OSS 120B allows TP=1, 2, 4, 8, not 3), or larger GPUs.
  • More concurrent users → more KV cache: lower context, quantise the KV cache, an additional instance – see Multiple inference instances.
  • Latency → less batching headroom, more hardware per user.
  • Inference can move to a separate GPU host without the platform changing – see Deployment topologies.

AISRV_LLM_CONTEXT_SIZE must match the context actually configured in the runtime; it bounds what users can load into a chat.

Storage and network

  • Storage grows with documents (media volume, ragsrv-db), model artefacts (200 GB+ for the inference cache is common) and databases. Chart guideline: 200 GB+ minimum, 500 GB+ SSD/NVMe recommended, 1 TB+ for model storage.
  • Network is rarely the bottleneck; relevant with separate application and inference hosts (prompts and context travel between them) and with tensor parallelism (NVLink/NVSwitch between the inference GPUs).
  • Shared memory (/dev/shm, 64 GB in the verified configuration) for loading large models.

When a second node

A single node covers the reference configurations. A second node becomes sensible when inference needs more GPUs than one server holds, when platform and inference belong in different security zones, or when high availability is required. What becomes possible then and what harder: Multi-node.

Observe, then scale

Scale on measurements, not on guesses: GPU utilisation and memory (DCGM), inference latency and tokens/s, document processing queues, pod CPU/RAM (kubectl top), database size. What to watch: Monitoring; the operations view of scaling: Scaling.

Next step: Reference configurations