Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Sizing

Applies to

Product: Server · Audience: Platform Operator · Decision-maker

Which hardware do you need for how many users and which models? This page summarises the guideline values from the detail pages and links to them. The concurrent-user values are estimates from VRAM arithmetic, not load measurements. Customer-specific sizing is completed with basebox in the solution design.

Size the three layers separately

basebox consists of three layers that grow independently of each other. Size each one on its own:

Layer Grows with Mainly needs
basebox platform Users, concurrent sessions, API traffic CPU, RAM, database storage – no GPU
Service models (RAG, OCR, speech-to-text) Document volume, OCR share, audio minutes A dedicated GPU or MIG slices; alternatively CPU (slower)
Inference (language model) Model size, context length, concurrent requests GPU memory for weights and KV cache

More users does not mean more inference GPU as long as concurrency does not rise; more documents means storage and service GPU, not inference. Details: Resources & scaling.

Platform

  • CPU and RAM for the whole stack without GPU workloads: minimum 5+ cores and 12 GB+ RAM, recommended 10+ cores and 32 GB+ RAM.
  • Storage: minimum 200 GB+, recommended 500 GB+ SSD/NVMe, 1 TB+ with model storage. 200 GB+ is common for the inference model cache.
  • Shared memory (/dev/shm): 64 GB in the verified configuration.

Service models

  • As orientation, not a minimum requirement: around 48 GB GPU memory, better about 96 GB for larger usage scenarios.
  • Two approaches from the reference configurations: a dedicated GPU (RTX PRO 6000) or MIG slices (4 × 1g.35gb on one H200; 2 × 3g.40gb on each of two H100s).
  • Speech-to-text (Whisper) needs about 8 GB GPU memory.
  • Measured on 2 × H200: 120-page PDF extracted in 28.47 s, 60 minutes of audio transcribed in 31.31 s.

Do not place service models on the inference GPU: uploads and transcriptions then compete with chat.

Inference: model and concurrent users

Rule of thumb:

Available VRAM = model weights + (context size × concurrent users × KV cache per token)

Guideline values by hardware class (estimates, FP16 KV cache):

Hardware class Total VRAM Recommended model Context Concurrent users
Enterprise – 4 × H100 320 GB GPT-OSS 120B MXFP4, TP=4 / TP=2 128k / 32–64k 15–20 / 8–12
Enterprise – alternative 320 GB Llama 3.3 70B Instruct FP8 32k–65k 20–25 / 10–12
Professional – 4 × L40S 192 GB GPT-OSS 120B AWQ, TP=4 32–64k 5–8
Workstation – 2 × RTX 4090 48 GB GPT-OSS 20B MXFP4, TP=2 64k 2–3
Entry – 1 × 24 GB GPU (L4 or similar) 24 GB GPT-OSS 20B MXFP4 32k 1

The H200 class (141 GB per GPU) sits between Professional and Enterprise: one H200 carries Llama 3.3 70B FP8 with long context, two H200 with TP=2 carry GPT-OSS 120B MXFP4. The RTX PRO 6000 (96 GB per card) follows the L40S row.

Context versus concurrency: example Llama 3.3 70B FP8 on 4 × H100 – 65k context → 10–12 concurrent users, 32k → 20–25, 16k → 40–50. KV cache quantisation can roughly double capacity. For RAG workloads, context takes priority over concurrency.

Details: Recommended models · LLM recommendations (sizing reference) with a VRAM table per model size · Benchmarks for what has been measured.

Reference configurations

Known hardware setups in which the three layers are already allocated:

Configuration Status Memory
3 × NVIDIA RTX PRO 6000 Supported 128 GB / 512 GB
2 × NVIDIA H200 141 GB Validated 128 GB / 512 GB
2 × NVIDIA H200 + 1 × RTX PRO 6000 Supported 128 GB / 512 GB
4 × NVIDIA H100 SXM 80 GB Validated see page
Custom hardware Custom –

Status and page contents: Reference configurations.

Procedure

  1. Define the demand: concurrent users (not total users), required context length, document and audio volume.
  2. Choose the model by hardware class and scenario – Recommended models.
  3. Add the service GPU: dedicated GPU or MIG slices.
  4. Check the platform for CPU/RAM/storage.
  5. Match against a reference configuration – if none fits, the setup is Custom and is reviewed with basebox.

A second node becomes useful when inference needs more GPUs than one server holds, when platform and inference belong in different security zones, or when high availability is required: Multi-node.

Next step: Reference configurations or Installation paths