Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Server architecture

Applies to

Product: Server · Audience: Platform Operator

The shape of a basebox Server: a dedicated machine (or a small cluster) with the basebox platform, service models and – on the same node or a separate GPU host – inference. This page shows the three usual layouts and the pattern by which GPUs are distributed.

The three layers on a server

flowchart TB
  subgraph SRV["Dedicated customer server · Kubernetes"]
    P["<b>basebox platform</b><br/>frontend · AISRV · storesrv · Keycloak · PostgreSQL · ingress<br/><i>CPU, RAM, storage – no GPU</i>"]
    S["<b>Service models</b><br/>ragsrv · ragsrv-support: RAG, extraction, OCR, STT<br/><i>dedicated GPU or MIG slices, otherwise CPU mode</i>"]
    I["<b>Inference</b><br/>vLLM with the language model<br/><i>whole GPU(s), optionally tensor-parallel</i>"]
    P --> S
    P -->|OpenAI-compatible API| I
  end
  U["Users · API clients"] -->|HTTPS| P
  style P fill:#f4f2ee,stroke:#524e47,color:#1d1e1c
  style S fill:#dcefe2,stroke:#3a7a49,color:#1d1e1c
  style I fill:#dbeafe,stroke:#1e40af,color:#1d1e1c

The platform needs no GPU. A server's GPUs belong to the service models and inference – and they should not share the same GPU, so that an upload or a transcription never competes with the chat for VRAM. Why: Understand the architecture.

The three layouts

Layout Description When appropriate GPU preparation
One server, one cluster All three layers on one Kubernetes node Evaluation, compact appliance, the reference configurations Prepare this server for GPU workloads
One cluster with CPU and GPU nodes Platform on CPU nodes, service models and inference on GPU nodes Existing Kubernetes with node separation Prepare only the GPU nodes
Separate application and inference servers Platform (+ service models in CPU or GPU mode) on an application server; inference on a separate GPU host, addressed via the OpenAI-compatible API GPU system is managed separately or lies in a different security zone The application server can be CPU-only

The third layout with a worked vLLM example, secrets, network requirements and verification steps: Deployment topologies. In every layout, users and browsers talk only to the platform, never directly to inference.

The GPU allocation pattern

Most reference configurations follow a pattern:

  • Inference GPU(s): whole, unpartitioned GPUs for the language model; for large models two GPUs as a tensor-parallel pair (TP=2).
  • Service GPU: one dedicated GPU (e.g. RTX PRO 6000) or one large GPU split into MIG instances – one per service: GPU RAG, document extraction, OCR, speech-to-text.
  • Platform: CPU.
Configuration Inference Service models Status
2 × H200 141 GB 1 whole H200 1 H200 as 4 × MIG 1g.35gb Validated
4 × H100 SXM 80 GB 2 H100 tensor-parallel 2 H100 as 2 × MIG 3g.40gb each Validated
2 × H200 + RTX PRO 6000 2 H200 RTX PRO 6000 dedicated Supported
3 × RTX PRO 6000 typically 2 1 dedicated Supported

Kubernetes sees this split as resources (nvidia.com/gpu, nvidia.com/mig-1g.35gb, …) that the Helm values request per service. Details per configuration: Reference configurations.

The components

What runs on the server – frontend, AISRV, storesrv, Keycloak, PostgreSQL clusters (CloudNativePG), ragsrv, ragsrv-support, inference, optional MCP connectors – and how they communicate is described under basebox components. Everything is deployed via the umbrella chart basebox.ai.

Software stack from bottom to top

  1. Operating system – Ubuntu 24.04 LTS recommended
  2. NVIDIA driver, CUDA, container toolkit – on hosts running GPU workloads
  3. Kubernetes (1.33+ recommended, 1.23+ required by the chart) with GPU Operator, ingress controller, storage class, CloudNativePG
  4. basebox via Helm – platform, service models, bundled or external inference

The way there: Bare-metal installation.

What a server is not

  • Not a high-availability cluster out of the box. The reference configurations are single-node; HA across several nodes is a separate architecture – see Multi-node.
  • Not defined by hardware. A reference configuration documents a known setup; it is not a minimum requirement of basebox.
  • Not Cloud, even if it stands at basebox. See Hosting options.

Next step: Hardware options