Licensed to be used in conjunction with basebox, only.
// installation
Advanced architectures
Applies to
Product: Server · Audience: Platform Operator
Layouts beyond a single node with one inference GPU. These pages exist so the information architecture does not change when such setups get documented – and so it is clear which pattern is validated today, which is supported and which is future. The yardstick is the same as for the reference configurations: Validated · Supported · Experimental · Custom.
The four patterns
| Pattern | What it solves | Status today | Page |
|---|---|---|---|
| Multi-GPU – tensor parallelism across several GPUs for one model | Models or contexts that do not fit on one GPU | Validated in the 4 × H100 SXM configuration (TP=2) | Multi-GPU |
| Dedicated service GPU – one GPU or MIG slices exclusively for service models | Uploads and transcriptions never compete with inference | Validated as MIG variant (2 × H200, 4 × H100); Supported as its own GPU (2 × H200 + RTX PRO 6000) | Dedicated service GPU |
| Multiple inference instances – several models or instances side by side | Default plus reasoning model; more concurrency at the same context | Possible via endpoints offering several models; routing details pending DevOps confirmation | Multiple inference instances |
| Multi-node – distribute platform, service models and inference across nodes | Capacity beyond one server, security zones, high availability | Separate application and inference hosts documented and proven (Deployment topologies); high availability is a separate architecture | Multi-node |
When to read on here
- The desired model does not fit on one GPU → Multi-GPU.
- Users report latency spikes in the chat while documents are processed → Dedicated service GPU.
- Two models are to be offered in parallel, or one instance is saturated → Multiple inference instances.
- One server is not enough (GPU slots, power), inference belongs in another zone, or resilience is required → Multi-node.
Everything before that – one node, one inference GPU, service models on MIG or CPU – is covered by the reference configurations and the bare-metal path.
What applies to all patterns
- Assign GPUs by stable identity (UUID, PCI address), never by index numbers.
- GPU memory is local per card. Two 80 GB GPUs are not one 160 GB device; tensor parallelism distributes layers, it does not pool memory.
- Users never talk to inference directly – not even with several instances or nodes. Only AISRV.
- One model cache per node. Model artefacts live where inference runs.
- No silent fallback. If an instance or node fails, basebox shows the error.
- Measure first, then build. Every pattern starts with a capacity or latency measurement – Monitoring, Scaling.
Reading status honestly
Listing a pattern as Supported or Experimental means: basebox knows and supports the approach but has not measured every combination. If you run such a layout in production, document it following the Custom hardware template and report your experience to support@basebox.ai – that is how patterns become reference configurations.
Next step: Multi-GPU