Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Dedicated service GPU

Applies to

Product: Server · Audience: Platform Operator

Reserving one GPU (or MIG slices) exclusively for service models so that uploads and transcriptions never compete with inference. This is the architectural principle behind all reference configurations: inference gets whole GPUs, service models get their own capacity – and the platform needs none.

The problem it solves

Service models – GPU RAG, document extraction, OCR, speech-to-text – work in bursts: an upload, a 120-page PDF, an hour-long transcription. If they share the GPU with inference, they compete for VRAM and compute in exactly those moments, and users see latency spikes in the chat – or inference runs into an OOM error. Service models, by contrast, have a stable, predictable footprint; separated, they are easy to size.

Three ways

Way How Status Reference
A · MIG slices of a large GPU One H200/H100 in MIG mode, one instance per service; the inference GPU(s) remain whole Validated 2 × H200 (4 × 1g.35gb), 4 × H100 (2 × 2 × 3g.40gb)
B · Its own physical GPU An additional card (e.g. RTX PRO 6000, 96 GB) only for service models Supported 2 × H200 + RTX PRO 6000, 3 × RTX PRO 6000
C · Free GPU from TP constraints If the LLM uses e.g. TP=2 of four GPUs, one of the remaining ones serves the services Supported LLM recommendations → GPU allocation

Without free GPU capacity, CPU mode remains – functional for RAG and OCR, considerably slower (Deploy service models).

Way A: MIG

  1. Identify the service GPU by UUID (nvidia-smi -L); do not touch the inference GPU(s).
  2. Enable MIG on the service GPU, create instances in the intended profile (1g.35gb on H200, 3g.40gb on H100 – nvidia-smi mig -lgip shows what the GPU can do) and create compute instances.
  3. Set the GPU Operator to mixed discovery (mixed), reload the discovery components.
  4. Check that the node advertises whole GPUs and MIG resources: 
    nvidia.com/gpu:          1      (2 × H200)
nvidia.com/mig-1g.35gb:  4
  5. Every service workload requests one instance: 
    resources:
  requests: {nvidia.com/mig-1g.35gb: 1}
  limits:   {nvidia.com/mig-1g.35gb: 1}

Step by step and failure patterns: NVIDIA / GPU → MIG. MIG isolates compute and memory between the instances; services on the same physical GPU still share its power and thermal budget.

Way B: its own physical GPU

The service workloads must land on this card and leave the inference GPUs free:

  • Either MIG on the service GPU here as well (if the card supports it) – then way A applies, and inference cannot get the service GPU at all because it advertises only MIG resources.
  • Or device selection: place inference and service workloads deliberately via node labels or a device plugin configuration that exposes the service GPU as its own resource. Which variant basebox intends for the RTX PRO 6000 configurations is pending DevOps confirmation.

Without one of these measures, Kubernetes can put inference's nvidia.com/gpu: 1 onto the service card – and vice versa.

Sizing

As architectural orientation, not a minimum requirement: about 48 GB GPU memory, better around 96 GB for larger usage scenarios, depending on enabled services and volume. The MIG variants give 35 GB (H200) or 40 GB (H100) per service. Whisper needs about 8 GB. Measured data points of the 2 × H200: 120-page PDF in 28.47 s, 60 minutes of audio in 31.31 s.

Which services on the service GPU

Service Component
GPU RAG (embeddings, retrieval, reranking) ragsrv
Document extraction ragsrv-support
OCR ragsrv-support or a separate OCR model endpoint
Speech-to-text Whisper endpoint (AISRV_WHISPER_URL)

Mapping and values: Deploy service models · Service models.

Verify

  • The node advertises the expected resources (kubectl describe node).
  • The inference pod holds only whole GPU(s); every service pod holds its instance/card (describe pod | grep nvidia.com/).
  • Mixed load: while a large PDF is processed and an audio file transcribed, chat latency stays stable and the inference GPU's memory unchanged (nvidia-smi dmon).
  • After a reboot, MIG layout and allocation return.

Common failure patterns

Symptom Cause Solution
Chat slows down during uploads Services share the inference GPU Dedicated capacity per way A/B
Service pod Pending MIG resource not advertised or wrong name Discovery mixed, resource name exact
Inference occupies the service card No device selection (way B) MIG or device selection
MIG gone after reboot Layout not persistent Apply MIG configuration at boot (e.g. nvidia-mig-parted/GPU Operator MIG manager)

Next step: Multiple inference instances