Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

NVIDIA / GPU

Applies to

Product: Server · Audience: Platform Operator

Driver, CUDA runtime, container toolkit and GPU Operator; MIG where the reference configuration uses it; how to check that the node advertises the expected GPU resources. This step applies only to hosts running GPU workloads – a CPU-only application server skips it.

Official documentation: CUDA Installation Guide for Linux · NVIDIA Container Toolkit · GPU Operator · MIG User Guide

1. Identify the GPUs

Before installing anything, record which GPU gets which role – by stable identity, not by index numbers:

lspci | grep -i nvidia
nvidia-smi -L          # after the driver installation: UUID per GPU
nvidia-smi topo -m     # interconnect between the GPUs (NVLink, PCIe)

Note UUID, PCI address and NUMA node per GPU as well as the planned role (inference or service models). For tensor-parallel inference, pick the pair with the fastest peer path.

2. Driver

sudo ubuntu-drivers autoinstall
sudo reboot
nvidia-smi

Expected: driver 550+ (verified: 580.126.09), CUDA version 13.0+ in the output, all GPUs listed. Details and failure patterns: Server Preparation Guide → Step 2.

3. CUDA Toolkit

Install CUDA 13.0 (12.9 as fallback), set environment variables, check nvcc --version – commands under Server Preparation Guide → Step 3.

4. Container runtime for GPUs

Two paths:

  • With Kubernetes and GPU Operator (default): the GPU Operator installs and configures the NVIDIA Container Toolkit on the nodes itself. Step 5 then skips the manual toolkit installation.
  • Docker only (development/test): install the NVIDIA Container Toolkit manually, sudo nvidia-ctk runtime configure --runtime=docker, restart Docker and check:
sudo docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi

Commands: Server Preparation Guide → Step 5.

5. GPU Operator

After the Kubernetes installation (Kubernetes), install the GPU Operator via Helm. On a single node with a control-plane taint, the operator needs tolerations – the complete values file and command are under Server Preparation Guide → Step 7:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --timeout 15m --generate-name \
  -n gpu-operator --create-namespace \
  -f gpu-operator-values.yaml \
  nvidia/gpu-operator
kubectl get pods -n gpu-operator

If drivers are already installed on the host (step 2), configure the operator not to roll out driver containers; the options are in the GPU Operator documentation. For special cases – air-gapped environments, pre-installed drivers, MIG – contact support@basebox.ai when in doubt.

6. Set up MIG where the configuration provides for it

The Validated configurations split a service GPU via MIG: 2 × H200 uses four 1g.35gb instances on one H200; 4 × H100 SXM two 3g.40gb each on two H100s. The inference GPUs remain whole, MIG disabled.

Procedure (addressed via the UUID of the service GPU):

  1. Enable MIG mode on the service GPU; leave the inference GPU(s) untouched.
  2. Create GPU instances in the intended profile (1g.35gb or 3g.40gb) and the corresponding compute instances.
  3. Configure the GPU Operator for mixed MIG discovery (MIG strategy mixed) so that whole GPUs and MIG instances are advertised together; reload the GPU discovery components.
  4. Check that Kubernetes advertises the resources under Capacity and Allocatable.

The exact nvidia-smi mig commands and profile names are in the MIG User Guide; nvidia-smi mig -lgip lists which profiles a GPU supports. The NVIDIA GPU system components may need elevated host privileges for mixed discovery.

7. Verify

nvidia-smi -L
kubectl describe node <node-name> | grep -A8 -E "Capacity|Allocatable"

Expected resources per configuration:

Configuration nvidia.com/gpu MIG resource
2 × H200 1 nvidia.com/mig-1g.35gb: 4
4 × H100 SXM 2 nvidia.com/mig-3g.40gb: 4
Configurations with a dedicated service GPU without MIG Number of all GPUs –

Test run in the cluster:

kubectl run gpu-test --image=nvidia/cuda:12.9.0-base-ubuntu22.04 --rm -it --restart=Never -- nvidia-smi

Common failure patterns

Symptom Cause Solution
nvidia-smi: command not found Driver not installed Step 2
"couldn't communicate with the NVIDIA driver" Installation incomplete sudo apt-get purge nvidia-*, ubuntu-drivers autoinstall again, reboot
GPU Operator pods Pending, "untolerated taint" Control-plane taint on single node Tolerations in the values (step 5)
helm install "context deadline exceeded" Large images, slow download Delete and recreate the namespace, --timeout 15m, watch pods
Node advertises no nvidia.com/gpu Operator not ready or driver conflict kubectl get pods -n gpu-operator, logs, check node labels
MIG resources missing Discovery not set to mixed, components not reloaded Check MIG strategy, restart device plugin/GFD

Complete decision trees: Server Preparation Guide → Troubleshooting.

Record

Driver version, CUDA version, GPU Operator version, GPU UUIDs with role, MIG layout. That belongs in the installation manifest.

Next step: Kubernetes