Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

NVIDIA / GPU

Applies to

Product: Server · Audience: Platform Operator

Driver, CUDA runtime, container toolkit and GPU Operator; MIG where the reference configuration uses it; how to check that the node advertises the expected GPU resources. This step applies only to hosts running GPU workloads – a CPU-only application server skips it.

Official documentation: CUDA Installation Guide for Linux · NVIDIA Container Toolkit · GPU Operator · MIG User Guide

1. Identify the GPUs

Before installing anything, record which GPU gets which role – by stable identity, not by index numbers:

lspci | grep -i nvidia
nvidia-smi -L          # after the driver installation: UUID per GPU
nvidia-smi topo -m     # interconnect between the GPUs (NVLink, PCIe)

Note UUID, PCI address and NUMA node per GPU as well as the planned role (inference or service models). For tensor-parallel inference, pick the pair with the fastest peer path.

2. Driver

sudo ubuntu-drivers autoinstall
sudo reboot
nvidia-smi

Expected: driver 550+ (verified: 580.126.09), CUDA version 13.0+ in the output, all GPUs listed. Failure patterns are listed below under "Common failure patterns".

3. CUDA Toolkit

Install CUDA 13.0, set the environment variables and check with nvcc --version:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y cuda-toolkit-13-0

echo 'export PATH=/usr/local/cuda-13.0/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
source ~/.bashrc
nvcc --version

If CUDA 13.0 is not available in your repository, use 12.9 as fallback:

sudo apt-get install -y cuda-toolkit-12-9
echo 'export PATH=/usr/local/cuda-12.9/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.9/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
source ~/.bashrc

4. Container runtime for GPUs

Two paths:

With Kubernetes and GPU Operator (default): the GPU Operator installs and configures the NVIDIA Container Toolkit on the nodes itself. Step 5 then skips the manual toolkit installation.
Docker only (development/test): install the NVIDIA Container Toolkit manually, configure Docker for the NVIDIA runtime, restart Docker and check:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
    sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi

5. GPU Operator

After the Kubernetes installation (Kubernetes), install the GPU Operator via Helm. On a single node with a control-plane taint, the operator needs tolerations in the values file gpu-operator-values.yaml:

# gpu-operator-values.yaml
node-feature-discovery:
  master:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
  worker:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
  gc:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
operator:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

Installation:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --timeout 15m --generate-name \
  -n gpu-operator --create-namespace \
  -f gpu-operator-values.yaml \
  nvidia/gpu-operator
kubectl get pods -n gpu-operator

On multi-node clusters with dedicated worker nodes without a control-plane taint, install without -f gpu-operator-values.yaml.

If drivers are already installed on the host (step 2), configure the operator not to roll out driver containers; the options are in the GPU Operator documentation. For special cases – air-gapped environments, pre-installed drivers, MIG – contact support@basebox.ai when in doubt.

6. Set up MIG where the configuration provides for it

The Validated configurations split a service GPU via MIG: 2 × H200 uses four 1g.35gb instances on one H200; 4 × H100 SXM two 3g.40gb each on two H100s. The inference GPUs remain whole, MIG disabled.

Procedure (addressed via the UUID of the service GPU):

Enable MIG mode on the service GPU; leave the inference GPU(s) untouched.
Create GPU instances in the intended profile (1g.35gb or 3g.40gb) and the corresponding compute instances.
Configure the GPU Operator for mixed MIG discovery (MIG strategy mixed) so that whole GPUs and MIG instances are advertised together; reload the GPU discovery components.
Check that Kubernetes advertises the resources under Capacity and Allocatable.

The exact nvidia-smi mig commands and profile names are in the MIG User Guide; nvidia-smi mig -lgip lists which profiles a GPU supports. The NVIDIA GPU system components may need elevated host privileges for mixed discovery.

7. Verify

nvidia-smi -L
kubectl describe node <node-name> | grep -A8 -E "Capacity|Allocatable"

Expected resources per configuration:

Configuration	`nvidia.com/gpu`	MIG resource
2 × H200	1	`nvidia.com/mig-1g.35gb: 4`
4 × H100 SXM	2	`nvidia.com/mig-3g.40gb: 4`
Configurations with a dedicated service GPU without MIG	Number of all GPUs	–

Test run in the cluster:

kubectl run gpu-test --image=nvidia/cuda:12.9.0-base-ubuntu22.04 --rm -it --restart=Never -- nvidia-smi

Common failure patterns

Symptom	Cause	Solution
`nvidia-smi: command not found`	Driver not installed	Step 2
"couldn't communicate with the NVIDIA driver"	Installation incomplete	`sudo apt-get purge nvidia-*`, `ubuntu-drivers autoinstall` again, reboot
GPU Operator pods `Pending`, "untolerated taint"	Control-plane taint on single node	Tolerations in the values (step 5)
`helm install` "context deadline exceeded"	Large images, slow download	Delete and recreate the namespace, `--timeout 15m`, watch pods
Node advertises no `nvidia.com/gpu`	Operator not ready or driver conflict	`kubectl get pods -n gpu-operator`, logs, check node labels
MIG resources missing	Discovery not set to `mixed`, components not reloaded	Check MIG strategy, restart device plugin/GFD

Detailed decision trees (GPU not detected, CUDA version conflict, GPU Operator): Prepare the server → Troubleshooting and Troubleshooting.

Record

Driver version, CUDA version, GPU Operator version, GPU UUIDs with role, MIG layout. That belongs in the installation manifest.

Back: Prepare the server · Next: Kubernetes