Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Multi-GPU

Applies to

Product: Server · Audience: Platform Operator

Tensor parallelism across several GPUs for one model, GPU pinning by UUID, NUMA and interconnect aspects. Validated in the reference configuration 4 × H100 SXM 80 GB, where two H100s serve the language model as a tensor-parallel pair (TP=2).

When tensor parallelism

When the model with the desired context and concurrency does not fit on one GPU – or when two GPUs are to deliver more throughput than one. Tensor parallelism (TP) distributes a model's layers across several GPUs that exchange data with every generation. It does not pool memory: two 80 GB GPUs do not behave like one 160 GB device; each holds its share of weights plus KV cache.

Examples from the LLM recommendations: GPT-OSS 120B MXFP4 with TP=4 on 4 × H100 for 128k context, TP=2 for 32–64k; Llama 3.3 70B FP8 with TP=2 or TP=4; GPT-OSS 120B AWQ needs all four GPUs on 4 × L40S.

Constraints

• TP must divide the attention heads evenly. GPT-OSS 120B (64 heads) allows TP=1, 2, 4, 8 – not 3. Check the model's num_attention_heads.
• GPT-OSS 120B: tensor, not data parallelism. Data parallelism produces garbled output.
• More TP = more interconnect traffic. On PCIe without NVLink the benefit shrinks; H100 SXM with NVLink (900 GB/s) is built for it, workstation cards such as the RTX PRO 6000 are not.
• All GPUs of a TP pair must be identical (model, VRAM, firmware).

Configuration

Bundled inference with two GPUs:

inference:
  resources:
    requests: {cpu: 8000m, memory: 128Gi, nvidia.com/gpu: 2}
    limits:   {cpu: 16000m, memory: 256Gi, nvidia.com/gpu: 2}
  env:
    NUM_GPUS: "2"                 # tensor parallelism = number of GPUs
    MODEL_ID: "openai/gpt-oss-120b"
    MAX_INPUT_TOKENS: "64000"
    SHM_SIZE: "64gb"
  nodeSelector:
    nvidia.com/gpu: "true"
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"

Requests and limits must match. SHM_SIZE generous – TP uses shared memory for the exchange; on the host /dev/shm correspondingly large (64 GB in the verified configuration).

Choosing the right GPU pair

Kubernetes assigns nvidia.com/gpu: 2 to any two free GPUs. In a server with several GPUs you have to ensure the TP pair has the fastest peer path and the service GPUs remain untouched:

1. Capture the topology:

   nvidia-smi -L                 # UUIDs
   nvidia-smi topo -m            # NV# = NVLink, PIX/PXB/PHB = PCIe proximity, SYS = across CPU sockets
   

   Choose the pair with NVLink (NV*) or the closest PCIe relationship; avoid pairs across NUMA nodes (SYS).
2. Take the service GPUs out of the pool: on MIG GPUs the node advertises only MIG resources, no nvidia.com/gpu – inference cannot get them. With a dedicated service GPU without MIG, use node labels/device selection to ensure inference does not occupy it.
3. Test peer communication before deploying basebox: peer-to-peer and NCCL checks (e.g. nccl-tests) between the chosen GPUs.
4. Align NUMA: pin the inference pod to the CPU socket the GPUs hang off (topology manager / CPU pinning) if the server has several NUMA nodes.
5. Record: UUIDs, PCI addresses, NUMA nodes, topology in the manifest. Index numbers change after reboots or driver updates.

What the 4 × H100 demonstrates

GPU	Mode	Workload
GPU 0, GPU 1	Whole	LLM inference, TP shards 0 and 1
GPU 2	2 × MIG 3g.40gb	GPU RAG, document extraction
GPU 3	2 × MIG 3g.40gb	OCR, speech-to-text

The node advertises nvidia.com/gpu: 2 and nvidia.com/mig-3g.40gb: 4. Deployment order, checks and acceptance: 4 × H100 SXM.

Verify

kubectl -n basebox describe pod -l app.kubernetes.io/name=inference | grep -A3 "nvidia.com/gpu"
kubectl -n basebox exec -it <inference-pod> -- nvidia-smi        # two GPUs visible
kubectl -n basebox logs -l app.kubernetes.io/name=inference | grep -iE "tensor|parallel|nccl"

Then trigger a long generation and watch both GPUs show load (nvidia-smi dmon). If inference does not start due to memory or collective errors, usually the TP count, head division or peer path is wrong.

Common failure patterns

Symptom	Cause	Solution
Inference does not start, "attention heads not divisible"	TP does not divide the heads	TP to 1, 2, 4, 8 (model-dependent)
NCCL timeouts, collective errors	Peer path slow/blocked, IOMMU, ACS	nvidia-smi topo -m; NCCL tests; BIOS/kernel parameters per NVIDIA docs
Only one GPU shows load	NUM_GPUS ≠ requests, or runtime ignores TP	Align values; runtime logs
Considerably slower than expected	PCIe instead of NVLink, pair across NUMA boundary	Choose the pair anew
Different pair after reboot	Index numbers instead of UUID	Assignment via stable identity

Next step: Dedicated service GPU