Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

Basebox 4 x NVIDIA H100 SXM 80 GB Hardware Profile

Profile overview

This profile describes a single-node Basebox deployment on a customer-owned Kubernetes server with four NVIDIA H100 SXM 80 GB GPUs. Two complete GPUs serve the language model as a tensor-parallel pair. The other two GPUs are divided into four MIG instances for retrieval, document extraction, OCR, and speech-to-text.

The layout keeps language-model inference on complete GPUs while assigning each support workload its own GPU instance. Models, documents, and processed output remain on your infrastructure during normal operation.

This profile applies specifically to NVIDIA H100 SXM 80 GB. H100 PCIe and H100 NVL systems have different form factors, interconnects, power envelopes, and topology requirements and require their own hardware profiles.

At a glance

Item	This profile
Physical GPUs	4 x NVIDIA H100 SXM 80 GB in one server
Inference allocation	2 complete GPUs, tensor parallelism 2
Service allocation	2 GPUs in MIG mode, each with 2 x `3g.40gb`
Kubernetes GPU resources	`nvidia.com/gpu: 2` and `nvidia.com/mig-3g.40gb: 4`
GPU workloads	Inference, GPU RAG, extraction, OCR, speech-to-text
Kubernetes topology	Single node containing all four GPUs
Storage	Persistent storage for model artifacts, documents, and platform data
Artifact access	Registry and model-repository access, or approved internal mirrors

MIG (Multi-Instance GPU) divides one physical GPU into isolated compute and memory instances. NVIDIA documents a maximum of two 3g.40gb instances on an H100 80 GB GPU, which gives this profile four service instances across two physical GPUs.

Published GPU characteristics

The following figures are NVIDIA device specifications. They describe each H100 SXM GPU and are not Basebox performance or capacity results.

Characteristic	H100 SXM
GPU memory	80 GB
Memory bandwidth	3.35 TB/s
NVLink bandwidth	900 GB/s
PCIe interface	Gen5, 128 GB/s
Maximum thermal design power	Up to 700 W, configurable
Maximum `3g.40gb` MIG instances per GPU	2

Sources: NVIDIA H100 specifications and NVIDIA supported MIG profiles.

GPU memory remains local to each physical GPU. Two 80 GB inference GPUs do not behave as one transparently shared 160 GB device. The selected model, precision, context length, KV-cache requirements, runtime configuration, tensor-parallel implementation, and GPU interconnect are qualified together.

Hardware layout

Physical GPU	Mode	Workload
H100 GPU 0	Complete GPU	LLM inference, tensor-parallel shard 0
H100 GPU 1	Complete GPU	LLM inference, tensor-parallel shard 1
H100 GPU 2	2 x `3g.40gb` MIG	GPU RAG and document extraction
H100 GPU 3	2 x `3g.40gb` MIG	OCR and speech-to-text

GPU numbers identify roles for illustration only. Deployment binds roles using stable GPU UUIDs and PCI addresses because device index numbers can change across reboots and driver updates.

graph TB
  subgraph SERVER["Customer server - 4 x NVIDIA H100 SXM 80 GB"]
    subgraph NODE["Single Kubernetes node"]
      subgraph INF["Inference pair - complete GPUs"]
        G0["H100 GPU 0<br/>tensor-parallel shard 0"]
        G1["H100 GPU 1<br/>tensor-parallel shard 1"]
      end
      subgraph SVC2["Service GPU 2 - MIG mode"]
        M20["3g.40gb - GPU RAG"]
        M21["3g.40gb - extraction"]
      end
      subgraph SVC3["Service GPU 3 - MIG mode"]
        M30["3g.40gb - OCR"]
        M31["3g.40gb - speech-to-text"]
      end
    end
  end
  G0 <-->|"tensor-parallel traffic"| G1

  style G0 fill:#dbeafe,stroke:#1e40af,color:#111827
  style G1 fill:#dbeafe,stroke:#1e40af,color:#111827
  style M20 fill:#dcfce7,stroke:#166534,color:#111827
  style M21 fill:#dcfce7,stroke:#166534,color:#111827
  style M30 fill:#dcfce7,stroke:#166534,color:#111827
  style M31 fill:#dcfce7,stroke:#166534,color:#111827

Workload allocation

GPU allocation	Basebox workload	Purpose
2 complete H100 GPUs	Language-model inference	Tensor-parallel model serving
Service GPU 2, instance 1	GPU RAG	Embedding, retrieval, and reranking
Service GPU 2, instance 2	Document extraction	Converts uploaded files into machine-readable text
Service GPU 3, instance 1	OCR	Reads text from images and scanned pages
Service GPU 3, instance 2	Speech-to-text	Transcribes audio recordings

MIG isolates GPU compute and memory between service instances. The service workloads cannot consume the two complete GPUs reserved for inference. All workloads still share host CPU, system RAM, storage, and networking. Services on the same physical GPU also share its power and thermal envelope.

Inference interconnect and NUMA

Tensor parallelism exchanges data between inference shards during every generation. The two inference GPUs must therefore be selected from the actual server topology rather than from their displayed index numbers.

Record the stable GPU UUID, PCI address, and NUMA node for each GPU.
Confirm the peer path with nvidia-smi topo -m.
Run peer-to-peer and NCCL communication checks before deploying Basebox.
Keep the inference pair aligned with its CPU, storage, and network locality where the server exposes multiple NUMA nodes.
Confirm supported GPU population, NVLink or NVSwitch topology, power, and cooling with the server vendor.

Delivery model

The profile combines the Basebox Helm release with the GPU model endpoints used by document and audio processing.

Workload	Delivery
Inference	Basebox inference chart with two GPU resources and tensor parallelism 2
GPU RAG	Basebox `ragsrv` chart configured for one MIG resource
Document extraction	Basebox `ragsrv-support` chart configured for one MIG resource
OCR model service	Separately managed model endpoint connected to `ragsrv-support`
Speech-to-text model service	Separately managed vLLM-compatible Whisper endpoint connected to Basebox

The complete deployment is therefore the Basebox Helm release plus the OCR and speech-to-text model endpoints. Their image versions, endpoint configuration, resource requests, and lifecycle are recorded in the deployment manifest.

Platform prerequisites

Exact host sizing and software versions are selected during solution design for the chosen server, model, document workload, and platform environment.

Kubernetes resource contract

The node must advertise two complete GPUs and four 3g.40gb instances:

nvidia.com/gpu:          2
nvidia.com/mig-3g.40gb:  4

Inference requests both complete GPUs:

resources:
  requests:
    nvidia.com/gpu: 2
  limits:
    nvidia.com/gpu: 2

Each support workload requests one MIG instance:

resources:
  requests:
    nvidia.com/mig-3g.40gb: 1
  limits:
    nvidia.com/mig-3g.40gb: 1

Requests and limits must match, and the resource names must exactly match those advertised by the NVIDIA GPU Operator or device plugin.

Deployment sequence

Record the hardware and software manifest, stable GPU identities, topology, and NUMA layout.
Select and test the two-GPU inference pair.
Keep the inference pair as complete GPUs with MIG disabled.
Enable MIG on the two service GPUs.
Create two 3g.40gb GPU instances and compute instances on each service GPU.
Enable mixed MIG discovery and refresh the GPU discovery components.
Confirm Kubernetes advertises exactly two complete GPUs and four MIG resources.
Deploy inference with two GPU resources and tensor parallelism 2.
Deploy the four support workloads with one MIG resource each.
Complete warm-up and application-level functional checks.
Run a bounded mixed workload across inference and all support services.
Test a planned reboot and confirm that GPU resources and workloads return.
Record the accepted image digests, model revisions, chart version, driver, CUDA runtime, GPU Operator or device-plugin version, and firmware.

flowchart LR
  HW["Hardware<br/>inventory"] --> PEER["Peer and<br/>NCCL checks"]
  PEER --> MIG["MIG<br/>configuration"]
  MIG --> RES["Kubernetes<br/>resources"]
  RES --> DEPLOY["Basebox<br/>deployment"]
  DEPLOY --> TEST["Functional and<br/>mixed-load checks"]
  TEST --> REBOOT["Reboot and<br/>recovery check"]
  REBOOT --> ACCEPT["Accepted<br/>configuration"]

Operator verification

Inspect the physical topology and Kubernetes resources:

nvidia-smi -L
nvidia-smi topo -m
kubectl describe node <node-name>
kubectl get pods -n <basebox-namespace> -o wide

Confirm Capacity and Allocatable both show the expected GPU resource counts. After the workloads are ready, use approved synthetic fixtures to complete one authenticated chat, document ingestion and RAG search, extraction, OCR, and speech-to-text request through the application path.

Acceptance checks

Qualification fixtures must not contain customer, patient, medical, personal, confidential, or production data. Store operational metadata and redacted result summaries rather than credentials, tokens, complete prompts, source documents, recordings, transcripts, or model responses.

Scope

This profile covers one Kubernetes node with 4 x NVIDIA H100 SXM 80 GB.
It does not apply automatically to H100 PCIe or H100 NVL systems.
Model and context fit are established on the selected server configuration.
Capacity depends on model precision, context length, document workload, and traffic pattern.
Multi-node high availability and disaster recovery require a separate architecture.
Customer-specific CPU, RAM, storage, network, and capacity sizing is completed during solution design.
Normal inference and document processing run on customer infrastructure.
Installation and updates require approved artifact access or internal mirrors.