Licensed to be used in conjunction with basebox, only.
// installation
Benchmarks
Applies to
Product: Server · Audience: Platform Operator
Measured results on reference configurations: throughput, latency, behaviour with long context and under mixed load. Only results that basebox has measured and documented are listed. Estimates – such as the concurrency guidelines of the sizing reference – are marked as such.
Measured: reference workload 2 × H200 141 GB
Single node; one H200 entirely for inference, the second as 4 × MIG 1g.35gb for GPU RAG, document extraction, OCR and speech-to-text. Language model traffic and all four auxiliary services simultaneously on the same server.
| Measurement | Result |
|---|---|
| Duration of continuous mixed load | 60 minutes |
| Short language model requests | 720/720 with expected result |
| Language model requests with long context | 12/12 with expected result |
| Input size for long context | about 258,000 tokens |
| Combined RAG, extraction, OCR and STT load spikes | 12/12 completed |
| Restarts of the inference workload during the run | 0 |
| Document processing | 120-page PDF extracted and made searchable in 28.47 s |
| Speech-to-text | 60-minute synthetic audio file transcribed in 31.31 s |
| Cold-start tolerance of inference | about 6 minutes from restart to warm-up |
Document and audio times are service-level measurements; actual time varies with file format, content, model choice and configuration. Source and topology: Reference configuration 2 × H200.
Measured: individual values from the sizing reference
| Model | Hardware | Measurement |
|---|---|---|
| Qwen3 4B Instruct FP16 | 1 × 24 GB GPU | ~23 tokens/s; practical maximum context ~30k |
| Qwen2.5 72B Instruct Int8 vs. Llama 3.3 70B FP8 | 4 × H100 | Qwen2.5 72B with faster response times in testing |
Source: LLM recommendations. Without documented version and date – read as orientation.
Estimated (no measurement): concurrency by context
The guidelines for concurrent users in the sizing reference are estimates from VRAM arithmetic (weights + KV cache with FP16 cache), not load measurements:
| Model / hardware | Context | Concurrent users (estimated) |
|---|---|---|
| Llama 3.3 70B FP8 / 4 × H100 (320 GB) | 65k | 10–12 |
| 32k | 20–25 | |
| 16k | 40–50 | |
| GPT-OSS 120B MXFP4 TP=4 / 4 × H100 | 128k | 15–20 |
| GPT-OSS 120B AWQ TP=4 / 4 × L40S | 32–64k | 5–8 |
| GPT-OSS 20B MXFP4 / 2 × RTX 4090 | 64k | 2–3 |
KV-cache quantisation can roughly double capacity. Formula and further tables: LLM recommendations → VRAM requirements.
Methodology
What basebox lists as measured fulfils:
- Defined configuration: reference configuration, model, quantisation, backend version, basebox version, context, TP.
- Synthetic test data: no customer, patient or production data.
- Mixed load, not just isolated inference: chat plus document ingestion, OCR and STT simultaneously – this shows whether service models and inference interfere.
- Stability criteria: expected results per request, no workload restarts, recovery after a planned reboot.
- Recorded metadata: operational metadata and redacted summaries, no prompts, documents or answers.
Your own measurement
Measure on your hardware before committing to capacity:
- Configure model and context as in production; wait for the warm-up generation.
- Inference in isolation: with a load tool (e.g. the benchmark script of your vLLM version) measure tokens/s, time to first token and latency at 1, 4, 8, 16 concurrent requests with realistic prompt length.
- Mixed load: meanwhile load documents into a knowledge base, one OCR and one STT job – chat latency must not collapse.
- Long context: one request close to
AISRV_LLM_CONTEXT_SIZE. - Record GPU metrics (DCGM: utilisation, memory), pod restarts and error rates.
- Report the result with configuration to basebox – template under Using other models.
What to monitor permanently: Monitoring.
Next step: Using other models