Licensed to be used in conjunction with basebox, only.
// installation
Recommended models
Applies to
Product: Server · Audience: Platform Operator
What basebox suggests for common scenarios – separate from what merely works. Recommendations change with new models and runtimes; they are based on the tests in the LLM recommendations (sizing reference) and the reference configurations. A recommendation is not a guarantee: verify the model on your hardware with your workload.
Quick selection by hardware class
| Your situation | Recommended model | Quantisation | Context | Concurrent users (guideline) |
|---|---|---|---|---|
| Enterprise – 4 × H100 (320 GB) | GPT-OSS 120B | MXFP4, TP=4 (128k) or TP=2 (32–64k) | 128k / 32–64k | 15–20 / 8–12 |
| Enterprise alternative | Llama 3.3 70B Instruct | FP8 | 32k–65k | 20–25 / 10–12 |
| Professional – 4 × L40S (192 GB) | GPT-OSS 120B | AWQ, TP=4 | 32–64k | 5–8 |
| Professional alternative | Llama 3.3 70B Instruct | FP8 | 32k–65k | 10–12 / 5–6 |
| Workstation – 2 × RTX 4090 (48 GB) | GPT-OSS 20B | MXFP4, TP=2 | 64k | 2–3 |
| Entry – 1 × 24 GB GPU (L4 or similar) | GPT-OSS 20B | MXFP4 | 32k | 1 |
| Faster response times | Qwen2.5 72B Instruct | Int8 | 32–65k | 10–12 |
| Minimal hardware, testing | Qwen3 4B Instruct | FP16 | 30k | 1 |
| Reasoning required | DeepSeek R1 Distill 8B / 70B | AWQ / FP8 | 32k | 1–2 / 10–12 |
The 141 GB class of the H200 (reference configurations 2 × H200) sits between Professional and Enterprise: one H200 carries Llama 3.3 70B FP8 with long context; two H200s with TP=2 GPT-OSS 120B MXFP4. The 96 GB class of the RTX PRO 6000 (2 cards = 192 GB) follows the L40S row. Both as orientation – Tested models says what has been measured.
Why GPT-OSS is the default recommendation
- Mixture of experts: 117B total with 5.1B active parameters (120B) or 21B/3.6B (20B) – fast inference for the model size.
- Fits with MXFP4 into ~60 GB (120B) or ~16 GB (20B) and leaves room for context.
- Reasoning-capable – thinking modes usable in basebox, provided the reasoning parser is configured in vLLM.
- Part of the basebox delivery and listed in the Model Register.
Prerequisites: vLLM 0.10.1+; tensor parallelism instead of data parallelism; TP must divide the 64 attention heads (1, 2, 4, 8).
Recommendations by use case
| Scenario | Recommendation | Rationale |
|---|---|---|
| RAG over large documents | As much context as possible (65k+) over concurrency; GPT-OSS 120B TP=4 or Llama 3.3 70B FP8 65k | More retrieved sections per question improve answer quality |
| Many concurrent users, short questions | Limit context to 16–32k; enable KV-cache quantisation | Capacity of concurrent users roughly doubles |
| Analyses, multi-step tasks | Reasoning model (GPT-OSS with reasoning parser, DeepSeek R1 Distill) | Thinking modes deliver more robust results; cost time and tokens |
| Connectors used intensively | Model with reliable tool calling (GPT-OSS, Llama 3.3 70B) | The assistant calls tools via function calls |
| Multilingual (DE/EN) | Llama 3.3 70B, Qwen2.5 72B, GPT-OSS | Solid German output; verify on your corpus |
| Two models side by side (default + reasoning or default + small) | Two inference instances on separate GPUs | See Multiple inference instances |
What is not recommended
- GGUF models in production – experimental in vLLM; prefer native FP8/AWQ/GPTQ.
- Data parallelism for GPT-OSS 120B – produces garbled output.
- Service models on the inference GPU – uploads and transcriptions then compete with the chat.
- Context larger than the runtime serves –
AISRV_LLM_CONTEXT_SIZEmust match the runtime context. - Choosing a model by benchmark alone – quality on your corpus, language and tone matter; test with real (synthetic) tasks.
Known quirks of the recommended models
| Model | Note |
|---|---|
| GPT-OSS | vLLM 0.10.1+; configure reasoning parser; tensor parallelism only |
| Llama 3.3 70B AWQ | May show high perplexity; adjust repetition penalty and temperature; prefer FP8 |
| DeepSeek-R1 | Backend must filter thinking tokens |
| Qwen 3 32B | Only BNB-4bit with vLLM; append /no_think to switch off reasoning |
In full: LLM recommendations → Known issues.
Recommendation ≠ verification
This page says where to start. Whether a combination was measured by basebox is under Tested models; numbers under Benchmarks. For models outside this list: Using other models.
Next step: Benchmarks