Licensed to be used in conjunction with basebox, only.
// installation
Using other models
Applies to
Product: Server · Audience: Platform Operator
Running a model that is not on the tested list: what to check (API compatibility, context, tool calling, reasoning, quantisation, licence), how to validate it and how to report the result to basebox so that Experimental may become Supported. Technically, any model the inference runtime serves can run; whether it fits basebox is decided by the checklist.
1. Check in advance
| Check | Question | Where to look |
|---|---|---|
| Runtime support | Does vLLM (or your backend) support the architecture and the quantisation format? | vLLM documentation "Supported Models" |
| Quantisation | Native format (FP8, AWQ, GPTQ, MXFP4)? GGUF is experimental in vLLM – not for production | Model card |
| VRAM | Do weights + KV cache for context × concurrency fit on the inference GPU(s)? | LLM recommendations → VRAM requirements |
| Tensor parallelism | Does TP divide the attention heads? (GPT-OSS 120B: 1, 2, 4, 8) | Model configuration (num_attention_heads) |
| Chat template | Does the model ship a chat template for system/user/assistant/tool? |
Model card, tokenizer_config.json |
| Tool calling | Does the model support function calling reliably? Otherwise connectors do not work | Model card; vLLM tool parser |
| Reasoning | Reasoning model? Then configure the reasoning parser in vLLM, otherwise the train of thought ends up in the text | vLLM documentation "Reasoning Outputs" |
| Context | Maximum context of the model ≥ desired AISRV_LLM_CONTEXT_SIZE? |
Model card (max_position_embeddings) |
| Language | German of sufficient quality? | Own test with synthetic tasks |
| Licence | Commercial use allowed? Conditions (attribution, user count)? | Model licence |
2. Legal classification
- basebox ships only models that the original developer published and licensed, and modifies them only through quantisation (Model Register).
- If you connect another model, you are responsible for licence, data protection and deployment context (Disclaimer).
- Fine-tuning or retraining by you makes you a provider within the meaning of the EU AI Act – with the corresponding obligations.
- External providers (OpenAI, Anthropic): prompts and context leave your environment; clarify data protection and contract beforehand – Model providers.
3. Connect without disturbing production
- As a second inference instance on a free GPU or as an external endpoint – see Multiple inference instances and Connect inference, variant B.
- Configuration as under Configure models; model identifier exactly from
/v1/models. - Administrators assign the model at first only to a private test app (Advanced fields → Model), not as default model.
4. Validate
With synthetic test data, recorded as metadata:
| Check | Passed if |
|---|---|
GET /v1/models from the AISRV pod |
Model identifier appears |
| Chat, short questions (DE and EN) | Answers correct, style usable, streaming smooth |
Long context close to AISRV_LLM_CONTEXT_SIZE |
No aborts, content is taken into account |
| RAG app with knowledge base | Answer uses the retrieved sections and cites |
| Connector call (e.g. calculator) | Model calls the tool correctly, result flows into the answer |
| Reasoning (if the model supports it) | Train of thought appears in the reasoning panel, not in the answer text |
| Concurrency | Stable latency at 4–8 concurrent requests, no OOM errors |
| Failure behaviour | Stop the endpoint: error visible, no fallback; recovery after start |
| Reboot | Model loads after reboot without intervention (cache volume) |
Watch for typical quirks: missing stop tokens (endless output), repetitions (repetition penalty), thinking tokens in the text, wrong language.
5. Assign status and report
After a passed validation the model is Experimental for you – works, but not validated for production by basebox. Report the result to support@basebox.ai in the format of Tested models:
Model: <HuggingFace identifier>
Quantisation: <FP8 / AWQ / MXFP4 / …>
Backend: <vLLM x.y.z>
Hardware: <GPU model × count, TP>
Context: <verified AISRV_LLM_CONTEXT_SIZE>
Concurrency: <verified concurrent requests>
basebox: <chart / app version>
Date: <YYYY-MM-DD>
Verified features: chat · streaming · long context · tool calling · reasoning · RAG
Limitations: <…>
basebox may list the model as Supported after its own verification.
6. Take into production
Only then as default model or in widely shared apps – via the administrators' Model selection. Monitor latency and GPU memory in the first days; inform users about the change (the "About" page shows the active model).
Back: Models & inference