Licensed to be used in conjunction with basebox, only.
// installation
Models & inference
Applies to
Product: Server · Audience: Platform Operator
The inference layer is separate from basebox itself: AISRV talks to the language model via an OpenAI-compatible API – wherever it runs. This section keeps four things apart that are easily confused:
| Term | Meaning | Page |
|---|---|---|
| Compatible | What can run technically: any runtime implementing the OpenAI-compatible API in the form basebox uses | Supported inference backends |
| Tested | What basebox has actually verified – with version, backend, hardware, date | Tested models |
| Recommended | What basebox suggests for common scenarios – changes with new models | Recommended models |
| Measured | Numbers from reference configurations: throughput, latency, long context | Benchmarks |
A compatible model is not automatically tested; a tested one not automatically recommended. Each page says what it is.
In this section
- Inference architecture – where inference runs, how AISRV talks to it, where the data boundary lies
- Supported inference backends – vLLM as validated backend, what every endpoint must fulfil, external providers
- Configure models – the
AISRV_LLM_*settings and their counterparts in the inference runtime - Tested models
- Recommended models
- Benchmarks
- Using other models – checking, validating and reporting a model outside the lists
- LLM recommendations (sizing reference) – which model fits which GPU
The essentials in three sentences
The GPU question is an inference question. Model, quantisation, context length and concurrency determine VRAM – not basebox. The platform itself needs no GPU; the service models (RAG, OCR, STT) need their own predictable capacity that should not be shared with inference.
The model identifier must match. AISRV_LLM_MODEL must exactly match what the endpoint reports under /v1/models; otherwise "Model not found".
No silent fallback. If the endpoint is offline, basebox shows the error and recovers as soon as it is ready again. It does not switch to another model unnoticed – plan monitoring accordingly.
Who decides what
| Decision | Who |
|---|---|
| Which models are technically connected | Platform Operator (these pages) |
| Which model the organisation uses by default, which per app | Administrator – Model selection · Enabling models |
| Reasoning effort per chat | Users – Reasoning effort |
| Whether external model providers (OpenAI, Anthropic) are an option | The organisation, contractually and under data protection law – Model providers |
Legal framework
The models shipped by basebox are listed in the Model Register of the EU AI Act compliance package; basebox modifies models only through quantisation. Whoever retrains or fine-tunes a model assumes the role of provider – see Disclaimer. Closed-source models are not part of the delivery.
Next step: Inference architecture