Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Supported inference backends

Applies to

Product: Server · Audience: Platform Operator

Which inference runtimes basebox is validated against, which are expected to work and what every OpenAI-compatible endpoint must fulfil. In short: vLLM is the validated backend; everything else is compatible provided it meets the requirements below – that does not make it verified.

Status of the backends

Backend Status Note
vLLM Validated Bundled in the chart (inference); proven as an external endpoint with basebox 1.7.1 and vLLM 0.15.0. GPT-OSS models require vLLM 0.10.1+ and a configured reasoning parser
TGI (Text Generation Inference) Maintenance mode The inference engine up to basebox 1.5; replaced by vLLM since 1.6. Not for new installations
Other OpenAI-compatible runtimes (e.g. SGLang, TensorRT-LLM with OpenAI frontend, llama.cpp server, Ollama) Compatible, not tested Expected to work if the requirements below are met; status Experimental until verified
External providers (OpenAI, Anthropic, Vertex AI) Compatible, to be checked contractually AISRV knows provider-specific auth schemes; closed-source models are not part of the delivery, data protection and contract lie with the customer

What every endpoint must fulfil

Regardless of backend, an endpoint AISRV is to use must provide:

Requirement Why
GET /v1/models returns the model identifier set in AISRV_LLM_MODEL AISRV checks and addresses the model through it
POST /v1/chat/completions in OpenAI format, with streaming (SSE) Interface and API stream answers
Message roles system, user, assistant, tool Connectors and API use tool messages
Tool calling (function calling), if connectors are to be used The assistant calls tools via the model's function calls
Reasoning output separate from the answer text (reasoning parser), if thinking modes are to be used Otherwise the train of thought ends up in the answer text
Context and output limits matching AISRV_LLM_CONTEXT_SIZE / AISRV_LLM_MAX_TOKENS Otherwise aborts with long context
API credential (bearer) No open endpoint
TLS when traffic crosses hosts or security zones Prompts and context travel over this path
Reachable only by AISRV Users never talk to inference directly
Health endpoint and runtime metrics Monitoring, probes, alerting
Stable latency under concurrency (batching) Several users at the same time

Also sensible: KV-cache quantisation (roughly doubles the capacity of concurrent users) and support for the native quantisation formats FP8, AWQ, GPTQ – GGUF is experimental in vLLM (single file only, limited features) and not recommended for production.

Providers in AISRV

AISRV_LLM_PROVIDER determines the authentication scheme towards the endpoint. Known values from the documentation examples:

Value Use
vLLM vLLM endpoints; current releases derive VLLM_API_KEY from it – set both key variables to the same secret
openai-compatible Generic OpenAI-compatible endpoint with bearer key

For Anthropic models, basebox adds no artificial instructions; a Vertex AI mode with tool call support has existed since 1.7. The full provider list is available from basebox.

What "validated" means for vLLM

  • The bundled inference component is a vLLM image from the basebox registry; model, GPUs, context and cache are set via MODEL_ID, NUM_GPUS, MAX_INPUT_TOKENS, HF_HOME – see Inference Server.
  • External vLLM endpoints are proven with the documented values example – see Deployment topologies.
  • The reference configurations and the LLM recommendations refer to vLLM.

Using another backend

  1. Check the requirements above against the runtime's documentation – especially tool calling, reasoning parser, streaming.
  2. Connect it as an external endpoint (Connect inference, variant B) so the bundled component stays untouched.
  3. Check GET /v1/models from the AISRV pod; test chat, long context, streaming, a tool call via a connector, reasoning display.
  4. Report the result with backend version, model and hardware to basebox – that is how Experimental may become Supported. Template: Using other models.

Next step: Configure models