Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Inference architecture

Applies to

Product: Server · Audience: Platform Operator

Where inference runs (same node, separate GPU host, separate cluster, external provider), how AISRV talks to it (OpenAI-compatible API) and where the data boundary lies. The architecture is deliberately built so that you can change or move the model without touching basebox.

The interface

AISRV calls inference via an OpenAI-compatible API:

  • GET /v1/models – which models the endpoint offers; the identifier must match AISRV_LLM_MODEL
  • POST /v1/chat/completions – chat completions, with streaming (Server-Sent Events) and tool calling if the model supports it

The endpoint requires an API key (bearer) that AISRV takes from a Kubernetes secret. What else AISRV needs to know about the model – context size, output limit, sampling defaults – is in the AISRV_LLM_* variables: Configure models.

Where inference can run

flowchart LR
  AISRV["AISRV<br/>basebox platform"]
  AISRV -->|"http://inference:8000"| A["A · Bundled vLLM<br/>same Kubernetes node"]
  AISRV -->|"https://…  TLS + API key"| B["B · Separate GPU host<br/>in the customer network"]
  AISRV -->|"https://…  TLS + API key"| C["C · Other cluster<br/>or other security zone"]
  AISRV -.->|"https://…  only where approved"| D["D · External provider"]
  style AISRV fill:#f4f2ee,stroke:#524e47,color:#1d1e1c
  style A fill:#dbeafe,stroke:#1e40af,color:#1d1e1c
  style B fill:#dbeafe,stroke:#1e40af,color:#1d1e1c
  style C fill:#dbeafe,stroke:#1e40af,color:#1d1e1c
  style D fill:#e5e7eb,stroke:#6b7280,color:#1d1e1c
Variant Description Typical for Data boundary
A – Bundled The chart deploys inference (vLLM) on the same node; traffic stays in the cluster Reference configurations, single node Within the server
B – Separate GPU host CPU-only application server + GPU host; AISRV talks to the endpoint over TLS Existing GPU system, separate management Both hosts in the same data processing boundary
C – Other cluster / security zone Like B, with a cluster or zone boundary; firewall only AISRV → endpoint Larger organisations Both zones approved
D – External provider An approved API provider with an OpenAI-compatible API Only where data protection and contract allow it Prompts and context leave your environment

A is the default and described under Connect inference; B and C with a worked vLLM example under Deployment topologies. D is a data protection and contractual decision of the organisation – see Model providers; closed-source models are not part of the basebox delivery.

What crosses the boundary

With every request, the following travels from the application server to the inference server: the prompt (system message, organisation-wide system prompt, personal instructions, user question), the history of the conversation, retrieved context from knowledge bases (the most relevant sections), content of uploaded documents and tool results from connectors. Tokens of the answer and, where applicable, reasoning content come back.

Therefore: if application server and inference are on different hosts, both belong in the same approved data processing boundary, the path is TLS-protected, and only AISRV may reach the endpoint. Users and browsers never talk to inference directly.

What inference does not see

  • No credentials of users or connectors – those stay in AISRV.
  • No identities beyond what is in the prompt.
  • No documents that were not retrieved for the question – RAG delivers sections, not knowledge bases.

Behaviour in operation

  • Cold start: models take several minutes to load on first start (2 × H200: about 6 minutes to warm-up). Probes with a long initialDelaySeconds; a warm-up generation before releasing to users.
  • No silent fallback: if the endpoint is offline, basebox shows the error and recovers as soon as the endpoint is ready again.
  • Health and metrics: the endpoint provides /health and runtime metrics; GPU metrics via DCGM – see Monitoring.
  • Streaming: AISRV streams tokens on to the interface and the API; ingress and proxies on the path must not buffer.

GPU layouts of inference

  • One whole GPU for one model – the normal case.
  • Tensor parallelism across two or more GPUs for large models or long contexts (NUM_GPUS); TP must divide the attention heads; choose the GPU pair with the fastest peer path – see Multi-GPU.
  • Several models or instances side by side – see Multiple inference instances.
  • Never MIG for the inference GPU in the reference configurations; MIG serves the service models.

Distinction from the service models

Embeddings, RAG processing, OCR and speech-to-text are not inference in the sense of this page. They run in ragsrv/ragsrv-support and their own model endpoints on a dedicated service GPU or MIG slices – see Service models. Changing the language model does not affect them.

Next step: Supported inference backends