Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Multiple inference instances

Applies to

Product: Server · Audience: Platform Operator

Running several models, or several instances of one model, side by side – for example a fast default model plus a reasoning model, or two instances of the same model for more concurrency. This page describes what applies from the platform's point of view, which patterns are technically sound and where the line between documented and still open lies.

What AISRV sees

For the basebox platform, inference is one OpenAI-compatible endpoint: AISRV_LLM_URL, an API key from a secret, and the model identifiers the endpoint reports under /v1/models. Which models administrators can enable under Enable models follows from that list; the default model applies wherever an app sets nothing else.

Hence the principle of this page: several models means one endpoint that reports several models. How that endpoint is built internally – one runtime, several runtimes behind a router, several replicas – is a matter for the inference layer, not the platform.

Three patterns

Pattern What it solves Setup Status
A · Two models, two runtimes, one router Default model + reasoning model; small model for bulk tasks, large one for hard ones One vLLM instance per model on its own GPU(s); in front an OpenAI-compatible router that dispatches by model and reports all models under /v1/models Supported – routing details pending DevOps confirmation
B · One model, several replicas More concurrent users at the same context Several identical instances behind a Kubernetes Service or load balancer; each instance its own GPU(s) Supported
C · One model, cut larger More context or throughput from one instance Tensor parallelism across several GPUs – see Multi-GPU Validated (4 × H100, TP=2)

Pattern C is often the better answer to "too slow" than pattern B: vLLM batches requests continuously, and one large instance uses GPU memory better than two small ones with weights loaded twice.

Pattern A: two models

Prerequisites

  • Enough GPU memory per model for weights and KV cache – two models on one GPU compete for the same memory; better one GPU (or set of GPUs) per model. The LLM recommendations list the need per model and quantization.
  • A router that speaks the OpenAI API, forwards requests by the model field and returns the combined model list. It must require an API credential and be reachable only by AISRV – the same requirements as for any external endpoint (Connect inference).
  • The identifiers under /v1/models must match exactly what administrators enable in basebox.

Setup

  1. Second runtime as its own deployment with its own GPU request (nvidia.com/gpu or a MIG profile), its own model cache and its own MODEL_ID. The chart's bundled inference carries one model; a second instance goes alongside it.
  2. Router in front; point AISRV at the router (AISRV_LLM_URL), key from secret.
  3. Check from the AISRV pod: 
    curl -s -H "Authorization: Bearer $KEY" $AISRV_LLM_URL/v1/models | jq '.data[].id'
    Both identifiers must appear.
  4. Enable both models in the administration, set the default, assign the reasoning model to individual apps (Default models).

Context size and output limit apply in AISRV per endpoint (AISRV_LLM_CONTEXT_SIZE, AISRV_LLM_WORD_LIMIT). With two models of different context, set the value to the smaller one – or clarify with basebox whether it can be configured per model.

Pattern B: replicas

  • Each instance needs its own GPU(s) and loads the weights itself – memory is not shared.
  • A Kubernetes Service in front of identical pods distributes requests; streaming responses run over one connection, so simple distribution without session affinity suffices.
  • The model cache per node must be present (Storage); cold starts take several minutes per instance – probes and warm-up accordingly.
  • Measure first: if one instance shows neither GPU saturation nor a queue under your load, a replica gains nothing – see Scaling.

What applies to all patterns

  • No silent fallback. If an instance fails, basebox shows the error for requests to that model; it does not switch unnoticed to another one.
  • Users never talk to an instance directly – only to AISRV.
  • Data processing boundary. All instances and the router sit inside the same approved boundary as the application server; paths across hosts via TLS (Inference architecture).
  • Service models are not inference. RAG, OCR and speech-to-text run separately on the dedicated service GPU and are unaffected by these patterns.
  • Keep a manifest: which instance holds which GPU(s), which model, which version.

Verify

  • /v1/models returns all expected identifiers.
  • One request per model from the AISRV pod answers; the load appears on the right GPU (nvidia-smi dmon).
  • In basebox both models can be enabled and chosen per app; the switch is visible in the chat.
  • Failure test: stop one instance → requests to that model report an error, the other model keeps working.

Common failure patterns

Symptom Cause Solution
Model missing in the administration Identifier not under /v1/models or spelled differently Check the router list; identifier exact
Second instance does not start (OOM) Both models on one GPU Own GPU(s) or MIG per instance
Answers come from the wrong model Router ignores the model field Router configuration; test with curl per model
Slow despite second instance Load was not GPU-bound Measure first, then scale

Next step: Multi-node