Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Configure models

Applies to

Product: Server · Audience: Platform Operator

The AISRV settings that bind basebox to a model – URL, provider, model identifier, context size, output limits, sampling defaults, handling of API keys – and their counterparts in the inference runtime. Both sides must match; the most common mistakes are a mismatched model identifier and a context size the runtime does not serve.

The AISRV side

Variable	Meaning	Example
`AISRV_LLM_URL`	Base URL of the endpoint	`http://inference:8000` (bundled) · `https://inference.customer.internal` (external)
`AISRV_LLM_CHAT_ENDPOINT`	Path of the chat completions	`/v1/chat/completions`
`AISRV_LLM_PROVIDER`	Determines the auth scheme	`vLLM` · `openai-compatible`
`AISRV_LLM_MODEL`	Model identifier – exactly as under `/v1/models`	`openai/gpt-oss-20b`
`AISRV_LLM_API_KEY`	API key of the endpoint, from a secret	`valueFrom.secretKeyRef`
`VLLM_API_KEY`	Provider-specific derivation for `vLLM`; the same secret	`valueFrom.secretKeyRef`
`AISRV_LLM_CONTEXT_SIZE`	Maximum context size in tokens; bounds what a chat can load; shown under "About"	`32768`
`AISRV_LLM_MAX_TOKENS`	Maximum output tokens (default 8000)	`8000`
`AISRV_LLM_WORD_LIMIT`	Estimated word limit per request	`2000`
`AISRV_LLM_TEMPERATURE`	Sampling default (0–1)	`0.7`
`AISRV_LLM_TOP_P`	Nucleus sampling (0–1)	–
`AISRV_LLM_REPETITION_PENALTY`	> 1.0 discourages repetition (relevant e.g. for Llama 3.3 70B AWQ)	–

All values go in aisrv.env of the Helm values. Reference: AISRV → LLM configuration.

The runtime side (bundled vLLM)

Variable	Meaning	Example
`MODEL_ID`	HuggingFace identifier of the model; must match `AISRV_LLM_MODEL`	`openai/gpt-oss-20b`
`MAX_INPUT_TOKENS`	Maximum input length; ≥ `AISRV_LLM_CONTEXT_SIZE`	`32000`
`NUM_GPUS`	Tensor parallelism	`1`, `2`, `4`
`SHM_SIZE`	Shared memory for GPU operations	`32gb`
`API_KEY`	Credential of the endpoint; the same secret as AISRV	–
`HF_TOKEN`	HuggingFace token for downloads (secret)	–
`HF_HOME`, `TRANSFORMERS_CACHE`	Cache paths on the model volume	`/data/.cache/huggingface`
`HF_HUB_OFFLINE`	`1` in air-gapped operation with pre-loaded models	–

Reference and examples (production, offline, small model, multi-GPU): Inference Server. With an external endpoint you configure the runtime there; AISRV only needs URL, provider, model, context and key.

What must match

AISRV	Runtime	Check
`AISRV_LLM_MODEL`	`MODEL_ID` or `/v1/models`	`curl $AISRV_LLM_URL/v1/models` from the AISRV pod
`AISRV_LLM_CONTEXT_SIZE`	Context the runtime actually serves (`MAX_INPUT_TOKENS`, `--max-model-len`)	Have a long synthetic text summarised
`AISRV_LLM_API_KEY` / `VLLM_API_KEY`	`API_KEY`	No 401
Tool calling expected (connectors)	Model and runtime support function calling	Call a connector in the chat
Thinking modes expected	Reasoning parser configured (e.g. for GPT-OSS)	Reasoning panel appears separately from the answer text

Choosing the context size

Context is a trade-off with concurrency: KV cache = context × concurrent users × cache per token. Example Llama 3.3 70B FP8 on 4 × H100: 65k → 10–12 users, 32k → 20–25, 16k → 40–50. For RAG workloads prioritise context (more sections per question); KV-cache quantisation can roughly double capacity. Tables: LLM recommendations → Context size vs. concurrent users.

AISRV_LLM_CONTEXT_SIZE must not exceed the runtime context; otherwise long requests abort. Users see the value under "About" and understand the context window through it.

Keys and secrets

All keys as Kubernetes secrets, referenced via valueFrom.secretKeyRef – never in clear text in values, never in Git. Rotation: set a new secret, helm upgrade, revoke the old key in the runtime. With vLLM as provider, both variables (AISRV_LLM_API_KEY, VLLM_API_KEY) to the same secret.

What administrators set beyond this

At application level, not in values: the organisation's default model, model per app, thinking effort and temperature per app – see Model configuration (administration). Your runtime values are the upper bounds within which the application works.

Changing the model

Size the new model (LLM recommendations); check GPU memory and cache volume.
Change MODEL_ID (or the external endpoint) and AISRV_LLM_MODEL, AISRV_LLM_CONTEXT_SIZE together.
helm upgrade in a maintenance window; wait for model download and warm-up (minutes).
Verify as under Connect inference → Verify; inform administrators that "About" shows the new model.
Mind the model's known quirks: Known issues.

Next step: Tested models