Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Connect inference

Applies to

Product: Server · Audience: Platform Operator

Pointing AISRV at the inference endpoint – bundled vLLM on the same node or an external OpenAI-compatible endpoint on a separate GPU host. In both cases: users never talk to inference directly; AISRV calls it via the OpenAI-compatible API, and the model identifier must match on both sides.

Variant A: bundled vLLM

The umbrella chart deploys inference. You set model, GPUs and cache:

inference:
  enabled: true
  resources:
    requests: {cpu: 4000m, memory: 32Gi, nvidia.com/gpu: 1}   # 2 with TP=2
    limits:   {cpu: 8000m, memory: 64Gi, nvidia.com/gpu: 1}
  env:
    MODEL_ID: "openai/gpt-oss-20b"        # HuggingFace identifier
    MAX_INPUT_TOKENS: "32000"
    NUM_GPUS: "1"                          # = tensor parallelism
    SHM_SIZE: "32gb"
    HF_HOME: "/data/.cache/huggingface"
    HF_TOKEN:
      valueFrom:
        secretKeyRef: {name: hf-credentials, key: token}
    API_KEY:
      valueFrom:
        secretKeyRef: {name: llm-credentials, key: api-key}
  livenessProbe:
    httpGet: {path: /health, port: http}
    initialDelaySeconds: 300               # model load time
    periodSeconds: 30
  readinessProbe:
    httpGet: {path: /health, port: http}
    initialDelaySeconds: 180
    periodSeconds: 10
  volumes:
    - name: model-cache
      persistentVolumeClaim: {claimName: inference-models}
  volumeMounts:
    - name: model-cache
      mountPath: /data/.cache/huggingface

Point AISRV at the service:

aisrv:
  env:
    AISRV_LLM_URL: "http://inference:8000"
    AISRV_LLM_CHAT_ENDPOINT: "/v1/chat/completions"
    AISRV_LLM_PROVIDER: "vLLM"
    AISRV_LLM_MODEL: "openai/gpt-oss-20b"   # exactly as MODEL_ID
    AISRV_LLM_CONTEXT_SIZE: "32768"
    AISRV_LLM_MAX_TOKENS: "8000"
    AISRV_LLM_API_KEY:
      valueFrom:
        secretKeyRef: {name: llm-credentials, key: api-key}
    VLLM_API_KEY:
      valueFrom:
        secretKeyRef: {name: llm-credentials, key: api-key}

Current releases derive VLLM_API_KEY from the provider name vLLM; set both keys to the same secret. Further parameters (temperature, top-p, repetition penalty, word limit): Configure models. Offline operation with pre-loaded models (HF_HUB_OFFLINE: "1"): Inference Server.

Variant B: external endpoint

For a separate GPU host, another cluster or – where approved – an external provider: switch off bundled inference and point AISRV at the endpoint.

  1. Create the credential secret without writing the key into a file: 
    read -s VLLM_API_KEY
kubectl -n basebox create secret generic external-inference-api \
  --from-literal=api-key="$VLLM_API_KEY" --dry-run=client -o yaml | kubectl apply -f -
unset VLLM_API_KEY
  2. Values (values.external-inference.yaml): inference.enabled: false, AISRV_LLM_URL: "https://inference.customer.internal", provider, model, context, both API key entries from the secret; ragsrv/ragsrv-support in the desired mode.
  3. Render and check that no inference deployment is included.
  4. Install with both values files.
  5. Check the boundary: kubectl -n basebox get deployment inference → NotFound.

Requirements for the endpoint: OpenAI-compatible API (/v1/models, chat completions), API credential, TLS across host boundaries, same model identifier, matching context/output limits, health and metrics endpoints; reachable only by AISRV. Import a private CA into the basebox workload. The complete example, proven with basebox 1.7.1 / vLLM 0.15.0 / openai/gpt-oss-20b: Deployment topologies.

Choosing a model

Which model fits which GPU, with context and concurrency guidelines: LLM recommendations. What is verified, recommended or merely compatible: Models & inference. Known quirks (GPT-OSS needs vLLM 0.10.1+ and a reasoning parser; Qwen 3 /no_think; DeepSeek-R1 thinking tokens): Known issues.

Tensor parallelism (NUM_GPUS) must divide the attention heads – GPT-OSS 120B allows 1, 2, 4, 8. For two GPUs choose the pair with the fastest peer path (nvidia-smi topo -m).

Verify

Watch the load – several minutes on first start, cold-start tolerance of the 2 × H200 about 6 minutes:

kubectl -n basebox logs -l app.kubernetes.io/name=inference -f | grep -iE "download|loading|ready"

Check the model from inside AISRV without displaying credentials:

kubectl -n basebox exec deploy/aisrv -c aisrv -- sh -c '
  curl --fail --max-time 15 -H "Authorization: Bearer $VLLM_API_KEY" "$AISRV_LLM_URL/v1/models"'

The returned model identifier must match AISRV_LLM_MODEL. Then sign in to the interface, make a chat request and check under user menu → "About" that model and context size are correct.

Test failure behaviour: stop or block inference in a maintenance window – basebox shows the error and recovers as soon as the endpoint is ready again. There is no silent fallback to another model.

Common failure patterns

Symptom Cause Solution
Inference pod Pending GPU resource not free or wrong resource name kubectl describe node; check requests
Pod starts, then probe restarts initialDelaySeconds too short for the model download Probes at 300/180 s; PVC for model cache
"Model not found" in AISRV AISRV_LLM_MODEL ≠ identifier under /v1/models Copy exactly (case-sensitive)
401 from the endpoint Keys differ Both sides from the same secret
"data transmission failed (SSE)" in the chat Connection AISRV ↔ inference or ragsrv aisrv logs; network/TLS between the hosts
Long context aborts AISRV_LLM_CONTEXT_SIZE > runtime context Align the values

Next step: Validate installation