Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

Deployment Topologies

basebox and the LLM do not have to run on the same server.

The standard Helm installation places the basebox application services and the bundled inference service in one Kubernetes environment. You can also run basebox on a CPU-only application server and connect AISRV to a protected OpenAI-compatible inference endpoint on a separate GPU server.

Choose A Topology

Topology Use when GPU preparation
One server and one cluster Evaluation or compact appliance Prepare that server for GPU workloads
One cluster with CPU and GPU nodes Kubernetes already provides node separation Prepare only the GPU nodes
Separate application and inference servers The GPU system is managed separately or belongs to another security zone The basebox application server can be CPU-only

The application server still runs basebox, identity, databases, storage, and RAG services. Only LLM inference moves to the GPU server.

Users
  |
  | HTTPS
  v
basebox application server
  - frontend
  - AISRV
  - Keycloak
  - storesrv
  - ragsrv and databases
  |
  | authenticated OpenAI-compatible API
  v
Dedicated GPU inference server
  - vLLM or another compatible runtime
  - model weights and model cache

Users and browsers connect to basebox, not directly to the inference server.

CPU-Only Application Server

The Server Preparation Guide describes a combined server that runs GPU workloads. For an application-only basebox server:

  • install Kubernetes, an ingress controller, storage, and CloudNativePG;
  • do not install NVIDIA drivers, CUDA, NVIDIA Container Toolkit, or GPU Operator unless that server will run another GPU workload;
  • run ragsrv and ragsrv-support in CPU mode, or place their accelerated services on a GPU host separately;
  • make sure AISRV can resolve and reach the inference endpoint;
  • import the inference endpoint's private CA into the basebox workload if it uses a private TLS certificate.

CPU mode is functional for RAG and OCR, but processing is slower than a dedicated GPU configuration. Size CPU, RAM, and storage for the number and size of documents you expect to process.

Network And Security Requirements

For separate servers, the inference endpoint must:

  • implement the OpenAI-compatible /v1/models and chat-completions API used by the selected basebox release;
  • be reachable from AISRV, but not exposed directly to end users;
  • require an API credential;
  • use TLS when traffic crosses hosts or security zones;
  • return the same model identifier configured in AISRV;
  • support compatible context and output limits;
  • expose health and runtime metrics to the operations team.

Treat prompts, retrieved context, and uploaded document content as data that crosses from the application server to the inference server. Both systems must therefore be inside the approved data-processing boundary.

Configure External vLLM Inference

The example below was exercised with basebox 1.7.1, vLLM 0.15.0, and openai/gpt-oss-20b. Replace the endpoint, model, and limits with values that the target runtime actually reports.

1. Create The Credential Secret

Create the namespace and secret before installing basebox. Read the key without putting it in a values file:

kubectl create namespace basebox --dry-run=client -o yaml | kubectl apply -f -

read -s VLLM_API_KEY
kubectl -n basebox create secret generic external-inference-api \
  --from-literal=api-key="$VLLM_API_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -
unset VLLM_API_KEY

2. Add The Helm Values

Create values.external-inference.yaml:

inference:
  enabled: false

aisrv:
  env:
    AISRV_LLM_URL: "https://inference.customer.internal"
    AISRV_LLM_CHAT_ENDPOINT: "v1/chat/completions"
    AISRV_LLM_PROVIDER: "vLLM"
    AISRV_LLM_MODEL: "openai/gpt-oss-20b"
    AISRV_LLM_CONTEXT_SIZE: "32768"
    AISRV_LLM_WORD_LIMIT: "2000"
    AISRV_LLM_API_KEY:
      valueFrom:
        secretKeyRef:
          name: external-inference-api
          key: api-key
    VLLM_API_KEY:
      valueFrom:
        secretKeyRef:
          name: external-inference-api
          key: api-key

ragsrv:
  enabled: true
  mode: cpu

ragsrv-support:
  enabled: true
  mode: cpu

Current basebox releases derive VLLM_API_KEY from the vLLM provider name. Set both API-key entries to the same Secret so the values remain compatible with the general AISRV setting and the provider-specific runtime lookup.

Combine this file with the domain, TLS, storage, and administrator settings from Using Helm Charts.

3. Render Before Installing

helm template basebox \
  oci://gitea.basebox.health/basebox-distribution/helm/basebox.ai \
  --version 0.3.19 \
  --namespace basebox \
  -f values.customer.yaml \
  -f values.external-inference.yaml \
  > /tmp/basebox-rendered.yaml

rg -n 'AISRV_LLM_URL|VLLM_API_KEY|name: inference' \
  /tmp/basebox-rendered.yaml

Review the render. It must contain the external endpoint and secret references, and it must not contain an inference Deployment.

4. Install

helm upgrade --install basebox \
  oci://gitea.basebox.health/basebox-distribution/helm/basebox.ai \
  --version 0.3.19 \
  --namespace basebox \
  --wait \
  --timeout 120m \
  -f values.customer.yaml \
  -f values.external-inference.yaml

5. Validate The Boundary

kubectl -n basebox get pods
kubectl -n basebox get deployment inference

The second command must return NotFound. Then validate the remote model from AISRV without displaying the credential:

kubectl -n basebox exec deploy/aisrv -c aisrv -- sh -c '
  curl --fail --max-time 15 \
    -H "Authorization: Bearer $VLLM_API_KEY" \
    "$AISRV_LLM_URL/v1/models" >/dev/null
'

Complete the normal login and chat smoke test. Also stop or block the inference endpoint during an approved maintenance window and confirm that basebox exposes the failure and recovers after inference becomes ready again. basebox does not silently fall back to another model when the configured endpoint is offline.

TLS Without Internet Access

Let's Encrypt is not required for a private basebox installation.

  • Use global.tls.mode=existing-secret when your organization supplies a certificate from its internal CA.
  • Use global.tls.mode=local for evaluation or self-managed local trust. Each client device must trust the issuing CA.
  • Keep the basebox ingress certificate and inference-endpoint certificate as separate trust decisions. AISRV must trust both paths it uses.

See Using Helm Charts and the Local Quick Start for the corresponding installation paths.

Production Checklist

Before using a separated deployment in production, verify:

  • private DNS and certificate ownership;
  • firewall rules from AISRV to inference only;
  • secret rotation without writing credentials to Git or Helm values;
  • model identity, context limit, output limit, streaming, and timeouts;
  • inference health, GPU metrics, logs, alerting, and capacity limits;
  • basebox and database backup and rollback procedures;
  • failure and recovery behavior;
  • RAG, OCR, and STT placement and resource ownership;
  • the documented data-processing boundary.

The same topology works when the inference endpoint is provided by another Kubernetes cluster or an approved API provider. Security, latency, capacity, and data-governance requirements still apply.