Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Deploy service models

Applies to

Product: Server · Audience: Platform Operator

Bringing up ragsrv, ragsrv-support, OCR, STT and embeddings on their GPU (or in CPU mode) and confirming their health. Service models are the middle layer: they need dedicated, predictable GPU capacity – separate from the inference GPU – but also run without a GPU.

What belongs to the service models

Service Task GPU
ragsrv Document ingestion, chunking, embeddings, semantic search; PostgreSQL with pgvector GPU or CPU (COMPUTE)
ragsrv-support Model service: document extraction, OCR, embeddings, speech-to-text GPU or CPU
OCR model service In some configurations a separately managed endpoint connected to ragsrv-support GPU (MIG instance)
Speech-to-text (Whisper) vLLM-compatible Whisper endpoint, connected to AISRV via AISRV_WHISPER_URL GPU (~8 GB) or MIG instance

Which services run as endpoints of their own depends on the reference configuration – the 4 × H100 describes OCR and Whisper endpoints as separately managed model endpoints per MIG instance. Component mapping: Service models.

Where they run: three variants

A – Dedicated physical GPU (e.g. RTX PRO 6000 in 2 × H200 + RTX PRO 6000): the service workloads request GPU resources on this card; the inference GPUs remain untouched.

B – MIG slices (the Validated configurations): each service requests one MIG instance:

# 2 × H200: one 1g.35gb slice each for GPU RAG, extraction, OCR, STT
resources:
  requests: {nvidia.com/mig-1g.35gb: 1}
  limits:   {nvidia.com/mig-1g.35gb: 1}
# 4 × H100 SXM: one 3g.40gb slice each
resources:
  requests: {nvidia.com/mig-3g.40gb: 1}
  limits:   {nvidia.com/mig-3g.40gb: 1}

Prerequisite: MIG set up and mixed discovery active – NVIDIA / GPU.

C – CPU mode (CPU-only application server, no GPU free):

ragsrv:
  enabled: true
  mode: cpu
  image:
    tag: cpu-latest
  env:
    COMPUTE: "cpu"
  resources:
    requests: {cpu: 4000m, memory: 16Gi}
    limits:   {cpu: 8000m, memory: 32Gi}

ragsrv-support:
  enabled: true
  mode: cpu

RAG and OCR remain functional but considerably slower. Size CPU and RAM by document volume.

Values for GPU mode (default)

ragsrv:
  enabled: true
  resources:
    requests: {cpu: 2000m, memory: 8Gi,  nvidia.com/gpu: 1}   # or MIG resource
    limits:   {cpu: 4000m, memory: 16Gi, nvidia.com/gpu: 1}
  env:
    COMPUTE: "gpu"
    SUPPORT_SERVICE_URL: "http://ragsrv-support:8000"
    WEBHOOK_STATE_URL: "http://aisrv:8888/rag/v1/state"
    REQUIRE_AUTH: "true"
    LOG_LEVEL: "info"
  nodeSelector:
    nvidia.com/gpu: "true"

ragsrv-support:
  resources:
    requests: {cpu: 4000m, memory: 16Gi, nvidia.com/gpu: 1}   # or MIG resource
    limits:   {cpu: 8000m, memory: 32Gi, nvidia.com/gpu: 1}
  env:
    LOG_LEVEL: "info"
    REQUIRE_AUTH: "true"

Important:

  • The API_KEY of ragsrv and ragsrv-support must match; take both from the same Kubernetes secret. AISRV talks to ragsrv with AISRV_RAG_API_KEY (header X-API-KEY).
  • ragsrv-support is enabled automatically when ragsrv is enabled.
  • Model cache for ragsrv-support as a PVC under /models (20–50 Gi); temporary files under /tmp/ragsrv – an RWX volume if pods run on different nodes.
  • Probes: ragsrv-support needs time for model downloads on first start; set initialDelaySeconds generously (60–90 s).
  • LOG_LEVEL at info in production – the default trace would log prompt content.

Full parameters: Ragsrv · Ragsrv-Support.

Connecting speech-to-text

AISRV reaches the Whisper service via AISRV_WHISPER_URL (default http://localhost:6000/inference; in the cluster the service of the Whisper endpoint) and optionally AISRV_WHISPER_API_KEY. Whisper needs about 8 GB GPU memory and can use a MIG instance or – if the LLM does not occupy all GPUs – a shared GPU (default split 85 % LLM / 10 % Whisper / 5 % buffer on an 80 GB GPU). The spoken language is detected automatically; recordings up to one hour.

Deploy

The service models are part of the umbrella chart: add the values and run helm upgrade --install as under Install basebox. Then:

kubectl -n basebox get pods -l 'app.kubernetes.io/name in (ragsrv,ragsrv-support)'
kubectl -n basebox logs -l app.kubernetes.io/name=ragsrv-support --tail=50 -f   # watch model download
kubectl -n basebox get cluster ragsrv-db
kubectl -n basebox describe pod -l app.kubernetes.io/name=ragsrv | grep -A5 "nvidia.com/"

Check health

kubectl -n basebox port-forward svc/ragsrv 3001:3001 &
curl -s http://localhost:3001/health
kubectl -n basebox exec -it <ragsrv-pod> -- nvidia-smi        # GPU mode: GPU/MIG visible

Functional check through the application: create an app with a knowledge base, upload a PDF, watch the processing status until "complete", ask a question and check the source chips; upload a scanned PDF or image (OCR); have a short audio file transcribed (STT). Reference values of the 2 × H200: 120-page PDF in 28.47 s, 60 minutes of audio in 31.31 s.

Common failure patterns

Symptom Cause Solution
ragsrv-support Pending No matching GPU/MIG resource free kubectl describe node – compare resources and requests
"Temporary error in name resolution" k3s DNS FAQ
Uploads stay in processing ragsrv cannot reach ragsrv-support, API_KEY mismatch Logs of both services; check the secret
Processing status does not arrive in the interface WEBHOOK_STATE_URL wrong Set to http://aisrv:8888/rag/v1/state
OCR/STT very slow CPU mode or shared inference GPU Provide dedicated GPU/MIG

Next step: Connect inference