Licensed to be used in conjunction with basebox, only.
// installation
Monitoring
Applies to
Product: Server · Audience: Platform Operator
What to watch: pod health, GPU utilisation and memory, inference latency, document processing queues, storage, certificates. From the services' health endpoints to Prometheus and NVIDIA DCGM – and which alerts you should define at minimum. The most important reason: when inference fails, basebox does not silently fall back to another model; you have to see it.
What you watch
| Area | Signal | Why |
|---|---|---|
| Pods | State, restart count, Pending/CrashLoopBackOff |
First symptom of almost every incident |
| Inference | /health, latency, time to first token, tokens/s, error rate, queue |
Users notice inference problems immediately; no fallback |
| GPU | Utilisation, memory usage, temperature, power per GPU/MIG instance | Memory pressure → OOM; heat → throttling |
| Service models | ragsrv/ragsrv-support healthy; processing status of uploads; duration per document/audio | Stuck processing blocks knowledge bases |
| Platform | AISRV /health, Keycloak /health/live and /health/ready, frontend /, ingress error rates |
Login and interface |
| Databases | CloudNativePG clusters healthy, replication lag, size, connection pool | Grow with documents and audit log |
| Storage | Fill level of PVCs (databases, media, model caches), /dev/shm |
Full volumes stop uploads and model downloads |
| Certificates | Remaining validity of the ingress certificate; with cert-manager the state of Certificate resources |
Expired certificate = no login |
| Nodes | CPU, RAM, disk, kubelet, network | Foundation for everything |
| Backups | Success and age of the last backup | See Backup & restore |
Health endpoints
| Service | Endpoint | Port |
|---|---|---|
| AISRV | /health |
8888 |
| Inference (vLLM) | /health |
8000 |
| ragsrv | /health |
3001 |
| ragsrv-support | /health |
8000 |
| Keycloak | /health/live, /health/ready (with KC_HEALTH_ENABLED=true) |
8080 |
| Frontend | / |
3000 |
Quick check:
kubectl -n basebox get pods -o wide
kubectl -n basebox port-forward svc/aisrv 8888:8888 & curl -s http://localhost:8888/health
kubectl -n basebox port-forward svc/inference 8000:8000 & curl -s http://localhost:8000/health
kubectl -n basebox get events --sort-by='.lastTimestamp' | tail -20
The Kubernetes probes use the same endpoints; for inference with a long initialDelaySeconds (300/180 s) because model loading takes minutes.
Metrics with Prometheus
Enable the exporters in the values:
aisrv:
env:
AISRV_METRICS_PORT: "9090" # Prometheus metrics from AISRV
aisrv-db:
cluster:
monitoring:
enablePodMonitor: true # CloudNativePG metrics; likewise storesrv-db, ragsrv-db, idp-db
idp:
env:
KC_METRICS_ENABLED: "true" # Keycloak /metrics
QUARKUS_MICROMETER_ENABLED: "true"
- GPU: NVIDIA DCGM Exporter (part of the GPU Operator or separate) provides utilisation, memory, temperature, power per GPU and MIG instance.
- Inference: vLLM exposes runtime metrics (requests, latency, tokens, queue, KV-cache usage) – take the endpoint from your runtime version.
- Kubernetes: kube-state-metrics and node-exporter for pods, restarts, node resources.
- Visualisation: Grafana; the standard dashboards of DCGM, CloudNativePG and Kubernetes are enough to start.
Without Prometheus: kubectl top pods -n basebox, kubectl top nodes, nvidia-smi dmon on the host or in the pod.
Alerts you need at minimum
| Alert | Threshold (suggestion) | Why |
|---|---|---|
Inference pod not Ready or /health failed |
> 2 minutes | Chat does not work; no fallback |
| GPU memory | > 90 % sustained | OOM on the next long request |
| GPU temperature | Manufacturer limit − 10 °C | Throttling |
| Inference latency (p95) or queue | Above your target | Capacity |
| PVC fill level | > 80 % | Full volume stops uploads/downloads |
| CloudNativePG cluster not healthy | Immediately | Risk of data loss |
| Certificate expiring | < 14 days | No login after expiry |
| Backup older than | 1 day (or your target) | Recoverability |
| Node NotReady | Immediately | Single node = everything |
What you do not need to monitor
The application level – who asks what, which apps are used, consumption in tokens – is what administrators see in the Dashboard and Audit log. That is their tool, not your monitoring.
Hosting or operation by basebox
With hosting at basebox, basebox monitors the physical side (power, cooling, network); the software stack follows the operating model. With operation by basebox, basebox monitors what is contractually agreed and reports alerts via the agreed path – see Operation by basebox.
Next step: Logging