Licensed to be used in conjunction with basebox, only.
// installation
Backup & restore
Applies to
Product: Server · Audience: Platform Operator
What must be backed up (databases, media, configuration, secrets), what can be recreated (model artefacts, caches), how often, where to – and how to restore. The one rule that outranks everything else: back up the AISRV database before every upgrade. Database migrations are forward-only; without a backup there is no way back.
What is backed up – and what is not
| Back up | Location | Why |
|---|---|---|
aisrv-db |
CloudNativePG cluster | Users, organisations, apps, settings, audit log, connector credentials – the core |
idp-db |
CloudNativePG cluster | Identities, realms, clients – no login without them |
storesrv-db |
CloudNativePG cluster | Store server persistence |
ragsrv-db |
CloudNativePG cluster (pgvector) | Document sections and embeddings; alternatively re-ingest knowledge bases (takes time) |
| Media volume | PVC under AISRV_MEDIA_ROOT |
Uploaded files |
| Kubernetes secrets | aisrv-database, storesrv-database, ragsrv-database, idp-database, keycloak-admin-secret, basebox-admin-secret, TLS secret, inference/registry keys |
Without them a restored database is unusable |
| Values files | Your Git (without secrets) | Reproducible installation |
| Manifest | Versions, image digests, GPU layout | For rebuild and support |
| Do not back up (recreatable) | Note |
|---|---|
Model weights (inference-models, ragsrv-support-models) |
Reloadable – keep the mirror in air-gapped operation |
Caches (Numba, Triton, /tmp/ragsrv) |
Rebuilt |
| Container images | From the registry or the mirror |
Method 1: CloudNativePG backups (recommended)
Automated backups per cluster to S3-compatible storage, in the values:
aisrv:
aisrv-db:
cluster:
backup:
barmanObjectStore:
destinationPath: s3://backups/aisrv
s3Credentials:
accessKeyId:
name: backup-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: backup-s3-creds
key: SECRET_ACCESS_KEY
retentionPolicy: "30d"
Likewise for idp.idp-db, storesrv.storesrv-db, ragsrv.ragsrv-db. CloudNativePG allows scheduled backups (ScheduledBackup), continuous WAL archiving and point-in-time recovery; the procedure is in the CloudNativePG documentation. The S3 target can be an internal object store (e.g. MinIO) – in air-gapped operation the only path.
Method 2: dumps
For individual backups and before upgrades:
kubectl exec -n basebox aisrv-db-1 -- pg_dump -U aisrv aisrv > aisrv-backup.sql
kubectl exec -n basebox idp-db-1 -- pg_dump -U idp idp > idp-backup.sql
kubectl exec -n basebox storesrv-db-1 -- pg_dump -U storesrv storesrv > storesrv-backup.sql
kubectl exec -n basebox ragsrv-db-1 -- pg_dump -U ragsrv ragsrv > ragsrv-backup.sql
Dumps contain personal data and (write-only stored) credentials: store encrypted, restrict access.
Media and secrets
# media volume: with your volume snapshot tool or a copy from the pod
kubectl -n basebox exec deploy/aisrv -- tar czf - -C "$AISRV_MEDIA_ROOT" . > media-backup.tgz
# export secrets (store encrypted!)
kubectl -n basebox get secret -o yaml > basebox-secrets.yaml
Before every upgrade
- Dump of
aisrv-db(and the other databases) or a fresh CloudNativePG backup. - Optionally in addition
AISRV_DB_MIGRATE_BACKUP: "true"withAISRV_DB_MIGRATE_BACKUP_DIRon a persistent volume – AISRV then backs up before migrations itself. - Record values files and
helm list -n basebox.
Concretely for basebox 1.8.6: the AISRV migration V32 cannot be undone without the prior backup. Procedure: Updates.
Frequency and retention (suggestion)
| What | Frequency | Retention |
|---|---|---|
| Databases (CNPG) | Daily full, WAL continuously | 30 days; longer per your policies |
| Media volume | Daily (snapshot) | 30 days |
| Secrets, values, manifest | On every change | Versioned |
| Before upgrades | Each time | At least until the upgrade is accepted |
Agree retention with data protection: backups contain the audit log and – depending on detail level – conversation content; deletion deadlines also apply to backups (Retention, Deletion).
Restore
Single database from dump:
kubectl -n basebox scale deploy/aisrv --replicas=0
kubectl exec -i -n basebox aisrv-db-1 -- psql -U aisrv aisrv < aisrv-backup.sql
kubectl -n basebox scale deploy/aisrv --replicas=1
From CloudNativePG backup: create a new cluster with bootstrap.recovery from the object store (optionally point-in-time), then point the services at it – following the CloudNativePG documentation.
Entire server (rebuild):
- Build the cluster along the bare-metal path up to and including Kubernetes.
- Apply the secrets (
kubectl apply -f basebox-secrets.yaml) before installing basebox. - Install basebox with the same values files and the same chart version (
helm upgrade --install … --version <old>). - Restore databases (dump or CNPG recovery), restore the media volume.
- Let models load (or from the mirror), acceptance check per Validate installation.
Keep data on uninstall: helm uninstall basebox -n basebox leaves the PVCs in place; a reinstall with the same values continues to use them. Delete PVCs only if you want the data loss.
Rehearse the restore
A backup that has never been restored is a hope. Monthly (or before every major upgrade) restore into a test environment or a test namespace, check login and one RAG answer, record the duration. That is also the evidence for audits.
Hosting or operation by basebox
With operation by basebox, backup target, frequency, retention and restore tests belong in the contract – see Operation by basebox. With hosting without an operations engagement, backups remain your task.
Next step: Updates