Skip to content

// installation

Backup & restore

Applies to

Product: Server · Audience: Platform Operator

What must be backed up (databases, media, configuration, secrets), what can be recreated (model artefacts, caches), how often, where to – and how to restore. The one rule that outranks everything else: back up the AISRV database before every upgrade. Database migrations are forward-only; without a backup there is no way back.

What is backed up – and what is not

Back up Location Why
aisrv-db CloudNativePG cluster Users, organisations, apps, settings, audit log, connector credentials – the core
idp-db CloudNativePG cluster Identities, realms, clients – no login without them
storesrv-db CloudNativePG cluster Store server persistence
ragsrv-db CloudNativePG cluster (pgvector) Document sections and embeddings; alternatively re-ingest knowledge bases (takes time)
Media volume PVC under AISRV_MEDIA_ROOT Uploaded files
Kubernetes secrets aisrv-database, storesrv-database, ragsrv-database, idp-database, keycloak-admin-secret, basebox-admin-secret, TLS secret, inference/registry keys Without them a restored database is unusable
Values files Your Git (without secrets) Reproducible installation
Manifest Versions, image digests, GPU layout For rebuild and support
Do not back up (recreatable) Note
Model weights (inference-models, ragsrv-support-models) Reloadable – keep the mirror in air-gapped operation
Caches (Numba, Triton, /tmp/ragsrv) Rebuilt
Container images From the registry or the mirror

Automated backups per cluster to S3-compatible storage, in the values:

aisrv:
  aisrv-db:
    cluster:
      backup:
        barmanObjectStore:
          destinationPath: s3://backups/aisrv
          s3Credentials:
            accessKeyId:
              name: backup-s3-creds
              key: ACCESS_KEY_ID
            secretAccessKey:
              name: backup-s3-creds
              key: SECRET_ACCESS_KEY
        retentionPolicy: "30d"

Likewise for idp.idp-db, storesrv.storesrv-db, ragsrv.ragsrv-db. CloudNativePG allows scheduled backups (ScheduledBackup), continuous WAL archiving and point-in-time recovery; the procedure is in the CloudNativePG documentation. The S3 target can be an internal object store (e.g. MinIO) – in air-gapped operation the only path.

Method 2: dumps

For individual backups and before upgrades:

kubectl exec -n basebox aisrv-db-1    -- pg_dump -U aisrv    aisrv    > aisrv-backup.sql
kubectl exec -n basebox idp-db-1      -- pg_dump -U idp      idp      > idp-backup.sql
kubectl exec -n basebox storesrv-db-1 -- pg_dump -U storesrv storesrv > storesrv-backup.sql
kubectl exec -n basebox ragsrv-db-1   -- pg_dump -U ragsrv   ragsrv   > ragsrv-backup.sql

Dumps contain personal data and (write-only stored) credentials: store encrypted, restrict access.

Media and secrets

# media volume: with your volume snapshot tool or a copy from the pod
kubectl -n basebox exec deploy/aisrv -- tar czf - -C "$AISRV_MEDIA_ROOT" . > media-backup.tgz

# export secrets (store encrypted!)
kubectl -n basebox get secret -o yaml > basebox-secrets.yaml

Before every upgrade

  1. Dump of aisrv-db (and the other databases) or a fresh CloudNativePG backup.
  2. Optionally in addition AISRV_DB_MIGRATE_BACKUP: "true" with AISRV_DB_MIGRATE_BACKUP_DIR on a persistent volume – AISRV then backs up before migrations itself.
  3. Record values files and helm list -n basebox.

Concretely for basebox 1.8.6: the AISRV migration V32 cannot be undone without the prior backup. Procedure: Updates.

Frequency and retention (suggestion)

What Frequency Retention
Databases (CNPG) Daily full, WAL continuously 30 days; longer per your policies
Media volume Daily (snapshot) 30 days
Secrets, values, manifest On every change Versioned
Before upgrades Each time At least until the upgrade is accepted

Agree retention with data protection: backups contain the audit log and – depending on detail level – conversation content; deletion deadlines also apply to backups (Retention, Deletion).

Restore

Single database from dump:

kubectl -n basebox scale deploy/aisrv --replicas=0
kubectl exec -i -n basebox aisrv-db-1 -- psql -U aisrv aisrv < aisrv-backup.sql
kubectl -n basebox scale deploy/aisrv --replicas=1

From CloudNativePG backup: create a new cluster with bootstrap.recovery from the object store (optionally point-in-time), then point the services at it – following the CloudNativePG documentation.

Entire server (rebuild):

  1. Build the cluster along the bare-metal path up to and including Kubernetes.
  2. Apply the secrets (kubectl apply -f basebox-secrets.yaml) before installing basebox.
  3. Install basebox with the same values files and the same chart version (helm upgrade --install … --version <old>).
  4. Restore databases (dump or CNPG recovery), restore the media volume.
  5. Let models load (or from the mirror), acceptance check per Validate installation.

Keep data on uninstall: helm uninstall basebox -n basebox leaves the PVCs in place; a reinstall with the same values continues to use them. Delete PVCs only if you want the data loss.

Rehearse the restore

A backup that has never been restored is a hope. Monthly (or before every major upgrade) restore into a test environment or a test namespace, check login and one RAG answer, record the duration. That is also the evidence for audits.

Hosting or operation by basebox

With operation by basebox, backup target, frequency, retention and restore tests belong in the contract – see Operation by basebox. With hosting without an operations engagement, backups remain your task.

Next step: Updates