// docs
deploy
Everything operational lives in deploy/: one compose stack, three shapes (local, TLS,
tunnel), and a set of scripts with deliberately distinct verbs. This page is the digest; the
runbook in deploy/README.md is the authority.
topology
host
│ :80 (the ONLY published port; :443 in prod)
┌──────┴───────┐
│ traefik │ sole ingress, TLS/ACME + WS passthrough
└──┬───────┬───┘
edge network │ │
┌──────┴───┐ ┌┴─────────┐
│ relay │ │ minio │ s3.localhost: CK-encrypted artifact bytes
└──┬─────┬─┘ └──────────┘ written client→minio; relay never sees bytes
│ │
internal net │ │ (internal: true, no NAT, no host ports)
┌───────┴──┐ ┌┴──────────┐ ┌─────────────────┐ ┌──────────────┐
│ postgres │ │ openbao │ │hermes-companion │ │ embeddings │
└──────────┘ └───────────┘ │ (also on edge) │ │ (no egress) │
Transit KMS └─────────────────┘ └──────────────┘
Two networks carry the security boundary: everything reachable arrives through Traefik on
edge; Postgres, OpenBao, and the embeddings service join only internal and are unreachable
from the host or internet (SO-6). MinIO joins edge for direct browser access at s3.localhost;
artifact bytes never pass through the relay. The companion host (hermes-companion) joins both
networks: internal to reach the data plane, edge for external model API calls.
the three shapes
| shape | compose files | when |
|---|---|---|
| local | docker-compose.yml | validation on *.localhost, HTTP |
| production TLS | + docker-compose.prod.yml | three public origins (app/relay/s3), Let’s Encrypt, :443 |
| tunnel | + docker-compose.tunnel.yml | one origin behind Cloudflare Tunnel + Access, zero published ports; see remote coordination |
Production is the local stack plus overrides: one extra -f flag each, nothing forked.
production OpenBao, not -dev
The compose ships OpenBao in dev mode for local validation only. Production runs a real sealed
server on the baodata volume: bao operator init once (record the unseal shares and root
token out-of-band, shown exactly once), unseal on every start (auto-unseal preferred), then
mint the relay a scoped token allowing only encrypt/decrypt on the relay-atrest Transit
key. preflight-prod.sh rejects a deployment still carrying the dev root token.
backups: the part you must not skip
Three durable off-device stores hold your data:
- Postgres (
pgdata): every member’s sealed keystore and all ciphertext events. The relay cannot reconstruct any of it. - OpenBao
baodata: therelay-atrestTransit key that decrypts every at-rest row. Postgres rows without the matching key are permanently undecryptable. - MinIO
miniodata: the CK-encrypted artifact bytes for every managed file. These bytes never pass through the relay, so they are not in the Postgres dump.
A complete backup is three things, paired:
- the Postgres dump:
./deploy/backup.sh /mnt/backups - the OpenBao
baodatasnapshot (therelay-atrestkey), taken automatically by the same run, with a matching timestamp - the MinIO
miniodatavolume: artifact bytes; snapshot separately with a matching timestamp (e.g.mc mirrorto off-host storage)
A Postgres-and-OpenBao-only restore recovers keystores and events but loses all managed files and artifacts permanently. Prune all three together, store all three off-host, and treat the OpenBao snapshot as more sensitive than the ciphertext dump.
(Product-side note: deploy/backup.sh currently snapshots Postgres and OpenBao but does not
yet snapshot miniodata. Until that is added, MinIO must be backed up separately.)
0 3 * * * /path/to/repo/deploy/backup.sh /mnt/backups >> /var/log/cozylabs-backup.log 2>&1
Targets: RPO 24h by default (tighten to hourly if your write volume warrants), RTO 1h from fresh host to validated stack. On full host loss, restore OpenBao first, then Postgres, then MinIO.
upgrades & rollback
Migrations are forward-only and auto-apply on boot: there are no down-migrations, so you do not roll back by redeploying old code onto a newer schema (ADR-0008). The upgrade dance takes its own rollback point first:
./deploy/backup.sh /mnt/backups # 1. backup FIRST, this is your rollback point
git pull # 2. the new release
docker compose -f deploy/docker-compose.yml up -d --build # 3. rebuild; migrations apply
./deploy/validate-prod.sh # 4. confirm (prod)
Rollback = previous checkout + restore the pre-upgrade dump. The Transit key is untouched by app upgrades.
the scripts, by verb
| script | kind | run it |
|---|---|---|
bootstrap.sh | setup | once, first run: generates deploy/.env secrets |
render-prod-config.sh | setup (prod) | after setting domains; SHAPE=tunnel for single-origin |
preflight-prod.sh | static check | before up: validates .env, no running stack needed |
validate-prod.sh | dynamic check | after up: read-only against the live deployment |
backup.sh / restore.sh | ops | before every upgrade; restore to roll back |
disaster-drill.sh | proof / beta gate | rehearse full host-loss recovery on a throwaway stack |
e2e-local.sh | ⚠ destructive, dev/CI | ephemeral boot→smoke→teardown; wipes volumes, never on a live host |