Updates, backups, and retention

A normal deployment rebuild preserves named volumes. Keep the Compose project name and environment file stable: changing the project name creates a different set of volumes and can look like an empty deployment.

Update static sites

Push the reviewed frontend or docs change to main. Cloudflare Pages builds and deploys both sites using the configuration managed in the infra repo. Check both Pages deployment results and verify the custom domains, docs navigation, and Terminal login against Mainframe. No server checkout or Docker command is needed for a static-only change.

Update the backend

bun run deploy backup
# Check out the reviewed release or commit.
bun run deploy config
bun run deploy up
bun run deploy ps

Mainframe’s image runs migrations before starting. Before an update, compare the deployed migration journal and SQL hashes with the release. When both match the applied database migration, the migration step is a no-op and existing data can be retained. The current schema is a single baseline migration: if that baseline changes, do not assume an in-place upgrade is supported; prepare and validate a data migration separately. A failed schema step should leave the service unhealthy rather than serving against an unexpected schema.

The live fragment ring uses only /data/media/.live as a disposable cache. An indexer restart clears that directory and refills it from publishing streams; database history and archived MP4 objects are preserved. Recordings from before the fragmented-MP4 release use native browser playback, while new recordings and the ring use MSE. Crossing an old recording boundary can briefly reload the video; it does not require rewriting stored media. Keep the same Compose project, credentials, and volumes, and never use down -v for a normal update.

After an update, verify login, module reconnects, telemetry, live video, and a new DVR segment. Check the logs and command failures. Restarting a process can interrupt live streams and pending commands even when all persisted data is intact.

Publish the Front app (optional)

<terminal>/get is the Android app’s download page: version, a Download button, and a QR code of the page itself, so a phone can pick it up from a screen or a printout. <mainframe>/get downloads the APK directly. The APK lives in the deployment’s bucket under downloads/front/; latest.json names the current one, older versions are kept. Publish only when the app changed — most backend updates skip this step; a fresh deployment publishes once.

From the machine that built the APK, with an administrator’s personal API key (Settings → Account → Keys):

MAINFRAME_KEY=... bun scripts/front/publish-apk.ts output/front/front-2.0-staging-arm64.apk \
  --mainframe https://mainframe.dev.mainline.inc

On the deployment host, without a key:

docker cp front-2.0.apk terminal-staging-mainframe-1:/tmp/front.apk
docker exec terminal-staging-mainframe-1 bun scripts/publish-front-apk.ts /tmp/front.apk 2.0

Or in Terminal: Settings → Users → Invites → Publish APK…. The same strip shows the published version, copies the /get link and shows its QR code. The invite page offers the download next to Open in Front. Check with bun scripts/front/publish-apk.ts --check --mainframe <url>. The APK carries no credentials (each phone gets its own when it joins), so the download is public. Build arm64 APKs (-PFRONT_ABIS=arm64-v8a): uploads stop at 120 MB.

Backups

bun run deploy backup writes a Postgres custom-format dump under the ignored deploy/backups/ directory with restricted permissions. Copy backups off the server; a local dump does not protect against losing the host.

For verified private off-host backups, run on the deployment host:

bun deploy/recovery/backup.ts --env-file deploy/.env.staging
bun deploy/recovery/restore.ts --env-file deploy/.env.staging --prefix backups/scheduled/PROJECT/BACKUP_ID
# After the restore drill passes, install the daily host timer as root:
sh deploy/recovery/install.sh

The backup command prints its prefix. It streams the custom-format database dump into the private bucket, downloads it to verify SHA-256 and size, and only then publishes a verified manifest. The scheduled job runs daily after 03:00 UTC, with up to 15 minutes of jitter. BACKUP_RETENTION_DAYS defaults to seven; cleanup only touches complete scheduled backups for this Compose project. Manual/initial backups and media objects are independent. A failed run preserves the previous success timestamp and its local working copy for diagnosis.

The manifest also inventories independently copied world-sim state, native Autel configuration/accounts, broker dynamic-security state, and deployed TAK server certificates (vanagas keeps no state beyond its captures). These artifacts contain credentials and belong only in the private bucket. The environment file, bucket bootstrap credentials, Terraform state and offline TAK CA signing key remain in the operator’s separate secret and infrastructure stores. Losing the host must not lose the only bucket key.

An online database dump is one database snapshot. Module files have separate capture times; this is not a coordinated application snapshot. Media stays in the private bucket, and its retention can invalidate old backup references. For a coordinated recovery point, quiesce application writers and preserve the database, module volumes, pending DVR files and object store together. Host backups do not replace backups of attached volumes or object storage.

The restore drill downloads and checks every backup object, restores into a temporary PostgreSQL container with no network and no exposed ports, counts restored tables, checks media metadata and twenty recent object references, and compares enabled controller accounts with broker clients. It writes a restricted JSON report and removes its temporary container/volume. It does not start modules or send aircraft commands. A module-state mismatch requires reconciliation before enabling automation in a recovered deployment.

Inspect systemctl status terminal-backup.timer terminal-backup.service and journalctl -u terminal-backup.service. The node-exporter textfile collector exposes last-success, latest-run and maintenance-success gauges; backup age alerts after 36 hours. If a killed process leaves deploy/backups/PROJECT/.offhost.lock, first verify that no backup process is running, then remove that empty lock directory before retrying. Retained failed upload prefixes have no successful manifest and require explicit operator cleanup.

Restore into a separate environment first. Restore database and object data before starting Core and modules, then verify a known historical replay and a newly recorded segment. Keep the previous environment available until that check succeeds.

Raw controller captures

Both Autel-protocol modules write raw captures, separate from operational logs. AUTEL_CAPTURE_RETENTION_HOURS and AUTEL_CAPTURE_MAX_BYTES bound the native host’s recognized hourly JSONL files, VANAGAS_CAPTURE_* the observer’s; zero disables each bound. The base deployment leaves them disabled; staging uses seven days and 2 GiB when no explicit environment override is present. Maintenance checks every five minutes and removes at most 24 closed files per run. The actual open file, the current hour, changed files, symlinks and unrelated files are protected. An oversized active file can exceed the limit until rotation. An incomplete inventory pauses deletion and reports failure instead of guessing.

Preview without deleting:

bun packages/autel-protocol/scripts/capture-retention.ts --directory /PATH/TO/CAPTURE --hours 168 --max-bytes 2147483648 --active-file /PATH/FROM/MODULE/HEALTH

The CLI never deletes; the running module owns deletion because it knows which file it is writing. Health and metrics expose the last successful inspection, retained bytes and failure status. Capture/maintenance failure is contained and does not interrupt MQTT telemetry or aircraft control.

Retention settings

Data	Default	Staging example
State history	90 days	90 days
Telemetry	14 days	30 days
Persisted bulk	48 hours	48 hours
Recorded media	168 hours	720 hours
Events, command audits, document history	No automatic age limit	No automatic age limit

MAINFRAME_STATE_RETENTION_DAYS, MAINFRAME_TELEMETRY_RETENTION_DAYS, MAINFRAME_BULK_RETENTION_HOURS and MAINFRAME_MEDIA_RETENTION_HOURS set the initial values on the first boot; afterwards change them in Terminal → Settings → Retention and Media, which the daily run reads. State partitions are monthly, so their actual horizon can exceed the configured age by up to a partition. Telemetry and bulk use daily partitions.

Retention runs during startup and daily. Media retention removes both indexed segments and their objects. To shorten the recording horizon immediately, use the existing media:retain tool with the deployment environment and review its output. Bucket lifecycle policies should not delete objects sooner than Core’s configured retention, or the database will retain references to missing recordings.

Validate container profile isolation without building images with bun run check:images --no-build (equivalently bun scripts/check-images.ts --no-build). The Compose CI job runs the same command.