Skip to content
Terminal

Production and releases

Production runs on the Mainline Hetzner account: CPX32 (4 vCPU, 8 GB), Helsinki, Ubuntu 24.04, protected server with daily Hetzner backups. Docker state stays on included NVMe. Infrastructure is managed by infra/terraform/terminal-production*.tf through reviewed Atlantis plans and applies.

Surface Address
Terminal https://terminal.mainline.inc
Mainframe https://mainframe.mainline.inc
Documentation https://docs.terminal.mainline.inc
Grafana https://mainframe.mainline.inc/grafana/
Private S3 mainline-terminal-production at https://hel1.your-objectstorage.com

Configuration and first boot

Section titled “Configuration and first boot”

bun run deploy init --production creates deploy/.env.production with independent secrets and mode 0600. It refuses to replace an existing file. Set the production IP, S3 credentials and external-provider credentials before use. Never commit it.

Production uses compose.production.yaml and Caddyfile.production. The deploy launcher rejects the world and automation profiles; neither world-sim nor auto-operator runs. It also rejects staging/archive storage credentials by bucket and endpoint. All other application modules, monitoring, voice recording and the forensic recorder are enabled by the production example. MinIO is unnecessary because production uses managed S3.

Provision fresh native Autel configuration with no imported controller accounts, a separate TAK CA/server certificate, and a dedicated Generic Drone database role. Use the module guides for account enrollment. Shared read-only Vanagas observers must have a production-specific MQTT client ID so staging stays connected.

Start Core first (bun run deploy up --production --profiles core), then provision the Generic Drone role/schema against the initialized default workspace. Fill in its workspace ID and database URL in the private environment. Copy the public Mainframe TLS certificate into the Generic Drone TLS directory and install its renewal timer. Configure the other modules, then:

Terminal window
bun run deploy config --production
bun run deploy up --production
bun run deploy seed --production
bun run deploy seed --production --apply

The production seed creates shared layouts, airframe definitions and a Briefing voice room. It creates no simulated aircraft, exercise missions, rules or map markings, and preserves existing documents and tombstones on repeat runs.

Explicit releases

Section titled “Explicit releases”

Staging’s Terminal and documentation Pages projects continue to build on pushes to main. Production has separate Git-integrated Pages projects, configured to build only the production branch, with preview deployments disabled. Production builds use VITE_MAINFRAME_URL=https://mainframe.mainline.inc.

  1. Merge and validate the chosen commit on main.

  2. Back up production. Export that exact commit to the server, preserve deploy/.env.production, bootstrap files and Docker volumes, and run bun run deploy up --production. Record the deployed SHA in .release-commit.

  3. Verify backend health, authentication, enabled providers and migrations.

  4. Preview and explicitly promote that commit:

    Terminal window
    bun deploy/release.ts --ref COMMIT --tag production/VERSION
    bun deploy/release.ts --ref COMMIT --tag production/VERSION --apply

The helper only accepts commits reachable from origin/main. It creates an immutable annotated release tag and atomically advances production, using an explicit lease to reject concurrent promotions. Both production Pages projects build that commit. Verify both builds and their live assets before announcing the release. No backend deploy is triggered by Git pushes.

To roll back the frontend, promote a previously verified main commit with a new release tag through the same helper. Check backend/API compatibility first. Database migrations need their own recovery decision; reverting source does not reverse schema or data changes.

Backups and restart recovery

Section titled “Backups and restart recovery”
Terminal window
bun deploy/recovery/backup.ts --env-file deploy/.env.production
bun deploy/recovery/restore.ts --env-file deploy/.env.production --prefix BACKUP_PREFIX
sudo sh deploy/recovery/install.sh deploy/.env.production

The backup verifies S3 readback checksums; the restore drill uses an isolated Postgres instance and never replaces the live database. The timer runs daily. Store the private environment and TAK CA signing key separately: they are not included in the application backup. Hetzner backups do not cover S3.

Install terminal-recording.service with its ExecStart passing deploy/.env.production to deploy/recording-up.sh. This reconciles recording containers that share the ingress network namespace after a reboot. Keep the Generic Drone TLS renewal timer enabled as well.