Production and releases
Production runs on the Mainline Hetzner account: CPX32 (4 vCPU, 8 GB), Helsinki,
Ubuntu 24.04, protected server with daily Hetzner backups. Docker state stays on
included NVMe. Infrastructure is managed by infra/terraform/terminal-production*.tf
through reviewed Atlantis plans and applies.
| Surface | Address |
|---|---|
| Terminal | https://terminal.mainline.inc |
| Mainframe | https://mainframe.mainline.inc |
| Documentation | https://docs.terminal.mainline.inc |
| Grafana | https://mainframe.mainline.inc/grafana/ |
| Private S3 | mainline-terminal-production at https://hel1.your-objectstorage.com |
Configuration and first boot
Section titled “Configuration and first boot”Production is the installation deploy/installations/production/installation.ts:
its domain, IP, bucket, account email and the packages and modules it runs
(Configuration and modules). It installs no
Simulation; were it ever added, bun run deploy check production says so on a
line of its own. bun run deploy init production writes
deploy/installations/production/secrets.toml with independent generated
credentials and mode 0600, and a SET_… placeholder for each credential a person
supplies (S3 keys, OpenRouter, the Generic Drone database URL). It refuses to
replace an existing file. Never commit it. A host that still runs from
deploy/.env.production moves with the recipe in
Move a host from its env file,
with production for staging.
Production uses compose.production.yaml and Caddyfile.production. The
launcher refuses a Compose project other than terminal-production, and
staging or archive storage by bucket and endpoint. Every application module but
the simulator, monitoring and voice recording are enabled by the installation.
The bundled RustFS (base.storage.bundled) is unnecessary because production
uses managed S3.
Provision fresh native Autel configuration with no imported controller accounts
(deploy/autel/production/native.json; check refuses the installation without
it), a separate TAK CA/server certificate, and a dedicated Generic Drone database
role. Use the module guides for account enrollment. check also refuses the
committed placeholder organizationId: "SET_WORKSPACE_ID": the Generic Drone
workspace ID is set in the installation file once the workspace exists.
On a fresh host the workspace does not exist yet, so the first up runs
without Generic Drone: take its key out of installation.ts on the host,
bun run deploy up production, then provision the Generic Drone role/schema
against the initialized default workspace. Put the module back with that
workspace ID (and commit it), and set its database URL in secrets.toml. Copy the public
Mainframe TLS certificate into the Generic Drone TLS directory and install its
renewal timer. Configure the other modules, then:
bun run deploy check productionbun run deploy up productionbun run deploy seed productionbun run deploy seed production --applyMainframe seeds Feeds, Pilot, Operations and Analysis itself at first boot, with Feeds as the initial workspace layout; the production seed adds airframe definitions and a Briefing voice room. It creates no simulated aircraft, exercise missions or map markings, and preserves existing documents and tombstones on repeat runs.
Explicit releases
Section titled “Explicit releases”Staging’s Terminal and documentation Pages projects continue to build on pushes
to main. Production has separate Git-integrated Pages projects, configured to
build only the production branch, with preview deployments disabled. Production
builds use VITE_MAINFRAME_URL=https://mainframe.mainline.inc.
-
Merge and validate the chosen commit on
main. -
Back up production. Export that exact commit to the server, preserve
deploy/installations/production/secrets.toml, bootstrap files and Docker volumes, and runbun run deploy up production. Record the deployed SHA in.release-commit. -
Verify backend health, authentication, enabled providers and migrations.
-
Preview and explicitly promote that commit:
Terminal window bun deploy/release.ts --ref COMMIT --tag production/VERSIONbun deploy/release.ts --ref COMMIT --tag production/VERSION --apply
The helper only accepts commits reachable from origin/main. It creates an
immutable annotated release tag and atomically advances production, using an
explicit lease to reject concurrent promotions. Both production Pages projects
build that commit. Verify both builds and their live assets before announcing the
release. No backend deploy is triggered by Git pushes.
To roll back the frontend, promote a previously verified main commit with a new release tag through the same helper. Check backend/API compatibility first. Database migrations need their own recovery decision; reverting source does not reverse schema or data changes.
Backups and restart recovery
Section titled “Backups and restart recovery”bun deploy/recovery/backup.ts --installation productionbun deploy/recovery/restore.ts --installation production --prefix BACKUP_PREFIXsudo sh deploy/recovery/install.sh productionThe backup verifies S3 readback checksums; the restore drill uses an isolated
Postgres instance and never replaces the live database. The timer runs daily.
Store secrets.toml and the TAK CA signing key separately: they are not
included in the application backup. Hetzner backups do not cover S3.
Install deploy/systemd/terminal-recording.service and set
Environment=TERMINAL_INSTALLATION=production in an override
(systemctl edit terminal-recording.service); the unit’s default is staging. It
runs bun run deploy up production --only recording after Docker starts, which
reconciles the recording containers that share the ingress network namespace
after a reboot. Keep the
Generic Drone TLS renewal timer enabled as well.