Skip to content

Production and releases

Production runs on the Mainline Hetzner account: CPX32 (4 vCPU, 8 GB), Helsinki, Ubuntu 24.04, protected server with daily Hetzner backups. Docker state stays on included NVMe. Infrastructure is managed by infra/terraform/terminal-production*.tf through reviewed Atlantis plans and applies.

Surface Address
Terminal https://terminal.mainline.inc
Mainframe https://mainframe.mainline.inc
Documentation https://docs.terminal.mainline.inc
Grafana https://mainframe.mainline.inc/grafana/
Private S3 mainline-terminal-production at https://hel1.your-objectstorage.com

Production is the installation deploy/installations/production/installation.ts: its domain, IP, bucket, account email and the packages and modules it runs (Configuration and modules). It installs no Simulation; were it ever added, bun run deploy check production says so on a line of its own. bun run deploy init production writes deploy/installations/production/secrets.toml with independent generated credentials and mode 0600, and a SET_… placeholder for each credential a person supplies (S3 keys, OpenRouter, the Generic Drone database URL). It refuses to replace an existing file. Never commit it. A host that still runs from deploy/.env.production moves with the recipe in Move a host from its env file, with production for staging.

Production uses compose.production.yaml and Caddyfile.production. The launcher refuses a Compose project other than terminal-production, and staging or archive storage by bucket and endpoint. Every application module but the simulator, monitoring and voice recording are enabled by the installation. The bundled RustFS (base.storage.bundled) is unnecessary because production uses managed S3.

Provision fresh native Autel configuration with no imported controller accounts (deploy/autel/production/native.json; check refuses the installation without it), a separate TAK CA/server certificate, and a dedicated Generic Drone database role. Use the module guides for account enrollment. check also refuses the committed placeholder organizationId: "SET_WORKSPACE_ID": the Generic Drone workspace ID is set in the installation file once the workspace exists.

On a fresh host the workspace does not exist yet, so the first up runs without Generic Drone: take its key out of installation.ts on the host, bun run deploy up production, then provision the Generic Drone role/schema against the initialized default workspace. Put the module back with that workspace ID (and commit it), and set its database URL in secrets.toml. Copy the public Mainframe TLS certificate into the Generic Drone TLS directory and install its renewal timer. Configure the other modules, then:

Terminal window
bun run deploy check production
bun run deploy up production
bun run deploy seed production
bun run deploy seed production --apply

Mainframe seeds Feeds, Pilot, Operations and Analysis itself at first boot, with Feeds as the initial workspace layout; the production seed adds airframe definitions and a Briefing voice room. It creates no simulated aircraft, exercise missions or map markings, and preserves existing documents and tombstones on repeat runs.

Staging’s Terminal and documentation Pages projects continue to build on pushes to main. Production has separate Git-integrated Pages projects, configured to build only the production branch, with preview deployments disabled. Production builds use VITE_MAINFRAME_URL=https://mainframe.mainline.inc.

  1. Merge and validate the chosen commit on main.

  2. Back up production. Export that exact commit to the server, preserve deploy/installations/production/secrets.toml, bootstrap files and Docker volumes, and run bun run deploy up production. Record the deployed SHA in .release-commit.

  3. Verify backend health, authentication, enabled providers and migrations.

  4. Preview and explicitly promote that commit:

    Terminal window
    bun deploy/release.ts --ref COMMIT --tag production/VERSION
    bun deploy/release.ts --ref COMMIT --tag production/VERSION --apply

The helper only accepts commits reachable from origin/main. It creates an immutable annotated release tag and atomically advances production, using an explicit lease to reject concurrent promotions. Both production Pages projects build that commit. Verify both builds and their live assets before announcing the release. No backend deploy is triggered by Git pushes.

To roll back the frontend, promote a previously verified main commit with a new release tag through the same helper. Check backend/API compatibility first. Database migrations need their own recovery decision; reverting source does not reverse schema or data changes.

Terminal window
bun deploy/recovery/backup.ts --installation production
bun deploy/recovery/restore.ts --installation production --prefix BACKUP_PREFIX
sudo sh deploy/recovery/install.sh production

The backup verifies S3 readback checksums; the restore drill uses an isolated Postgres instance and never replaces the live database. The timer runs daily. Store secrets.toml and the TAK CA signing key separately: they are not included in the application backup. Hetzner backups do not cover S3.

Install deploy/systemd/terminal-recording.service and set Environment=TERMINAL_INSTALLATION=production in an override (systemctl edit terminal-recording.service); the unit’s default is staging. It runs bun run deploy up production --only recording after Docker starts, which reconciles the recording containers that share the ingress network namespace after a reboot. Keep the Generic Drone TLS renewal timer enabled as well.