Skip to content

Deploy staging

Terminal and this documentation are static Cloudflare Pages sites. Mainframe staging runs Core, Postgres, SRS, the media indexer, Caddy, world-sim (with its operator), and monitoring on its own Hetzner server. DVR and review objects use a private Hetzner S3 bucket. It uses real credentials and the normal authentication path.

Surface Address
Terminal https://terminal.dev.mainline.inc
Mainframe https://mainframe.dev.mainline.inc
Documentation https://docs.terminal.dev.mainline.inc/
Grafana https://mainframe.dev.mainline.inc/grafana/
Simulator administration https://mainframe.dev.mainline.inc/sim/admin

These are the configured targets. DNS and TLS must be provisioned before the addresses will work.

The separate infra repository owns the Git-integrated Pages projects (terraform/terminal-pages.tf), custom domains, Hetzner server, firewall, and object storage (terraform/terminal-staging.tf). Its Terminal staging resources reuse the account, SSH keys, and providers used by the CC clusters while creating a separate server and private bucket.

Review the Terraform plan before applying it. The resulting server IP is host.publicIp in deploy/installations/staging/installation.ts. Follow capacity planning when choosing server and disk sizes. The server needs Docker with Compose 2.24.4 or newer and Bun 1.4.2 for the deployment launcher.

Public traffic uses TCP 80 and 443 for HTTP/TLS, UDP and TCP 8000 for WebRTC in staging (MEDIA_RTC_PORT; the default is 8000), and TCP 1935 for authenticated RTMP publishing. SSH access should match the administrator CIDRs configured in Terraform. Postgres, SRS’s API, Prometheus, and exporters are not public services.

Both Pages projects use the repository root, Bun 1.4.2, SKIP_DEPENDENCY_INSTALL=1, and the main production branch. Every push to main rebuilds and publishes both sites without touching Docker. Pull requests receive separate preview builds.

Project Build command Output
terminal-staging bun install --frozen-lockfile && bun run --cwd apps/web build apps/web/dist
terminal-docs-staging bun install --frozen-lockfile && bun run --cwd docs build docs/dist

Terminal’s build environment sets VITE_MAINFRAME_URL=https://mainframe.dev.mainline.inc. Mainframe continues to allow the Terminal origin through its configured CORS/auth settings; preview origins are not automatically trusted for staging login. The old Terminal /docs/ URLs redirect to the documentation domain.

The Hetzner console name is mainframe-staging. Its Terraform resource address, /opt/terminal checkout, terminal-staging Compose project, and existing volumes retain their original identities. Renaming the console label does not change the public IP or DNS.

Place the reviewed Terminal checkout on the server. What staging runs is its installation file, deploy/installations/staging/installation.ts, committed in the repository: the server IP, domains, bucket, account email and the packages and modules (Configuration and modules). From the checkout’s root:

Terminal window
bun install --frozen-lockfile
bun run deploy init staging

This writes deploy/installations/staging/secrets.toml with mode 0600: every credential the launcher can generate, and a SET_… placeholder for each one a person supplies (the S3 keys). It refuses to overwrite an existing file. Fill the placeholders and keep the file outside version control. A host that still runs from deploy/.env.staging moves with init staging --from deploy/.env.staging instead (Move a host from its env file).

The bucket endpoint and region must match, for example https://hel1.your-objectstorage.com and hel1. The bucket must already exist. Leave base.storage.presign unset (false) to serve authenticated playback through Mainframe without configuring public bucket CORS.

Terminal window
bun run deploy check staging
bun run deploy up staging
bun run deploy ps staging

check validates the installation and its secrets and parses every container’s rendered environment with that module’s own parser. up renders deploy/installations/staging/.rendered/ (one environment file per container, the Compose profiles, the Prometheus targets and the edge’s module routes), builds backend images, and waits for service health checks. Caddy uses the stock image and serves only Mainframe, module routes, and monitoring; it does not build Terminal or docs. A normal rebuild preserves named volumes.

If local map assets are required, build them before deployment using the repository’s map tooling. The stack can otherwise use the frontend’s configured online basemap.

Terminal window
bun run deploy seed staging
bun run deploy seed staging --apply

The first command shows missing records. The second creates them through Mainframe’s authenticated document and fleet APIs. It adds ten SIM-STG-001 through SIM-STG-010 aircraft, one staging flight, four short patrol loops, and the AO area marking that Mainframe Settings → Configuration → Workspace names as the area of operations. Re-running preserves existing documents and fleet identities.

The simulator’s operator dispatches only world-sim’s own aircraft, and Mainframe refuses a module a command for another provider’s entity. Real aircraft from another provider are never eligible. The aircraft start at different battery levels; four missions leave six aircraft available for charging and rotation.

Sign in to Terminal with the admin email of the installation file and the [base.admin] password of secrets.toml, then open Profile… in the Terminal menu and enable two-factor authentication before anyone else signs in; Configuration → Security → Two-factor requirement set to Admins enforces it for every later administrator. Create the people’s accounts in Mainframe Settings → Users: each gets a temporary password shown once and must change it at the first sign-in. Confirm the virtual fleet appears, missions become assigned, telemetry advances, and a live feed plays. After at least one DVR segment closes, enter replay and confirm video and positions match the selected time.

In Grafana, inspect service availability, module connections, S3 uploads, simulator publishers, automation freshness, and mission coverage. Check an actual charge → dispatch → return → recharge cycle before calling the continuous run accepted. A long replay history requires a real soak run; an initial healthy boot proves only that the stack started.

Use updates and backups for subsequent releases and real aircraft before enabling the autel module.

The map directory is excluded from Git and must be populated on each new host:

Terminal window
bun scripts/mapctl.ts fetch --area lt --name lt
bun scripts/mapctl.ts assets

Run from /opt/terminal. This installs the full Lithuania extract plus the fonts and sprites in deploy/map, mounted read-only at Caddy’s /srv/map. --area lt matters: the default ao only covers the southeast operating area. Caddy serves /map/* directly with Range and CORS support; native Autel /map/api/v1/* routes continue to go to Autel.

Verify a range request to /map/lt.pmtiles returns 206 with PMTiles magic, and that /map/fonts/Noto%20Sans%20Regular/0-255.pbf and /map/sprites/v4/black.json return 200. A frontend rebuild is unnecessary when installing basemap assets; reload the browser to retry a cached fallback.

Park the simulator for real test operations

Section titled “Park the simulator for real test operations”

On the staging host, from /opt/terminal:

Terminal window
bun deploy/park-world.ts
bun deploy/park-world.ts --apply

--installation <name> names another installation; staging is the default. The first command previews the affected world-sim entities. Apply saves a Postgres dump, a copy of installation.ts and secrets.toml and an identity inventory in deploy/backups/park-world-<timestamp>/ and stops world-sim with its operator. It retires the provider’s entities through Mainframe’s provider protocol, updating connected clients without restarting Mainframe. No real provider’s entities are changed. The command can be rerun after a partial failure.

Retired aircraft are hidden by default in Assets; simulator sites, radars and detections also leave the live registry. Missions, markings, airframes, recordings and the simulator volume are preserved. Existing history/media retention still applies; parking is not an indefinite archival hold.

The world stays parked until the next bun run deploy up staging, which starts world-sim again while simulation is in the installation file. To keep it parked, remove the simulation key from deploy/installations/staging/installation.ts (its secrets may stay; they only warn). To bring the simulator back, put the key back and run:

Terminal window
bun run deploy up staging
bun run deploy logs staging -- world-sim

World-sim reads its retained volume and announces its assets again. Mainframe reactivates existing identities (or creates them if normal retention has already pruned them). Do not clear volumes or reseed merely to restart. Leave world-sim’s operator setting out for an idle simulator. Add it back only when automatic simulator mission dispatch is wanted. Verify world-sim health, live assets and video after restart; restarting was intentionally not part of parking the simulator.

Staging is the standing demo source for the air picture. World-sim runs the standing air world: radars of every type, RF, EO/IR, acoustic, ADS-B, Remote ID, FLARM and Network Remote ID sensors, the IXL-4 launcher and the RJ-120 jammer near VNO, live ADS-B traffic from adsb.lol (80 nm, at most 80 aircraft, with staging’s adsb.lol key) and the real wind from the weather module (Open-Meteo’s paid plan, with staging’s key). The continuous demo keeps the picture moving without resets:

  • cooperative traffic: the live airliners, which the simulated EYVI PSR/SSR and C-UAS radars see as well as ADS-B; Remote ID quads; a permitted inspection drone; a FLARM glider;
  • non-cooperative traffic: quads, a fixed-wing, a bird flock and a balloon, staggered, each with a bounded lifetime;
  • about every 10 minutes a radio-silent quad or fibre FPV flies into the VNO zone and earns a high threat and an effector recommendation;
  • nothing engages by itself. A launcher reloads to full 60 s after its last engagement ends; a jammer is ready again when its engagement ends.

The same demo runs on a laptop with bun run demo (Laptop demo).

The seeded live fusion configuration confirms tracks by AP4’s fast rules (MOGS’ 2-second clock). For a customer demo where the bird flock and the ground clutter should stay quiet, set Mainframe Settings → Fusion → Confirmation → Track confirmation to Conservative (before AP4) and save; it is a revision of the live document and survives redeploys (the seed only creates missing documents); Fast reverts it (the two sets).

Fusion runs the Rust core on the DEM. The simulator’s sensors are seeded sources (apps/mainframe/seeds/staging/surveillance.sources.json) that send to the fusion container. Real aircraft and phones (autel, vanagas, vanagas-phones, front) are not touched: the operator dispatches only world-sim’s own aircraft, and the demo’s targets exist only inside world-sim.

Run everything below on the staging host as root, from /opt/terminal. The host runs the rendered Compose command with two release overrides appended, not bun run deploy up. .rendered/dc is one line: docker compose with the rendered compose.env and the Compose files the installation selects.

Terminal window
cd /opt/terminal
SHA=<full release commit>; TAG=release-${SHA:0:8}; NAME=air-picture-$(date -u +%Y%m%dT%H%MZ)
I=deploy/installations/staging
DC="$(cat $I/.rendered/dc) -f deploy/compose.ux-release.json -f deploy/compose.fleet-release.json"
curl -s https://mainframe.dev.mainline.inc/api/deployment; echo # retentionHold: see "Preservation hold" below
df -h / /srv/terminal-data; nproc; free -g
Terminal window
umask 077; B=deploy/backups/$NAME; mkdir -p $B
cp -a $I/installation.ts $I/secrets.toml $I/.rendered deploy/compose*.yaml deploy/compose*.json deploy/Caddyfile.staging $B/
$DC ps --format json > $B/ps.json
docker exec terminal-staging-postgres-1 pg_dump -U mainframe -d mainframe -Fc > $B/mainframe.dump
docker exec -i terminal-staging-postgres-1 pg_restore --list < $B/mainframe.dump | head -5

From a machine with the repository:

Terminal window
git archive --format=tar $SHA | ssh root@89.167.55.244 "mkdir -p /opt/terminal/releases/$SHA && tar -x -C /opt/terminal/releases/$SHA"

On the host, copy the release over the checkout and record it. Host-only files (deploy/installations/staging/secrets.toml and .rendered/, the two override JSON files, deploy/backups/, deploy/map/) are not in the archive and stay; a release carries the committed installation.ts, so keep host edits to it committed; files a release dropped stay too and are harmless:

Terminal window
cp -a releases/$SHA/. /opt/terminal/
echo $SHA > releases/current-revision

One at a time, at low priority, from the release directory. Every Dockerfile builds with Docker’s legacy builder, so the host needs no buildx. Fusion’s first build compiles the Rust core (a clean emulated x86_64 build took 4 minutes); its dependencies are a layer of their own, so a release that changes only the core’s sources rebuilds only the core.

Terminal window
cd /opt/terminal/releases/$SHA
nice -n 10 docker build -f apps/mainframe/Dockerfile --build-arg MAINFRAME_BUILD=${SHA:0:8} -t terminal-staging-mainframe:$TAG .
nice -n 10 docker build -f services/fusion/Dockerfile -t terminal-staging-fusion:$TAG .
nice -n 10 docker build -f services/world-sim/Dockerfile -t terminal-staging-world-sim:$TAG .
nice -n 10 docker build -f services/weather/Dockerfile -t terminal-staging-weather:$TAG .
nice -n 10 docker build -f services/radar-net/Dockerfile -t terminal-staging-radar-net:$TAG .
nice -n 10 docker build -f services/tak/Dockerfile -t terminal-staging-tak:$TAG .
cd /opt/terminal

The Copernicus DEM GLO-30 tile set (about 23 MB, free with attribution; dem/manifest.json carries the attribution) goes to a host directory that fusion and world-sim mount read-only:

Terminal window
mkdir -p /opt/terminal/shared/terrain
(cd releases/$SHA && bun install --frozen-lockfile && bun scripts/terrain/fetch-dem.ts --out /opt/terminal/shared/terrain)
python3 -c 'import json;m=json.load(open("/opt/terminal/shared/terrain/dem/manifest.json"));print(m["levels"],len(m["tiles"]))'

Or copy a set fetched elsewhere: rsync -a data/terrain/ root@89.167.55.244:/opt/terminal/shared/terrain/.

What staging runs is deploy/installations/staging/installation.ts; a release that changes it carries the change. Edit it only through a commit, then render:

Terminal window
cp -a $I/secrets.toml $I/secrets.toml.pre-$NAME
bun install --frozen-lockfile
bun run deploy render staging
grep -cE '^WORLD_SIM_(AIR_WORLD|DEMO|DOMAINS)=' $I/.rendered/env/world-sim.env # 3

render prints the profiles and runs check first and refuses a missing or unfilled secret, then writes the environment files, the profiles, targets/services.json (Prometheus reads it within 15 seconds) and caddy/routes.caddy. FUSION_THREADS=4 (packages["air-picture"].modules.fusion.threads) on the 8-vCPU host leaves four cores to ingest, Mainframe, Postgres and media; the default rule would take six.

The two supplied keys live in secrets.toml: Open-Meteo’s paid plan under [base.modules.weather] openMeteoApiKey and adsb.lol’s under [packages.simulation.modules.world-sim] liveTrafficApiKey. They are kept outside git, on Jonas’s Mac, in /Users/j/Code/terminal/deploy/.env.staging.air-picture-keys (mode 600, git-ignored), one KEY=value line each (WEATHER_API_KEY, WORLD_SIM_LIVE_TRAFFIC_API_KEY). Copy each value into its line of secrets.toml by hand, never print them, and render again. Check that both reach their containers by counting, not reading:

Terminal window
grep -c "^WEATHER_API_KEY='.\+'" $I/.rendered/env/weather.env # 1
grep -c "^WORLD_SIM_LIVE_TRAFFIC_API_KEY='.\+'" $I/.rendered/env/world-sim.env # 1

With the key the weather module asks customer-api.open-meteo.com with &apikey=, and world-sim sends adsb.lol’s key as the x-api-key header. Neither is logged; the health checks in step 8 say keyed instead.

Reload Prometheus so it reads the release’s alert rules as well:

Terminal window
$DC kill -s SIGHUP prometheus

Grafana finds the Mainframe · Air picture dashboard in the release’s dashboard folder by itself.

The two override files pin images to releases. Print them, then point each rebuilt service at $TAG, adding the service where it is missing:

Terminal window
for f in deploy/compose.ux-release.json deploy/compose.fleet-release.json; do cp -a $f $f.pre-$NAME; python3 -c "import json;d=json.load(open('$f'));print('$f',{k:v.get('image') for k,v in d.get('services',{}).items()})"; done
cat > /root/air-pin.py <<'PY'
import json, sys
p, tag = sys.argv[1], sys.argv[2]; d = json.load(open(p)); svc = d.setdefault("services", {})
for name in ["mainframe", "media-indexer", "fusion", "world-sim", "weather", "radar-net", "tak"]:
image = "terminal-staging-mainframe" if name == "media-indexer" else f"terminal-staging-{name}"
svc.setdefault(name, {})["image"] = f"{image}:{tag}"
json.dump(d, open(p, "w"), indent=2)
PY
python3 /root/air-pin.py deploy/compose.fleet-release.json $TAG && rm /root/air-pin.py
$DC config --quiet && $DC config --format json | python3 -c 'import json,sys;[print(k,v.get("image")) for k,v in sorted(json.load(sys.stdin)["services"].items())]'

The later file wins where both pin a service. Check the printed list: the seven services above on $TAG, every other service on its old tag. If RECORDING_MAINFRAME_IMAGE is set in the environment it names the Mainframe image as well; set it to terminal-staging-mainframe:$TAG.

Mainframe first (it applies the additive migration 0019 at boot), then the documents, then the air world, then the fleet check:

Terminal window
$DC up -d --no-deps --no-build mainframe media-indexer
until [ "$(docker inspect -f '{{.State.Health.Status}}' terminal-staging-mainframe-1)" = healthy ]; do sleep 5; done
$DC exec -T mainframe bun src/data-tool.ts seed --profile staging --documents-only
$DC exec -T mainframe bun src/data-tool.ts seed --profile staging --documents-only --apply
$DC up -d --no-deps --no-build weather fusion world-sim radar-net tak
$DC exec -T mainframe bun src/data-tool.ts seed --profile staging
# the release's Caddyfile adds <mainframe>/licenses; a reload is graceful
$DC exec -T caddy caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
$DC exec -T caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile

The preview lists only missing documents; check it before --apply. Expect the effector types, the IX-1 interceptor airframe, the sensor types, the live fusion configuration, the whitelist, the UTM permits, the 17 simulator sources and the VNO zone marking (0–150 m AGL, high priority). Existing documents and tombstones are kept. The last command checks the virtual fleet once world-sim is online and should have nothing to create.

Terminal window
curl -s https://mainframe.dev.mainline.inc/api/health | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d["status"],d["version"]);[print(k,v) for k,v in d["checks"].items()]'
$DC exec -T fusion bun -e 'const h=await (await fetch("http://127.0.0.1:8799/api/health")).json();console.log(h.status,JSON.stringify(h.core),JSON.stringify(h.heights))'
T=$(sed -n "s/^WORLD_SIM_ADMIN_TOKEN='\(.*\)'$/\1/p" $I/.rendered/env/world-sim.env)
for p in demo live wind; do curl -s -H "authorization: Bearer $T" http://127.0.0.1:8787/api/sim/air/$p | head -c 600; echo; done
$DC exec -T weather bun -e 'const h=await (await fetch("http://127.0.0.1:8791/api/health")).json();console.log(h.source,JSON.stringify(h.openMeteo),JSON.stringify(h.forecast))'
$DC logs --tail 100 world-sim | grep -E 'air world|world config|terrain'
curl -s https://mainframe.dev.mainline.inc/api/licenses; echo
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://mainframe.dev.mainline.inc/licenses
$DC exec -T prometheus wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=up{job="mainframe-services"}' | python3 -c 'import json,sys;[print(r["metric"]["service"],r["value"][1]) for r in json.load(sys.stdin)["data"]["result"]]'

Expect:

  • Mainframe ok, the providers check listing world-sim, weather and fusion beside the real providers;
  • fusion core.impl rust, threads 4, heights.terrain dem-… (not null) and no fallback;
  • the demo enabled: true with targets, live traffic enabled with aircraft and its feed.keyed: true, lastError null; wind mode real;
  • weather open-meteo, openMeteo.keyed: true, the forecast’s lastError null;
  • /api/licenses: terrain the Copernicus attribution, liveTraffic ["adsb.lol"], openMeteo: true; /licenses answers 302 to https://terminal.dev.mainline.inc/licenses?mainframe=…;
  • air world: standing with the continuous demo in world-sim’s log;
  • Prometheus up 1 for fusion, weather and world-sim, and Grafana’s Mainframe · Air picture dashboard filled: fusion memory against its 4 GB limit, the Rust core with 4 threads, step times, tracks, the demo’s spawns and the live ADS-B feed.

If the demo, live traffic or wind is off, the world-sim volume holds a setting saved earlier (it wins over the environment); turn it on with the POST under “Pause the demo”.

Then set staging’s state history to 30 days (Jonas, 2026-11-08): sign in to Terminal as an admin, Mainframe Settings → Configuration → Retention, state 30 days. The running value lives there; a new database seeds the code’s 90 days. It applies to every asset, the real hardware’s history included. See “Preservation hold, retention and growth”.

In Terminal, signed in:

  1. Airliners near Vilnius are fused tracks whose composition names both eyvi-adsb and eyvi-psr.
  2. Within 3 minutes of world-sim’s start the demo’s quads, the bird flock, the balloon, the glider (FLARM) and Remote ID drones are tracks.
  3. A threat appears, the first about 3 minutes in, then about every 10 minutes. Tracks puts it first; Selection → Threat reads High with the time to VNO; the recommendation offers IXL-4 VNO.
  4. Engage With… → IXL-4 VNO: the interceptor launches and flies to it. The launcher reads 3/4, then reloading a minute after the engagement ends, then 4/4.
  5. Assets shows the real aircraft and phones as before, and no mission was dispatched to anything but SIM-STG-….
  6. About Terminal → Licences opens /licenses, which lists the Copernicus DEM (its notice “Produced using Copernicus WorldDEM-30 …”), adsb.lol (ODbL), Open-Meteo (CC BY 4.0) and EGM2008, then the map and fonts.
  7. The map’s attribution (ⓘ) carries the basemap’s credits only: no Copernicus line and no live-traffic notice, with Air tracks and Sensor coverage on.
Terminal window
cp -a $I/secrets.toml.pre-$NAME $I/secrets.toml
cp -a $B/.rendered/. $I/.rendered/
for f in deploy/compose.ux-release.json deploy/compose.fleet-release.json; do cp -a $f.pre-$NAME $f; done
cp -a $B/Caddyfile.staging deploy/Caddyfile.staging && $DC exec -T caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
$DC stop fusion world-sim
$DC up -d --no-deps --no-build mainframe media-indexer weather radar-net tak

The old release’s installation.ts comes back with its checkout; $B/.rendered is what that release rendered. The old release runs on migration 0019, which is additive. Seeded documents and fusion’s fusion schema stay and are harmless. Restoring $B/mainframe.dump loses everything since the backup and is Jonas’s call.

To stop the simulator, use bun deploy/park-world.ts and then --apply (above); fusion keeps running with silent sources. To keep the world but stop the demo’s traffic and threats:

Terminal window
curl -s -X POST -H "authorization: Bearer $T" -H 'content-type: application/json' -d '{"enabled":false}' http://127.0.0.1:8787/api/sim/air/demo

The setting persists in world-sim’s volume and wins over the environment at the next start. {"enabled":true} turns the demo back on; POST /api/sim/air/live and /api/sim/air/wind take {"enabled":true} and {"mode":"real"} the same way.

A preservation hold (/api/deployment reads retentionHold: true, or the mode is archive) stops Mainframe’s history retention and the pruning of retired entities, and a boot path that would delete blobs makes Mainframe refuse to start. This release adds no such path: the seed only creates documents, and fusion’s raw-log retention (FUSION_RAW_RETENTION_DAYS) reads the same hold and deletes nothing while it is on, or while it cannot read it. With the hold on, the demo’s tracks grow Postgres and the bucket without bound. Watch them:

Terminal window
df -h /srv/terminal-data
docker exec terminal-staging-postgres-1 psql -U mainframe -d mainframe -c "select pg_size_pretty(sum(pg_total_relation_size(oid))) from pg_class where relname like 'samples_state%' and relkind = 'r'"
docker stats --no-stream terminal-staging-fusion-1 terminal-staging-world-sim-1 terminal-staging-mainframe-1 terminal-staging-postgres-1

Without a hold, Mainframe Settings → Configuration → Retention bounds history; a new database seeds the code’s values (90 days of state, 14 of telemetry), so change retention there. Staging keeps 30 days of state history (Jonas, 2026-11-08): the continuous demo makes about 1,800 track entities a day and, at about 50 tracks, a few GB of samples_state a day, so 90 days would approach the disk. The shorter value shortens the real hardware’s history too.

Before installation files a host ran from one env file, deploy/.env.staging (production: deploy/.env.production; an archive: its own). The installation file and secrets.toml replace it. Nothing in the database changes, so the move is a recreation of containers, not a migration. On the host, from /opt/terminal:

  1. Update the checkout to a release with deploy/installations/. Restart nothing.

  2. Write the secrets from the env file:

    Terminal window
    bun run deploy init staging --from deploy/.env.staging

    It takes every credential from the env file and prints three lists: variables whose value differs from what the committed installation.ts renders (NAME: env has X, installation renders Y), variables that no longer exist (the auto-operator key and serial selection, the first-boot retention values), and variables of modules the installation does not enable.

  3. Resolve each difference in installation.ts and commit it, or accept it. Repeat until only intended differences are left.

  4. Compare what every container receives today with what it would receive:

    Terminal window
    bun deploy/launcher/env-diff.ts --old-ref <the deployed commit> --old-env deploy/.env.staging --installation staging

    It runs docker compose config on the deployed commit’s deploy/ tree with the env file and on the rendered installation, and prints each difference with the rule that allows it; it exits non-zero on any difference no rule allows. Secret values print as hashes. If the host ran the vanagas observers, they appear as removed services: Legacy is not rendered, and this is the moment they stop.

  5. bun run deploy render staging, then append the release override files to $(cat deploy/installations/staging/.rendered/dc) as above.

  6. Recreate the containers in the order of the release steps above. The first recreation of weather and radar-net uses their own profile names, which the rendered compose.env already has.

  7. Keep deploy/.env.staging until the installation has run for a day, then delete it.

Rollback is the previous checkout with the old env file. Staging’s first-boot retention values (30 days of telemetry, 720 hours of media in the old example) no longer exist: a new database seeds the code’s defaults, an existing one keeps its Settings documents.