Deploy staging
Terminal and this documentation are static Cloudflare Pages sites. Mainframe staging runs Core, Postgres, SRS, the media indexer, Caddy, world-sim (with its operator), and monitoring on its own Hetzner server. DVR and review objects use a private Hetzner S3 bucket. It uses real credentials and the normal authentication path.
| Surface | Address |
|---|---|
| Terminal | https://terminal.dev.mainline.inc |
| Mainframe | https://mainframe.dev.mainline.inc |
| Documentation | https://docs.terminal.dev.mainline.inc/ |
| Grafana | https://mainframe.dev.mainline.inc/grafana/ |
| Simulator administration | https://mainframe.dev.mainline.inc/sim/admin |
These are the configured targets. DNS and TLS must be provisioned before the addresses will work.
1. Provision Pages, the host, and bucket
Section titled “1. Provision Pages, the host, and bucket”The separate infra repository owns the Git-integrated Pages projects (terraform/terminal-pages.tf), custom domains, Hetzner server, firewall, and object storage (terraform/terminal-staging.tf). Its Terminal staging resources reuse the account, SSH keys, and providers used by the CC clusters while creating a separate server and private bucket.
Review the Terraform plan before applying it. The resulting server IP is host.publicIp in deploy/installations/staging/installation.ts. Follow capacity planning when choosing server and disk sizes. The server needs Docker with Compose 2.24.4 or newer and Bun 1.4.2 for the deployment launcher.
Public traffic uses TCP 80 and 443 for HTTP/TLS, UDP and TCP 8000 for WebRTC in staging (MEDIA_RTC_PORT; the default is 8000), and TCP 1935 for authenticated RTMP publishing. SSH access should match the administrator CIDRs configured in Terraform. Postgres, SRS’s API, Prometheus, and exporters are not public services.
Static builds
Section titled “Static builds”Both Pages projects use the repository root, Bun 1.4.2, SKIP_DEPENDENCY_INSTALL=1, and the main production branch. Every push to main rebuilds and publishes both sites without touching Docker. Pull requests receive separate preview builds.
| Project | Build command | Output |
|---|---|---|
terminal-staging |
bun install --frozen-lockfile && bun run --cwd apps/web build |
apps/web/dist |
terminal-docs-staging |
bun install --frozen-lockfile && bun run --cwd docs build |
docs/dist |
Terminal’s build environment sets VITE_MAINFRAME_URL=https://mainframe.dev.mainline.inc. Mainframe continues to allow the Terminal origin through its configured CORS/auth settings; preview origins are not automatically trusted for staging login. The old Terminal /docs/ URLs redirect to the documentation domain.
The Hetzner console name is mainframe-staging. Its Terraform resource address, /opt/terminal checkout, terminal-staging Compose project, and existing volumes retain their original identities. Renaming the console label does not change the public IP or DNS.
2. Prepare the backend checkout
Section titled “2. Prepare the backend checkout”Place the reviewed Terminal checkout on the server. What staging runs is its installation file, deploy/installations/staging/installation.ts, committed in the repository: the server IP, domains, bucket, account email and the packages and modules (Configuration and modules). From the checkout’s root:
bun install --frozen-lockfilebun run deploy init stagingThis writes deploy/installations/staging/secrets.toml with mode 0600: every credential the launcher can generate, and a SET_… placeholder for each one a person supplies (the S3 keys). It refuses to overwrite an existing file. Fill the placeholders and keep the file outside version control. A host that still runs from deploy/.env.staging moves with init staging --from deploy/.env.staging instead (Move a host from its env file).
The bucket endpoint and region must match, for example https://hel1.your-objectstorage.com and hel1. The bucket must already exist. Leave base.storage.presign unset (false) to serve authenticated playback through Mainframe without configuring public bucket CORS.
3. Validate and launch
Section titled “3. Validate and launch”bun run deploy check stagingbun run deploy up stagingbun run deploy ps stagingcheck validates the installation and its secrets and parses every container’s rendered environment with that module’s own parser. up renders deploy/installations/staging/.rendered/ (one environment file per container, the Compose profiles, the Prometheus targets and the edge’s module routes), builds backend images, and waits for service health checks. Caddy uses the stock image and serves only Mainframe, module routes, and monitoring; it does not build Terminal or docs. A normal rebuild preserves named volumes.
If local map assets are required, build them before deployment using the repository’s map tooling. The stack can otherwise use the frontend’s configured online basemap.
4. Seed the virtual fleet
Section titled “4. Seed the virtual fleet”bun run deploy seed stagingbun run deploy seed staging --applyThe first command shows missing records. The second creates them through Mainframe’s authenticated document and fleet APIs. It adds ten SIM-STG-001 through SIM-STG-010 aircraft, one staging flight, four short patrol loops, and the AO area marking that Mainframe Settings → Configuration → Workspace names as the area of operations. Re-running preserves existing documents and fleet identities.
The simulator’s operator dispatches only world-sim’s own aircraft, and Mainframe refuses a module a command for another provider’s entity. Real aircraft from another provider are never eligible. The aircraft start at different battery levels; four missions leave six aircraft available for charging and rotation.
5. Verify the deployment
Section titled “5. Verify the deployment”Sign in to Terminal with the admin email of the installation file and the [base.admin] password of secrets.toml, then open Profile… in the Terminal menu and enable two-factor authentication before anyone else signs in; Configuration → Security → Two-factor requirement set to Admins enforces it for every later administrator. Create the people’s accounts in Mainframe Settings → Users: each gets a temporary password shown once and must change it at the first sign-in. Confirm the virtual fleet appears, missions become assigned, telemetry advances, and a live feed plays. After at least one DVR segment closes, enter replay and confirm video and positions match the selected time.
In Grafana, inspect service availability, module connections, S3 uploads, simulator publishers, automation freshness, and mission coverage. Check an actual charge → dispatch → return → recharge cycle before calling the continuous run accepted. A long replay history requires a real soak run; an initial healthy boot proves only that the stack started.
Use updates and backups for subsequent releases and real aircraft before enabling the autel module.
Lithuania basemap assets
Section titled “Lithuania basemap assets”The map directory is excluded from Git and must be populated on each new host:
bun scripts/mapctl.ts fetch --area lt --name ltbun scripts/mapctl.ts assetsRun from /opt/terminal. This installs the full Lithuania extract plus the
fonts and sprites in deploy/map, mounted read-only at Caddy’s /srv/map.
--area lt matters: the default ao only covers the southeast operating area.
Caddy serves /map/* directly with Range and CORS support; native Autel
/map/api/v1/* routes continue to go to Autel.
Verify a range request to /map/lt.pmtiles returns 206 with PMTiles magic,
and that /map/fonts/Noto%20Sans%20Regular/0-255.pbf and
/map/sprites/v4/black.json return 200. A frontend rebuild is unnecessary
when installing basemap assets; reload the browser to retry a cached fallback.
Park the simulator for real test operations
Section titled “Park the simulator for real test operations”On the staging host, from /opt/terminal:
bun deploy/park-world.tsbun deploy/park-world.ts --apply--installation <name> names another installation; staging is the default.
The first command previews the affected world-sim entities. Apply saves a
Postgres dump, a copy of installation.ts and secrets.toml and an identity inventory in
deploy/backups/park-world-<timestamp>/ and stops world-sim with its operator.
It retires the provider’s entities through Mainframe’s provider protocol,
updating connected clients without restarting Mainframe. No real provider’s
entities are changed. The command can be rerun after a partial failure.
Retired aircraft are hidden by default in Assets; simulator sites, radars and detections also leave the live registry. Missions, markings, airframes, recordings and the simulator volume are preserved. Existing history/media retention still applies; parking is not an indefinite archival hold.
The world stays parked until the next bun run deploy up staging, which starts
world-sim again while simulation is in the installation file. To keep it
parked, remove the simulation key from deploy/installations/staging/installation.ts
(its secrets may stay; they only warn). To bring the simulator back, put the key
back and run:
bun run deploy up stagingbun run deploy logs staging -- world-simWorld-sim reads its retained volume and announces its assets again. Mainframe
reactivates existing identities (or creates them if normal retention has
already pruned them). Do not clear volumes or reseed merely to restart.
Leave world-sim’s operator setting out for an idle simulator. Add it back
only when automatic simulator mission dispatch is wanted.
Verify world-sim health, live assets and video after restart; restarting was
intentionally not part of parking the simulator.
Air picture and the continuous demo
Section titled “Air picture and the continuous demo”Staging is the standing demo source for the air picture. World-sim runs the standing air world: radars of every type, RF, EO/IR, acoustic, ADS-B, Remote ID, FLARM and Network Remote ID sensors, the IXL-4 launcher and the RJ-120 jammer near VNO, live ADS-B traffic from adsb.lol (80 nm, at most 80 aircraft, with staging’s adsb.lol key) and the real wind from the weather module (Open-Meteo’s paid plan, with staging’s key). The continuous demo keeps the picture moving without resets:
- cooperative traffic: the live airliners, which the simulated EYVI PSR/SSR and C-UAS radars see as well as ADS-B; Remote ID quads; a permitted inspection drone; a FLARM glider;
- non-cooperative traffic: quads, a fixed-wing, a bird flock and a balloon, staggered, each with a bounded lifetime;
- about every 10 minutes a radio-silent quad or fibre FPV flies into the VNO zone and earns a high threat and an effector recommendation;
- nothing engages by itself. A launcher reloads to full 60 s after its last engagement ends; a jammer is ready again when its engagement ends.
The same demo runs on a laptop with bun run demo (Laptop demo).
The seeded live fusion configuration confirms tracks by AP4’s fast rules
(MOGS’ 2-second clock). For a customer demo where the bird flock and the
ground clutter should stay quiet, set Mainframe Settings → Fusion →
Confirmation → Track confirmation to Conservative (before AP4) and
save; it is a revision of the live document and survives redeploys (the
seed only creates missing documents); Fast reverts it (the two sets).
Fusion runs the Rust core on the DEM. The simulator’s sensors are seeded
sources (apps/mainframe/seeds/staging/surveillance.sources.json) that send
to the fusion container. Real aircraft and phones (autel, vanagas,
vanagas-phones, front) are not touched: the operator dispatches only
world-sim’s own aircraft, and the demo’s targets exist
only inside world-sim.
Run everything below on the staging host as root, from /opt/terminal. The
host runs the rendered Compose command with two release overrides appended, not
bun run deploy up. .rendered/dc is one line: docker compose with the
rendered compose.env and the Compose files the installation selects.
cd /opt/terminalSHA=<full release commit>; TAG=release-${SHA:0:8}; NAME=air-picture-$(date -u +%Y%m%dT%H%MZ)I=deploy/installations/stagingDC="$(cat $I/.rendered/dc) -f deploy/compose.ux-release.json -f deploy/compose.fleet-release.json"curl -s https://mainframe.dev.mainline.inc/api/deployment; echo # retentionHold: see "Preservation hold" belowdf -h / /srv/terminal-data; nproc; free -g1. Back up
Section titled “1. Back up”umask 077; B=deploy/backups/$NAME; mkdir -p $Bcp -a $I/installation.ts $I/secrets.toml $I/.rendered deploy/compose*.yaml deploy/compose*.json deploy/Caddyfile.staging $B/$DC ps --format json > $B/ps.jsondocker exec terminal-staging-postgres-1 pg_dump -U mainframe -d mainframe -Fc > $B/mainframe.dumpdocker exec -i terminal-staging-postgres-1 pg_restore --list < $B/mainframe.dump | head -52. Unpack the release
Section titled “2. Unpack the release”From a machine with the repository:
git archive --format=tar $SHA | ssh root@89.167.55.244 "mkdir -p /opt/terminal/releases/$SHA && tar -x -C /opt/terminal/releases/$SHA"On the host, copy the release over the checkout and record it. Host-only files
(deploy/installations/staging/secrets.toml and .rendered/, the two override JSON
files, deploy/backups/, deploy/map/) are not in the archive and stay; a release
carries the committed installation.ts, so keep host edits to it committed; files a release dropped stay
too and are harmless:
cp -a releases/$SHA/. /opt/terminal/echo $SHA > releases/current-revision3. Build the changed images
Section titled “3. Build the changed images”One at a time, at low priority, from the release directory. Every Dockerfile builds with Docker’s legacy builder, so the host needs no buildx. Fusion’s first build compiles the Rust core (a clean emulated x86_64 build took 4 minutes); its dependencies are a layer of their own, so a release that changes only the core’s sources rebuilds only the core.
cd /opt/terminal/releases/$SHAnice -n 10 docker build -f apps/mainframe/Dockerfile --build-arg MAINFRAME_BUILD=${SHA:0:8} -t terminal-staging-mainframe:$TAG .nice -n 10 docker build -f services/fusion/Dockerfile -t terminal-staging-fusion:$TAG .nice -n 10 docker build -f services/world-sim/Dockerfile -t terminal-staging-world-sim:$TAG .nice -n 10 docker build -f services/weather/Dockerfile -t terminal-staging-weather:$TAG .nice -n 10 docker build -f services/radar-net/Dockerfile -t terminal-staging-radar-net:$TAG .nice -n 10 docker build -f services/tak/Dockerfile -t terminal-staging-tak:$TAG .cd /opt/terminal4. Fetch the DEM
Section titled “4. Fetch the DEM”The Copernicus DEM GLO-30 tile set (about 23 MB, free with attribution;
dem/manifest.json carries the attribution) goes to a host directory that fusion
and world-sim mount read-only:
mkdir -p /opt/terminal/shared/terrain(cd releases/$SHA && bun install --frozen-lockfile && bun scripts/terrain/fetch-dem.ts --out /opt/terminal/shared/terrain)python3 -c 'import json;m=json.load(open("/opt/terminal/shared/terrain/dem/manifest.json"));print(m["levels"],len(m["tiles"]))'Or copy a set fetched elsewhere:
rsync -a data/terrain/ root@89.167.55.244:/opt/terminal/shared/terrain/.
5. Edit the installation and render
Section titled “5. Edit the installation and render”What staging runs is deploy/installations/staging/installation.ts; a release
that changes it carries the change. Edit it only through a commit, then render:
cp -a $I/secrets.toml $I/secrets.toml.pre-$NAMEbun install --frozen-lockfilebun run deploy render staginggrep -cE '^WORLD_SIM_(AIR_WORLD|DEMO|DOMAINS)=' $I/.rendered/env/world-sim.env # 3render prints the profiles and runs check first and refuses a missing or unfilled secret, then writes
the environment files, the profiles, targets/services.json (Prometheus reads
it within 15 seconds) and caddy/routes.caddy. FUSION_THREADS=4
(packages["air-picture"].modules.fusion.threads) on the 8-vCPU host leaves
four cores to ingest, Mainframe, Postgres and media; the default rule would
take six.
The two supplied keys live in secrets.toml: Open-Meteo’s paid plan under
[base.modules.weather] openMeteoApiKey and adsb.lol’s under
[packages.simulation.modules.world-sim] liveTrafficApiKey. They are kept
outside git, on Jonas’s Mac, in
/Users/j/Code/terminal/deploy/.env.staging.air-picture-keys (mode 600,
git-ignored), one KEY=value line each (WEATHER_API_KEY,
WORLD_SIM_LIVE_TRAFFIC_API_KEY). Copy each value into its line of
secrets.toml by hand, never print them, and render again. Check that both
reach their containers by counting, not reading:
grep -c "^WEATHER_API_KEY='.\+'" $I/.rendered/env/weather.env # 1grep -c "^WORLD_SIM_LIVE_TRAFFIC_API_KEY='.\+'" $I/.rendered/env/world-sim.env # 1With the key the weather module asks customer-api.open-meteo.com with
&apikey=, and world-sim sends adsb.lol’s key as the x-api-key header.
Neither is logged; the health checks in step 8 say keyed instead.
Reload Prometheus so it reads the release’s alert rules as well:
$DC kill -s SIGHUP prometheusGrafana finds the Mainframe · Air picture dashboard in the release’s dashboard folder by itself.
6. Pin the images
Section titled “6. Pin the images”The two override files pin images to releases. Print them, then point each
rebuilt service at $TAG, adding the service where it is missing:
for f in deploy/compose.ux-release.json deploy/compose.fleet-release.json; do cp -a $f $f.pre-$NAME; python3 -c "import json;d=json.load(open('$f'));print('$f',{k:v.get('image') for k,v in d.get('services',{}).items()})"; donecat > /root/air-pin.py <<'PY'import json, sysp, tag = sys.argv[1], sys.argv[2]; d = json.load(open(p)); svc = d.setdefault("services", {})for name in ["mainframe", "media-indexer", "fusion", "world-sim", "weather", "radar-net", "tak"]: image = "terminal-staging-mainframe" if name == "media-indexer" else f"terminal-staging-{name}" svc.setdefault(name, {})["image"] = f"{image}:{tag}"json.dump(d, open(p, "w"), indent=2)PYpython3 /root/air-pin.py deploy/compose.fleet-release.json $TAG && rm /root/air-pin.py$DC config --quiet && $DC config --format json | python3 -c 'import json,sys;[print(k,v.get("image")) for k,v in sorted(json.load(sys.stdin)["services"].items())]'The later file wins where both pin a service. Check the printed list: the
seven services above on $TAG, every other service on its old tag. If
RECORDING_MAINFRAME_IMAGE is set in the environment it names the Mainframe
image as well; set it to terminal-staging-mainframe:$TAG.
7. Start, seed, start the air world
Section titled “7. Start, seed, start the air world”Mainframe first (it applies the additive migration 0019 at boot), then the
documents, then the air world, then the fleet check:
$DC up -d --no-deps --no-build mainframe media-indexeruntil [ "$(docker inspect -f '{{.State.Health.Status}}' terminal-staging-mainframe-1)" = healthy ]; do sleep 5; done$DC exec -T mainframe bun src/data-tool.ts seed --profile staging --documents-only$DC exec -T mainframe bun src/data-tool.ts seed --profile staging --documents-only --apply$DC up -d --no-deps --no-build weather fusion world-sim radar-net tak$DC exec -T mainframe bun src/data-tool.ts seed --profile staging# the release's Caddyfile adds <mainframe>/licenses; a reload is graceful$DC exec -T caddy caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile$DC exec -T caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfileThe preview lists only missing documents; check it before --apply. Expect
the effector types, the IX-1 interceptor airframe, the sensor types, the
live fusion configuration, the whitelist, the UTM permits, the 17 simulator
sources and the VNO zone marking (0–150 m AGL, high priority). Existing
documents and tombstones are kept. The last command checks the virtual fleet
once world-sim is online and should have nothing to create.
8. Check health
Section titled “8. Check health”curl -s https://mainframe.dev.mainline.inc/api/health | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d["status"],d["version"]);[print(k,v) for k,v in d["checks"].items()]'$DC exec -T fusion bun -e 'const h=await (await fetch("http://127.0.0.1:8799/api/health")).json();console.log(h.status,JSON.stringify(h.core),JSON.stringify(h.heights))'T=$(sed -n "s/^WORLD_SIM_ADMIN_TOKEN='\(.*\)'$/\1/p" $I/.rendered/env/world-sim.env)for p in demo live wind; do curl -s -H "authorization: Bearer $T" http://127.0.0.1:8787/api/sim/air/$p | head -c 600; echo; done$DC exec -T weather bun -e 'const h=await (await fetch("http://127.0.0.1:8791/api/health")).json();console.log(h.source,JSON.stringify(h.openMeteo),JSON.stringify(h.forecast))'$DC logs --tail 100 world-sim | grep -E 'air world|world config|terrain'curl -s https://mainframe.dev.mainline.inc/api/licenses; echocurl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://mainframe.dev.mainline.inc/licenses$DC exec -T prometheus wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=up{job="mainframe-services"}' | python3 -c 'import json,sys;[print(r["metric"]["service"],r["value"][1]) for r in json.load(sys.stdin)["data"]["result"]]'Expect:
- Mainframe
ok, theproviderscheck listingworld-sim,weatherandfusionbeside the real providers; - fusion
core.implrust,threads4,heights.terraindem-…(notnull) and nofallback; - the demo
enabled: truewith targets, live traffic enabled with aircraft and itsfeed.keyed: true,lastErrornull; wind modereal; - weather
open-meteo,openMeteo.keyed: true, the forecast’slastErrornull; /api/licenses:terrainthe Copernicus attribution,liveTraffic["adsb.lol"],openMeteo: true;/licensesanswers 302 tohttps://terminal.dev.mainline.inc/licenses?mainframe=…;air world: standing with the continuous demoin world-sim’s log;- Prometheus
up1 forfusion,weatherandworld-sim, and Grafana’s Mainframe · Air picture dashboard filled: fusion memory against its 4 GB limit, the Rust core with 4 threads, step times, tracks, the demo’s spawns and the live ADS-B feed.
If the demo, live traffic or wind is off, the world-sim volume holds a
setting saved earlier (it wins over the environment); turn it on with the
POST under “Pause the demo”.
Then set staging’s state history to 30 days (Jonas, 2026-11-08): sign in to Terminal as an admin, Mainframe Settings → Configuration → Retention, state 30 days. The running value lives there; a new database seeds the code’s 90 days. It applies to every asset, the real hardware’s history included. See “Preservation hold, retention and growth”.
9. Five minutes: is the demo alive?
Section titled “9. Five minutes: is the demo alive?”In Terminal, signed in:
- Airliners near Vilnius are fused tracks whose composition names both
eyvi-adsbandeyvi-psr. - Within 3 minutes of world-sim’s start the demo’s quads, the bird flock, the balloon, the glider (FLARM) and Remote ID drones are tracks.
- A threat appears, the first about 3 minutes in, then about every
10 minutes. Tracks puts it first; Selection → Threat reads
Highwith the time to VNO; the recommendation offers IXL-4 VNO. - Engage With… → IXL-4 VNO: the interceptor launches and flies to it. The launcher reads 3/4, then reloading a minute after the engagement ends, then 4/4.
- Assets shows the real aircraft and phones as before, and no mission was
dispatched to anything but
SIM-STG-…. - About Terminal → Licences opens
/licenses, which lists the Copernicus DEM (its notice “Produced using Copernicus WorldDEM-30 …”), adsb.lol (ODbL), Open-Meteo (CC BY 4.0) and EGM2008, then the map and fonts. - The map’s attribution (ⓘ) carries the basemap’s credits only: no Copernicus line and no live-traffic notice, with Air tracks and Sensor coverage on.
10. Roll back
Section titled “10. Roll back”cp -a $I/secrets.toml.pre-$NAME $I/secrets.tomlcp -a $B/.rendered/. $I/.rendered/for f in deploy/compose.ux-release.json deploy/compose.fleet-release.json; do cp -a $f.pre-$NAME $f; donecp -a $B/Caddyfile.staging deploy/Caddyfile.staging && $DC exec -T caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile$DC stop fusion world-sim$DC up -d --no-deps --no-build mainframe media-indexer weather radar-net takThe old release’s installation.ts comes back with its checkout; $B/.rendered
is what that release rendered. The old release runs on migration 0019, which is additive. Seeded
documents and fusion’s fusion schema stay and are harmless. Restoring
$B/mainframe.dump loses everything since the backup and is Jonas’s call.
11. Pause the demo
Section titled “11. Pause the demo”To stop the simulator, use bun deploy/park-world.ts and then --apply
(above); fusion keeps running with silent sources. To keep the world but
stop the demo’s traffic and threats:
curl -s -X POST -H "authorization: Bearer $T" -H 'content-type: application/json' -d '{"enabled":false}' http://127.0.0.1:8787/api/sim/air/demoThe setting persists in world-sim’s volume and wins over the environment at
the next start. {"enabled":true} turns the demo back on;
POST /api/sim/air/live and /api/sim/air/wind take {"enabled":true} and
{"mode":"real"} the same way.
Preservation hold, retention and growth
Section titled “Preservation hold, retention and growth”A preservation hold (/api/deployment reads retentionHold: true, or the
mode is archive) stops Mainframe’s history retention and the pruning of
retired entities, and a boot path that would delete blobs makes Mainframe
refuse to start. This release adds no such path: the seed only creates
documents, and fusion’s raw-log retention (FUSION_RAW_RETENTION_DAYS) reads
the same hold and deletes nothing while it is on, or while it cannot read
it. With the hold on, the demo’s tracks grow Postgres and the bucket without
bound. Watch them:
df -h /srv/terminal-datadocker exec terminal-staging-postgres-1 psql -U mainframe -d mainframe -c "select pg_size_pretty(sum(pg_total_relation_size(oid))) from pg_class where relname like 'samples_state%' and relkind = 'r'"docker stats --no-stream terminal-staging-fusion-1 terminal-staging-world-sim-1 terminal-staging-mainframe-1 terminal-staging-postgres-1Without a hold, Mainframe Settings → Configuration → Retention bounds history; a new
database seeds the code’s values (90 days of state, 14 of telemetry), so change retention there.
Staging keeps 30 days of state history (Jonas, 2026-11-08): the continuous
demo makes about 1,800 track entities a day and, at about 50 tracks, a few
GB of samples_state a day, so 90 days would approach the disk. The shorter
value shortens the real hardware’s history too.
Move a host from its env file
Section titled “Move a host from its env file”Before installation files a host ran from one env file, deploy/.env.staging (production:
deploy/.env.production; an archive: its own). The installation file and
secrets.toml replace it. Nothing in the database changes, so the move is a
recreation of containers, not a migration. On the host, from /opt/terminal:
-
Update the checkout to a release with
deploy/installations/. Restart nothing. -
Write the secrets from the env file:
Terminal window bun run deploy init staging --from deploy/.env.stagingIt takes every credential from the env file and prints three lists: variables whose value differs from what the committed
installation.tsrenders (NAME: env has X, installation renders Y), variables that no longer exist (the auto-operator key and serial selection, the first-boot retention values), and variables of modules the installation does not enable. -
Resolve each difference in
installation.tsand commit it, or accept it. Repeat until only intended differences are left. -
Compare what every container receives today with what it would receive:
Terminal window bun deploy/launcher/env-diff.ts --old-ref <the deployed commit> --old-env deploy/.env.staging --installation stagingIt runs
docker compose configon the deployed commit’sdeploy/tree with the env file and on the rendered installation, and prints each difference with the rule that allows it; it exits non-zero on any difference no rule allows. Secret values print as hashes. If the host ran thevanagasobservers, they appear as removed services: Legacy is not rendered, and this is the moment they stop. -
bun run deploy render staging, then append the release override files to$(cat deploy/installations/staging/.rendered/dc)as above. -
Recreate the containers in the order of the release steps above. The first recreation of
weatherandradar-netuses their own profile names, which the renderedcompose.envalready has. -
Keep
deploy/.env.staginguntil the installation has run for a day, then delete it.
Rollback is the previous checkout with the old env file. Staging’s first-boot retention values (30 days of telemetry, 720 hours of media in the old example) no longer exist: a new database seeds the code’s defaults, an existing one keeps its Settings documents.