Troubleshooting
Start with bun run deploy ps, Mainframe’s /api/health, and the Deployment dashboard. Then read the relevant service logs with bun run deploy logs mainframe world-sim or Grafana’s log panel.
| Symptom | What to check |
|---|---|
| Login works on one origin but not the other | MAINFRAME_BASE_URL, trusted origins, HTTPS, and browser cookie behavior |
| Terminal shows no link | Its saved Mainframe URL, DNS/TLS, /ws, and Core health |
| A module starts but never connects | Enabled module identity, stable API key, provider URL, and protocol-error metrics |
| No virtual aircraft appear | World profile health, seed preview, and persisted world identities |
| Auto-operator is connected but does nothing | Serial/provider selection, fresh state, battery thresholds, existing assignments/proposals, and mission coverage |
| Video never becomes live | Publisher connectivity to RTMP, signed assignments, SRS hooks, and SRS stream traffic |
| WebRTC fails after signaling succeeds | Reachability of UDP/TCP 8000 and the advertised PUBLIC_IP |
| Live video works but replay is missing | Closed DVR files, indexer failures, S3 access, and recent media manifests |
| Database queue grows | Postgres health, transaction failures, disk latency/capacity, and incoming rate |
| All historical data seems missing after a change | Compose project name and selected environment/volumes |
| Optional module shows as down | Selected profiles and regenerated Prometheus targets |
Voice panel reads Voice unavailable |
The voice profile is selected (the launcher sets LIVEKIT_URL only then) and /api/health lists livekit |
| Voice joins but nobody hears anybody, or audio is one-way | 7882/udp and 7881/tcp open to PUBLIC_IP on the host firewall; PUBLIC_IP is the address the browsers can reach; LIVEKIT_STUN_SERVERS='[]' on a LAN box without internet; chrome://webrtc-internals shows the selected candidate pair |
| Voice joins listen-only although the role talks | The page is not a secure context (plain http:// by IP has no microphone; serve over HTTPS), or the browser denied the microphone (Microphone denied · Retry) |
Audio blocked · Enable |
The browser’s autoplay policy: click Enable once per page |
| A removed user is still in a room | Mainframe reaches http://livekit:7880 (the Twirp API) on the Compose network; voice: cannot remove in its log |
An S3 outage leaves failed recordings locally for retry. Do not delete the DVR volume to make the error disappear. Resolve the storage problem and watch failed segments drain after the next recovery scan.
An initial connection refusal can be startup ordering. Wait for the health checks and provider reconnect before diagnosing a regression. If a dependency remains unhealthy, use its specific metrics and logs rather than repeatedly rebuilding all services.
For an incident involving hardware, preserve the module’s capture files and the relevant timestamps. Avoid broad resets: they can remove the history needed to understand the failure.
Inspect and retry DVR jobs
Section titled “Inspect and retry DVR jobs”The SRS worker keeps durable local upload and archive job journals beside original FLV chunks. Check /health and /metrics on the private worker port, controller status, and the Recording reliability dashboard. srs_preservation_pending_bytes, srs_preservation_oldest_seconds, srs_archive_pending and failure counts distinguish upload and packaging problems. A stale controller or failed stream inventory is an explicit health failure.
An original closes only after the controller verifies the stock recorder’s reopen/exit and writes its journal. The uploader retries independently with bounded multipart memory, verifies size and SHA-256 in the bucket, then lets archive processing and verified cleanup proceed. An unavailable decoder context is reported without discarding the original. Do not delete local originals, pending capture chunks or journals to clear an alert.
Recent MP4s are disposable derived cache entries: two running jobs, sixteen pending streams and 512 MiB including scratch reservations. Cache pressure returns unavailable rather than blocking recording. A stopped video stream with continued silent Autel AAC is not video coverage.
The old media:jobs command and native-recorder quarantine sidecars are historical audit tools; they do not control the SRS worker. See the checked-in deploy/srs/README.md for current runtime and rollback boundaries.