Skip to content

Monitoring and logs

Prometheus collects numeric measurements over time. Grafana queries those measurements and presents dashboards. Loki stores container logs, and Alloy forwards logs from this Compose project. The Terminal browser is not instrumented by this stack.

Open /grafana/ on the Mainframe domain and sign in with the separate Grafana credentials from the deployment environment. Self-registration and anonymous access are disabled. Metrics endpoints and the SRS API remain private.

Dashboard Use it to answer
Deployment Which service is down, and is the host under pressure?
Core Are samples arriving and persisting? Are commands, tasking, documents, and HTTP requests working?
Modules Is each module connected, healthy upstream, publishing, and handling commands?
Media and replay Is SRS receiving video? Are DVR segments reaching S3 and Mainframe?
Simulator and automation Where are virtual aircraft? Are batteries cycling and missions covered?
Air picture Is fusion’s memory bounded under its limit? Is the core keeping up (step time, slow steps, loop lag, queues)? Are tracks published and the raw log uploaded and pruned? Is the continuous demo spawning threats, engaging and reloading, and are live ADS-B and the weather answering?
Host and services Are CPU, RAM, disk, network, Caddy, and Postgres healthy?

A process being reachable is only the first check. An available radar-net module may still report stale upstream data. An available indexer may still have failed segments. A running simulator operator should also have recent successful evaluations and mission coverage.

Core measures validated sample ingest, committed samples, storage queues and write failures, HTTP latency and errors, command lifecycle updates, tasking phases, entity counts, client sessions, leases, documents, and dependency checks.

The provider SDK measures module connections, reconnects, upstream health, queued/dropped samples, publishing rate, protocol errors, and provider command outcomes. Modules can expose numeric workload statistics such as stale records, active analysis sessions, or broker message counts.

Fusion adds which core runs (fusion_core_info: rust or ts, and fusion_core_threads), the step and tick time (fusion_stage_latency_seconds, fusion_tick_seconds: p50, p95 and the largest), slow and failed steps, the main thread’s event-loop lag (fusion_loop_lag_seconds, health loop), the ingest, quarantine and publish queues, measurements, decodes and quarantines per source, tracks by existence, the publish rate and stretch, raw-log upload failures and the spool backlog, and the raw-log retention’s runs, deletions and failures. Its process memory is the standard process_resident_memory_bytes (which includes the Rust core) and nodejs_heap_size_used_bytes; the container’s working set against its limit comes from cAdvisor. World-sim adds the continuous demo (world_sim_demo_enabled, spawns per lane), the simulated effectors’ engagements, outcomes, reloads and rounds, the live ADS-B feed (world_sim_live_traffic_*: aircraft, polls, the last answer, back-off) and the real wind; the weather module adds its polls, sites by fresh, stale or missing, and Open-Meteo requests.

World-sim also adds camera rendering time, rendered frame count, active encoders, encoder failures, aircraft states, battery levels, and coordinates. Aircraft serials are labels because the fleet is small and bounded; temporary detection IDs are not labels. The map requires an internet basemap, while position measurements remain in the local monitoring store.

Prometheus rules detect unavailable services, disconnected modules, unhealthy dependencies, storage failure/backlog, DVR stalls, repeated encoder failures, stalled automation, prolonged mission gaps, and low disk or memory. The air-picture group covers fusion down, fusion’s memory over 80 % of its limit for 15 minutes, fusion restarted or OOM-killed, sustained slow steps, raw-log uploads failing, the continuous demo spawning no threat for 30 minutes while it is on, and the live ADS-B feed silent for an hour (a warning). The incidents group covers what production once had unseen: a container restarted without a deploy, any kernel OOM kill, replay cursors lost, slow seek snapshots, a full replay gate, late phone positions, video refused while the phone reports it live, Front refusing phones, one query hogging Postgres, long transactions and a disk filling within a day.

Prometheus hands firing alerts to Alertmanager, which groups them by deployment, service and alert name, waits 30 seconds for the first notification, batches updates every five minutes and repeats an unresolved alert every four hours. Every alert carries a deployment label (the Compose project name), so one chat can serve several hosts.

Prometheus retains up to 30 days or 15 GB, whichever limit is reached first. Loki retains 30 days. Application replay has its own retention settings and does not depend on Prometheus.

Delivery is set in the deployment environment; nothing is sent until a destination is set:

Variable Use
ALERTMANAGER_WEBHOOK_URL POSTs Alertmanager’s webhook JSON to this URL. A chat service that needs its own format needs an adapter in between.
ALERTMANAGER_TELEGRAM_BOT_TOKEN, ALERTMANAGER_TELEGRAM_CHAT_ID Sends one message per group to a Telegram chat: severity, alert, deployment, service and summary. Both are required; the chat id is numeric (-100… for a group).

Either, both or none can be set. With none, alerts still group in Alertmanager and show in Grafana under Alerting (the Alertmanager data source). The Alertmanager UI listens on 127.0.0.1:9093 of the host only (ALERTMANAGER_PORT); open it through an SSH tunnel. Secrets are written to files inside the container, never into the rendered configuration. A setting that does not parse stops the container with the reason in its log.

After changing a destination, restart the alertmanager service and send a test alert from the host:

Terminal window
curl -XPOST 127.0.0.1:9093/api/v2/alerts -H 'content-type: application/json' \
-d '[{"labels":{"alertname":"DeliveryTest","severity":"warning","service":"alertmanager"},"annotations":{"summary":"Delivery test"}}]'

It arrives after the 30-second group wait. AlertDeliveryFailing fires when a destination refuses notifications; AlertmanagerUnreachable shows in Grafana only, since nothing can deliver it.

Dashboards, data sources, scrape configuration, and alert rules are versioned under deploy/monitoring/. Edit the source files to make durable changes; provisioned dashboards are read-only in Grafana. bun run deploy render <name> regenerates the target list for the installation’s modules (.rendered/targets/services.json); it touches no container, and Prometheus reads it again within 15 seconds.

Recording health is checked per active SRS stream whose entity is online and whose policy permits recording. Freshness uses the deployment’s 60-second segment cadence. The media dashboard distinguishes trailing delay, missing intervals between indexed segments, pending original bytes, upload failures, archive backlog and recent-cache pressure. An intentional publisher stop removes its expected recording series.

Gap checks cover the observed continuous active session, capped at one hour. After a Core restart, observation starts again; these checks do not certify the older archive. Inventory failures retain the previous observations and raise a stale-monitor alert. Each inventory is capped at 128 stream labels; exceeding that cap is also visible and alerts.

The node exporter uses the Linux host network namespace, following the exporter’s container guidance. Its listener binds only to the private Docker bridge gateway. The launcher discovers this address; set NODE_EXPORTER_HOST_GATEWAY only when using a different private bridge. With raw Compose, supply the gateway explicitly when it differs from 172.17.0.1. Prometheus resolves node-exporter to the same private address via Compose extra hosts.

Host network charts exclude loopback, Docker bridges, and veth interfaces to avoid double-counting traffic. On macOS, these are Linux VM metrics. Network history collected before the host namespace correction must not be used for bandwidth estimates; record the rollout time in the deployment evidence.

The PostgreSQL dashboard shows database/table/index and snapshot growth, query calls, execution time, rows, and temporary-block use. The monitoring startup job installs pg_stat_statements idempotently after PostgreSQL becomes healthy; the server preloads the extension. This is required by PostgreSQL. The exporter limits statement results to 100 and disables SQL-text labels. Its pinned custom-query support supplies temporary-block counts; revisit that configuration when upgrading the exporter.

For a bounded, read-only diagnosis from the deployment checkout:

Terminal window
bun run deploy compose staging -- exec -T postgres psql -U mainframe -d mainframe < deploy/monitoring/postgres/diagnose.sql

The result includes query IDs, timings and storage sizes. If SQL text is needed to identify a specific query, inspect pg_stat_statements through an authorized database session and keep the result private; normalized query text can still contain identifying data. Do not reset statistics during a before/after measurement.

  • Command outcome failed/unresolved: inspect the command audit and its child launch. An unresolved outcome is not evidence that the aircraft stayed on the ground. Reconcile device telemetry before retrying. Lease and not-own-entity refusals are counted separately and do not page as provider failures.
  • Fusion memory or restarts: open the Air picture dashboard. Bounded means the working set flattens after warm-up; a climb through the limit ends in a kernel kill and a restart (FusionRestarted). Note the time, the curve and the track count; raising FUSION_MEMORY_LIMIT is a deliberate decision, not a fix.
  • Demo stalled or live traffic down: world-sim’s /api/sim/air/demo and /api/sim/air/live (admin bearer) show the settings, the lanes and the last error. A setting saved through the API wins over the environment at the next start.
  • Container memory: compare working set, process RSS, replay active/queued transfers and post-load idle periods. Capture a before/after sample before restarting; a restart alone does not establish the cause.
  • Dropped operational events: inspect event kind/reason counters. Command audit lookup remains available even when the events feed is limited.
  • Recording alerts: open the media dashboard for the specific stream, then inspect pending/quarantined jobs using the troubleshooting commands. An acknowledged quarantine remains a gap in the recording.
  • Backup freshness: inspect the most recent backup manifest and maintenance result. Success means the off-host upload was verified, not merely that a local dump was started.
  • OOM kill or restart: journalctl -k | grep -i oom names the victim; read the service’s log from before the restart. A Mainframe restart loses every replay cursor; clients re-create theirs, so ReplayCursorsLost firing afterwards means one did not.
  • Replay snapshots or gate: compare the Postgres dashboard’s query time with the replay transfer panels. A seek waits for the state snapshot; a full gate shows as NO RECORDING or a stalled video.
  • Phones late, refused, or video refused while live: Front’s phones_refused, request_failed and store_write_failed lines name the reason and a sample of phones; Mainframe logs media_offer_refused with the entity.

Run the alert regression checks with the pinned Prometheus image:

Terminal window
docker run --rm --network none --entrypoint /bin/promtool -v "$PWD/deploy/monitoring/prometheus:/rules:ro" -w /rules prom/prometheus:v3.14.0 test rules media-rules.test.yml operational-rules.test.yml srs-rules.test.yml srs-health-rules.test.yml incident-rules.test.yml air-picture-rules.test.yml
bun deploy/monitoring/check-postgres.ts

A delivered test (above) is required before calling notifications operational. Check the delivery configuration with the pinned Alertmanager image, and the log pipeline with the pinned Alloy and Loki images:

Terminal window
bun deploy/monitoring/check-alertmanager.ts
bun deploy/monitoring/check-alloy.ts

Access logs retain every status, duration, byte count and original path, but omit request/response headers and query strings. The proxy generates a request ID and forwards it to Mainframe for replay correlation. Runtime proxy logs also strip headers and query parameters. Search the Caddy request_id and Core requestId fields together.

Mainframe, the media indexer, the derivative worker, the provider SDK in every module, and Front write one JSON object per line: ts, level, component, event, message, then fields; errors keep their name, a redacted message, a code and a few stack frames. Alloy makes level a Loki label (debug, info, warn, error) and component, event and code structured metadata, so a search is a label plus filters:

{service="mainframe", level="error"}
{service="mainframe"} | event="cursor_replay_failed"
{service=~".+"} | component="provider" | event="provider_rejected"
{service="front"} | event="phones_refused"

A failure that repeats (a timer, a retry loop, a reconnecting provider) is logged once when it starts, with a count each minute while it lasts, and once when it ends (state is started, ongoing or ended).

SRS logs connection and recorder events. Mainframe reports refused media access without credentials. Periodic operations and indexing jobs report failures and recovery counts; counters include every failure. Raw controller captures (autel, vanagas) remain separate protected files, not operational logs.