Operations runbook¶

Normal service chain¶

flowchart LR
    USB[USB receiver] --> R[readsb]
    R --> AL[airplanes.live]
    R --> C[collector]
    C --> U[uploader]
    U --> V[remote relay]
    T[Cloudflare tunnel] --> V
    V --> B[browser]

Check from left to right. A downstream failure does not always affect an upstream service: the dashboard uplink can fail while airplanes.live continues receiving directly from readsb.

Mac service checks¶

for label in \
  local.airplanes-live.readsb \
  local.antenna-observatory.web \
  local.antenna-observatory.uplink \
  local.antenna-observatory.keepawake
do
  launchctl print "gui/$(id -u)/$label" | grep -E 'state =|pid =|last exit code'
done

state = running or an active PID indicates the job is loaded. KeepAlive restarts an unexpected exit after the configured throttle interval.

Logs¶

On Linux, the unprivileged supervisors write relay and tunnel logs under ~/.local/state/antenna-observatory/.

Restart a Mac job¶

Use kickstart for a routine restart:

launchctl kickstart -k "gui/$(id -u)/local.airplanes-live.readsb"

Use the same command with another label for the collector, uploader, or keep-awake job. Avoid loading two decoder jobs because only one can claim the USB device.

Deployments and rollback¶

Production releases are immutable directories named by the full Git commit. The current symlink changes atomically after extraction and validation. The deployment script restarts only the relay, checks its loopback dashboard, and restores the previous symlink if health does not recover.

The five newest releases are retained. To inspect the active target:

ssh antenna-observatory 'readlink "$HOME/antenna-observatory/current"'

State and backup¶

Back up the remote state directory separately from releases. It contains:

• relay and tunnel token files;
• the SQLite history database and WAL files;
• latest accepted relay snapshot and bounded log tail;
• public-origin configuration.

Stop or quiesce the relay before making a raw SQLite filesystem copy, or use SQLite’s online backup command. Protect backups with the same care as production state.

Routine maintenance¶