Recorded so the same shape can't recur silently. Newest first. If an incident isn't here with a permanent fix, it isn't closed.
Both incidents below are the same disease, and it's why v100 is retired: detection theatre. The published page and the on‑box agent reported health while the services that do the work weren't loaded — and nothing had authority to restart or the reach to raise an alarm. A page that says "operated by Stevie · 1.73 TB free" is worse than no page when the box is hung and 100% full. v101's whole job is to close the gap between what's published and what's actually loaded and responding.
| Impact | All ~30 friend accounts could not stream. Total Famflix outage. |
| Window | Last healthy telemetry 02:00; first failed probe 02:10 → API died ≈02:05. Recovered 07:25 on manual restart. ≈5h15m down. |
| Signature | Process alive (PID 1451, 0% CPU, 87 MB RSS), port 32400 still LISTEN — but every socket stuck in CLOSE_WAIT and curl /identity returned 000. Worker loop deadlocked; the log only spammed GDM broadcast errors. Not a crash — a hang. |
| Why it stayed down | (1) StevieNicks was up and read Plex m‑1 every tick from 02:10, but has no "Plex unreachable" branch — she misread a total outage as a done‑not‑imported handoff and fired forced‑import 30+ times into a dead Plex. (2) The one watchdog that probes readiness correctly — plex‑protector.sh (curl :32400/identity) — was not loaded in launchd; it never ran. (3) Even loaded, it's alert‑only by design and pings a topic separate from the ss‑delta feed. Nothing could restart Plex; no page reached Anthony. He found it ~5h later. |
| Fix applied | SIGKILL the wedged PID (SIGTERM was ignored), launchd KeepAlive relaunched Plex (PID 82282). Verified /identity = 200, RSS 290 MB, CPU active, CLOSE_WAIT cleared to 0. |
| Permanent fix | Reload plex‑protector and upgrade it from alert‑only to act: 2 consecutive /identity≠200 → restart, 3 → hard page. Stevie gains a checkpoint 0 = Plex readiness above import work. One guard, one Priority‑5 channel Anthony watches. owner: Scotty |
| Impact | 0 albums reached Plex all night; 8 completed downloads stuck done‑not‑imported. |
| Signature | tick.log shows a0/150 and Δ a+0 every tick since midnight. slskd is active — completed and .incomplete FLACs sitting in /Volumes/PERSONAL/slskd. Acquisition is working; the shelf has no room. |
| Root cause | MEDIA is 100% full — 3.8 GB free on 8 TB. slskd downloads land on PERSONAL/slskd + the 2 TB SSD but cannot import into the Plex libraries on MEDIA — no space. Same wall holds the 8 done‑not‑imported. Music is not retired — the v100 page claimed it was; the disk is the block. |
| Decision | Use MEDIA‑MIRROR (2026‑07‑01). It has 2.1 TB free — route imports + new music there as working capacity until the 20 TB drive lands. Trade‑off: it was the nightly backup target, so while it doubles as primary, MEDIA is not being mirrored to it; media‑mirror.sh (00:00) stays NOT‑LOADED under this plan. |
| Permanent fix | Free‑space gate — under 25 GB free on any media volume pages Anthony and pauses acquisition before the drive wedges. Honest storage panel on the "How it runs" page, measured not asserted. owner: Scotty |
Measured live via SSH to Studio, 2026‑07‑01. The v100 page claimed MEDIA "79% · 1.73 TB free" — the actual figure was 100% · 3.8 GB.
| Drive | Used | Free | State |
|---|---|---|---|
| MEDIA — 8 TB | 100% | 3.8 GB | FULL — blocking every import |
| MEDIA‑MIRROR — 8 TB | 72% | 2.1 TB | backup → working capacity (decision 2026‑07‑01) |
| PERSONAL — 4 TB | 33% | 2.5 TB | media cleared — music dir 0 B; 1.1 TB is data/downloads, 20 GB slskd |
| 2 TB SSD | 16% | 1.5 TB | downloads + transcode |
Not an outage, but it slowed tonight's diagnosis and belongs on the record: the sandbox → Studio Tailscale tunnel flakes several times per session. Cause: run‑on‑macpro.sh cold‑starts a userspace tailscaled on every bash call (the daemon dies between calls), joins as a fresh ephemeral node, and runs tailscale up with no retry — so a start‑race or ephemeral‑node churn drops the whole call. Fix: wrap up in a retry loop that polls BackendState=Running (as the SSH smoke test already does), stop the ephemeral churn (stable hostname, no --reset each call), and longer‑term add an always‑on command channel on the box so ops don't depend on a cold tunnel. owner: Scotty · Kendall flagged