StudioServer · full debug playbook · v1.0 · companion to v6.0 · 2026-08-14
Every step, in order, to take Famflix from “nothing is landing” to proof it is.
This is not theory. Every step below is one that actually found a fault on
14 August 2026, when the pipeline had been dead for 34 days and four separate monitors all
reported healthy. Run them in order — each layer’s answer changes what the next layer means.
Zeroth law
- Measure before you theorise. Every wrong diagnosis today came from reasoning about the system instead of reading it. “Out of memory”, “IPv6”, “tailnet blocked” — all three were invented, all three were wrong, all three cost time.
- A green monitor is not evidence. Four monitors said healthy through a 34-day outage. A monitor that can report
+0without also reporting why is decoration. - Work bottom-up. Disk → path → app → indexer → client → import → library. A fault at a lower layer makes every diagnosis above it meaningless.
- Prove the fix with a number that moved. Not “should work now”. A grab timestamp, a byte count, a queue depth.
A · Reach the box
- Bring up the tailnet and SSH in the same shell invocation — a daemon started in one call is not visible to the next.
- Run
tailscaledunprivileged. Undersudoit loses the proxy environment and every request 403s — which reads exactly like “blocked”, and isn’t. - Check binary architecture before blaming the network. Vault binaries are
aarch64; the cloud sandbox isx86_64. - Docker is not on the default
PATHon the Studio. Export~/.orbstack/binfirst or everydockercall dies with “command not found” and looks like a dead daemon. - Long commands must be detached (
nohup … &) and polled. The SSH channel caps at ~2 minutes; a timeout is not a failure.
B · Storage — always first
df -hevery volume. Note free, not used.- Identify which volume each arr root folder resolves to, and read the free space the app itself reports (
/api/v3/rootfolder). This is the number the app makes decisions on. - The killer check: if any active root has less headroom than a typical release, every search silently returns zero. Radarr rejects with “Importing after download will exceed available disk space” and Seerr shows the request as merely processing. Nothing surfaces as an error anywhere.
- Run a live test search and read the rejection reasons, not the result count. 18 releases found / 0 accepted is a disk fault, not an indexer fault.
duthe library roots and compare against the app’ssizeOnDisk. Untracked duplicates are invisible to the API — 351 GB of them were found this way (Outlander 235 GB, S.W.A.T. 63 GB, Billions 53 GB).- Look for a stale rsync source. 278 GB of orphaned copies sat at an old download path.
- Check for a second, unused disk before deleting anything. There was 1.4 TB free on one nobody was using.
C · Paths — the single largest source of faults
- Reconcile four layers, in this order, and expect drift between all of them:
- Plex library Locations
- arr root folders (
/data,/data2,/data3) - Container volume mounts in
docker-compose.yml - Hard-coded constants inside
.pyhelper scripts
- A volume the container cannot see cannot be written to, however much free space it has.
/Volumes/PERSONALhad 3.6 TB free and was mounted into nothing. - Download client path and *arr path must agree, or you get “downloaded, never imported”. Set remote path mappings if they can’t.
- Check every root the app knows about, not just the default — movies were scattered across three roots with no rule.
- Never move an in-flight torrent with
transmission-remote --move. It corrupts pieces and leaves completed torrents reporting “No data found”. - After any compose edit: back up, edit,
up -d, then re-read the root folders from the API to confirm the app actually sees the new mount.
D · Host performance
- Free RAM is not available RAM on macOS. Inactive pages are reclaimable. A dashboard reading “swap pressure 100%” was measuring allocation, not activity — a false alarm that nearly triggered a pointless restart.
- Use
memory_pressureand the swapout rate over a window. A cumulative swap counter tells you nothing about now. - Check the app’s own thread health. Kestrel logging “heartbeat has been running for 00:00:07” means thread-pool starvation inside Radarr, not a sick host.
- Watch OrbStack import I/O separately — a large manual import can saturate it while the host looks idle.
E · Seerr (Overseerr)
- Pull every request via the API, not the web UI — the public URL sits behind Cloudflare and 403s to anything scripted.
- Read
request.statusandmedia.status. They disagree constantly. Approved + processing can mean “nothing has ever been searched for this”. - Bucket the whole list: available / processing / partial / failed / declined. The interesting number is approved but not available — 137 movies and 77 TV, some over a year old.
- Check Seerr’s
activeDirectoryfor each *arr. This was the root cause of the 34-day stall: it pointed at a volume with 126 MB free, so every single request was born into a folder that could not accept it. - Verify the default quality profile and root folder Seerr pushes on new requests — fixing existing ones doesn’t fix the next one.
- Cross-reference every non-available request against Radarr/Sonarr by TMDB id. Expect three failure classes: not in the app at all, present but unmonitored, present and monitored but never searched.
- Audit who is actually using it. 785 of 786 requests came from one account — the “friends and family” assumption was wrong.
- Deduplicate. Six titles were requested twice.
- Decline what you don’t actually want before searching. Four network shows would have pulled 1,500+ episodes.
- Retry failures only after the underlying cause is fixed — a retry into a full disk just re-fails and burns the record.
F · Radarr / Sonarr
- Read
/api/v3/healthfirst. It names failing indexers and broken paths in one call. - Read
/api/v3/command. The command queue is single-threaded. One 199-fileManualImportmonopolised it for 40+ minutes, stacked 24 commands behind it, and left every completed download atimportPending. That, not the downloads, was “nothing is landing”. - If the queue is jammed, cancel the backlog rather than waiting. Queued commands do not time out.
- Compare library count against files on disk. 978 movie folders / 2.6 TB were untracked by Radarr entirely — “Radarr can only see half your library” was literally true.
- Check
monitoredandhasFileseparately. Unmonitored-and-missing is invisible to every wanted list. - Check
isAvailablebefore firing searches. Announced-but-unreleased titles waste indexer budget forever — 22 of the outstanding requests were unreleased. - Batch
MoviesSearchin groups of ~20 with a pause. One command with 115 ids blocks the queue you just cleared. - On
PUT /movie/{id},idis read-only in the body. Sending it returnsrequest.body.id is read-only. - Sonarr: read
episodeCountvsepisodeFileCountper series. The gap is real missing episodes;totalEpisodeCountincludes specials and unaired and will terrify you for no reason.
G · Indexers — check the count, not the config
- The line that matters is in the Radarr log:
Searching indexers for [X]. N active indexers. Configured ≠ active. Three were configured; one was working. - Read
/api/v3/healthon each *arr for “Indexers unavailable due to failures” and “…for more than 6 hours”. - Read Prowlarr
/api/v1/indexerstatusfordisabledTill. An indexer can be enabled in the UI and disabled in reality until tomorrow. POST /indexer/teston every indexer individually. Failures sort into four kinds, each with a different fix:- Cloudflare protection → route through FlareSolverr
- Name does not resolve → dead domain, needs a live mirror
- Forbidden → IP or UA blocked
- Definition does not exist → remove and re-add
- FlareSolverr only works on tagged indexers. It was running and solving challenges all along; not one indexer carried the tag. Create the tag, attach it to the proxy, then attach it to each Cloudflare-blocked indexer.
- Add indexers by definition name, test, and delete the ones that fail the test — a broken indexer in the pool slows every search.
- Confirm Prowlarr → app sync actually happened. An indexer can be healthy in Prowlarr and absent from Radarr. Fire
ApplicationIndexerSyncand re-read the app’s indexer list. - Sync is per-app: an indexer with TV-only categories will appear in Sonarr and not Radarr. That is correct, not a fault.
- Watch which indexer supplies your junk. Persistent
.nfo-only torrents traced to one source, surfacing via an aggregator.
H · Transmission & throughput
- Do not trust the speed cap as the explanation for slow. Compute MB per active slot:
speed ÷ active torrents. - If MB/slot is low (0.4–0.6) and slots are pinned at the ceiling with torrents pending, the ceiling is the bottleneck, not bandwidth. Raise
SLOT_MAXand the peer limits. - Establish a single owner of Transmission settings. An agent writing
session-setevery 60 s will silently overwrite anything you change insettings.jsonor the UI. - Transmission RPC rejects floats.
20.0 × 1024 = 20480.0is refused; cast toint. - Never
except: passaround an RPC call. One swallowed exception hid a rejected speed setting for hours — the cap read 20, the box ran 8. - Check the zombie-prune window. Torrents delivering no video get pruned after 30 min, so the failure never surfaces as an error — it surfaces as silence.
- Verify torrents actually contain video. A completed torrent of
.nfoand.txtimports as nothing and reports success.
I · Import & naming
- Anything stuck at
importPendingis a command-queue or path problem, never a download problem. - Run manual import in small batches. One giant batch is the single-threaded queue jam described in F.
- Verify content matches title with
ffprobeonformat_tags.title— that is how “Shame is actually Submarine” was proven rather than guessed. - Hunt tracker junk in filenames (
www.SITE.org - Title). It is the main cause of Plex refusing to match an item, which presents to you as a missing poster. - Look for TV episodes sitting under the movie roots. They import as nothing and inflate the untracked count.
- Clear metadata-only folders — a folder with an
.nfoand a poster and no video reads as “have it” to a human and “missing” to the app. - When deleting by title match, print the exact list and count rows the way the human is counting them. An off-by-one on a header row cost eight series; an over-broad title match deleted two good films.
J · Plex & Kometa
- Missing posters are almost never a network fault. Check unmatched items first — the cause is filename junk, not IPv6, not DNS.
- Kometa with
restart: "no"and a config error dies in under a second and leaves nothing behind. It had been dead since 18 July on a blank TMDB apikey. - Read the Kometa log for
Config Errorbefore assuming it ran. Success and never-started look identical from outside. - Validate the config actually has: Plex URL + token, TMDB apikey, a generous
timeout, and no unsupported mass-update keys. - Prove Kometa worked with a collection count, not a log line. IMDb Top 250 went 39 → 236 items; collections 88 → 99.
- A collection that reports a small number is usually a library-matching failure, not a missing-media failure.
K · Music
- Check the importer’s hard-coded source directory matches where slskd is actually writing. It didn’t.
- Gate imports on a completeness ratio (tracks present ÷ tracks expected), with a staleness escape hatch, or you import half-albums forever.
- Add a no-manifest fallback for albums with no track count, or they never import at all.
- Prove throughput with the Plex album count before and after, not with the importer log.
L · Scripts & monitors
- Grep every
.pyfor hard-coded paths after any storage change. Thirteen still referenced the old download directory. - Back up before editing, with a dated suffix, every time.
- Audit the monitors themselves: does each one report a reason alongside a zero? If not, it cannot detect this class of outage.
- Counters worth having that were missing: Transmission active/completed, *arr queue depth, *arr import failures, swapout rate.
- Beware Python footguns hit today:
urllib.request.quote(it’surllib.parse.quote), globs in redirect targets, and parsingtransmission-remote -lwith awk instead of using RPC.
M · Close the loop — the proof step
- Nothing is fixed until a number moves. Collect, in this order:
- A grab timestamp in
/api/v3/history?eventType=1for a title you specifically chose - That title present in the download queue with a non-zero percentage
- An import into the library
- The item visible in Plex
- A grab timestamp in
- Re-run the same measurement an hour later. A one-off grab is not a working pipeline.
- Flag anything you cannot explain rather than smoothing it over — 93.6 GB of disk use vanished during the proof window and still has no cause.
- Write down what is still broken. A runbook that only lists wins is a marketing document.