diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index 70873e5..112d617 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -628,7 +628,7 @@ Regression-checked against the real `logs/I-gbkey.json` that caused this: the ol logic matches the quota pattern, the new logic yields SUCCESS and feeds the detector an empty blob. -## Wave 8 — J-crashsafe: IN FLIGHT, launched 2026-09-12 17:50Z +## Wave 8 — J-crashsafe: launched 2026-09-12 17:50Z — COMPLETE, see "accepted" below Prompt: `tasks/J-crashsafe.txt`. Session id in `logs/J-crashsafe.sid`. Lease `bookshelf-wave` held by `tasks/wave-guard.sh`, sentinel `logs/WAVE8-DONE`. @@ -682,3 +682,57 @@ chain leaves the process hung instead of dying. **Not in scope, deliberately:** finding the original throwing line. Nobody knows what it was and the prompt says not to hunt for it. If the guard lands and the user ever sees "unexpected: SomeException" on the scan sheet, THAT is when we learn the answer. + +## Wave 8 — J-crashsafe: COMPLETE, verified by the orchestrator 2026-09-13 +Commit `3409726`. Worker finished 09-12 18:20Z: 1 attempt, 0 quota waits, 111 turns, +$4.02. Its report was honest and every claim below re-checked. + +| Check | Result | +|---|---| +| `./tasks/gw assembleDebug --rerun-tasks` | exit 0 | +| `./tasks/gw testDebugUnitTest` | exit 0 — **205 tests**, 2 skipped, 0 failures (was 190), from `TEST-*.xml` | +| `./tasks/gw verifyPaparazziDebug` | exit 0 | +| `grep "always 'false'"` on the rerun build | **0 hits** | +| boundary check | clean — data/metadata, ui/scan, ui/settings, new `diagnostics/`, CrashReporter wiring in AppContainer + BookshelfApplication; no build files | + +Read, not just trusted: `CancellationException` is caught and rethrown AHEAD of the +`Throwable` catch in both clients and in `ScanViewModel.runLookup` (the import is +`kotlinx.coroutines.CancellationException`, the same type). Reasons carry +`e.javaClass.simpleName` only. `RetryPolicy` does not retry UNEXPECTED. +`CrashReporter.install()` calls `previous?.uncaughtException` in a `finally`, so the +process still dies even if writing the report throws. The Diagnostics PNGs (empty and +populated, light and dark) match the rest of settings. One nit: outlined mahogany +buttons on the dark ground are low-contrast, but "Sign out" already looked like that. + +**Open, flagged by the worker, deliberately out of scope:** `performSave` / +`performSaveManualEntry` make the same unguarded Room calls `runLookup` used to. A +disk failure while SAVING can still crash the process. Same failure class; small +follow-up if wanted. + +**The payoff is on the phone, not here.** If the original crash recurs, it now lands +either as "unexpected: SomeException" on the scan sheet (caught) or as a stored trace +under Settings → Diagnostics → Share (uncaught). Either one finally tells us the class. + +### HAZARD #10 — the wave-guard held the sprite hot for 17 hours +The guard's loop was `while pgrep -f 'run-task\.sh|run-resume\.sh'`. The orchestrator +launched the wave from ONE `bash -c '... setsid nohup ./tasks/run-task.sh ... & +... wave-guard.sh ... & sleep 8; ...'`, and that shell (re-parented to PID 1) was +still alive the next day. Its command line contains `run-task.sh`, so the guard saw +"workers still running" from 18:20Z on 09-12 until 11:16Z on 09-13, renewing the lease +every 15 min. This is HAZARD #6 again, inside the guard itself. Worse than hazard +#8: #8 fails SAFE (the sprite sleeps); this fails EXPENSIVE, and silently, because the +guard log says "workers still running" either way. + +Resolved by killing that one stale shell by PID. The guard then exited normally and +wrote `WAVE8-DONE` and released the lease. + +**Fixed in `tasks/wave-guard.sh`** (safe: nothing was running). `workers_running()` +now matches argv SHAPE: argv[0] is bash, argv[1] IS `run-task.sh`/`run-resume.sh`, +and argv[2] is one of THIS wave's task names. A `bash -c` has argv[1] = `-c` and +cannot match. Tested with no worker, with a decoy `bash -c` containing +`./tasks/run-task.sh J-fake`, with a real-shaped detached worker (and with a +non-matching task name), and after killing it. Only the real worker counts as running. + +**When checking a wave, compare the guard log with the worker's `.state`:** a +`workers still running` renewal timestamped AFTER `.state` says SUCCESS means +the guard is stuck. It is not a slow worker. diff --git a/logs/I-gbkey.sid b/logs/I-gbkey.sid new file mode 100644 index 0000000..8986fb9 --- /dev/null +++ b/logs/I-gbkey.sid @@ -0,0 +1 @@ +7f72ab05-2e62-49a0-858f-638646c68767 diff --git a/logs/WAVE7-DONE b/logs/WAVE7-DONE new file mode 100644 index 0000000..2e8cbaa --- /dev/null +++ b/logs/WAVE7-DONE @@ -0,0 +1,10 @@ +=== WAVE7-DONE written 2026-09-11T23:25:00+00:00 === +Workers finished. The orchestrator was NOT necessarily alive for this. + +--- I-gbkey --- +[2026-09-11T23:23:43+00:00] I-gbkey: QUOTA hit (wait #1). sleeping 600s then probing again. +cost=$1.5929609999999998 turns=51 + +NEXT: orchestrator must independently verify before accepting: + cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest + git status --porcelain # boundary check: who touched what diff --git a/tasks/wave-guard.sh b/tasks/wave-guard.sh index 4e393d3..9686b63 100755 --- a/tasks/wave-guard.sh +++ b/tasks/wave-guard.sh @@ -57,7 +57,20 @@ else fi last_renew=$(date +%s) -while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do +# WAVE 8 POST-MORTEM — why this is not `pgrep -f 'run-task\.sh'`: +# pgrep -f matches ANY command line containing that text. The orchestrator's own +# launching `bash -c '... setsid nohup ./tasks/run-task.sh ...'` stayed alive after +# the worker finished, so the guard saw "workers still running" for 17 hours and +# held the sprite hot the whole time (HAZARD #6). Match on argv SHAPE instead: the +# interpreter is bash, argv[1] IS the runner script, and argv[2] is one of THIS +# wave's task names. A `bash -c` has argv[1] = "-c" and can never match. +workers_running() { + ps -eo args= | awk -v tasks=" ${TASKS[*]} " ' + $1 ~ /(^|\/)bash$/ && $2 ~ /(^|\/)run-(task|resume)\.sh$/ && index(tasks, " " $3 " ") { found=1 } + END { exit !found }' +} + +while workers_running; do sleep "$POLL" now=$(date +%s) if [ $(( now - last_renew )) -ge "$RENEW" ]; then