wave-guard: match workers by argv shape; accept wave 8; record HAZARD #10

The guard's `pgrep -f 'run-task\.sh'` matched the orchestrator's own stale
launching shell, whose command line contains that text, so it kept the sprite
awake for 17 hours after J-crashsafe finished. workers_running() now requires
argv[1] to be the runner script and argv[2] to be one of this wave's tasks.

Also commits the wave-7 sid/sentinel that were never added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AQb3d68LqWD4EdND3C8jkD
This commit is contained in:
Spriteandclaude committed 2026-09-13 11:33:21 +00:00
1 parent 34097269c9
commit 007a1f426d
4 files changed
+80 -2

No files matched your search

+55 -1
View File
@@ -628,7 +628,7 @@ Regression-checked against the real `logs/I-gbkey.json` that caused this: the ol
logic matches the quota pattern, the new logic yields SUCCESS and feeds the
detector an empty blob.
## Wave 8 — J-crashsafe: IN FLIGHT, launched 2026-09-12 17:50Z
## Wave 8 — J-crashsafe: launched 2026-09-12 17:50Z — COMPLETE, see "accepted" below
Prompt: `tasks/J-crashsafe.txt`. Session id in `logs/J-crashsafe.sid`. Lease
`bookshelf-wave` held by `tasks/wave-guard.sh`, sentinel `logs/WAVE8-DONE`.
@@ -682,3 +682,57 @@ chain leaves the process hung instead of dying.
**Not in scope, deliberately:** finding the original throwing line. Nobody knows what it
was and the prompt says not to hunt for it. If the guard lands and the user ever sees
"unexpected: SomeException" on the scan sheet, THAT is when we learn the answer.
## Wave 8 — J-crashsafe: COMPLETE, verified by the orchestrator 2026-09-13
Commit `3409726`. Worker finished 09-12 18:20Z: 1 attempt, 0 quota waits, 111 turns,
$4.02. Its report was honest and every claim below re-checked.
| Check | Result |
|---|---|
| `./tasks/gw assembleDebug --rerun-tasks` | exit 0 |
| `./tasks/gw testDebugUnitTest` | exit 0 — **205 tests**, 2 skipped, 0 failures (was 190), from `TEST-*.xml` |
| `./tasks/gw verifyPaparazziDebug` | exit 0 |
| `grep "always 'false'"` on the rerun build | **0 hits** |
| boundary check | clean — data/metadata, ui/scan, ui/settings, new `diagnostics/`, CrashReporter wiring in AppContainer + BookshelfApplication; no build files |
Read, not just trusted: `CancellationException` is caught and rethrown AHEAD of the
`Throwable` catch in both clients and in `ScanViewModel.runLookup` (the import is
`kotlinx.coroutines.CancellationException`, the same type). Reasons carry
`e.javaClass.simpleName` only. `RetryPolicy` does not retry UNEXPECTED.
`CrashReporter.install()` calls `previous?.uncaughtException` in a `finally`, so the
process still dies even if writing the report throws. The Diagnostics PNGs (empty and
populated, light and dark) match the rest of settings. One nit: outlined mahogany
buttons on the dark ground are low-contrast, but "Sign out" already looked like that.
**Open, flagged by the worker, deliberately out of scope:** `performSave` /
`performSaveManualEntry` make the same unguarded Room calls `runLookup` used to. A
disk failure while SAVING can still crash the process. Same failure class; small
follow-up if wanted.
**The payoff is on the phone, not here.** If the original crash recurs, it now lands
either as "unexpected: SomeException" on the scan sheet (caught) or as a stored trace
under Settings → Diagnostics → Share (uncaught). Either one finally tells us the class.
### HAZARD #10 — the wave-guard held the sprite hot for 17 hours
The guard's loop was `while pgrep -f 'run-task\.sh|run-resume\.sh'`. The orchestrator
launched the wave from ONE `bash -c '... setsid nohup ./tasks/run-task.sh ... &
... wave-guard.sh ... & sleep 8; ...'`, and that shell (re-parented to PID 1) was
still alive the next day. Its command line contains `run-task.sh`, so the guard saw
"workers still running" from 18:20Z on 09-12 until 11:16Z on 09-13, renewing the lease
every 15 min. This is HAZARD #6 again, inside the guard itself. Worse than hazard
#8: #8 fails SAFE (the sprite sleeps); this fails EXPENSIVE, and silently, because the
guard log says "workers still running" either way.
Resolved by killing that one stale shell by PID. The guard then exited normally and
wrote `WAVE8-DONE` and released the lease.
**Fixed in `tasks/wave-guard.sh`** (safe: nothing was running). `workers_running()`
now matches argv SHAPE: argv[0] is bash, argv[1] IS `run-task.sh`/`run-resume.sh`,
and argv[2] is one of THIS wave's task names. A `bash -c` has argv[1] = `-c` and
cannot match. Tested with no worker, with a decoy `bash -c` containing
`./tasks/run-task.sh J-fake`, with a real-shaped detached worker (and with a
non-matching task name), and after killing it. Only the real worker counts as running.
**When checking a wave, compare the guard log with the worker's `.state`:** a
`workers still running` renewal timestamped AFTER `.state` says SUCCESS means
the guard is stuck. It is not a slow worker.
+1
View File
@@ -0,0 +1 @@
7f72ab05-2e62-49a0-858f-638646c68767
+10
View File
@@ -0,0 +1,10 @@
=== WAVE7-DONE written 2026-09-11T23:25:00+00:00 ===
Workers finished. The orchestrator was NOT necessarily alive for this.
--- I-gbkey ---
[2026-09-11T23:23:43+00:00] I-gbkey: QUOTA hit (wait #1). sleeping 600s then probing again.
cost=$1.5929609999999998 turns=51
NEXT: orchestrator must independently verify before accepting:
cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest
git status --porcelain # boundary check: who touched what
+14 -1
View File
@@ -57,7 +57,20 @@ else
fi
last_renew=$(date +%s)
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
# WAVE 8 POST-MORTEM — why this is not `pgrep -f 'run-task\.sh'`:
# pgrep -f matches ANY command line containing that text. The orchestrator's own
# launching `bash -c '... setsid nohup ./tasks/run-task.sh ...'` stayed alive after
# the worker finished, so the guard saw "workers still running" for 17 hours and
# held the sprite hot the whole time (HAZARD #6). Match on argv SHAPE instead: the
# interpreter is bash, argv[1] IS the runner script, and argv[2] is one of THIS
# wave's task names. A `bash -c` has argv[1] = "-c" and can never match.
workers_running() {
ps -eo args= | awk -v tasks=" ${TASKS[*]} " '
$1 ~ /(^|\/)bash$/ && $2 ~ /(^|\/)run-(task|resume)\.sh$/ && index(tasks, " " $3 " ") { found=1 }
END { exit !found }'
}
while workers_running; do
sleep "$POLL"
now=$(date +%s)
if [ $(( now - last_renew )) -ge "$RENEW" ]; then