wave-guard: match workers by argv shape; accept wave 8; record HAZARD #10
The guard's `pgrep -f 'run-task\.sh'` matched the orchestrator's own stale launching shell, whose command line contains that text, so it kept the sprite awake for 17 hours after J-crashsafe finished. workers_running() now requires argv[1] to be the runner script and argv[2] to be one of this wave's tasks. Also commits the wave-7 sid/sentinel that were never added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AQb3d68LqWD4EdND3C8jkD
This commit is contained in:
1 parent
34097269c9
commit
007a1f426d
4 files changed
+80
-2
No files matched your search
+55
-1
@@ -628,7 +628,7 @@ Regression-checked against the real `logs/I-gbkey.json` that caused this: the ol
|
||||
logic matches the quota pattern, the new logic yields SUCCESS and feeds the
|
||||
detector an empty blob.
|
||||
|
||||
## Wave 8 — J-crashsafe: IN FLIGHT, launched 2026-09-12 17:50Z
|
||||
## Wave 8 — J-crashsafe: launched 2026-09-12 17:50Z — COMPLETE, see "accepted" below
|
||||
Prompt: `tasks/J-crashsafe.txt`. Session id in `logs/J-crashsafe.sid`. Lease
|
||||
`bookshelf-wave` held by `tasks/wave-guard.sh`, sentinel `logs/WAVE8-DONE`.
|
||||
|
||||
@@ -682,3 +682,57 @@ chain leaves the process hung instead of dying.
|
||||
**Not in scope, deliberately:** finding the original throwing line. Nobody knows what it
|
||||
was and the prompt says not to hunt for it. If the guard lands and the user ever sees
|
||||
"unexpected: SomeException" on the scan sheet, THAT is when we learn the answer.
|
||||
|
||||
## Wave 8 — J-crashsafe: COMPLETE, verified by the orchestrator 2026-09-13
|
||||
Commit `3409726`. Worker finished 09-12 18:20Z: 1 attempt, 0 quota waits, 111 turns,
|
||||
$4.02. Its report was honest and every claim below re-checked.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `./tasks/gw assembleDebug --rerun-tasks` | exit 0 |
|
||||
| `./tasks/gw testDebugUnitTest` | exit 0 — **205 tests**, 2 skipped, 0 failures (was 190), from `TEST-*.xml` |
|
||||
| `./tasks/gw verifyPaparazziDebug` | exit 0 |
|
||||
| `grep "always 'false'"` on the rerun build | **0 hits** |
|
||||
| boundary check | clean — data/metadata, ui/scan, ui/settings, new `diagnostics/`, CrashReporter wiring in AppContainer + BookshelfApplication; no build files |
|
||||
|
||||
Read, not just trusted: `CancellationException` is caught and rethrown AHEAD of the
|
||||
`Throwable` catch in both clients and in `ScanViewModel.runLookup` (the import is
|
||||
`kotlinx.coroutines.CancellationException`, the same type). Reasons carry
|
||||
`e.javaClass.simpleName` only. `RetryPolicy` does not retry UNEXPECTED.
|
||||
`CrashReporter.install()` calls `previous?.uncaughtException` in a `finally`, so the
|
||||
process still dies even if writing the report throws. The Diagnostics PNGs (empty and
|
||||
populated, light and dark) match the rest of settings. One nit: outlined mahogany
|
||||
buttons on the dark ground are low-contrast, but "Sign out" already looked like that.
|
||||
|
||||
**Open, flagged by the worker, deliberately out of scope:** `performSave` /
|
||||
`performSaveManualEntry` make the same unguarded Room calls `runLookup` used to. A
|
||||
disk failure while SAVING can still crash the process. Same failure class; small
|
||||
follow-up if wanted.
|
||||
|
||||
**The payoff is on the phone, not here.** If the original crash recurs, it now lands
|
||||
either as "unexpected: SomeException" on the scan sheet (caught) or as a stored trace
|
||||
under Settings → Diagnostics → Share (uncaught). Either one finally tells us the class.
|
||||
|
||||
### HAZARD #10 — the wave-guard held the sprite hot for 17 hours
|
||||
The guard's loop was `while pgrep -f 'run-task\.sh|run-resume\.sh'`. The orchestrator
|
||||
launched the wave from ONE `bash -c '... setsid nohup ./tasks/run-task.sh ... &
|
||||
... wave-guard.sh ... & sleep 8; ...'`, and that shell (re-parented to PID 1) was
|
||||
still alive the next day. Its command line contains `run-task.sh`, so the guard saw
|
||||
"workers still running" from 18:20Z on 09-12 until 11:16Z on 09-13, renewing the lease
|
||||
every 15 min. This is HAZARD #6 again, inside the guard itself. Worse than hazard
|
||||
#8: #8 fails SAFE (the sprite sleeps); this fails EXPENSIVE, and silently, because the
|
||||
guard log says "workers still running" either way.
|
||||
|
||||
Resolved by killing that one stale shell by PID. The guard then exited normally and
|
||||
wrote `WAVE8-DONE` and released the lease.
|
||||
|
||||
**Fixed in `tasks/wave-guard.sh`** (safe: nothing was running). `workers_running()`
|
||||
now matches argv SHAPE: argv[0] is bash, argv[1] IS `run-task.sh`/`run-resume.sh`,
|
||||
and argv[2] is one of THIS wave's task names. A `bash -c` has argv[1] = `-c` and
|
||||
cannot match. Tested with no worker, with a decoy `bash -c` containing
|
||||
`./tasks/run-task.sh J-fake`, with a real-shaped detached worker (and with a
|
||||
non-matching task name), and after killing it. Only the real worker counts as running.
|
||||
|
||||
**When checking a wave, compare the guard log with the worker's `.state`:** a
|
||||
`workers still running` renewal timestamped AFTER `.state` says SUCCESS means
|
||||
the guard is stuck. It is not a slow worker.
|
||||
@@ -0,0 +1 @@
|
||||
7f72ab05-2e62-49a0-858f-638646c68767
|
||||
@@ -0,0 +1,10 @@
|
||||
=== WAVE7-DONE written 2026-09-11T23:25:00+00:00 ===
|
||||
Workers finished. The orchestrator was NOT necessarily alive for this.
|
||||
|
||||
--- I-gbkey ---
|
||||
[2026-09-11T23:23:43+00:00] I-gbkey: QUOTA hit (wait #1). sleeping 600s then probing again.
|
||||
cost=$1.5929609999999998 turns=51
|
||||
|
||||
NEXT: orchestrator must independently verify before accepting:
|
||||
cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest
|
||||
git status --porcelain # boundary check: who touched what
|
||||
+14
-1
@@ -57,7 +57,20 @@ else
|
||||
fi
|
||||
last_renew=$(date +%s)
|
||||
|
||||
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
|
||||
# WAVE 8 POST-MORTEM — why this is not `pgrep -f 'run-task\.sh'`:
|
||||
# pgrep -f matches ANY command line containing that text. The orchestrator's own
|
||||
# launching `bash -c '... setsid nohup ./tasks/run-task.sh ...'` stayed alive after
|
||||
# the worker finished, so the guard saw "workers still running" for 17 hours and
|
||||
# held the sprite hot the whole time (HAZARD #6). Match on argv SHAPE instead: the
|
||||
# interpreter is bash, argv[1] IS the runner script, and argv[2] is one of THIS
|
||||
# wave's task names. A `bash -c` has argv[1] = "-c" and can never match.
|
||||
workers_running() {
|
||||
ps -eo args= | awk -v tasks=" ${TASKS[*]} " '
|
||||
$1 ~ /(^|\/)bash$/ && $2 ~ /(^|\/)run-(task|resume)\.sh$/ && index(tasks, " " $3 " ") { found=1 }
|
||||
END { exit !found }'
|
||||
}
|
||||
|
||||
while workers_running; do
|
||||
sleep "$POLL"
|
||||
now=$(date +%s)
|
||||
if [ $(( now - last_renew )) -ge "$RENEW" ]; then
|
||||
|
||||
Reference in new issue
Block a user