wave-guard: match workers by argv shape; accept wave 8; record HAZARD #10
The guard's `pgrep -f 'run-task\.sh'` matched the orchestrator's own stale launching shell, whose command line contains that text, so it kept the sprite awake for 17 hours after J-crashsafe finished. workers_running() now requires argv[1] to be the runner script and argv[2] to be one of this wave's tasks. Also commits the wave-7 sid/sentinel that were never added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AQb3d68LqWD4EdND3C8jkD
This commit is contained in:
1 parent
34097269c9
commit
007a1f426d
4 files changed
+80
-2
No files matched your search
+55
-1
@@ -628,7 +628,7 @@ Regression-checked against the real `logs/I-gbkey.json` that caused this: the ol
|
|||||||
logic matches the quota pattern, the new logic yields SUCCESS and feeds the
|
logic matches the quota pattern, the new logic yields SUCCESS and feeds the
|
||||||
detector an empty blob.
|
detector an empty blob.
|
||||||
|
|
||||||
## Wave 8 — J-crashsafe: IN FLIGHT, launched 2026-09-12 17:50Z
|
## Wave 8 — J-crashsafe: launched 2026-09-12 17:50Z — COMPLETE, see "accepted" below
|
||||||
Prompt: `tasks/J-crashsafe.txt`. Session id in `logs/J-crashsafe.sid`. Lease
|
Prompt: `tasks/J-crashsafe.txt`. Session id in `logs/J-crashsafe.sid`. Lease
|
||||||
`bookshelf-wave` held by `tasks/wave-guard.sh`, sentinel `logs/WAVE8-DONE`.
|
`bookshelf-wave` held by `tasks/wave-guard.sh`, sentinel `logs/WAVE8-DONE`.
|
||||||
|
|
||||||
@@ -682,3 +682,57 @@ chain leaves the process hung instead of dying.
|
|||||||
**Not in scope, deliberately:** finding the original throwing line. Nobody knows what it
|
**Not in scope, deliberately:** finding the original throwing line. Nobody knows what it
|
||||||
was and the prompt says not to hunt for it. If the guard lands and the user ever sees
|
was and the prompt says not to hunt for it. If the guard lands and the user ever sees
|
||||||
"unexpected: SomeException" on the scan sheet, THAT is when we learn the answer.
|
"unexpected: SomeException" on the scan sheet, THAT is when we learn the answer.
|
||||||
|
|
||||||
|
## Wave 8 — J-crashsafe: COMPLETE, verified by the orchestrator 2026-09-13
|
||||||
|
Commit `3409726`. Worker finished 09-12 18:20Z: 1 attempt, 0 quota waits, 111 turns,
|
||||||
|
$4.02. Its report was honest and every claim below re-checked.
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| `./tasks/gw assembleDebug --rerun-tasks` | exit 0 |
|
||||||
|
| `./tasks/gw testDebugUnitTest` | exit 0 — **205 tests**, 2 skipped, 0 failures (was 190), from `TEST-*.xml` |
|
||||||
|
| `./tasks/gw verifyPaparazziDebug` | exit 0 |
|
||||||
|
| `grep "always 'false'"` on the rerun build | **0 hits** |
|
||||||
|
| boundary check | clean — data/metadata, ui/scan, ui/settings, new `diagnostics/`, CrashReporter wiring in AppContainer + BookshelfApplication; no build files |
|
||||||
|
|
||||||
|
Read, not just trusted: `CancellationException` is caught and rethrown AHEAD of the
|
||||||
|
`Throwable` catch in both clients and in `ScanViewModel.runLookup` (the import is
|
||||||
|
`kotlinx.coroutines.CancellationException`, the same type). Reasons carry
|
||||||
|
`e.javaClass.simpleName` only. `RetryPolicy` does not retry UNEXPECTED.
|
||||||
|
`CrashReporter.install()` calls `previous?.uncaughtException` in a `finally`, so the
|
||||||
|
process still dies even if writing the report throws. The Diagnostics PNGs (empty and
|
||||||
|
populated, light and dark) match the rest of settings. One nit: outlined mahogany
|
||||||
|
buttons on the dark ground are low-contrast, but "Sign out" already looked like that.
|
||||||
|
|
||||||
|
**Open, flagged by the worker, deliberately out of scope:** `performSave` /
|
||||||
|
`performSaveManualEntry` make the same unguarded Room calls `runLookup` used to. A
|
||||||
|
disk failure while SAVING can still crash the process. Same failure class; small
|
||||||
|
follow-up if wanted.
|
||||||
|
|
||||||
|
**The payoff is on the phone, not here.** If the original crash recurs, it now lands
|
||||||
|
either as "unexpected: SomeException" on the scan sheet (caught) or as a stored trace
|
||||||
|
under Settings → Diagnostics → Share (uncaught). Either one finally tells us the class.
|
||||||
|
|
||||||
|
### HAZARD #10 — the wave-guard held the sprite hot for 17 hours
|
||||||
|
The guard's loop was `while pgrep -f 'run-task\.sh|run-resume\.sh'`. The orchestrator
|
||||||
|
launched the wave from ONE `bash -c '... setsid nohup ./tasks/run-task.sh ... &
|
||||||
|
... wave-guard.sh ... & sleep 8; ...'`, and that shell (re-parented to PID 1) was
|
||||||
|
still alive the next day. Its command line contains `run-task.sh`, so the guard saw
|
||||||
|
"workers still running" from 18:20Z on 09-12 until 11:16Z on 09-13, renewing the lease
|
||||||
|
every 15 min. This is HAZARD #6 again, inside the guard itself. Worse than hazard
|
||||||
|
#8: #8 fails SAFE (the sprite sleeps); this fails EXPENSIVE, and silently, because the
|
||||||
|
guard log says "workers still running" either way.
|
||||||
|
|
||||||
|
Resolved by killing that one stale shell by PID. The guard then exited normally and
|
||||||
|
wrote `WAVE8-DONE` and released the lease.
|
||||||
|
|
||||||
|
**Fixed in `tasks/wave-guard.sh`** (safe: nothing was running). `workers_running()`
|
||||||
|
now matches argv SHAPE: argv[0] is bash, argv[1] IS `run-task.sh`/`run-resume.sh`,
|
||||||
|
and argv[2] is one of THIS wave's task names. A `bash -c` has argv[1] = `-c` and
|
||||||
|
cannot match. Tested with no worker, with a decoy `bash -c` containing
|
||||||
|
`./tasks/run-task.sh J-fake`, with a real-shaped detached worker (and with a
|
||||||
|
non-matching task name), and after killing it. Only the real worker counts as running.
|
||||||
|
|
||||||
|
**When checking a wave, compare the guard log with the worker's `.state`:** a
|
||||||
|
`workers still running` renewal timestamped AFTER `.state` says SUCCESS means
|
||||||
|
the guard is stuck. It is not a slow worker.
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
7f72ab05-2e62-49a0-858f-638646c68767
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
=== WAVE7-DONE written 2026-09-11T23:25:00+00:00 ===
|
||||||
|
Workers finished. The orchestrator was NOT necessarily alive for this.
|
||||||
|
|
||||||
|
--- I-gbkey ---
|
||||||
|
[2026-09-11T23:23:43+00:00] I-gbkey: QUOTA hit (wait #1). sleeping 600s then probing again.
|
||||||
|
cost=$1.5929609999999998 turns=51
|
||||||
|
|
||||||
|
NEXT: orchestrator must independently verify before accepting:
|
||||||
|
cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest
|
||||||
|
git status --porcelain # boundary check: who touched what
|
||||||
+14
-1
@@ -57,7 +57,20 @@ else
|
|||||||
fi
|
fi
|
||||||
last_renew=$(date +%s)
|
last_renew=$(date +%s)
|
||||||
|
|
||||||
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
|
# WAVE 8 POST-MORTEM — why this is not `pgrep -f 'run-task\.sh'`:
|
||||||
|
# pgrep -f matches ANY command line containing that text. The orchestrator's own
|
||||||
|
# launching `bash -c '... setsid nohup ./tasks/run-task.sh ...'` stayed alive after
|
||||||
|
# the worker finished, so the guard saw "workers still running" for 17 hours and
|
||||||
|
# held the sprite hot the whole time (HAZARD #6). Match on argv SHAPE instead: the
|
||||||
|
# interpreter is bash, argv[1] IS the runner script, and argv[2] is one of THIS
|
||||||
|
# wave's task names. A `bash -c` has argv[1] = "-c" and can never match.
|
||||||
|
workers_running() {
|
||||||
|
ps -eo args= | awk -v tasks=" ${TASKS[*]} " '
|
||||||
|
$1 ~ /(^|\/)bash$/ && $2 ~ /(^|\/)run-(task|resume)\.sh$/ && index(tasks, " " $3 " ") { found=1 }
|
||||||
|
END { exit !found }'
|
||||||
|
}
|
||||||
|
|
||||||
|
while workers_running; do
|
||||||
sleep "$POLL"
|
sleep "$POLL"
|
||||||
now=$(date +%s)
|
now=$(date +%s)
|
||||||
if [ $(( now - last_renew )) -ge "$RENEW" ]; then
|
if [ $(( now - last_renew )) -ge "$RENEW" ]; then
|
||||||
|
|||||||
Reference in new issue
Block a user