run-task.sh: stop reading the worker's own prose as a quota error
The quota detector grepped `cat "$LOG" "$ERR"` for strings like "429" and "usage
limit". $LOG holds the worker's final JSON, including its .result prose — so a
worker whose TASK was about HTTP 429 finished successfully, wrote a report saying
so, and the runner read its own worker's words, logged "QUOTA hit", slept 600s
and was about to --resume a session that had already succeeded. That would have
burned quota redoing finished work with a fresh worker loose on a completed tree.
Two independent defences, because either alone suffices:
- the blob is now stderr plus the result text ONLY when is_error is true,
falling back to the whole log when it isn't valid JSON (a hard crash writes
no JSON, and has no prose to be confused by).
- the success check runs BEFORE the quota and session-vanished checks. A
finished worker is finished regardless of what strings its output contains.
Regression-checked against the real logs/I-gbkey.json that triggered this: the
old logic matches the quota pattern, the new logic yields SUCCESS.
Recorded as HAZARD #9 in docs/HANDOFF.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
486f6ebc48
commit
26f76bf0f7
2 files changed
+35
-15
No files matched your search
+13
-6
@@ -555,9 +555,16 @@ the sleep elapsed, and appended a note to `logs/<name>.state` saying why.
|
||||
finished:** a non-empty `logs/<name>.json` with `is_error:false` means it
|
||||
SUCCEEDED and the runner is about to waste a session.
|
||||
|
||||
The real fix, for whoever next touches the runner (write a NEW file; never edit
|
||||
`run-task.sh` while workers are running): the detector must read only the
|
||||
transport-level error stream, not `.result` prose — e.g. grep `$ERR` alone, or
|
||||
`jq -r 'select(.is_error==true) | .result'`, rather than `cat "$LOG" "$ERR"`.
|
||||
Leaving it as-is means any future wave whose subject matter mentions rate limits
|
||||
or 429 will loop this way.
|
||||
**FIXED 2026-09-12 in `run-task.sh` itself** (safe: no workers were running —
|
||||
the standing rule is only about editing it *while* a wave is live). Two defences,
|
||||
because either alone would have prevented this:
|
||||
1. The detector blob is now stderr plus `jq -r 'select(.is_error==true) | .result'`
|
||||
— the worker's prose reaches it ONLY when the run actually errored. If the log
|
||||
isn't valid JSON at all (a hard crash), it falls back to the whole log, where
|
||||
there is no prose to be confused by.
|
||||
2. The success check moved ABOVE the quota and session-vanished checks. A finished
|
||||
worker is finished regardless of what strings appear in its output.
|
||||
|
||||
Regression-checked against the real `logs/I-gbkey.json` that caused this: the old
|
||||
logic matches the quota pattern, the new logic yields SUCCESS and feeds the
|
||||
detector an empty blob.
|
||||
Reference in new issue
Block a user