run-task.sh: stop reading the worker's own prose as a quota error

The quota detector grepped `cat "$LOG" "$ERR"` for strings like "429" and "usage
limit". $LOG holds the worker's final JSON, including its .result prose — so a
worker whose TASK was about HTTP 429 finished successfully, wrote a report saying
so, and the runner read its own worker's words, logged "QUOTA hit", slept 600s
and was about to --resume a session that had already succeeded. That would have
burned quota redoing finished work with a fresh worker loose on a completed tree.

Two independent defences, because either alone suffices:
  - the blob is now stderr plus the result text ONLY when is_error is true,
    falling back to the whole log when it isn't valid JSON (a hard crash writes
    no JSON, and has no prose to be confused by).
  - the success check runs BEFORE the quota and session-vanished checks. A
    finished worker is finished regardless of what strings its output contains.

Regression-checked against the real logs/I-gbkey.json that triggered this: the
old logic matches the quota pattern, the new logic yields SUCCESS.

Recorded as HAZARD #9 in docs/HANDOFF.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Spriteandclaude committed 2026-09-12 11:08:05 +00:00
1 parent 486f6ebc48
commit 26f76bf0f7
2 files changed
+34 -14

No files matched your search

+13 -6
View File
@@ -555,9 +555,16 @@ the sleep elapsed, and appended a note to `logs/<name>.state` saying why.
finished:** a non-empty `logs/<name>.json` with `is_error:false` means it finished:** a non-empty `logs/<name>.json` with `is_error:false` means it
SUCCEEDED and the runner is about to waste a session. SUCCEEDED and the runner is about to waste a session.
The real fix, for whoever next touches the runner (write a NEW file; never edit **FIXED 2026-09-12 in `run-task.sh` itself** (safe: no workers were running —
`run-task.sh` while workers are running): the detector must read only the the standing rule is only about editing it *while* a wave is live). Two defences,
transport-level error stream, not `.result` prose — e.g. grep `$ERR` alone, or because either alone would have prevented this:
`jq -r 'select(.is_error==true) | .result'`, rather than `cat "$LOG" "$ERR"`. 1. The detector blob is now stderr plus `jq -r 'select(.is_error==true) | .result'`
Leaving it as-is means any future wave whose subject matter mentions rate limits — the worker's prose reaches it ONLY when the run actually errored. If the log
or 429 will loop this way. isn't valid JSON at all (a hard crash), it falls back to the whole log, where
there is no prose to be confused by.
2. The success check moved ABOVE the quota and session-vanished checks. A finished
worker is finished regardless of what strings appear in its output.
Regression-checked against the real `logs/I-gbkey.json` that caused this: the old
logic matches the quota pattern, the new logic yields SUCCESS and feeds the
detector an empty blob.
+21 -8
View File
@@ -57,7 +57,27 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
fi fi
rc=$? rc=$?
# HAZARD #9 — the detectors below must NEVER see the worker's own prose. The
# wave-7 worker's task was ABOUT HTTP 429; it finished successfully, wrote a
# report mentioning 429, and this runner read its own worker's words, declared
# a quota hit, and was about to --resume a session that had already SUCCEEDED.
# So: stderr always, plus the result text ONLY when the run actually errored.
# A hard crash may write no JSON at all, in which case fall back to the whole
# log — there is no prose to be confused by in that case.
if jq -e . "$LOG" >/dev/null 2>&1; then
blob="$( { cat "$ERR"; jq -r 'select(.is_error==true) | (.result // "")' "$LOG"; } 2>/dev/null | head -c 20000)"
else
blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)" blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)"
fi
# 0) success -> accept it before any error-string matching can second-guess it.
# This ordering is itself a guard: a finished worker is finished no matter what
# strings appear anywhere in its output.
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
break
fi
# 1) quota / rate limit -> wait it out, do NOT burn a hard-fail # 1) quota / rate limit -> wait it out, do NOT burn a hard-fail
if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then
@@ -74,14 +94,7 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
continue continue
fi fi
# 3) success # 3) genuine failure
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
break
fi
# 4) genuine failure
hard=$((hard+1)) hard=$((hard+1))
say "hard failure #$hard (rc=$rc is_error=$isErr)" say "hard failure #$hard (rc=$rc is_error=$isErr)"
if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi