run-task.sh: stop reading the worker's own prose as a quota error

The quota detector grepped `cat "$LOG" "$ERR"` for strings like "429" and "usage
limit". $LOG holds the worker's final JSON, including its .result prose — so a
worker whose TASK was about HTTP 429 finished successfully, wrote a report saying
so, and the runner read its own worker's words, logged "QUOTA hit", slept 600s
and was about to --resume a session that had already succeeded. That would have
burned quota redoing finished work with a fresh worker loose on a completed tree.

Two independent defences, because either alone suffices:
  - the blob is now stderr plus the result text ONLY when is_error is true,
    falling back to the whole log when it isn't valid JSON (a hard crash writes
    no JSON, and has no prose to be confused by).
  - the success check runs BEFORE the quota and session-vanished checks. A
    finished worker is finished regardless of what strings its output contains.

Regression-checked against the real logs/I-gbkey.json that triggered this: the
old logic matches the quota pattern, the new logic yields SUCCESS.

Recorded as HAZARD #9 in docs/HANDOFF.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Spriteandclaude committed 2026-09-12 11:08:05 +00:00
1 parent 486f6ebc48
commit 26f76bf0f7
2 files changed
+35 -15

No files matched your search

+22 -9
View File
@@ -57,7 +57,27 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
fi
rc=$?
blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)"
# HAZARD #9 — the detectors below must NEVER see the worker's own prose. The
# wave-7 worker's task was ABOUT HTTP 429; it finished successfully, wrote a
# report mentioning 429, and this runner read its own worker's words, declared
# a quota hit, and was about to --resume a session that had already SUCCEEDED.
# So: stderr always, plus the result text ONLY when the run actually errored.
# A hard crash may write no JSON at all, in which case fall back to the whole
# log — there is no prose to be confused by in that case.
if jq -e . "$LOG" >/dev/null 2>&1; then
blob="$( { cat "$ERR"; jq -r 'select(.is_error==true) | (.result // "")' "$LOG"; } 2>/dev/null | head -c 20000)"
else
blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)"
fi
# 0) success -> accept it before any error-string matching can second-guess it.
# This ordering is itself a guard: a finished worker is finished no matter what
# strings appear anywhere in its output.
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
break
fi
# 1) quota / rate limit -> wait it out, do NOT burn a hard-fail
if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then
@@ -74,14 +94,7 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
continue
fi
# 3) success
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
break
fi
# 4) genuine failure
# 3) genuine failure
hard=$((hard+1))
say "hard failure #$hard (rc=$rc is_error=$isErr)"
if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi