run-task.sh: stop reading the worker's own prose as a quota error
The quota detector grepped `cat "$LOG" "$ERR"` for strings like "429" and "usage
limit". $LOG holds the worker's final JSON, including its .result prose — so a
worker whose TASK was about HTTP 429 finished successfully, wrote a report saying
so, and the runner read its own worker's words, logged "QUOTA hit", slept 600s
and was about to --resume a session that had already succeeded. That would have
burned quota redoing finished work with a fresh worker loose on a completed tree.
Two independent defences, because either alone suffices:
- the blob is now stderr plus the result text ONLY when is_error is true,
falling back to the whole log when it isn't valid JSON (a hard crash writes
no JSON, and has no prose to be confused by).
- the success check runs BEFORE the quota and session-vanished checks. A
finished worker is finished regardless of what strings its output contains.
Regression-checked against the real logs/I-gbkey.json that triggered this: the
old logic matches the quota pattern, the new logic yields SUCCESS.
Recorded as HAZARD #9 in docs/HANDOFF.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
486f6ebc48
commit
26f76bf0f7
2 files changed
+35
-15
No files matched your search
+22
-9
@@ -57,7 +57,27 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
fi
|
||||
rc=$?
|
||||
|
||||
blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)"
|
||||
# HAZARD #9 — the detectors below must NEVER see the worker's own prose. The
|
||||
# wave-7 worker's task was ABOUT HTTP 429; it finished successfully, wrote a
|
||||
# report mentioning 429, and this runner read its own worker's words, declared
|
||||
# a quota hit, and was about to --resume a session that had already SUCCEEDED.
|
||||
# So: stderr always, plus the result text ONLY when the run actually errored.
|
||||
# A hard crash may write no JSON at all, in which case fall back to the whole
|
||||
# log — there is no prose to be confused by in that case.
|
||||
if jq -e . "$LOG" >/dev/null 2>&1; then
|
||||
blob="$( { cat "$ERR"; jq -r 'select(.is_error==true) | (.result // "")' "$LOG"; } 2>/dev/null | head -c 20000)"
|
||||
else
|
||||
blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)"
|
||||
fi
|
||||
|
||||
# 0) success -> accept it before any error-string matching can second-guess it.
|
||||
# This ordering is itself a guard: a finished worker is finished no matter what
|
||||
# strings appear anywhere in its output.
|
||||
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
|
||||
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
|
||||
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
|
||||
break
|
||||
fi
|
||||
|
||||
# 1) quota / rate limit -> wait it out, do NOT burn a hard-fail
|
||||
if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then
|
||||
@@ -74,14 +94,7 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
continue
|
||||
fi
|
||||
|
||||
# 3) success
|
||||
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
|
||||
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
|
||||
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
|
||||
break
|
||||
fi
|
||||
|
||||
# 4) genuine failure
|
||||
# 3) genuine failure
|
||||
hard=$((hard+1))
|
||||
say "hard failure #$hard (rc=$rc is_error=$isErr)"
|
||||
if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi
|
||||
|
||||
Reference in new issue
Block a user