run-task.sh: stop reading the worker's own prose as a quota error
The quota detector grepped `cat "$LOG" "$ERR"` for strings like "429" and "usage
limit". $LOG holds the worker's final JSON, including its .result prose — so a
worker whose TASK was about HTTP 429 finished successfully, wrote a report saying
so, and the runner read its own worker's words, logged "QUOTA hit", slept 600s
and was about to --resume a session that had already succeeded. That would have
burned quota redoing finished work with a fresh worker loose on a completed tree.
Two independent defences, because either alone suffices:
- the blob is now stderr plus the result text ONLY when is_error is true,
falling back to the whole log when it isn't valid JSON (a hard crash writes
no JSON, and has no prose to be confused by).
- the success check runs BEFORE the quota and session-vanished checks. A
finished worker is finished regardless of what strings its output contains.
Regression-checked against the real logs/I-gbkey.json that triggered this: the
old logic matches the quota pattern, the new logic yields SUCCESS.
Recorded as HAZARD #9 in docs/HANDOFF.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
486f6ebc48
commit
26f76bf0f7
2 files changed
+35
-15
No files matched your search
+13
-6
@@ -555,9 +555,16 @@ the sleep elapsed, and appended a note to `logs/<name>.state` saying why.
|
|||||||
finished:** a non-empty `logs/<name>.json` with `is_error:false` means it
|
finished:** a non-empty `logs/<name>.json` with `is_error:false` means it
|
||||||
SUCCEEDED and the runner is about to waste a session.
|
SUCCEEDED and the runner is about to waste a session.
|
||||||
|
|
||||||
The real fix, for whoever next touches the runner (write a NEW file; never edit
|
**FIXED 2026-09-12 in `run-task.sh` itself** (safe: no workers were running —
|
||||||
`run-task.sh` while workers are running): the detector must read only the
|
the standing rule is only about editing it *while* a wave is live). Two defences,
|
||||||
transport-level error stream, not `.result` prose — e.g. grep `$ERR` alone, or
|
because either alone would have prevented this:
|
||||||
`jq -r 'select(.is_error==true) | .result'`, rather than `cat "$LOG" "$ERR"`.
|
1. The detector blob is now stderr plus `jq -r 'select(.is_error==true) | .result'`
|
||||||
Leaving it as-is means any future wave whose subject matter mentions rate limits
|
— the worker's prose reaches it ONLY when the run actually errored. If the log
|
||||||
or 429 will loop this way.
|
isn't valid JSON at all (a hard crash), it falls back to the whole log, where
|
||||||
|
there is no prose to be confused by.
|
||||||
|
2. The success check moved ABOVE the quota and session-vanished checks. A finished
|
||||||
|
worker is finished regardless of what strings appear in its output.
|
||||||
|
|
||||||
|
Regression-checked against the real `logs/I-gbkey.json` that caused this: the old
|
||||||
|
logic matches the quota pattern, the new logic yields SUCCESS and feeds the
|
||||||
|
detector an empty blob.
|
||||||
+22
-9
@@ -57,7 +57,27 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
|
|||||||
fi
|
fi
|
||||||
rc=$?
|
rc=$?
|
||||||
|
|
||||||
blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)"
|
# HAZARD #9 — the detectors below must NEVER see the worker's own prose. The
|
||||||
|
# wave-7 worker's task was ABOUT HTTP 429; it finished successfully, wrote a
|
||||||
|
# report mentioning 429, and this runner read its own worker's words, declared
|
||||||
|
# a quota hit, and was about to --resume a session that had already SUCCEEDED.
|
||||||
|
# So: stderr always, plus the result text ONLY when the run actually errored.
|
||||||
|
# A hard crash may write no JSON at all, in which case fall back to the whole
|
||||||
|
# log — there is no prose to be confused by in that case.
|
||||||
|
if jq -e . "$LOG" >/dev/null 2>&1; then
|
||||||
|
blob="$( { cat "$ERR"; jq -r 'select(.is_error==true) | (.result // "")' "$LOG"; } 2>/dev/null | head -c 20000)"
|
||||||
|
else
|
||||||
|
blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 0) success -> accept it before any error-string matching can second-guess it.
|
||||||
|
# This ordering is itself a guard: a finished worker is finished no matter what
|
||||||
|
# strings appear anywhere in its output.
|
||||||
|
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
|
||||||
|
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
|
||||||
|
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
# 1) quota / rate limit -> wait it out, do NOT burn a hard-fail
|
# 1) quota / rate limit -> wait it out, do NOT burn a hard-fail
|
||||||
if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then
|
if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then
|
||||||
@@ -74,14 +94,7 @@ while [ "$(date +%s)" -lt "$deadline" ]; do
|
|||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# 3) success
|
# 3) genuine failure
|
||||||
isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)"
|
|
||||||
if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then
|
|
||||||
say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)"
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
|
|
||||||
# 4) genuine failure
|
|
||||||
hard=$((hard+1))
|
hard=$((hard+1))
|
||||||
say "hard failure #$hard (rc=$rc is_error=$isErr)"
|
say "hard failure #$hard (rc=$rc is_error=$isErr)"
|
||||||
if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi
|
if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi
|
||||||
|
|||||||
Reference in new issue
Block a user