From 26f76bf0f7bc51fabe49cec3fc4634fa2b8d869b Mon Sep 17 00:00:00 2001 From: Sprite Date: Sat, 12 Sep 2026 11:08:05 +0000 Subject: [PATCH] run-task.sh: stop reading the worker's own prose as a quota error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The quota detector grepped `cat "$LOG" "$ERR"` for strings like "429" and "usage limit". $LOG holds the worker's final JSON, including its .result prose — so a worker whose TASK was about HTTP 429 finished successfully, wrote a report saying so, and the runner read its own worker's words, logged "QUOTA hit", slept 600s and was about to --resume a session that had already succeeded. That would have burned quota redoing finished work with a fresh worker loose on a completed tree. Two independent defences, because either alone suffices: - the blob is now stderr plus the result text ONLY when is_error is true, falling back to the whole log when it isn't valid JSON (a hard crash writes no JSON, and has no prose to be confused by). - the success check runs BEFORE the quota and session-vanished checks. A finished worker is finished regardless of what strings its output contains. Regression-checked against the real logs/I-gbkey.json that triggered this: the old logic matches the quota pattern, the new logic yields SUCCESS. Recorded as HAZARD #9 in docs/HANDOFF.md. Co-Authored-By: Claude Opus 5 --- docs/HANDOFF.md | 19 +++++++++++++------ tasks/run-task.sh | 31 ++++++++++++++++++++++--------- 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index dea52ba..c8ddc2f 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -555,9 +555,16 @@ the sleep elapsed, and appended a note to `logs/.state` saying why. finished:** a non-empty `logs/.json` with `is_error:false` means it SUCCEEDED and the runner is about to waste a session. -The real fix, for whoever next touches the runner (write a NEW file; never edit -`run-task.sh` while workers are running): the detector must read only the -transport-level error stream, not `.result` prose — e.g. grep `$ERR` alone, or -`jq -r 'select(.is_error==true) | .result'`, rather than `cat "$LOG" "$ERR"`. -Leaving it as-is means any future wave whose subject matter mentions rate limits -or 429 will loop this way. +**FIXED 2026-09-12 in `run-task.sh` itself** (safe: no workers were running — +the standing rule is only about editing it *while* a wave is live). Two defences, +because either alone would have prevented this: +1. The detector blob is now stderr plus `jq -r 'select(.is_error==true) | .result'` + — the worker's prose reaches it ONLY when the run actually errored. If the log + isn't valid JSON at all (a hard crash), it falls back to the whole log, where + there is no prose to be confused by. +2. The success check moved ABOVE the quota and session-vanished checks. A finished + worker is finished regardless of what strings appear in its output. + +Regression-checked against the real `logs/I-gbkey.json` that caused this: the old +logic matches the quota pattern, the new logic yields SUCCESS and feeds the +detector an empty blob. diff --git a/tasks/run-task.sh b/tasks/run-task.sh index 7ed8280..a21e6d4 100755 --- a/tasks/run-task.sh +++ b/tasks/run-task.sh @@ -57,7 +57,27 @@ while [ "$(date +%s)" -lt "$deadline" ]; do fi rc=$? - blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)" + # HAZARD #9 — the detectors below must NEVER see the worker's own prose. The + # wave-7 worker's task was ABOUT HTTP 429; it finished successfully, wrote a + # report mentioning 429, and this runner read its own worker's words, declared + # a quota hit, and was about to --resume a session that had already SUCCEEDED. + # So: stderr always, plus the result text ONLY when the run actually errored. + # A hard crash may write no JSON at all, in which case fall back to the whole + # log — there is no prose to be confused by in that case. + if jq -e . "$LOG" >/dev/null 2>&1; then + blob="$( { cat "$ERR"; jq -r 'select(.is_error==true) | (.result // "")' "$LOG"; } 2>/dev/null | head -c 20000)" + else + blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)" + fi + + # 0) success -> accept it before any error-string matching can second-guess it. + # This ordering is itself a guard: a finished worker is finished no matter what + # strings appear anywhere in its output. + isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)" + if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then + say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)" + break + fi # 1) quota / rate limit -> wait it out, do NOT burn a hard-fail if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then @@ -74,14 +94,7 @@ while [ "$(date +%s)" -lt "$deadline" ]; do continue fi - # 3) success - isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)" - if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then - say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)" - break - fi - - # 4) genuine failure + # 3) genuine failure hard=$((hard+1)) say "hard failure #$hard (rc=$rc is_error=$isErr)" if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi