diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index dea52ba..c8ddc2f 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -555,9 +555,16 @@ the sleep elapsed, and appended a note to `logs/.state` saying why. finished:** a non-empty `logs/.json` with `is_error:false` means it SUCCEEDED and the runner is about to waste a session. -The real fix, for whoever next touches the runner (write a NEW file; never edit -`run-task.sh` while workers are running): the detector must read only the -transport-level error stream, not `.result` prose — e.g. grep `$ERR` alone, or -`jq -r 'select(.is_error==true) | .result'`, rather than `cat "$LOG" "$ERR"`. -Leaving it as-is means any future wave whose subject matter mentions rate limits -or 429 will loop this way. +**FIXED 2026-09-12 in `run-task.sh` itself** (safe: no workers were running — +the standing rule is only about editing it *while* a wave is live). Two defences, +because either alone would have prevented this: +1. The detector blob is now stderr plus `jq -r 'select(.is_error==true) | .result'` + — the worker's prose reaches it ONLY when the run actually errored. If the log + isn't valid JSON at all (a hard crash), it falls back to the whole log, where + there is no prose to be confused by. +2. The success check moved ABOVE the quota and session-vanished checks. A finished + worker is finished regardless of what strings appear in its output. + +Regression-checked against the real `logs/I-gbkey.json` that caused this: the old +logic matches the quota pattern, the new logic yields SUCCESS and feeds the +detector an empty blob. diff --git a/tasks/run-task.sh b/tasks/run-task.sh index 7ed8280..a21e6d4 100755 --- a/tasks/run-task.sh +++ b/tasks/run-task.sh @@ -57,7 +57,27 @@ while [ "$(date +%s)" -lt "$deadline" ]; do fi rc=$? - blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)" + # HAZARD #9 — the detectors below must NEVER see the worker's own prose. The + # wave-7 worker's task was ABOUT HTTP 429; it finished successfully, wrote a + # report mentioning 429, and this runner read its own worker's words, declared + # a quota hit, and was about to --resume a session that had already SUCCEEDED. + # So: stderr always, plus the result text ONLY when the run actually errored. + # A hard crash may write no JSON at all, in which case fall back to the whole + # log — there is no prose to be confused by in that case. + if jq -e . "$LOG" >/dev/null 2>&1; then + blob="$( { cat "$ERR"; jq -r 'select(.is_error==true) | (.result // "")' "$LOG"; } 2>/dev/null | head -c 20000)" + else + blob="$(cat "$LOG" "$ERR" 2>/dev/null | head -c 20000)" + fi + + # 0) success -> accept it before any error-string matching can second-guess it. + # This ordering is itself a guard: a finished worker is finished no matter what + # strings appear anywhere in its output. + isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)" + if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then + say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)" + break + fi # 1) quota / rate limit -> wait it out, do NOT burn a hard-fail if printf '%s' "$blob" | grep -qiE 'usage limit|limit will reset|limit resets|rate_limit_error|rate limit exceeded|429|too many requests|overloaded_error'; then @@ -74,14 +94,7 @@ while [ "$(date +%s)" -lt "$deadline" ]; do continue fi - # 3) success - isErr="$(jq -r '.is_error // false' "$LOG" 2>/dev/null)" - if [ "$rc" -eq 0 ] && [ "$isErr" != "true" ]; then - say "SUCCESS after $attempt attempt(s), $quota_waits quota wait(s)" - break - fi - - # 4) genuine failure + # 3) genuine failure hard=$((hard+1)) say "hard failure #$hard (rc=$rc is_error=$isErr)" if [ "$hard" -ge "$MAX_HARD_FAILS" ]; then say "GIVING UP after $hard hard failures"; break; fi