orchestration: run workers as a sprite service; record wave 4 completion
setsid nohup did not survive orchestrator teardown on 09-06 (uptime was continuous, so this was a teardown kill, not the hazard-#5 suspend). tasks/service-worker.sh runs a worker under the sprite service supervisor instead, which both outlives the orchestrator and holds the box awake, making the task-lease guard redundant. It is sentinel-guarded so a supervisor restart does not re-run a finished wave, and it stops its own service afterwards so the sprite can suspend. logs/WAVE4-DONE records that F3's files were all complete but the worker was stuck re-running verification it could not finish, so the orchestrator ran the verification itself and accepted the work. F3's own written report — including the design critique of the rendered screens it was asked for — was never produced; that gap is recorded in the sentinel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014TyzeWmdTqi7U85iYNGy7P
This commit is contained in:
@@ -0,0 +1 @@
|
|||||||
|
0bdf98e8-c28a-4069-ad84-87e674234136
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
=== WAVE4-DONE written 2026-09-08T21:34Z by the Opus orchestrator ===
|
||||||
|
NOT written by service-worker.sh. F3-release's FILES were all complete and
|
||||||
|
untouched for ~50 min; it was stuck re-running verification it could not finish
|
||||||
|
(600s background-wait ceiling, then a quota wait). The orchestrator ran the
|
||||||
|
verification itself, accepted the work, and stopped the service.
|
||||||
|
|
||||||
|
Independently verified by the orchestrator (not self-reported):
|
||||||
|
./tasks/gw assembleDebug exit 0
|
||||||
|
./tasks/gw testDebugUnitTest exit 0 -- 102 tests, 1 skipped, 0 failures, 0 errors
|
||||||
|
./tasks/gw assembleRelease exit 0
|
||||||
|
app-release.apk 41,777,344 bytes, V2-signed CN=Bookshelf,O=Montanaro (real
|
||||||
|
release key, not the debug cert)
|
||||||
|
keystore + keystore.properties confirmed gitignored, NOT in the commit
|
||||||
|
skipped test = LiveSyncTest (opt-in, needs the live server)
|
||||||
|
|
||||||
|
All four F3 tasks DONE: settings-email fix, 5 remaining screens' Paparazzi
|
||||||
|
shots (light+dark), release signing + signed APK, top-level README.
|
||||||
|
NOT delivered: F3's own written report, incl. its candid design critique of
|
||||||
|
the rendered screens. The orchestrator must eyeball the PNGs instead.
|
||||||
Executable
+63
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# service-worker.sh <task-name> <prompt-file> <sentinel-name>
|
||||||
|
#
|
||||||
|
# Runs ONE worker under the sprite service supervisor instead of `setsid nohup`.
|
||||||
|
#
|
||||||
|
# WHY THIS EXISTS (wave-4 post-mortem, second failure):
|
||||||
|
# `setsid nohup ./tasks/run-task.sh ...` does NOT reliably outlive the orchestrator.
|
||||||
|
# On 09-06 at 19:04 the orchestrator's session was torn down and the detached worker
|
||||||
|
# AND its wave-guard died with it -- no reboot (uptime was continuous), so this was a
|
||||||
|
# teardown kill, not the hazard-#5 suspend. `setsid` did not save it.
|
||||||
|
#
|
||||||
|
# /.sprite/llm.txt: services are "long-running processes that persist across reboots"
|
||||||
|
# and "services and sessions keep sprites alive". A service therefore fixes BOTH
|
||||||
|
# failure modes at once -- it survives orchestrator teardown AND holds the box awake --
|
||||||
|
# which makes the task-lease guard redundant.
|
||||||
|
#
|
||||||
|
# IDEMPOTENCE MATTERS HERE: the supervisor restarts services. Without the sentinel
|
||||||
|
# guard below, a finished wave would be re-run on every restart and burn quota
|
||||||
|
# redoing completed work.
|
||||||
|
set -u
|
||||||
|
NAME="$1"; PROMPT="$2"; SENT="$3"
|
||||||
|
L="$HOME/bookshelf/logs"; DONE="$L/$SENT"
|
||||||
|
SVC="${SVC_NAME:-bookshelf-worker}"
|
||||||
|
|
||||||
|
log() { echo "[$(date -Is)] service-worker: $*" >> "$L/service-worker.log"; }
|
||||||
|
|
||||||
|
# Already finished -> do not re-run. Idle briefly so a restart loop stays slow.
|
||||||
|
if [ -f "$DONE" ]; then
|
||||||
|
log "$SENT already exists; nothing to do (service can be deleted)"
|
||||||
|
sleep 300
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "starting worker $NAME (prompt=$PROMPT, sentinel=$SENT)"
|
||||||
|
"$HOME/bookshelf/tasks/run-task.sh" "$NAME" "$PROMPT"
|
||||||
|
rc=$?
|
||||||
|
log "worker $NAME exited rc=$rc; writing $SENT"
|
||||||
|
|
||||||
|
{
|
||||||
|
echo "=== $SENT written $(date -Is) ==="
|
||||||
|
echo "Worker finished under the service supervisor. The orchestrator was NOT"
|
||||||
|
echo "necessarily alive for this. run-task.sh exit code: $rc"
|
||||||
|
echo
|
||||||
|
echo "--- $NAME ---"
|
||||||
|
grep -hE 'SUCCESS|GIVING UP|WALL CLOCK|QUOTA' "$L/$NAME.state" 2>/dev/null | tail -5
|
||||||
|
if [ -s "$L/$NAME.json" ]; then
|
||||||
|
jq -r '"cost=$" + ((.total_cost_usd//0)|tostring) + " turns=" + ((.num_turns//0)|tostring)' "$L/$NAME.json" 2>/dev/null
|
||||||
|
jq -r '"is_error=" + ((.is_error//false)|tostring)' "$L/$NAME.json" 2>/dev/null
|
||||||
|
else
|
||||||
|
echo "!! $NAME.json is 0 bytes -> worker was KILLED, not finished (hazard #3)"
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
echo "NEXT: orchestrator must independently verify before accepting:"
|
||||||
|
echo " cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest"
|
||||||
|
echo " git status --porcelain # what did the worker actually touch?"
|
||||||
|
} > "$DONE"
|
||||||
|
sync
|
||||||
|
|
||||||
|
# Sentinel is durable now, so it is safe to stop ourselves. This lets the sprite
|
||||||
|
# suspend instead of idling hot on the owner's dime.
|
||||||
|
log "stopping service $SVC so the sprite can suspend"
|
||||||
|
sprite-env services stop "$SVC" >/dev/null 2>&1
|
||||||
|
sleep 60
|
||||||
Reference in New Issue
Block a user