orchestration: sprite task lease + durable wave-completion record
The wave-3 loss was caused by sprite auto-suspend, not nohup process-group semantics. /.sprite/llm.txt: 'When idle, sprites pause automatically. Services and sessions keep sprites alive.' Detached processes are on neither list, so setsid is necessary but not sufficient. tasks/wave-guard.sh holds a /v1/tasks lease (max 3600s, renewal is DELETE+POST since re-POST returns 409), renews every 15 min while workers run, writes logs/WAVE<N>-DONE, then releases the lease so the sprite can suspend rather than idle hot. Also documents hazard #6 (pgrep -f / pkill -f matching the orchestrator's own shell) and adds the wave-3 worker prompts and shared screen contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Executable
+58
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env bash
|
||||
# wave-guard.sh <sentinel-name> <task> [<task>...]
|
||||
#
|
||||
# Solves the real cause of the wave-3 loss: the sprite AUTO-SUSPENDS when idle
|
||||
# ("When idle, sprites pause automatically" — /.sprite/llm.txt). Detached workers
|
||||
# do NOT keep it awake; only services, live sessions, or a task lease do.
|
||||
#
|
||||
# Holds a sprite task lease (POST /v1/tasks, max expire 3600s) and renews it while
|
||||
# workers run, so the box stays hot with no console attached. When the workers stop
|
||||
# it writes the durable completion record AND releases the lease, so the sprite is
|
||||
# free to suspend instead of burning money idling.
|
||||
set -u
|
||||
SENT="$1"; shift
|
||||
TASKS=("$@")
|
||||
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"; LEASE="bookshelf-wave"
|
||||
RENEW="${RENEW:-900}" # renew every 15 min against a 60 min lease
|
||||
GUARD_LOG="$L/wave-guard.log"
|
||||
|
||||
lease_hold() {
|
||||
sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1
|
||||
sprite-env curl -X POST /v1/tasks -H 'Content-Type: application/json' \
|
||||
-d "{\"name\":\"$LEASE\",\"expire\":\"3600s\"}" >/dev/null 2>&1
|
||||
}
|
||||
lease_release() { sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1; }
|
||||
log() { echo "[$(date -Is)] $*" >> "$GUARD_LOG"; }
|
||||
|
||||
trap 'lease_release; log "guard exiting, lease released"; exit 0' TERM INT
|
||||
|
||||
log "guard start: sentinel=$SENT tasks=${TASKS[*]} renew=${RENEW}s"
|
||||
lease_hold; log "lease '$LEASE' acquired (3600s)"
|
||||
|
||||
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
|
||||
sleep "$RENEW"
|
||||
lease_hold
|
||||
log "lease renewed; workers still running"
|
||||
done
|
||||
|
||||
log "workers stopped; writing $SENT"
|
||||
{
|
||||
echo "=== $SENT written $(date -Is) ==="
|
||||
echo "Workers finished. The orchestrator was NOT necessarily alive for this."
|
||||
echo
|
||||
for t in "${TASKS[@]}"; do
|
||||
echo "--- $t ---"
|
||||
grep -hE 'SUCCESS|GIVING UP|WALL CLOCK|QUOTA' "$L/$t.state" 2>/dev/null | tail -3
|
||||
if [ -s "$L/$t.json" ]; then
|
||||
jq -r '"cost=$" + ((.total_cost_usd//0)|tostring) + " turns=" + ((.num_turns//0)|tostring)' "$L/$t.json" 2>/dev/null
|
||||
else
|
||||
echo "!! $t.json is 0 bytes -> worker was KILLED, not finished (hazard #3/#5)"
|
||||
fi
|
||||
echo
|
||||
done
|
||||
echo "NEXT: orchestrator must independently verify before accepting:"
|
||||
echo " cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest"
|
||||
echo " git status --porcelain # boundary check: who touched what"
|
||||
} > "$OUT"
|
||||
|
||||
lease_release; log "lease released; sprite may suspend"
|
||||
Reference in New Issue
Block a user