orchestration: sprite task lease + durable wave-completion record
The wave-3 loss was caused by sprite auto-suspend, not nohup process-group semantics. /.sprite/llm.txt: 'When idle, sprites pause automatically. Services and sessions keep sprites alive.' Detached processes are on neither list, so setsid is necessary but not sufficient. tasks/wave-guard.sh holds a /v1/tasks lease (max 3600s, renewal is DELETE+POST since re-POST returns 409), renews every 15 min while workers run, writes logs/WAVE<N>-DONE, then releases the lease so the sprite can suspend rather than idle hot. Also documents hazard #6 (pgrep -f / pkill -f matching the orchestrator's own shell) and adds the wave-3 worker prompts and shared screen contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Executable
+27
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# wave-sentinel.sh <sentinel-name> <task> [<task>...]
|
||||
# Detached watcher: waits for the named workers to stop, then writes a durable
|
||||
# completion record. Survives the orchestrator exiting, so a FRESH session can
|
||||
# learn a wave finished without having been present when it happened.
|
||||
set -u
|
||||
SENT="$1"; shift
|
||||
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"
|
||||
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do sleep 30; done
|
||||
{
|
||||
echo "=== $SENT written $(date -Is) ==="
|
||||
echo "Workers finished. The orchestrator was NOT necessarily alive for this."
|
||||
echo
|
||||
for t in "$@"; do
|
||||
echo "--- $t ---"
|
||||
grep -hE 'SUCCESS|GIVING UP|WALL CLOCK|QUOTA' "$L/$t.state" 2>/dev/null | tail -3
|
||||
if [ -s "$L/$t.json" ]; then
|
||||
jq -r '"cost=$" + ((.total_cost_usd//0)|tostring) + " turns=" + ((.num_turns//0)|tostring)' "$L/$t.json" 2>/dev/null
|
||||
else
|
||||
echo "!! $t.json is 0 bytes -> worker was KILLED, not finished (hazard #3/#5)"
|
||||
fi
|
||||
echo
|
||||
done
|
||||
echo "NEXT: orchestrator must independently verify before accepting:"
|
||||
echo " cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest"
|
||||
echo " git status --porcelain # boundary check: who touched what"
|
||||
} > "$OUT"
|
||||
Reference in New Issue
Block a user