The wave-3 loss was caused by sprite auto-suspend, not nohup process-group semantics. /.sprite/llm.txt: 'When idle, sprites pause automatically. Services and sessions keep sprites alive.' Detached processes are on neither list, so setsid is necessary but not sufficient. tasks/wave-guard.sh holds a /v1/tasks lease (max 3600s, renewal is DELETE+POST since re-POST returns 409), renews every 15 min while workers run, writes logs/WAVE<N>-DONE, then releases the lease so the sprite can suspend rather than idle hot. Also documents hazard #6 (pgrep -f / pkill -f matching the orchestrator's own shell) and adds the wave-3 worker prompts and shared screen contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
28 lines
1.2 KiB
Bash
Executable File
28 lines
1.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# wave-sentinel.sh <sentinel-name> <task> [<task>...]
|
|
# Detached watcher: waits for the named workers to stop, then writes a durable
|
|
# completion record. Survives the orchestrator exiting, so a FRESH session can
|
|
# learn a wave finished without having been present when it happened.
|
|
set -u
|
|
SENT="$1"; shift
|
|
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"
|
|
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do sleep 30; done
|
|
{
|
|
echo "=== $SENT written $(date -Is) ==="
|
|
echo "Workers finished. The orchestrator was NOT necessarily alive for this."
|
|
echo
|
|
for t in "$@"; do
|
|
echo "--- $t ---"
|
|
grep -hE 'SUCCESS|GIVING UP|WALL CLOCK|QUOTA' "$L/$t.state" 2>/dev/null | tail -3
|
|
if [ -s "$L/$t.json" ]; then
|
|
jq -r '"cost=$" + ((.total_cost_usd//0)|tostring) + " turns=" + ((.num_turns//0)|tostring)' "$L/$t.json" 2>/dev/null
|
|
else
|
|
echo "!! $t.json is 0 bytes -> worker was KILLED, not finished (hazard #3/#5)"
|
|
fi
|
|
echo
|
|
done
|
|
echo "NEXT: orchestrator must independently verify before accepting:"
|
|
echo " cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest"
|
|
echo " git status --porcelain # boundary check: who touched what"
|
|
} > "$OUT"
|