orchestration: sprite task lease + durable wave-completion record

The wave-3 loss was caused by sprite auto-suspend, not nohup process-group
semantics. /.sprite/llm.txt: 'When idle, sprites pause automatically. Services and
sessions keep sprites alive.' Detached processes are on neither list, so setsid is
necessary but not sufficient.

tasks/wave-guard.sh holds a /v1/tasks lease (max 3600s, renewal is DELETE+POST since
re-POST returns 409), renews every 15 min while workers run, writes logs/WAVE<N>-DONE,
then releases the lease so the sprite can suspend rather than idle hot.

Also documents hazard #6 (pgrep -f / pkill -f matching the orchestrator's own shell)
and adds the wave-3 worker prompts and shared screen contract.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-06 10:48:53 +00:00
parent d1a73a1193
commit c18649c726
6 changed files with 419 additions and 2 deletions
+64 -2
View File
@@ -1,6 +1,28 @@
# Bookshelf — session handoff
Written 2026-09-06 by the Opus orchestrator, after a sprite restart forced a fresh session.
## FIRST COMMAND OF A NEW SESSION — is a wave already finished?
Workers run fully detached (hazard #5) and OUTLIVE the orchestrator. A wave can finish
while no orchestrator exists. Nothing will "notify" a session that was not running, so
completion is recorded ON DISK. Run this before anything else:
cd ~/bookshelf && cat logs/WAVE*-DONE 2>/dev/null; \
pgrep -fc 'run-task\.sh' ; tail -2 logs/*.state
- `logs/WAVE<N>-DONE` exists -> that wave's workers have STOPPED. Read it, then
independently verify (assembleDebug + testDebugUnitTest + `git status --porcelain`)
before accepting anything. Workers self-report optimistically; two of three waves so
far over-claimed.
- no sentinel + `pgrep` count > 0 -> still running; arm a Monitor and wait.
- no sentinel + count 0 -> workers were KILLED. Check `logs/<name>.json`: 0 bytes means
killed, not failed (hazard #3). Relaunch with `setsid` per hazard #5; run-task.sh will
RESUME the existing session id rather than restart, so context/quota is preserved.
The sentinel is written by `tasks/wave-sentinel.sh`, itself launched detached:
setsid nohup ./tasks/wave-sentinel.sh WAVE3-DONE E1-shell E2-books </dev/null >/dev/null 2>&1 &
An in-process Monitor is only a convenience for a LIVE orchestrator; it dies with the
process and caps at 1h. Never rely on it as the record that a wave completed.
## Read these first, in order
1. `docs/SPEC.md` — the authoritative product/technical contract. Unchanged and still correct.
Every worker prompt must point at it. Do not restate it; do not let it drift.
@@ -17,10 +39,16 @@ The user is on the **$20/mo Pro plan** and wants Opus used sparingly.
Worker A alone cost **$2.40 / 85 turns**. Budget accordingly; prefer resuming a
session over restarting one.
### How to launch a worker
### How to launch a worker (BOTH steps — the guard is not optional)
```
cd ~/bookshelf && nohup ./tasks/run-task.sh <NAME> ./tasks/<NAME>.txt >/dev/null 2>&1 &
cd ~/bookshelf
setsid nohup ./tasks/run-task.sh <NAME> ./tasks/<NAME>.txt </dev/null >/dev/null 2>&1 &
setsid nohup ./tasks/wave-guard.sh WAVE<N>-DONE <NAME> [<NAME>...] </dev/null >/dev/null 2>&1 &
```
Without the guard the sprite auto-suspends as soon as the user's console goes idle and
the whole wave is lost (hazard #5). The guard holds a `/v1/tasks` lease, renews it every
15 min, writes `logs/WAVE<N>-DONE` at the end, and releases the lease so the box can
sleep. Check it with `sprite-env curl /v1/tasks` and `cat logs/wave-guard.log`.
`tasks/run-task.sh` is quota-aware: on a usage-limit error it sleeps `POLL` (600s) and
resumes the SAME session rather than restarting, up to `MAX_WALL` (24h), and does not
count quota waits against its 3-strike hard-failure budget. It writes:
@@ -132,3 +160,37 @@ Workers self-report optimistically. Before accepting any wave:
## Open questions for the user (not yet asked — deferred, not forgotten)
- Where the server will actually live (home box vs a sprite) — only affects the deploy README.
- Their two account emails, for `create-user.sh`. Not needed until the app can log in.
## HAZARD #5 — THE SPRITE AUTO-SUSPENDS; detached workers do NOT keep it awake
This is the real cause of the wave-3 loss on 09-06, and an earlier note in this file
blamed the wrong thing (it claimed `nohup` process-group semantics). Correct diagnosis,
credit to the user: `/.sprite/llm.txt` says "When idle, sprites pause automatically" and
"Services and sessions keep sprites alive." A detached background process is on NEITHER
list. Wave 3 was launched at 03:25 with `setsid nohup`, the user's console session went
away, the sprite went COLD, and all process state was lost. Evidence: at 10:07 `uptime`
read "up 2 min" (boot 10:05:40) while the workers' session files stopped at 03:25 — a
machine that stopped and rebooted, not a signalled process.
**FIX — hold a sprite task lease for the duration of the wave.**
Undocumented in /.sprite/docs but live on the API socket:
POST /v1/tasks {"name":"<lease>","expire":"3600s"} -> holds the sprite HOT
GET /v1/tasks -> list active leases
DELETE /v1/tasks/<lease> -> release
- max expire is 3600s (a 2h request is rejected: "exceeds maximum 3600 seconds")
- re-POSTing a live name returns 409, so RENEWAL = DELETE then POST
- the lease lives server-side, so it keeps the box up independently of any process;
a renewal loop running on the sprite therefore sustains itself
`tasks/wave-guard.sh` does all of this: acquires the lease, renews every 15 min while
workers run, then writes `logs/WAVE<N>-DONE` and RELEASES the lease so the sprite can
suspend instead of idling hot on the user's dime. Launch it detached alongside a wave:
setsid nohup ./tasks/wave-guard.sh WAVE3-DONE E1-shell E2-books </dev/null >/dev/null 2>&1 &
Still launch workers with `setsid` (needed so they survive the orchestrator exiting),
but understand that alone it does NOT survive a suspend. The lease is what does.
## HAZARD #6 — `pgrep -f` / `pkill -f` match YOUR OWN shell
Bitten three times in one session, once fatally: `pkill -f wave-sentinel.sh` killed the
orchestrator's own shell (exit 144) because the bash -c command line contained that
literal string. Same bug made `pgrep -f "claude -p"` report a phantom running worker.
Use the bracket trick (`ps aux | grep "[c]laude -p"`) or match on argv shape
(`ps -eo pid,args | grep -E "tasks/(wave-guard|run-task)" | grep -v grep`).