orchestration: sprite task lease + durable wave-completion record
The wave-3 loss was caused by sprite auto-suspend, not nohup process-group semantics. /.sprite/llm.txt: 'When idle, sprites pause automatically. Services and sessions keep sprites alive.' Detached processes are on neither list, so setsid is necessary but not sufficient. tasks/wave-guard.sh holds a /v1/tasks lease (max 3600s, renewal is DELETE+POST since re-POST returns 409), renews every 15 min while workers run, writes logs/WAVE<N>-DONE, then releases the lease so the sprite can suspend rather than idle hot. Also documents hazard #6 (pgrep -f / pkill -f matching the orchestrator's own shell) and adds the wave-3 worker prompts and shared screen contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+64
-2
@@ -1,6 +1,28 @@
|
||||
# Bookshelf — session handoff
|
||||
Written 2026-09-06 by the Opus orchestrator, after a sprite restart forced a fresh session.
|
||||
|
||||
## FIRST COMMAND OF A NEW SESSION — is a wave already finished?
|
||||
Workers run fully detached (hazard #5) and OUTLIVE the orchestrator. A wave can finish
|
||||
while no orchestrator exists. Nothing will "notify" a session that was not running, so
|
||||
completion is recorded ON DISK. Run this before anything else:
|
||||
|
||||
cd ~/bookshelf && cat logs/WAVE*-DONE 2>/dev/null; \
|
||||
pgrep -fc 'run-task\.sh' ; tail -2 logs/*.state
|
||||
|
||||
- `logs/WAVE<N>-DONE` exists -> that wave's workers have STOPPED. Read it, then
|
||||
independently verify (assembleDebug + testDebugUnitTest + `git status --porcelain`)
|
||||
before accepting anything. Workers self-report optimistically; two of three waves so
|
||||
far over-claimed.
|
||||
- no sentinel + `pgrep` count > 0 -> still running; arm a Monitor and wait.
|
||||
- no sentinel + count 0 -> workers were KILLED. Check `logs/<name>.json`: 0 bytes means
|
||||
killed, not failed (hazard #3). Relaunch with `setsid` per hazard #5; run-task.sh will
|
||||
RESUME the existing session id rather than restart, so context/quota is preserved.
|
||||
|
||||
The sentinel is written by `tasks/wave-sentinel.sh`, itself launched detached:
|
||||
setsid nohup ./tasks/wave-sentinel.sh WAVE3-DONE E1-shell E2-books </dev/null >/dev/null 2>&1 &
|
||||
An in-process Monitor is only a convenience for a LIVE orchestrator; it dies with the
|
||||
process and caps at 1h. Never rely on it as the record that a wave completed.
|
||||
|
||||
## Read these first, in order
|
||||
1. `docs/SPEC.md` — the authoritative product/technical contract. Unchanged and still correct.
|
||||
Every worker prompt must point at it. Do not restate it; do not let it drift.
|
||||
@@ -17,10 +39,16 @@ The user is on the **$20/mo Pro plan** and wants Opus used sparingly.
|
||||
Worker A alone cost **$2.40 / 85 turns**. Budget accordingly; prefer resuming a
|
||||
session over restarting one.
|
||||
|
||||
### How to launch a worker
|
||||
### How to launch a worker (BOTH steps — the guard is not optional)
|
||||
```
|
||||
cd ~/bookshelf && nohup ./tasks/run-task.sh <NAME> ./tasks/<NAME>.txt >/dev/null 2>&1 &
|
||||
cd ~/bookshelf
|
||||
setsid nohup ./tasks/run-task.sh <NAME> ./tasks/<NAME>.txt </dev/null >/dev/null 2>&1 &
|
||||
setsid nohup ./tasks/wave-guard.sh WAVE<N>-DONE <NAME> [<NAME>...] </dev/null >/dev/null 2>&1 &
|
||||
```
|
||||
Without the guard the sprite auto-suspends as soon as the user's console goes idle and
|
||||
the whole wave is lost (hazard #5). The guard holds a `/v1/tasks` lease, renews it every
|
||||
15 min, writes `logs/WAVE<N>-DONE` at the end, and releases the lease so the box can
|
||||
sleep. Check it with `sprite-env curl /v1/tasks` and `cat logs/wave-guard.log`.
|
||||
`tasks/run-task.sh` is quota-aware: on a usage-limit error it sleeps `POLL` (600s) and
|
||||
resumes the SAME session rather than restarting, up to `MAX_WALL` (24h), and does not
|
||||
count quota waits against its 3-strike hard-failure budget. It writes:
|
||||
@@ -132,3 +160,37 @@ Workers self-report optimistically. Before accepting any wave:
|
||||
## Open questions for the user (not yet asked — deferred, not forgotten)
|
||||
- Where the server will actually live (home box vs a sprite) — only affects the deploy README.
|
||||
- Their two account emails, for `create-user.sh`. Not needed until the app can log in.
|
||||
|
||||
## HAZARD #5 — THE SPRITE AUTO-SUSPENDS; detached workers do NOT keep it awake
|
||||
This is the real cause of the wave-3 loss on 09-06, and an earlier note in this file
|
||||
blamed the wrong thing (it claimed `nohup` process-group semantics). Correct diagnosis,
|
||||
credit to the user: `/.sprite/llm.txt` says "When idle, sprites pause automatically" and
|
||||
"Services and sessions keep sprites alive." A detached background process is on NEITHER
|
||||
list. Wave 3 was launched at 03:25 with `setsid nohup`, the user's console session went
|
||||
away, the sprite went COLD, and all process state was lost. Evidence: at 10:07 `uptime`
|
||||
read "up 2 min" (boot 10:05:40) while the workers' session files stopped at 03:25 — a
|
||||
machine that stopped and rebooted, not a signalled process.
|
||||
|
||||
**FIX — hold a sprite task lease for the duration of the wave.**
|
||||
Undocumented in /.sprite/docs but live on the API socket:
|
||||
POST /v1/tasks {"name":"<lease>","expire":"3600s"} -> holds the sprite HOT
|
||||
GET /v1/tasks -> list active leases
|
||||
DELETE /v1/tasks/<lease> -> release
|
||||
- max expire is 3600s (a 2h request is rejected: "exceeds maximum 3600 seconds")
|
||||
- re-POSTing a live name returns 409, so RENEWAL = DELETE then POST
|
||||
- the lease lives server-side, so it keeps the box up independently of any process;
|
||||
a renewal loop running on the sprite therefore sustains itself
|
||||
|
||||
`tasks/wave-guard.sh` does all of this: acquires the lease, renews every 15 min while
|
||||
workers run, then writes `logs/WAVE<N>-DONE` and RELEASES the lease so the sprite can
|
||||
suspend instead of idling hot on the user's dime. Launch it detached alongside a wave:
|
||||
setsid nohup ./tasks/wave-guard.sh WAVE3-DONE E1-shell E2-books </dev/null >/dev/null 2>&1 &
|
||||
Still launch workers with `setsid` (needed so they survive the orchestrator exiting),
|
||||
but understand that alone it does NOT survive a suspend. The lease is what does.
|
||||
|
||||
## HAZARD #6 — `pgrep -f` / `pkill -f` match YOUR OWN shell
|
||||
Bitten three times in one session, once fatally: `pkill -f wave-sentinel.sh` killed the
|
||||
orchestrator's own shell (exit 144) because the bash -c command line contained that
|
||||
literal string. Same bug made `pgrep -f "claude -p"` report a phantom running worker.
|
||||
Use the bracket trick (`ps aux | grep "[c]laude -p"`) or match on argv shape
|
||||
(`ps -eo pid,args | grep -E "tasks/(wave-guard|run-task)" | grep -v grep`).
|
||||
|
||||
Reference in New Issue
Block a user