orchestration: sprite task lease + durable wave-completion record
The wave-3 loss was caused by sprite auto-suspend, not nohup process-group semantics. /.sprite/llm.txt: 'When idle, sprites pause automatically. Services and sessions keep sprites alive.' Detached processes are on neither list, so setsid is necessary but not sufficient. tasks/wave-guard.sh holds a /v1/tasks lease (max 3600s, renewal is DELETE+POST since re-POST returns 409), renews every 15 min while workers run, writes logs/WAVE<N>-DONE, then releases the lease so the sprite can suspend rather than idle hot. Also documents hazard #6 (pgrep -f / pkill -f matching the orchestrator's own shell) and adds the wave-3 worker prompts and shared screen contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+64
-2
@@ -1,6 +1,28 @@
|
||||
# Bookshelf — session handoff
|
||||
Written 2026-09-06 by the Opus orchestrator, after a sprite restart forced a fresh session.
|
||||
|
||||
## FIRST COMMAND OF A NEW SESSION — is a wave already finished?
|
||||
Workers run fully detached (hazard #5) and OUTLIVE the orchestrator. A wave can finish
|
||||
while no orchestrator exists. Nothing will "notify" a session that was not running, so
|
||||
completion is recorded ON DISK. Run this before anything else:
|
||||
|
||||
cd ~/bookshelf && cat logs/WAVE*-DONE 2>/dev/null; \
|
||||
pgrep -fc 'run-task\.sh' ; tail -2 logs/*.state
|
||||
|
||||
- `logs/WAVE<N>-DONE` exists -> that wave's workers have STOPPED. Read it, then
|
||||
independently verify (assembleDebug + testDebugUnitTest + `git status --porcelain`)
|
||||
before accepting anything. Workers self-report optimistically; two of three waves so
|
||||
far over-claimed.
|
||||
- no sentinel + `pgrep` count > 0 -> still running; arm a Monitor and wait.
|
||||
- no sentinel + count 0 -> workers were KILLED. Check `logs/<name>.json`: 0 bytes means
|
||||
killed, not failed (hazard #3). Relaunch with `setsid` per hazard #5; run-task.sh will
|
||||
RESUME the existing session id rather than restart, so context/quota is preserved.
|
||||
|
||||
The sentinel is written by `tasks/wave-sentinel.sh`, itself launched detached:
|
||||
setsid nohup ./tasks/wave-sentinel.sh WAVE3-DONE E1-shell E2-books </dev/null >/dev/null 2>&1 &
|
||||
An in-process Monitor is only a convenience for a LIVE orchestrator; it dies with the
|
||||
process and caps at 1h. Never rely on it as the record that a wave completed.
|
||||
|
||||
## Read these first, in order
|
||||
1. `docs/SPEC.md` — the authoritative product/technical contract. Unchanged and still correct.
|
||||
Every worker prompt must point at it. Do not restate it; do not let it drift.
|
||||
@@ -17,10 +39,16 @@ The user is on the **$20/mo Pro plan** and wants Opus used sparingly.
|
||||
Worker A alone cost **$2.40 / 85 turns**. Budget accordingly; prefer resuming a
|
||||
session over restarting one.
|
||||
|
||||
### How to launch a worker
|
||||
### How to launch a worker (BOTH steps — the guard is not optional)
|
||||
```
|
||||
cd ~/bookshelf && nohup ./tasks/run-task.sh <NAME> ./tasks/<NAME>.txt >/dev/null 2>&1 &
|
||||
cd ~/bookshelf
|
||||
setsid nohup ./tasks/run-task.sh <NAME> ./tasks/<NAME>.txt </dev/null >/dev/null 2>&1 &
|
||||
setsid nohup ./tasks/wave-guard.sh WAVE<N>-DONE <NAME> [<NAME>...] </dev/null >/dev/null 2>&1 &
|
||||
```
|
||||
Without the guard the sprite auto-suspends as soon as the user's console goes idle and
|
||||
the whole wave is lost (hazard #5). The guard holds a `/v1/tasks` lease, renews it every
|
||||
15 min, writes `logs/WAVE<N>-DONE` at the end, and releases the lease so the box can
|
||||
sleep. Check it with `sprite-env curl /v1/tasks` and `cat logs/wave-guard.log`.
|
||||
`tasks/run-task.sh` is quota-aware: on a usage-limit error it sleeps `POLL` (600s) and
|
||||
resumes the SAME session rather than restarting, up to `MAX_WALL` (24h), and does not
|
||||
count quota waits against its 3-strike hard-failure budget. It writes:
|
||||
@@ -132,3 +160,37 @@ Workers self-report optimistically. Before accepting any wave:
|
||||
## Open questions for the user (not yet asked — deferred, not forgotten)
|
||||
- Where the server will actually live (home box vs a sprite) — only affects the deploy README.
|
||||
- Their two account emails, for `create-user.sh`. Not needed until the app can log in.
|
||||
|
||||
## HAZARD #5 — THE SPRITE AUTO-SUSPENDS; detached workers do NOT keep it awake
|
||||
This is the real cause of the wave-3 loss on 09-06, and an earlier note in this file
|
||||
blamed the wrong thing (it claimed `nohup` process-group semantics). Correct diagnosis,
|
||||
credit to the user: `/.sprite/llm.txt` says "When idle, sprites pause automatically" and
|
||||
"Services and sessions keep sprites alive." A detached background process is on NEITHER
|
||||
list. Wave 3 was launched at 03:25 with `setsid nohup`, the user's console session went
|
||||
away, the sprite went COLD, and all process state was lost. Evidence: at 10:07 `uptime`
|
||||
read "up 2 min" (boot 10:05:40) while the workers' session files stopped at 03:25 — a
|
||||
machine that stopped and rebooted, not a signalled process.
|
||||
|
||||
**FIX — hold a sprite task lease for the duration of the wave.**
|
||||
Undocumented in /.sprite/docs but live on the API socket:
|
||||
POST /v1/tasks {"name":"<lease>","expire":"3600s"} -> holds the sprite HOT
|
||||
GET /v1/tasks -> list active leases
|
||||
DELETE /v1/tasks/<lease> -> release
|
||||
- max expire is 3600s (a 2h request is rejected: "exceeds maximum 3600 seconds")
|
||||
- re-POSTing a live name returns 409, so RENEWAL = DELETE then POST
|
||||
- the lease lives server-side, so it keeps the box up independently of any process;
|
||||
a renewal loop running on the sprite therefore sustains itself
|
||||
|
||||
`tasks/wave-guard.sh` does all of this: acquires the lease, renews every 15 min while
|
||||
workers run, then writes `logs/WAVE<N>-DONE` and RELEASES the lease so the sprite can
|
||||
suspend instead of idling hot on the user's dime. Launch it detached alongside a wave:
|
||||
setsid nohup ./tasks/wave-guard.sh WAVE3-DONE E1-shell E2-books </dev/null >/dev/null 2>&1 &
|
||||
Still launch workers with `setsid` (needed so they survive the orchestrator exiting),
|
||||
but understand that alone it does NOT survive a suspend. The lease is what does.
|
||||
|
||||
## HAZARD #6 — `pgrep -f` / `pkill -f` match YOUR OWN shell
|
||||
Bitten three times in one session, once fatally: `pkill -f wave-sentinel.sh` killed the
|
||||
orchestrator's own shell (exit 144) because the bash -c command line contained that
|
||||
literal string. Same bug made `pgrep -f "claude -p"` report a phantom running worker.
|
||||
Use the bracket trick (`ps aux | grep "[c]laude -p"`) or match on argv shape
|
||||
(`ps -eo pid,args | grep -E "tasks/(wave-guard|run-task)" | grep -v grep`).
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
You are Worker E1 on the Bookshelf project (~/bookshelf). Wave 3, part 1:
|
||||
APP SHELL + SETUP + LOCATIONS + SETTINGS screens.
|
||||
|
||||
FIRST, READ THESE — the contract. Follow exactly; do not invent alternative names,
|
||||
do not restate them back to me:
|
||||
~/bookshelf/docs/SPEC.md (authoritative — see "Screens", "Design language")
|
||||
~/bookshelf/docs/HANDOFF.md (operational state, gotchas already paid for)
|
||||
|
||||
Waves 1-2 are DONE and verified green: theme + shared components (ui.theme,
|
||||
ui.components), the whole data layer (data.local/remote/repo/prefs, AppContainer),
|
||||
and metadata/scanning plumbing. REUSE them. Read `AppContainer.kt` and the repository
|
||||
classes to learn the real API before you write anything against them.
|
||||
|
||||
## Your scope
|
||||
ui.nav Routes.kt + BookshelfNavHost — the whole navigation graph, including
|
||||
routes to E2's screens (see the contract block below)
|
||||
ui.setup SetupScreen: server URL (https validation, trailing-slash strip,
|
||||
reachability probe), email, password. MUST distinguish "bad/unreachable
|
||||
URL" from "bad credentials" in its error text — SPEC calls this out.
|
||||
ui.locations LocationsScreen: bookcases -> shelves tree, CRUD + reorder, book counts
|
||||
per shelf, tap-a-shelf -> library filtered to it, "Move books" bulk action.
|
||||
ui.settings SettingsScreen: server, account, sign out, manual sync + last-sync time,
|
||||
book/cover counts.
|
||||
MainActivity: host BookshelfNavHost, pick the start route (setup vs library) based
|
||||
on whether a server URL + auth token already exist.
|
||||
|
||||
## HARD BOUNDARIES — Worker E2 is running RIGHT NOW in this same repo
|
||||
- DO NOT create or edit anything under ui.library, ui.detail, ui.scan. E2 owns those.
|
||||
You WILL reference E2's screen composables from BookshelfNavHost — that is expected;
|
||||
use exactly the signatures in the contract block and do not open those files.
|
||||
- DO NOT edit ui.theme or ui.components (wave 1B, verified green). Reuse, don't modify.
|
||||
- DO NOT edit `app/build.gradle.kts` or `gradle/libs.versions.toml`. Everything needed
|
||||
is already declared and wired. If you think something is missing, report it, do not add it.
|
||||
- Data layer: you MAY extend LocationRepository / ShelfDao / BookcaseDao (e.g. reorder,
|
||||
bulk-move helpers) if a screen genuinely needs it. You may NOT touch BookRepository or
|
||||
BookDao — E2 owns those. If you need a book-side query, report it instead.
|
||||
|
||||
## SHARED SCREEN CONTRACT — fixed by the orchestrator, IDENTICAL in both wave-3
|
||||
## prompts. Do NOT change these signatures. E1 writes BookshelfNavHost against them;
|
||||
## E2 writes the screens to match. If you deviate, the other worker's code stops
|
||||
## compiling and the wave fails.
|
||||
|
||||
Routes (string constants live in ui/nav/Routes.kt, owned by E1):
|
||||
"setup" | "library" | "library?shelfId={shelfId}" | "detail/{bookId}" |
|
||||
"scan" | "locations" | "settings"
|
||||
|
||||
Screen composable signatures:
|
||||
|
||||
// E2 owns
|
||||
@Composable fun LibraryScreen(
|
||||
shelfIdFilter: String?,
|
||||
onBookClick: (String) -> Unit,
|
||||
onScanClick: () -> Unit,
|
||||
onLocationsClick: () -> Unit,
|
||||
onSettingsClick: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun DetailScreen(
|
||||
bookId: String,
|
||||
onBack: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun ScanScreen(
|
||||
onBack: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
|
||||
// E1 owns
|
||||
@Composable fun SetupScreen(onSetupComplete: () -> Unit, container: AppContainer)
|
||||
@Composable fun LocationsScreen(
|
||||
onBack: () -> Unit,
|
||||
onShelfClick: (String) -> Unit, // navigates to library?shelfId=...
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun SettingsScreen(
|
||||
onBack: () -> Unit,
|
||||
onSignedOut: () -> Unit, // navigates back to "setup"
|
||||
container: AppContainer,
|
||||
)
|
||||
|
||||
All screens take `container: AppContainer` and construct their own ViewModel from it
|
||||
(manual DI per SPEC — NO Hilt). Package = ui.<screen>, e.g. ui.library.LibraryScreen.
|
||||
|
||||
## Build/verify — CRITICAL
|
||||
Never run `./gradlew`; E2 builds concurrently and you will corrupt each other's build.
|
||||
ALWAYS use the serialized wrapper (it takes a lock and may block — wait for it):
|
||||
~/bookshelf/tasks/gw assembleDebug
|
||||
~/bookshelf/tasks/gw testDebugUnitTest
|
||||
|
||||
## Definition of done — verified by YOU, actually run, not assumed
|
||||
1. `~/bookshelf/tasks/gw assembleDebug` exits 0.
|
||||
2. `~/bookshelf/tasks/gw testDebugUnitTest` exits 0 with ZERO failures. There are 68
|
||||
existing passing tests — you must not break any of them.
|
||||
3. The app must behave per SPEC with NO server configured (lands on setup) and must not
|
||||
crash when the server is unreachable. Sync failure = a quiet status line, never a
|
||||
crash or a blocking dialog.
|
||||
4. Every screen uses the wave-1B theme and components. Warm paper / mahogany / gold,
|
||||
Literata for titles. Do not introduce new colors or a second type scale.
|
||||
|
||||
## IMPORTANT — do not repeat Worker D's mistake
|
||||
A previous worker ended its turn saying "tests are running in the background, I'll
|
||||
report when done." That is a FAILURE. Run the commands, WAIT for them, and report real
|
||||
exit codes you actually observed. A truthfully reported gap is worth more than a false
|
||||
green; an unverified claim is worth less than nothing.
|
||||
|
||||
## Report back (short — read by a token-constrained orchestrator)
|
||||
- real exit codes of the two gradle commands, and the test failure count
|
||||
- files created, one line each
|
||||
- anything in SPEC.md you could NOT satisfy, and why
|
||||
@@ -0,0 +1,116 @@
|
||||
You are Worker E2 on the Bookshelf project (~/bookshelf). Wave 3, part 2:
|
||||
LIBRARY + DETAIL + SCAN screens — the three the user actually looks at most.
|
||||
|
||||
FIRST, READ THESE — the contract. Follow exactly; do not invent alternative names,
|
||||
do not restate them back to me:
|
||||
~/bookshelf/docs/SPEC.md (authoritative — see "Screens", "Design language")
|
||||
~/bookshelf/docs/HANDOFF.md (operational state, gotchas already paid for)
|
||||
|
||||
Waves 1-2 are DONE and verified green: theme + shared components (ui.theme,
|
||||
ui.components), the whole data layer (data.local/remote/repo/prefs, AppContainer),
|
||||
metadata lookup (data.metadata) and scanner plumbing (ui.scan: ScannerController,
|
||||
IsbnBarcodeAnalyzer, ScanCodeFilter). REUSE them. Read `AppContainer.kt`, the
|
||||
repository classes, and the existing ui.scan classes to learn the real API before
|
||||
writing against them.
|
||||
|
||||
## Your scope
|
||||
ui.library LibraryScreen: adaptive 2-3 column cover grid, search by title/author/
|
||||
ISBN, filter by bookcase/shelf, sort by title/author/added, empty state
|
||||
that invites the first scan, FAB -> scan, sync status line.
|
||||
Covers are the hero — let them carry the color.
|
||||
ui.detail DetailScreen: big cover, title/subtitle/authors/publisher/year/pages/ISBN,
|
||||
collapsible description, editable notes, location picker, edit,
|
||||
soft-delete WITH UNDO.
|
||||
ui.scan ScanScreen ONLY — the camera screen itself, built on the EXISTING
|
||||
ScannerController/IsbnBarcodeAnalyzer (do not rewrite them; extend only
|
||||
if genuinely necessary). Camera + reticle; on hit -> bottom sheet with
|
||||
the fetched book + shelf picker + Save/Skip. Duplicate-ISBN warning if
|
||||
already owned. CONTINUOUS mode: after a save, stay on camera for the next
|
||||
book, with a running "added this session" count. Handle permission denial,
|
||||
torch toggle, and a manual-ISBN-entry escape hatch.
|
||||
|
||||
## HARD BOUNDARIES — Worker E1 is running RIGHT NOW in this same repo
|
||||
- DO NOT create or edit anything under ui.nav, ui.setup, ui.locations, ui.settings,
|
||||
or MainActivity.kt. E1 owns those. E1 will call your screens from the NavHost using
|
||||
exactly the signatures in the contract block below — match them precisely.
|
||||
- DO NOT edit ui.theme or ui.components (wave 1B, verified green). Reuse, don't modify.
|
||||
- DO NOT edit `app/build.gradle.kts` or `gradle/libs.versions.toml`. Everything needed
|
||||
is already declared and wired. If you think something is missing, report it, do not add it.
|
||||
- Data layer: you MAY extend BookRepository / BookDao (e.g. sort-by-author, filter by
|
||||
bookcase) if a screen genuinely needs it. You may NOT touch LocationRepository,
|
||||
ShelfDao, or BookcaseDao — E1 owns those. If you need a location-side query, report it.
|
||||
|
||||
## SHARED SCREEN CONTRACT — fixed by the orchestrator, IDENTICAL in both wave-3
|
||||
## prompts. Do NOT change these signatures. E1 writes BookshelfNavHost against them;
|
||||
## E2 writes the screens to match. If you deviate, the other worker's code stops
|
||||
## compiling and the wave fails.
|
||||
|
||||
Routes (string constants live in ui/nav/Routes.kt, owned by E1):
|
||||
"setup" | "library" | "library?shelfId={shelfId}" | "detail/{bookId}" |
|
||||
"scan" | "locations" | "settings"
|
||||
|
||||
Screen composable signatures:
|
||||
|
||||
// E2 owns
|
||||
@Composable fun LibraryScreen(
|
||||
shelfIdFilter: String?,
|
||||
onBookClick: (String) -> Unit,
|
||||
onScanClick: () -> Unit,
|
||||
onLocationsClick: () -> Unit,
|
||||
onSettingsClick: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun DetailScreen(
|
||||
bookId: String,
|
||||
onBack: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun ScanScreen(
|
||||
onBack: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
|
||||
// E1 owns
|
||||
@Composable fun SetupScreen(onSetupComplete: () -> Unit, container: AppContainer)
|
||||
@Composable fun LocationsScreen(
|
||||
onBack: () -> Unit,
|
||||
onShelfClick: (String) -> Unit, // navigates to library?shelfId=...
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun SettingsScreen(
|
||||
onBack: () -> Unit,
|
||||
onSignedOut: () -> Unit, // navigates back to "setup"
|
||||
container: AppContainer,
|
||||
)
|
||||
|
||||
All screens take `container: AppContainer` and construct their own ViewModel from it
|
||||
(manual DI per SPEC — NO Hilt). Package = ui.<screen>, e.g. ui.library.LibraryScreen.
|
||||
|
||||
## Build/verify — CRITICAL
|
||||
Never run `./gradlew`; E1 builds concurrently and you will corrupt each other's build.
|
||||
ALWAYS use the serialized wrapper (it takes a lock and may block — wait for it):
|
||||
~/bookshelf/tasks/gw assembleDebug
|
||||
~/bookshelf/tasks/gw testDebugUnitTest
|
||||
|
||||
## Definition of done — verified by YOU, actually run, not assumed
|
||||
1. `~/bookshelf/tasks/gw assembleDebug` exits 0.
|
||||
2. `~/bookshelf/tasks/gw testDebugUnitTest` exits 0 with ZERO failures. There are 68
|
||||
existing passing tests — you must not break any of them.
|
||||
3. Add real unit tests with real assertions for the logic you add (search/sort/filter
|
||||
selection, duplicate-ISBN detection, scan-session state). SPEC: "Do not write
|
||||
assertion-free tests." Do not test-by-screenshot only.
|
||||
4. Offline-first per SPEC: every read comes from Room, no screen blocks on network,
|
||||
nothing crashes when the server is unreachable.
|
||||
5. Every screen uses the wave-1B theme and components. Warm paper / mahogany / gold,
|
||||
Literata for titles. Do not introduce new colors or a second type scale.
|
||||
|
||||
## IMPORTANT — do not repeat Worker D's mistake
|
||||
A previous worker ended its turn saying "tests are running in the background, I'll
|
||||
report when done." That is a FAILURE. Run the commands, WAIT for them, and report real
|
||||
exit codes you actually observed. A truthfully reported gap is worth more than a false
|
||||
green; an unverified claim is worth less than nothing.
|
||||
|
||||
## Report back (short — read by a token-constrained orchestrator)
|
||||
- real exit codes of the two gradle commands, and the test failure count
|
||||
- files created, one line each
|
||||
- anything in SPEC.md you could NOT satisfy, and why
|
||||
@@ -0,0 +1,45 @@
|
||||
## SHARED SCREEN CONTRACT — fixed by the orchestrator, IDENTICAL in both wave-3
|
||||
## prompts. Do NOT change these signatures. E1 writes BookshelfNavHost against them;
|
||||
## E2 writes the screens to match. If you deviate, the other worker's code stops
|
||||
## compiling and the wave fails.
|
||||
|
||||
Routes (string constants live in ui/nav/Routes.kt, owned by E1):
|
||||
"setup" | "library" | "library?shelfId={shelfId}" | "detail/{bookId}" |
|
||||
"scan" | "locations" | "settings"
|
||||
|
||||
Screen composable signatures:
|
||||
|
||||
// E2 owns
|
||||
@Composable fun LibraryScreen(
|
||||
shelfIdFilter: String?,
|
||||
onBookClick: (String) -> Unit,
|
||||
onScanClick: () -> Unit,
|
||||
onLocationsClick: () -> Unit,
|
||||
onSettingsClick: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun DetailScreen(
|
||||
bookId: String,
|
||||
onBack: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun ScanScreen(
|
||||
onBack: () -> Unit,
|
||||
container: AppContainer,
|
||||
)
|
||||
|
||||
// E1 owns
|
||||
@Composable fun SetupScreen(onSetupComplete: () -> Unit, container: AppContainer)
|
||||
@Composable fun LocationsScreen(
|
||||
onBack: () -> Unit,
|
||||
onShelfClick: (String) -> Unit, // navigates to library?shelfId=...
|
||||
container: AppContainer,
|
||||
)
|
||||
@Composable fun SettingsScreen(
|
||||
onBack: () -> Unit,
|
||||
onSignedOut: () -> Unit, // navigates back to "setup"
|
||||
container: AppContainer,
|
||||
)
|
||||
|
||||
All screens take `container: AppContainer` and construct their own ViewModel from it
|
||||
(manual DI per SPEC — NO Hilt). Package = ui.<screen>, e.g. ui.library.LibraryScreen.
|
||||
Executable
+58
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env bash
|
||||
# wave-guard.sh <sentinel-name> <task> [<task>...]
|
||||
#
|
||||
# Solves the real cause of the wave-3 loss: the sprite AUTO-SUSPENDS when idle
|
||||
# ("When idle, sprites pause automatically" — /.sprite/llm.txt). Detached workers
|
||||
# do NOT keep it awake; only services, live sessions, or a task lease do.
|
||||
#
|
||||
# Holds a sprite task lease (POST /v1/tasks, max expire 3600s) and renews it while
|
||||
# workers run, so the box stays hot with no console attached. When the workers stop
|
||||
# it writes the durable completion record AND releases the lease, so the sprite is
|
||||
# free to suspend instead of burning money idling.
|
||||
set -u
|
||||
SENT="$1"; shift
|
||||
TASKS=("$@")
|
||||
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"; LEASE="bookshelf-wave"
|
||||
RENEW="${RENEW:-900}" # renew every 15 min against a 60 min lease
|
||||
GUARD_LOG="$L/wave-guard.log"
|
||||
|
||||
lease_hold() {
|
||||
sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1
|
||||
sprite-env curl -X POST /v1/tasks -H 'Content-Type: application/json' \
|
||||
-d "{\"name\":\"$LEASE\",\"expire\":\"3600s\"}" >/dev/null 2>&1
|
||||
}
|
||||
lease_release() { sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1; }
|
||||
log() { echo "[$(date -Is)] $*" >> "$GUARD_LOG"; }
|
||||
|
||||
trap 'lease_release; log "guard exiting, lease released"; exit 0' TERM INT
|
||||
|
||||
log "guard start: sentinel=$SENT tasks=${TASKS[*]} renew=${RENEW}s"
|
||||
lease_hold; log "lease '$LEASE' acquired (3600s)"
|
||||
|
||||
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
|
||||
sleep "$RENEW"
|
||||
lease_hold
|
||||
log "lease renewed; workers still running"
|
||||
done
|
||||
|
||||
log "workers stopped; writing $SENT"
|
||||
{
|
||||
echo "=== $SENT written $(date -Is) ==="
|
||||
echo "Workers finished. The orchestrator was NOT necessarily alive for this."
|
||||
echo
|
||||
for t in "${TASKS[@]}"; do
|
||||
echo "--- $t ---"
|
||||
grep -hE 'SUCCESS|GIVING UP|WALL CLOCK|QUOTA' "$L/$t.state" 2>/dev/null | tail -3
|
||||
if [ -s "$L/$t.json" ]; then
|
||||
jq -r '"cost=$" + ((.total_cost_usd//0)|tostring) + " turns=" + ((.num_turns//0)|tostring)' "$L/$t.json" 2>/dev/null
|
||||
else
|
||||
echo "!! $t.json is 0 bytes -> worker was KILLED, not finished (hazard #3/#5)"
|
||||
fi
|
||||
echo
|
||||
done
|
||||
echo "NEXT: orchestrator must independently verify before accepting:"
|
||||
echo " cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest"
|
||||
echo " git status --porcelain # boundary check: who touched what"
|
||||
} > "$OUT"
|
||||
|
||||
lease_release; log "lease released; sprite may suspend"
|
||||
Executable
+27
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# wave-sentinel.sh <sentinel-name> <task> [<task>...]
|
||||
# Detached watcher: waits for the named workers to stop, then writes a durable
|
||||
# completion record. Survives the orchestrator exiting, so a FRESH session can
|
||||
# learn a wave finished without having been present when it happened.
|
||||
set -u
|
||||
SENT="$1"; shift
|
||||
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"
|
||||
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do sleep 30; done
|
||||
{
|
||||
echo "=== $SENT written $(date -Is) ==="
|
||||
echo "Workers finished. The orchestrator was NOT necessarily alive for this."
|
||||
echo
|
||||
for t in "$@"; do
|
||||
echo "--- $t ---"
|
||||
grep -hE 'SUCCESS|GIVING UP|WALL CLOCK|QUOTA' "$L/$t.state" 2>/dev/null | tail -3
|
||||
if [ -s "$L/$t.json" ]; then
|
||||
jq -r '"cost=$" + ((.total_cost_usd//0)|tostring) + " turns=" + ((.num_turns//0)|tostring)' "$L/$t.json" 2>/dev/null
|
||||
else
|
||||
echo "!! $t.json is 0 bytes -> worker was KILLED, not finished (hazard #3/#5)"
|
||||
fi
|
||||
echo
|
||||
done
|
||||
echo "NEXT: orchestrator must independently verify before accepting:"
|
||||
echo " cd ~/bookshelf && ./tasks/gw assembleDebug && ./tasks/gw testDebugUnitTest"
|
||||
echo " git status --porcelain # boundary check: who touched what"
|
||||
} > "$OUT"
|
||||
Reference in New Issue
Block a user