orchestration: verify lease acquisition in wave-guard; F3 prompt for wave 4 finish

wave-guard.sh post-mortem: lease_hold() did DELETE-then-POST with both results
discarded and logged "lease renewed" unconditionally, so a failed re-POST left
the box with no lease while the log claimed it was protected. That matches the
wave-4 loss exactly (last "renewal" 11:39, workers dead 11:46, reboot 12:55).

Now: every acquire is verified against GET /v1/tasks before it is believed, a
failed acquire retries and is logged as a failure, and the lease is re-checked
every POLL rather than only every RENEW so a lease lost between renewals is
caught in seconds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016mTs3kQXQsQwonXpEq7aEw
This commit is contained in:
2026-09-06 19:00:11 +00:00
parent cf36b27457
commit 36b46b4644
2 changed files with 142 additions and 15 deletions
+91
View File
@@ -0,0 +1,91 @@
You are Worker F3 on the Bookshelf project (~/bookshelf). You are FINISHING wave 4.
FIRST READ (contract — follow exactly, do not restate):
~/bookshelf/docs/SPEC.md ("Design language", "Screens", "Quality bar")
~/bookshelf/docs/HANDOFF.md (state + hazards already paid for — read the hazards)
## Where things actually stand (do not re-derive this; it is verified)
Waves 1-3 are DONE. Wave 4 was launched once and both workers were killed ~11 minutes
in by the 5h session quota plus a sprite suspend. Their partial work is COMMITTED and
verified green at `a4adf83`: `assembleDebug` + `testDebugUnitTest` pass, 94 tests, 0
failures.
Already done, DO NOT redo:
- The live-server sync test is FINISHED and PASSING against the real PocketBase --
auth, push with client-generated ids, pull, last-write-wins, tombstones, and the
cover byte-for-byte round-trip. That was wave 4's biggest risk and it is closed.
`server/live-sync-test.sh` + `org.modg.bookshelf.livesync.LiveSyncTest`. LEAVE BOTH
ALONE, and leave `server/` alone entirely.
- Library screen screenshots (populated + empty, light + dark) and the reusable
`app/app/src/test/java/org/modg/bookshelf/ui/screens/ScreenFixtures.kt`.
FOLLOW that file's patterns; do not invent a second approach.
## Your scope — four tasks, in this order
1. **Fix the settings-email gap FIRST** (it changes what task 2 renders).
Settings currently shows the PocketBase user id instead of the signed-in email,
because `AuthRepository`/`SettingsStore` never persist the email at login. Persist
it on successful login and show it in Settings. Small, surgical change -- do not
redesign auth. Add/extend a unit test that actually asserts the email is persisted.
2. **Paparazzi screenshots for the five REMAINING screens** -- setup, detail, scan,
locations, settings -- each in BOTH light and dark, with realistic populated state
(a plausible shelf of books, placeholder cover art), not empty states. The library
screen is already done. The scan screen cannot show a live camera in Paparazzi:
render its overlay/reticle and the result bottom sheet over a static backdrop, and
SAY SO in your report.
3. **Release signing + signed APK.** Generate a release keystore, wire a signingConfig
into `app/app/build.gradle.kts`, produce a signed release APK. The keystore and its
passwords MUST be gitignored -- credentials in `app/keystore.properties` (gitignored),
read from the build file, with a graceful fallback so the build still works for
anyone WITHOUT the keystore. State the passwords in your report so the owner can
rebuild. `minifyEnabled` is false today: if you enable R8 you must PROVE the signed
APK still builds and that Room/Retrofit/kotlinx-serialization/ML Kit survive it. If
you cannot prove it, leave R8 OFF and say so. Leaving it off is an acceptable answer.
4. **Top-level `README.md`** -- what this is, the two-person shared-library model,
architecture (offline-first Room + PocketBase, push-then-pull last-write-wins sync),
how to build, how to deploy the server (point at `server/deploy/`), how to install
the APK, and an honest "current limitations" section. SPEC requires the LWW conflict
rule be documented here. Reuse real content from `server/README.md` and `docs/SPEC.md`;
do not contradict them. Include that the app has never run on a physical device.
## Boundaries
- You are the ONLY worker running. No file-ownership conflicts this time.
- Do NOT touch `server/`, `LiveSyncTest.kt`, or `live-sync-test.sh`.
- Do NOT weaken or delete existing tests to get green. 94 tests pass now; that number
must not go DOWN.
- Do not modify the six screens' behavior just to make them easier to screenshot. If a
screen is hard to render in isolation, that is a finding to REPORT, not a licence to
restructure it. (Task 1 is the one deliberate exception, and it is a data-layer fix.)
## Build/verify — use the serialized wrapper, never ./gradlew directly
~/bookshelf/tasks/gw assembleDebug
~/bookshelf/tasks/gw testDebugUnitTest
~/bookshelf/tasks/gw recordPaparazziDebug
~/bookshelf/tasks/gw assembleRelease
## Definition of done — actually run these, WAIT for them, report REAL exit codes
1. All four commands exit 0. `testDebugUnitTest` has ZERO failures and >= 94 tests.
2. Screenshot PNGs exist on disk for all SIX screens x light/dark. List their paths.
3. A signed release APK exists. Give its exact path, size, and the output of
`~/toolchain/android-sdk/build-tools/37.0.0/apksigner verify --print-certs <apk>`
(or an explanation if that tool is unavailable).
4. `git status` shows NO keystore and NO keystore.properties as tracked/untracked-to-be
-committed. Verify it and say so explicitly.
5. Do NOT run `git commit` yourself. Leave the work in the working tree; the
orchestrator verifies and commits. (Hazard #7.)
## Do not repeat an earlier worker's mistake
A worker once ended its turn saying "tests are running in the background, I'll report
when done." That is a FAILURE. Run it, WAIT, report observed exit codes. A truthfully
reported gap is worth far more than a false green. If you run out of room, STOP and
report exactly which of the four tasks are complete and which are not -- a partial
result you describe accurately is recoverable; a false claim of completion is not.
## Report (short — read by a token-constrained orchestrator)
- real exit codes for all four commands, and the test count
- the screenshot PNG paths, the signed APK path + size + signer info
- which of the four tasks are DONE and which are not
- anything in SPEC's design language you think the screens get WRONG, now that you can
finally see them rendered. Be candid — the owner explicitly cares how this looks.
+51 -15
View File
@@ -9,37 +9,73 @@
# workers run, so the box stays hot with no console attached. When the workers stop # workers run, so the box stays hot with no console attached. When the workers stop
# it writes the durable completion record AND releases the lease, so the sprite is # it writes the durable completion record AND releases the lease, so the sprite is
# free to suspend instead of burning money idling. # free to suspend instead of burning money idling.
#
# WAVE 4 POST-MORTEM — why this file was rewritten:
# The previous lease_hold() did DELETE-then-POST with both results discarded, and
# logged "lease renewed" unconditionally. If that POST ever failed, the box was left
# with NO lease while the log claimed otherwise. That is exactly what wave 4 looks
# like: last "renewal" 11:39, workers dead by 11:46, box rebooted 12:55. So now:
# - every acquire is VERIFIED against GET /v1/tasks before it is believed
# - a failed acquire retries, and is logged as a failure, never as a success
# - the lease is re-checked every POLL (cheap), not only every RENEW, so a lease
# lost between renewals is noticed in seconds instead of never
set -u set -u
SENT="$1"; shift SENT="$1"; shift
TASKS=("$@") TASKS=("$@")
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"; LEASE="bookshelf-wave" L="$HOME/bookshelf/logs"; OUT="$L/$SENT"; LEASE="bookshelf-wave"
RENEW="${RENEW:-900}" # renew every 15 min against a 60 min lease RENEW="${RENEW:-900}" # proactively renew every 15 min against a 60 min lease
POLL="${POLL:-30}" # but CHECK for completion every 30s (see loop below) POLL="${POLL:-30}" # but CHECK completion + lease health every 30s
GUARD_LOG="$L/wave-guard.log" GUARD_LOG="$L/wave-guard.log"
lease_hold() {
sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1
sprite-env curl -X POST /v1/tasks -H 'Content-Type: application/json' \
-d "{\"name\":\"$LEASE\",\"expire\":\"3600s\"}" >/dev/null 2>&1
}
lease_release() { sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1; }
log() { echo "[$(date -Is)] $*" >> "$GUARD_LOG"; } log() { echo "[$(date -Is)] $*" >> "$GUARD_LOG"; }
lease_present() { sprite-env curl /v1/tasks 2>/dev/null | grep -q "\"$LEASE\""; }
# Acquire and VERIFY. Returns 0 only if the lease is actually live afterwards.
lease_acquire() {
local attempt
for attempt in 1 2 3; do
sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1
sprite-env curl -X POST /v1/tasks -H 'Content-Type: application/json' \
-d "{\"name\":\"$LEASE\",\"expire\":\"3600s\"}" >/dev/null 2>&1
if lease_present; then return 0; fi
log "WARNING: lease acquire attempt $attempt did not take; retrying"
sleep 5
done
return 1
}
lease_release() { sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1; }
trap 'lease_release; log "guard exiting, lease released"; exit 0' TERM INT trap 'lease_release; log "guard exiting, lease released"; exit 0' TERM INT
log "guard start: sentinel=$SENT tasks=${TASKS[*]} renew=${RENEW}s" log "guard start: sentinel=$SENT tasks=${TASKS[*]} renew=${RENEW}s poll=${POLL}s"
lease_hold; log "lease '$LEASE' acquired (3600s)" if lease_acquire; then
log "lease '$LEASE' acquired and VERIFIED (3600s)"
else
log "FATAL: could not acquire lease '$LEASE' -- the sprite may suspend mid-wave"
fi
last_renew=$(date +%s) last_renew=$(date +%s)
# Poll FREQUENTLY (POLL) but renew SLOWLY (RENEW). Coupling the two, as the first
# version did, meant a wave that finished right after a renewal sat undetected for a
# full RENEW interval with the sprite pinned hot the whole time.
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
sleep "$POLL" sleep "$POLL"
now=$(date +%s) now=$(date +%s)
if [ $(( now - last_renew )) -ge "$RENEW" ]; then if [ $(( now - last_renew )) -ge "$RENEW" ]; then
lease_hold; last_renew=$now if lease_acquire; then
log "lease renewed; workers still running" log "lease renewed and VERIFIED; workers still running"
else
log "ERROR: scheduled renewal FAILED; lease is not held"
fi
last_renew=$now
elif ! lease_present; then
# Lost between renewals (expiry race, API blip, someone else's DELETE).
log "ERROR: lease disappeared between renewals -- re-acquiring now"
if lease_acquire; then
log "lease re-acquired and VERIFIED"
last_renew=$now
else
log "ERROR: emergency re-acquire FAILED; sprite is unprotected"
fi
fi fi
done done