orchestration: verify lease acquisition in wave-guard; F3 prompt for wave 4 finish
wave-guard.sh post-mortem: lease_hold() did DELETE-then-POST with both results discarded and logged "lease renewed" unconditionally, so a failed re-POST left the box with no lease while the log claimed it was protected. That matches the wave-4 loss exactly (last "renewal" 11:39, workers dead 11:46, reboot 12:55). Now: every acquire is verified against GET /v1/tasks before it is believed, a failed acquire retries and is logged as a failure, and the lease is re-checked every POLL rather than only every RENEW so a lease lost between renewals is caught in seconds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016mTs3kQXQsQwonXpEq7aEw
This commit is contained in:
@@ -0,0 +1,91 @@
|
|||||||
|
You are Worker F3 on the Bookshelf project (~/bookshelf). You are FINISHING wave 4.
|
||||||
|
|
||||||
|
FIRST READ (contract — follow exactly, do not restate):
|
||||||
|
~/bookshelf/docs/SPEC.md ("Design language", "Screens", "Quality bar")
|
||||||
|
~/bookshelf/docs/HANDOFF.md (state + hazards already paid for — read the hazards)
|
||||||
|
|
||||||
|
## Where things actually stand (do not re-derive this; it is verified)
|
||||||
|
Waves 1-3 are DONE. Wave 4 was launched once and both workers were killed ~11 minutes
|
||||||
|
in by the 5h session quota plus a sprite suspend. Their partial work is COMMITTED and
|
||||||
|
verified green at `a4adf83`: `assembleDebug` + `testDebugUnitTest` pass, 94 tests, 0
|
||||||
|
failures.
|
||||||
|
|
||||||
|
Already done, DO NOT redo:
|
||||||
|
- The live-server sync test is FINISHED and PASSING against the real PocketBase --
|
||||||
|
auth, push with client-generated ids, pull, last-write-wins, tombstones, and the
|
||||||
|
cover byte-for-byte round-trip. That was wave 4's biggest risk and it is closed.
|
||||||
|
`server/live-sync-test.sh` + `org.modg.bookshelf.livesync.LiveSyncTest`. LEAVE BOTH
|
||||||
|
ALONE, and leave `server/` alone entirely.
|
||||||
|
- Library screen screenshots (populated + empty, light + dark) and the reusable
|
||||||
|
`app/app/src/test/java/org/modg/bookshelf/ui/screens/ScreenFixtures.kt`.
|
||||||
|
FOLLOW that file's patterns; do not invent a second approach.
|
||||||
|
|
||||||
|
## Your scope — four tasks, in this order
|
||||||
|
1. **Fix the settings-email gap FIRST** (it changes what task 2 renders).
|
||||||
|
Settings currently shows the PocketBase user id instead of the signed-in email,
|
||||||
|
because `AuthRepository`/`SettingsStore` never persist the email at login. Persist
|
||||||
|
it on successful login and show it in Settings. Small, surgical change -- do not
|
||||||
|
redesign auth. Add/extend a unit test that actually asserts the email is persisted.
|
||||||
|
|
||||||
|
2. **Paparazzi screenshots for the five REMAINING screens** -- setup, detail, scan,
|
||||||
|
locations, settings -- each in BOTH light and dark, with realistic populated state
|
||||||
|
(a plausible shelf of books, placeholder cover art), not empty states. The library
|
||||||
|
screen is already done. The scan screen cannot show a live camera in Paparazzi:
|
||||||
|
render its overlay/reticle and the result bottom sheet over a static backdrop, and
|
||||||
|
SAY SO in your report.
|
||||||
|
|
||||||
|
3. **Release signing + signed APK.** Generate a release keystore, wire a signingConfig
|
||||||
|
into `app/app/build.gradle.kts`, produce a signed release APK. The keystore and its
|
||||||
|
passwords MUST be gitignored -- credentials in `app/keystore.properties` (gitignored),
|
||||||
|
read from the build file, with a graceful fallback so the build still works for
|
||||||
|
anyone WITHOUT the keystore. State the passwords in your report so the owner can
|
||||||
|
rebuild. `minifyEnabled` is false today: if you enable R8 you must PROVE the signed
|
||||||
|
APK still builds and that Room/Retrofit/kotlinx-serialization/ML Kit survive it. If
|
||||||
|
you cannot prove it, leave R8 OFF and say so. Leaving it off is an acceptable answer.
|
||||||
|
|
||||||
|
4. **Top-level `README.md`** -- what this is, the two-person shared-library model,
|
||||||
|
architecture (offline-first Room + PocketBase, push-then-pull last-write-wins sync),
|
||||||
|
how to build, how to deploy the server (point at `server/deploy/`), how to install
|
||||||
|
the APK, and an honest "current limitations" section. SPEC requires the LWW conflict
|
||||||
|
rule be documented here. Reuse real content from `server/README.md` and `docs/SPEC.md`;
|
||||||
|
do not contradict them. Include that the app has never run on a physical device.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
- You are the ONLY worker running. No file-ownership conflicts this time.
|
||||||
|
- Do NOT touch `server/`, `LiveSyncTest.kt`, or `live-sync-test.sh`.
|
||||||
|
- Do NOT weaken or delete existing tests to get green. 94 tests pass now; that number
|
||||||
|
must not go DOWN.
|
||||||
|
- Do not modify the six screens' behavior just to make them easier to screenshot. If a
|
||||||
|
screen is hard to render in isolation, that is a finding to REPORT, not a licence to
|
||||||
|
restructure it. (Task 1 is the one deliberate exception, and it is a data-layer fix.)
|
||||||
|
|
||||||
|
## Build/verify — use the serialized wrapper, never ./gradlew directly
|
||||||
|
~/bookshelf/tasks/gw assembleDebug
|
||||||
|
~/bookshelf/tasks/gw testDebugUnitTest
|
||||||
|
~/bookshelf/tasks/gw recordPaparazziDebug
|
||||||
|
~/bookshelf/tasks/gw assembleRelease
|
||||||
|
|
||||||
|
## Definition of done — actually run these, WAIT for them, report REAL exit codes
|
||||||
|
1. All four commands exit 0. `testDebugUnitTest` has ZERO failures and >= 94 tests.
|
||||||
|
2. Screenshot PNGs exist on disk for all SIX screens x light/dark. List their paths.
|
||||||
|
3. A signed release APK exists. Give its exact path, size, and the output of
|
||||||
|
`~/toolchain/android-sdk/build-tools/37.0.0/apksigner verify --print-certs <apk>`
|
||||||
|
(or an explanation if that tool is unavailable).
|
||||||
|
4. `git status` shows NO keystore and NO keystore.properties as tracked/untracked-to-be
|
||||||
|
-committed. Verify it and say so explicitly.
|
||||||
|
5. Do NOT run `git commit` yourself. Leave the work in the working tree; the
|
||||||
|
orchestrator verifies and commits. (Hazard #7.)
|
||||||
|
|
||||||
|
## Do not repeat an earlier worker's mistake
|
||||||
|
A worker once ended its turn saying "tests are running in the background, I'll report
|
||||||
|
when done." That is a FAILURE. Run it, WAIT, report observed exit codes. A truthfully
|
||||||
|
reported gap is worth far more than a false green. If you run out of room, STOP and
|
||||||
|
report exactly which of the four tasks are complete and which are not -- a partial
|
||||||
|
result you describe accurately is recoverable; a false claim of completion is not.
|
||||||
|
|
||||||
|
## Report (short — read by a token-constrained orchestrator)
|
||||||
|
- real exit codes for all four commands, and the test count
|
||||||
|
- the screenshot PNG paths, the signed APK path + size + signer info
|
||||||
|
- which of the four tasks are DONE and which are not
|
||||||
|
- anything in SPEC's design language you think the screens get WRONG, now that you can
|
||||||
|
finally see them rendered. Be candid — the owner explicitly cares how this looks.
|
||||||
+51
-15
@@ -9,37 +9,73 @@
|
|||||||
# workers run, so the box stays hot with no console attached. When the workers stop
|
# workers run, so the box stays hot with no console attached. When the workers stop
|
||||||
# it writes the durable completion record AND releases the lease, so the sprite is
|
# it writes the durable completion record AND releases the lease, so the sprite is
|
||||||
# free to suspend instead of burning money idling.
|
# free to suspend instead of burning money idling.
|
||||||
|
#
|
||||||
|
# WAVE 4 POST-MORTEM — why this file was rewritten:
|
||||||
|
# The previous lease_hold() did DELETE-then-POST with both results discarded, and
|
||||||
|
# logged "lease renewed" unconditionally. If that POST ever failed, the box was left
|
||||||
|
# with NO lease while the log claimed otherwise. That is exactly what wave 4 looks
|
||||||
|
# like: last "renewal" 11:39, workers dead by 11:46, box rebooted 12:55. So now:
|
||||||
|
# - every acquire is VERIFIED against GET /v1/tasks before it is believed
|
||||||
|
# - a failed acquire retries, and is logged as a failure, never as a success
|
||||||
|
# - the lease is re-checked every POLL (cheap), not only every RENEW, so a lease
|
||||||
|
# lost between renewals is noticed in seconds instead of never
|
||||||
set -u
|
set -u
|
||||||
SENT="$1"; shift
|
SENT="$1"; shift
|
||||||
TASKS=("$@")
|
TASKS=("$@")
|
||||||
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"; LEASE="bookshelf-wave"
|
L="$HOME/bookshelf/logs"; OUT="$L/$SENT"; LEASE="bookshelf-wave"
|
||||||
RENEW="${RENEW:-900}" # renew every 15 min against a 60 min lease
|
RENEW="${RENEW:-900}" # proactively renew every 15 min against a 60 min lease
|
||||||
POLL="${POLL:-30}" # but CHECK for completion every 30s (see loop below)
|
POLL="${POLL:-30}" # but CHECK completion + lease health every 30s
|
||||||
GUARD_LOG="$L/wave-guard.log"
|
GUARD_LOG="$L/wave-guard.log"
|
||||||
|
|
||||||
lease_hold() {
|
|
||||||
sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1
|
|
||||||
sprite-env curl -X POST /v1/tasks -H 'Content-Type: application/json' \
|
|
||||||
-d "{\"name\":\"$LEASE\",\"expire\":\"3600s\"}" >/dev/null 2>&1
|
|
||||||
}
|
|
||||||
lease_release() { sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1; }
|
|
||||||
log() { echo "[$(date -Is)] $*" >> "$GUARD_LOG"; }
|
log() { echo "[$(date -Is)] $*" >> "$GUARD_LOG"; }
|
||||||
|
|
||||||
|
lease_present() { sprite-env curl /v1/tasks 2>/dev/null | grep -q "\"$LEASE\""; }
|
||||||
|
|
||||||
|
# Acquire and VERIFY. Returns 0 only if the lease is actually live afterwards.
|
||||||
|
lease_acquire() {
|
||||||
|
local attempt
|
||||||
|
for attempt in 1 2 3; do
|
||||||
|
sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1
|
||||||
|
sprite-env curl -X POST /v1/tasks -H 'Content-Type: application/json' \
|
||||||
|
-d "{\"name\":\"$LEASE\",\"expire\":\"3600s\"}" >/dev/null 2>&1
|
||||||
|
if lease_present; then return 0; fi
|
||||||
|
log "WARNING: lease acquire attempt $attempt did not take; retrying"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
lease_release() { sprite-env curl -X DELETE "/v1/tasks/$LEASE" >/dev/null 2>&1; }
|
||||||
|
|
||||||
trap 'lease_release; log "guard exiting, lease released"; exit 0' TERM INT
|
trap 'lease_release; log "guard exiting, lease released"; exit 0' TERM INT
|
||||||
|
|
||||||
log "guard start: sentinel=$SENT tasks=${TASKS[*]} renew=${RENEW}s"
|
log "guard start: sentinel=$SENT tasks=${TASKS[*]} renew=${RENEW}s poll=${POLL}s"
|
||||||
lease_hold; log "lease '$LEASE' acquired (3600s)"
|
if lease_acquire; then
|
||||||
|
log "lease '$LEASE' acquired and VERIFIED (3600s)"
|
||||||
|
else
|
||||||
|
log "FATAL: could not acquire lease '$LEASE' -- the sprite may suspend mid-wave"
|
||||||
|
fi
|
||||||
last_renew=$(date +%s)
|
last_renew=$(date +%s)
|
||||||
|
|
||||||
# Poll FREQUENTLY (POLL) but renew SLOWLY (RENEW). Coupling the two, as the first
|
|
||||||
# version did, meant a wave that finished right after a renewal sat undetected for a
|
|
||||||
# full RENEW interval with the sprite pinned hot the whole time.
|
|
||||||
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
|
while pgrep -f 'run-task\.sh|run-resume\.sh' >/dev/null; do
|
||||||
sleep "$POLL"
|
sleep "$POLL"
|
||||||
now=$(date +%s)
|
now=$(date +%s)
|
||||||
if [ $(( now - last_renew )) -ge "$RENEW" ]; then
|
if [ $(( now - last_renew )) -ge "$RENEW" ]; then
|
||||||
lease_hold; last_renew=$now
|
if lease_acquire; then
|
||||||
log "lease renewed; workers still running"
|
log "lease renewed and VERIFIED; workers still running"
|
||||||
|
else
|
||||||
|
log "ERROR: scheduled renewal FAILED; lease is not held"
|
||||||
|
fi
|
||||||
|
last_renew=$now
|
||||||
|
elif ! lease_present; then
|
||||||
|
# Lost between renewals (expiry race, API blip, someone else's DELETE).
|
||||||
|
log "ERROR: lease disappeared between renewals -- re-acquiring now"
|
||||||
|
if lease_acquire; then
|
||||||
|
log "lease re-acquired and VERIFIED"
|
||||||
|
last_renew=$now
|
||||||
|
else
|
||||||
|
log "ERROR: emergency re-acquire FAILED; sprite is unprotected"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user