jfmonty2/bookshelf
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move Open Library lookup to the Editions API; stop asking Google Books for its placeholder

Browse Source 
Two user-reported shelf-testing symptoms, both a third party answering
misleadingly and the app believing it.

1. Open Library's /api/books?bibkeys=... now 404s for EVERY ISBN, including
   books OL demonstrably still holds, with OL's own x-ol-stats header on the
   response while /isbn/, /search.json, /api/volumes/brief and covers.* all
   serve normally. OL's docs call it the "Legacy Books API" that "may be phased
   out" and it is gone from their API index, so this reads as a retirement
   rather than an outage. The app had been effectively single-sourced on Google
   Books since it broke: every failure the user saw was a book GB lacks.

   Lookup now uses /isbn/{isbn}.json — current, non-legacy, edition-level, and
   the only option of the three that carries a description. /api/volumes/brief
   is a near drop-in for the old response shape and was rejected precisely
   because it is also legacy.

   Its costs, all handled: authors are references, so AuthorNameCache resolves
   and caches them for the process lifetime (books by one author get scanned in
   runs off one shelf); an edition may carry NO authors, in which case they live
   on the work — 9780898707168 on the user's own shelf is exactly this, so
   without the work fallback the move would have silently dropped its author;
   author/work requests are best-effort and can only degrade a record, never
   turn Found into Unavailable.

   404 on this endpoint is authoritative NotFound. The legacy endpoint reported
   a miss as 200 with an empty object, which is why every non-2xx there was a
   failure. Every other non-2xx still is.

2. Google Books answers zoom=2 with a grey "image not available" PNG at HTTP
   200 — not a 404 — for any volume it holds no full preview of. Coil loads it
   as a success, so BookCover's placeholder never fires and the cover pipeline
   uploads Google's placeholder to PocketBase as the book's cover. Measured over
   18 real volumes: 11 placeholders at zoom=2, 0 at zoom=1&w=400. zoom=0/3/6 are
   placeholders too. normalizeCoverUrl now pins zoom=1, adds w=400 and strips
   edge=curl.

SPEC.md's "Book metadata lookup" is rewritten with both rules and the evidence
for them — it was the source of the zoom=2 instruction, and would otherwise be
the reason someone restores it.

Also fixed, because it blocked verification: LibraryViewModelTest never cleared
the view models it built, and LibraryViewModel's eleven WhileSubscribed(5_000)
flows kept running five seconds into later tests, racing resetMain(). It now
cancels each viewModelScope in tearDown.

NOT fixed, reported instead: AddBookViewModel.performSave's in-flight guard is a
check-then-act and two coroutines can both pass it. Unrelated to this change
(that VM has no metadata dependency) and out of scope. See HAZARD #13.

Verified: assembleDebug exit 0; testDebugUnitTest --rerun-tasks 378 tests,
2 skipped, 0 failures (was 355); verifyPaparazziDebug exit 0, no pixels moved;
0 "always 'false'" warnings; no build files touched. LIVE_METADATA=1 live test
ran (not skipped): 9/9 Found with cover art, with the GB key absent, so Open
Library alone answered through the new endpoint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Spriteandclaude committed 2026-09-20 02:13:01 +00:00
1 parent 4695731f91
commit ab83287b5d
25 files changed
+972 -193
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/HANDOFF.md
+63
View File
@@ -896,3 +896,66 @@ code: compare against the gate logs' "BUILD SUCCESSFUL in" times and ask the use
`tasks/wave-chain.sh` now reads its commit subject from `tasks/<task>.subject`
(it was hard-coded for waves 9/10).
## Wave 12 — Open Library endpoint move + Google Books cover fix: DONE 2026-09-20
Done **by the orchestrator directly, not a worker** — the user left the choice open
("do whatever you think will yield the best results"). The change is confined to
`data/metadata`, and the live API shapes had already been measured during diagnosis,
so a worker would have spent a session rediscovering them. Full write-up in
`docs/METADATA-SOURCES.md` § "The Open Library endpoint move".
**Two user-reported symptoms, both third parties answering misleadingly:**
1. `/api/books?bibkeys=...` now 404s for EVERY ISBN, including books OL still holds.
OL's docs call it the "Legacy Books API" that "may be phased out"; it is gone from
their API index. Treat as retired, not down. The app had been effectively
single-sourced on Google Books since it broke.
2. Google Books answers `zoom=2` with a grey "image not available" PNG at **HTTP 200**
for volumes with no full preview — 11 of 18 sampled. SPEC told us to force zoom=2.
**Changed:** `OpenLibraryClient` now uses `/isbn/{isbn}.json` (+ `/works/` fallback
for authors, + `/authors/` resolution via the new `AuthorNameCache`);
`normalizeCoverUrl` pins `zoom=1`, adds `w=400`, strips `edge=curl`. SPEC.md's
"Book metadata lookup" section rewritten with both rules and WHY, so nobody restores
either. Old-endpoint fixtures deleted; 9 new fixtures captured from the live API.
**Three things that would have been silent bugs** (all caught pre-ship, all pinned by
tests): an edition record can carry NO authors (9780898707168 — the user's own shelf —
has them only on the work, so a naive move drops the author); `covers` uses `-1` as a
no-cover sentinel; `description` is a bare string on some records and `{type,value}`
on others. Also: **on this endpoint 404 IS authoritative NotFound**, where on the old
one every non-2xx was a failure.
| Check | Result |
|---|---|
| `./tasks/gw assembleDebug` | exit 0 |
| `./tasks/gw testDebugUnitTest --rerun-tasks` | exit 0 — **378 tests**, 2 skipped, 0 failures (was 355) |
| `./tasks/gw verifyPaparazziDebug` | exit 0 — no pixels moved |
| `LIVE_METADATA=1` live test | **RAN, not skipped** — 9/9 Found with cover art, keyless GB, so OL alone answered |
| `grep "always 'false'"` | 0 hits |
### HAZARD #13 — two latent test races, surfaced by adding tests
Adding ~20 tests shifted suite timing and made two pre-existing races start flapping.
Neither was caused by the metadata change; both flake in whichever test happens to be
running when a window expires, NOT in the one at fault. Do not chase the named test.
1. **FIXED. `LibraryViewModelTest` leaked view models.** `LibraryViewModel` has eleven
`stateIn(viewModelScope, WhileSubscribed(5_000), ...)` flows, so its upstreams keep
running five seconds after the last collector. Nothing ever cleared the VM, so they
were still touching `Dispatchers.Main` while the NEXT test's tearDown called
`resetMain()` -> "Dispatchers.Main is used concurrently with setting it". The test
now tracks every VM it builds and cancels `viewModelScope` in tearDown (and guards
`db.close()` with `::db.isInitialized`, since a failed setUp otherwise masks the
real failure with an UninitializedPropertyAccessException).
2. **NOT FIXED — reported to the user, out of scope.** `AddBookViewModel.performSave`
has a check-then-act in-flight guard: `if (_formState.value.isSaving) return null`
and only then `update { isSaving = true }`. Two coroutines can both pass the check,
which is what `a second save while one is in flight does not create a second book`
catches when timing allows. The KDoc right above it claims a double-tap "still
can't create two books" — that claim is false today. `AddBookViewModel` takes no
metadata dependency at all, so this is provably unrelated to wave 12. One atomic
`getAndUpdate` fixes it.
**Lesson:** a green suite on this project is worth one re-run before you trust it, and
`--rerun-tasks` is mandatory — a plain `testDebugUnitTest` after a stash happily
reports BUILD SUCCESSFUL `FROM-CACHE` without executing a single test. That nearly
produced a false "pre-existing, not mine" conclusion here.
docs/METADATA-SOURCES.md
+119
View File
@@ -350,3 +350,122 @@ The 17% of books with no cover art anywhere is unchanged, and so is Open
Library's ~13% transport failure rate. What changes is that a failure of one
source is now much more likely to be covered by the other instead of surfacing
as `Unavailable`.
---
## The Open Library endpoint move — 2026-09-20
The user came back from a round of shelf testing with two symptoms. Both had the
same character: a third party answering 200-with-something-useless, or 404-without-
meaning-it, and the app believing it.
### 1. `/api/books` is answering 404 to everything
Reported symptom: intermittent "ISBN 9781328613042 — one or more sources couldn't be
reached … open library: http 404".
Measured from this box, and independently reproduced from the user's phone network:
| Request | Result |
|---|---|
| `/api/books?bibkeys=ISBN:9781328613042` (*The Fall of Gondolin*) | **404, 0 bytes** |
| `/api/books?bibkeys=ISBN:9780140328721` (*Fantastic Mr Fox*) | **404, 0 bytes** |
| `/api/books?bibkeys=OLID:OL1017798M` | **404, 0 bytes** |
| same, `jscmd=viewapi` / no `jscmd` | **404** |
| same, browser UA / descriptive UA / `okhttp` UA / HTTP/1.1 | **404** |
| `/isbn/9781328613042.json` | 200 — full edition record |
| `/api/volumes/brief/isbn/9781328613042.json` | 200 — edition `OL26961988M` |
| `/search.json?q=isbn:…` | 200 |
| `covers.openlibrary.org/b/isbn/…` | 200 |
The 404 carries `content-type: application/json` and OL's own `x-ol-stats` header, so
it is OL's application answering, not a CDN error page. **Open Library still holds the
book** — only that one door is shut.
**This was first written up as an outage. That was probably wrong.** OL's docs
(`/dev/docs/api/books`) call `/api/books` the "Legacy Books API" and say "Please
consider using the Book Search API above; this is a legacy endpoint and may be phased
out in the future", and it does not appear in their API index at all. A blanket,
header-independent 404 while every neighbouring endpoint serves normally fits a
retirement. There is no 410 and no `Sunset` header, so intent can't be proven from
outside — but "wait for it to come back" was never a plan either way.
**Consequence while it lasted: the app was single-sourced on Google Books.** Every
lookup that reached the user as a failure was a book Google Books doesn't have.
9781328613042 is exactly that — Google Books returns `totalItems: 0` for it, with the
key. So the error message named only Open Library because `combine()` names only
sources that *failed*, and Google Books had answered authoritatively.
### Which endpoint replaces it
| | `/api/volumes/brief` (legacy) | **`/isbn/{isbn}.json`** (chosen) | `search.json` (OL's own suggestion) |
|---|---|---|---|
| Requests per lookup | 1 | 1 + 1 per author | 1 |
| Title | edition | **edition** | *work* title |
| Author names | inline | **refs — needs a fetch** | inline |
| Publisher | edition | **edition** | all editions mashed together |
| Year | 2018 (this edition) | **2018** | 1985 (first ed. of the work) |
| Description | — | **yes, when present** | — |
| Cover | `cover` object | **`covers: [id]`** | `cover_i` |
| Status | **legacy** | current | current |
`search.json` is work-level: for *The Fall of Gondolin* it reports 1985 and eight
publishers, which is wrong for a catalogue of specific editions someone owns.
`/api/volumes/brief` is the least work — its `data` block is nearly the old
`jscmd=data` shape — and that is the trap: swapping a retired legacy endpoint for
another legacy endpoint buys one migration and no safety.
### What the Editions API costs, and two things that would have been silent bugs
- **`authors` are references, not names.** Hence `AuthorNameCache` (process-lifetime,
unbounded by design: hundreds of short strings at most). Books by one author get
scanned in runs off one shelf, so the cache hits constantly.
- **An edition can carry no authors at all.** 9780898707168 — *The Harp and Laurel
Wreath*, on the user's own shelf — has `authors: None`; they exist only on the work.
The legacy endpoint resolved that and returned "Laura M. Berquist". Without a
work-level fallback the migration would have silently dropped the author for books
like it. Caught before shipping, and pinned by a test.
- **`covers` uses `-1` as a "no cover" sentinel** (9780898707168 -> `[698345, -1]`,
9780140328721 -> `[15152634, 8739161, -1]`). Taking `.first()` would eventually
build `b/id/-1-L.jpg`. Take the first positive id.
- **`description` is a bare string on some records and `{type, value}` on others**,
and absent on most: of 15 real editions sampled, 1 string, 5 objects, 9 absent.
- **404 now means NotFound.** The legacy endpoint reported a miss as `200 {}`, so
every non-2xx there was rightly a failure. This endpoint 404s instead, and that is
authoritative — reporting it as Unavailable would offer a retry for a book no
amount of retrying will find. Every other non-2xx is still Unavailable.
### 2. Google Books serves a placeholder image at `zoom=2`
Reported symptom: two different cover placeholders — the app's own two-tone/gold one,
and "an ugly gray-text-on-white saying image not available".
The second is Google's, and the app was asking for it. SPEC said to force `zoom=2` on
GB `imageLinks`; Google's `thumbnail` is `zoom=1`. For a volume Google has no full
preview of — the metadata-only `…AAAACAAJ` records, i.e. most small-press and older
material — **`zoom=2` is not a valid rendition and Google substitutes a placeholder
with HTTP 200** instead of 404ing. Coil loads it as a success, so `BookCover`'s
placeholder never fires, and the cover pipeline uploads it to PocketBase as the cover.
Measured over 18 volumes that have `imageLinks`:
| URL form | real cover | "image not available" |
|---|---|---|
| `zoom=2` (what the app sent) | 7 | **11** |
| `zoom=1&w=400`, `edge=curl` stripped | **18** | 0 |
The 11 were byte-identical: 15,567 bytes, md5 `c96309220b9cbd205c36d879d09a3647`.
`zoom=0`, `3` and `6` return the same artwork at 575x750 and 1280x1670 — so there is
no bigger-zoom escape, and hash-detection would mean chasing renditions forever.
`w=` is what buys resolution: Google honours it up to the source scan's native width.
**Confirmed in the live library, not just on screen.** *Abraham Lincoln's World*
(`9plhmyj5s68f58e`) has a `zoom=2` `cover_source_url`, and the file stored on
PocketBase for it is that exact placeholder — 15,567 bytes, same md5, served as
`.jpg`. The user is handling the already-poisoned covers themselves.
### Verification
`LiveMetadataLookupTest` (opt-in, `LIVE_METADATA=1`) against the rewritten client:
9 lookups, all **Found with cover art**, 1.7s–4.9s — and with the Google Books key
absent, so that is Open Library alone answering through the new endpoint.
docs/SPEC.md
+36 -3
View File
@@ -102,14 +102,44 @@ Trigger: app start, manual pull-to-refresh, WorkManager periodic (~6h, network-c
Never let sync failure surface as a crash or a blocking dialog — a quiet status line only.
## Book metadata lookup
Primary Open Library: https://openlibrary.org/api/books?bibkeys=ISBN:{isbn}&format=json&jscmd=data
Fallback Google Books: https://www.googleapis.com/books/v1/volumes?q=isbn:{isbn} (no key)
Cover: https://covers.openlibrary.org/b/isbn/{isbn}-L.jpg else GB imageLinks (force https, zoom=2)
Primary Open Library EDITIONS API: https://openlibrary.org/isbn/{isbn}.json (302 -> the edition record)
Fallback Google Books: https://www.googleapis.com/books/v1/volumes?q=isbn:{isbn} (keyed)
Merge: prefer whichever has a title; fill blanks from the other.
DO NOT go back to `/api/books?bibkeys=...&jscmd=data`. On 2026-09-20 it was found
answering 404 to EVERY bibkey form for EVERY ISBN — including books Open Library
demonstrably still holds — while /isbn/, /search.json, /api/volumes/brief and
covers.openlibrary.org all served normally. OL's own docs call it the "Legacy Books
API" and say it "may be phased out in the future", and it is absent from their API
index, so treat it as retired. /api/volumes/brief (the "Legacy Partner API") is a
near drop-in for the old response shape and is ALSO legacy — that is why it was not
chosen. See docs/METADATA-SOURCES.md.
The Editions API costs extra requests, by design:
- `authors` are REFERENCES (`{"key": "/authors/OL26320A"}`), not names; each needs
a GET of /authors/{id}.json. `AuthorNameCache` holds resolved names for the
process lifetime — books by one author are scanned in runs off one shelf.
- an edition record may carry NO authors at all (real: 9780898707168), in which
case they are on the work; fall back to /works/{id}.json. Losing this fallback
silently drops the author for such books.
- author/work requests are BEST-EFFORT: the edition request already succeeded, so
a failure there degrades the record, it must never turn Found into Unavailable.
Cover: OL `covers: [id]` -> https://covers.openlibrary.org/b/id/{id}-L.jpg
else GB imageLinks (force https, **zoom=1**, add `w=400`, strip `edge=curl`)
Cover URL comes from a source that REPORTS one. Never synthesize the by-ISBN cover
URL as if it were evidence: for an edition with no art that endpoint returns 200 +
a 43-byte 1x1 transparent GIF, which loads "successfully" and paints nothing. As a
last resort it may be used only with `?default=false`, which makes a miss a 404.
OL's `covers` array uses -1 as a "no cover here" sentinel; take the first POSITIVE id.
DO NOT force `zoom=2` on a Google Books cover (this spec said to, and was wrong).
For a volume Google holds no full preview of, zoom=2 is not a valid rendition and
Google answers 200 with a grey "image not available" PNG rather than a 404 — which
loads successfully, so BookCover's placeholder never fires and the cover pipeline
uploads Google's placeholder to PocketBase as the book's cover. Measured 2026-09-20:
11 of 18 real volumes did this at zoom=2; 0 of 18 at zoom=1 with a width. zoom=0/3/6
are placeholders too, so there is no larger-zoom escape.
Lookup outcome is THREE-WAY, never a bare null. A source that could not be reached
must never be reported to the user as a book that does not exist:
@@ -117,6 +147,9 @@ must never be reported to the user as a book that does not exist:
NotFound - EVERY source answered authoritatively and none had it
Unavailable - no source could be reached (non-2xx, timeout, transport error)
and none of the reachable ones had it
On the Editions API a 404 IS authoritative NotFound — unlike the legacy endpoint,
which reported a miss as 200 with an empty object and so made every non-2xx a
failure. Every OTHER non-2xx remains Unavailable.
UI: Found -> the save sheet. NotFound -> manual entry pre-filled with the scanned
ISBN. Unavailable -> a retry affordance, with manual entry as the escape hatch;
it must NOT claim the book is unknown.