jfmonty2/bookshelf
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move Open Library lookup to the Editions API; stop asking Google Books for its placeholder

Browse Source 
Two user-reported shelf-testing symptoms, both a third party answering
misleadingly and the app believing it.

1. Open Library's /api/books?bibkeys=... now 404s for EVERY ISBN, including
   books OL demonstrably still holds, with OL's own x-ol-stats header on the
   response while /isbn/, /search.json, /api/volumes/brief and covers.* all
   serve normally. OL's docs call it the "Legacy Books API" that "may be phased
   out" and it is gone from their API index, so this reads as a retirement
   rather than an outage. The app had been effectively single-sourced on Google
   Books since it broke: every failure the user saw was a book GB lacks.

   Lookup now uses /isbn/{isbn}.json — current, non-legacy, edition-level, and
   the only option of the three that carries a description. /api/volumes/brief
   is a near drop-in for the old response shape and was rejected precisely
   because it is also legacy.

   Its costs, all handled: authors are references, so AuthorNameCache resolves
   and caches them for the process lifetime (books by one author get scanned in
   runs off one shelf); an edition may carry NO authors, in which case they live
   on the work — 9780898707168 on the user's own shelf is exactly this, so
   without the work fallback the move would have silently dropped its author;
   author/work requests are best-effort and can only degrade a record, never
   turn Found into Unavailable.

   404 on this endpoint is authoritative NotFound. The legacy endpoint reported
   a miss as 200 with an empty object, which is why every non-2xx there was a
   failure. Every other non-2xx still is.

2. Google Books answers zoom=2 with a grey "image not available" PNG at HTTP
   200 — not a 404 — for any volume it holds no full preview of. Coil loads it
   as a success, so BookCover's placeholder never fires and the cover pipeline
   uploads Google's placeholder to PocketBase as the book's cover. Measured over
   18 real volumes: 11 placeholders at zoom=2, 0 at zoom=1&w=400. zoom=0/3/6 are
   placeholders too. normalizeCoverUrl now pins zoom=1, adds w=400 and strips
   edge=curl.

SPEC.md's "Book metadata lookup" is rewritten with both rules and the evidence
for them — it was the source of the zoom=2 instruction, and would otherwise be
the reason someone restores it.

Also fixed, because it blocked verification: LibraryViewModelTest never cleared
the view models it built, and LibraryViewModel's eleven WhileSubscribed(5_000)
flows kept running five seconds into later tests, racing resetMain(). It now
cancels each viewModelScope in tearDown.

NOT fixed, reported instead: AddBookViewModel.performSave's in-flight guard is a
check-then-act and two coroutines can both pass it. Unrelated to this change
(that VM has no metadata dependency) and out of scope. See HAZARD #13.

Verified: assembleDebug exit 0; testDebugUnitTest --rerun-tasks 378 tests,
2 skipped, 0 failures (was 355); verifyPaparazziDebug exit 0, no pixels moved;
0 "always 'false'" warnings; no build files touched. LIVE_METADATA=1 live test
ran (not skipped): 9/9 Found with cover art, with the GB key absent, so Open
Library alone answered through the new endpoint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Spriteandclaude committed 2026-09-20 02:13:01 +00:00
1 parent 4695731f91
commit ab83287b5d
25 files changed
+972 -193
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/METADATA-SOURCES.md
+119
View File
@@ -350,3 +350,122 @@ The 17% of books with no cover art anywhere is unchanged, and so is Open
Library's ~13% transport failure rate. What changes is that a failure of one
source is now much more likely to be covered by the other instead of surfacing
as `Unavailable`.
---
## The Open Library endpoint move — 2026-09-20
The user came back from a round of shelf testing with two symptoms. Both had the
same character: a third party answering 200-with-something-useless, or 404-without-
meaning-it, and the app believing it.
### 1. `/api/books` is answering 404 to everything
Reported symptom: intermittent "ISBN 9781328613042 — one or more sources couldn't be
reached … open library: http 404".
Measured from this box, and independently reproduced from the user's phone network:
| Request | Result |
|---|---|
| `/api/books?bibkeys=ISBN:9781328613042` (*The Fall of Gondolin*) | **404, 0 bytes** |
| `/api/books?bibkeys=ISBN:9780140328721` (*Fantastic Mr Fox*) | **404, 0 bytes** |
| `/api/books?bibkeys=OLID:OL1017798M` | **404, 0 bytes** |
| same, `jscmd=viewapi` / no `jscmd` | **404** |
| same, browser UA / descriptive UA / `okhttp` UA / HTTP/1.1 | **404** |
| `/isbn/9781328613042.json` | 200 — full edition record |
| `/api/volumes/brief/isbn/9781328613042.json` | 200 — edition `OL26961988M` |
| `/search.json?q=isbn:…` | 200 |
| `covers.openlibrary.org/b/isbn/…` | 200 |
The 404 carries `content-type: application/json` and OL's own `x-ol-stats` header, so
it is OL's application answering, not a CDN error page. **Open Library still holds the
book** — only that one door is shut.
**This was first written up as an outage. That was probably wrong.** OL's docs
(`/dev/docs/api/books`) call `/api/books` the "Legacy Books API" and say "Please
consider using the Book Search API above; this is a legacy endpoint and may be phased
out in the future", and it does not appear in their API index at all. A blanket,
header-independent 404 while every neighbouring endpoint serves normally fits a
retirement. There is no 410 and no `Sunset` header, so intent can't be proven from
outside — but "wait for it to come back" was never a plan either way.
**Consequence while it lasted: the app was single-sourced on Google Books.** Every
lookup that reached the user as a failure was a book Google Books doesn't have.
9781328613042 is exactly that — Google Books returns `totalItems: 0` for it, with the
key. So the error message named only Open Library because `combine()` names only
sources that *failed*, and Google Books had answered authoritatively.
### Which endpoint replaces it
| | `/api/volumes/brief` (legacy) | **`/isbn/{isbn}.json`** (chosen) | `search.json` (OL's own suggestion) |
|---|---|---|---|
| Requests per lookup | 1 | 1 + 1 per author | 1 |
| Title | edition | **edition** | *work* title |
| Author names | inline | **refs — needs a fetch** | inline |
| Publisher | edition | **edition** | all editions mashed together |
| Year | 2018 (this edition) | **2018** | 1985 (first ed. of the work) |
| Description | — | **yes, when present** | — |
| Cover | `cover` object | **`covers: [id]`** | `cover_i` |
| Status | **legacy** | current | current |
`search.json` is work-level: for *The Fall of Gondolin* it reports 1985 and eight
publishers, which is wrong for a catalogue of specific editions someone owns.
`/api/volumes/brief` is the least work — its `data` block is nearly the old
`jscmd=data` shape — and that is the trap: swapping a retired legacy endpoint for
another legacy endpoint buys one migration and no safety.
### What the Editions API costs, and two things that would have been silent bugs
- **`authors` are references, not names.** Hence `AuthorNameCache` (process-lifetime,
unbounded by design: hundreds of short strings at most). Books by one author get
scanned in runs off one shelf, so the cache hits constantly.
- **An edition can carry no authors at all.** 9780898707168 — *The Harp and Laurel
Wreath*, on the user's own shelf — has `authors: None`; they exist only on the work.
The legacy endpoint resolved that and returned "Laura M. Berquist". Without a
work-level fallback the migration would have silently dropped the author for books
like it. Caught before shipping, and pinned by a test.
- **`covers` uses `-1` as a "no cover" sentinel** (9780898707168 -> `[698345, -1]`,
9780140328721 -> `[15152634, 8739161, -1]`). Taking `.first()` would eventually
build `b/id/-1-L.jpg`. Take the first positive id.
- **`description` is a bare string on some records and `{type, value}` on others**,
and absent on most: of 15 real editions sampled, 1 string, 5 objects, 9 absent.
- **404 now means NotFound.** The legacy endpoint reported a miss as `200 {}`, so
every non-2xx there was rightly a failure. This endpoint 404s instead, and that is
authoritative — reporting it as Unavailable would offer a retry for a book no
amount of retrying will find. Every other non-2xx is still Unavailable.
### 2. Google Books serves a placeholder image at `zoom=2`
Reported symptom: two different cover placeholders — the app's own two-tone/gold one,
and "an ugly gray-text-on-white saying image not available".
The second is Google's, and the app was asking for it. SPEC said to force `zoom=2` on
GB `imageLinks`; Google's `thumbnail` is `zoom=1`. For a volume Google has no full
preview of — the metadata-only `…AAAACAAJ` records, i.e. most small-press and older
material — **`zoom=2` is not a valid rendition and Google substitutes a placeholder
with HTTP 200** instead of 404ing. Coil loads it as a success, so `BookCover`'s
placeholder never fires, and the cover pipeline uploads it to PocketBase as the cover.
Measured over 18 volumes that have `imageLinks`:
| URL form | real cover | "image not available" |
|---|---|---|
| `zoom=2` (what the app sent) | 7 | **11** |
| `zoom=1&w=400`, `edge=curl` stripped | **18** | 0 |
The 11 were byte-identical: 15,567 bytes, md5 `c96309220b9cbd205c36d879d09a3647`.
`zoom=0`, `3` and `6` return the same artwork at 575x750 and 1280x1670 — so there is
no bigger-zoom escape, and hash-detection would mean chasing renditions forever.
`w=` is what buys resolution: Google honours it up to the source scan's native width.
**Confirmed in the live library, not just on screen.** *Abraham Lincoln's World*
(`9plhmyj5s68f58e`) has a `zoom=2` `cover_source_url`, and the file stored on
PocketBase for it is that exact placeholder — 15,567 bytes, same md5, served as
`.jpg`. The user is handling the already-poisoned covers themselves.
### Verification
`LiveMetadataLookupTest` (opt-in, `LIVE_METADATA=1`) against the rewritten client:
9 lookups, all **Found with cover art**, 1.7s–4.9s — and with the Google Books key
absent, so that is Open Library alone answering through the new endpoint.