Spriteandclaude
1ba22a9f36
The user supplied the failing ISBNs: 9781883937386 (The Hittite Warrior) and 9781883937676 (Shadow Hawk), both Bethlehem Books. Both resolve against the source the app already queries — title, author, publisher, page count, cover art — and both real responses parse correctly through the app's own OpenLibraryClient. Captured as fixtures with a regression test, because these are the specific books that motivated the research. That kills the coverage hypothesis for these two and demotes Harvard, which holds neither: its 93% in the sample table is inflated by construction (the sample was drawn from Harvard) and misleading in exactly the direction that matters — it is a research library and does not carry small-press children's historical fiction. The failure is upstream of the metadata sources. Documented the three candidates; the leading one is that the barcode never decoded into a valid ISBN-13, which ScanCodeFilter drops silently with no UI feedback whatsoever. Revised the recommendation accordingly: make the app say what happened before adding any source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDcPottghJXEvfYKqFM7zf
218 lines
11 KiB
Markdown
218 lines
11 KiB
Markdown
# Book metadata lookup — why scans miss, and what else we could ask
|
||
|
||
Research note, 2026-09-09. Written in response to "out of the 3 barcodes I've
|
||
scanned, only 1 has been discovered properly."
|
||
|
||
**Nothing in here has been implemented.** SPEC's two-source design (Open Library
|
||
primary, Google Books fallback) is unchanged. This is the evidence for deciding
|
||
whether to change it.
|
||
|
||
> **Update, same day — the coverage hypothesis is dead.** The user supplied the
|
||
> two ISBNs that failed: 9781883937386 (*The Hittite Warrior*) and 9781883937676
|
||
> (*Shadow Hawk*), both Bethlehem Books. **Both are fully present in Open
|
||
> Library** — title, author, publisher, page count and cover art — and the app's
|
||
> own parser handles both real responses correctly (regression test:
|
||
> `OpenLibraryClientTest.parses the real responses for the two books the app
|
||
> failed to identify`). Whatever went wrong on the phone was upstream of the
|
||
> metadata sources. Everything below still holds as background, but do not act
|
||
> on "add a third source" until we know why a book Open Library *has* did not
|
||
> reach the lookup. See [What actually failed](#what-actually-failed).
|
||
|
||
---
|
||
|
||
## Short version
|
||
|
||
Three separate defects were making lookups *look* far worse than the underlying
|
||
data actually is, and all three are now fixed (commit `356f639`). They are not
|
||
the same problem as "this book isn't in the database":
|
||
|
||
1. A cover that loaded fine still rendered as nothing, so a **successful** lookup
|
||
looked like a failed one. That alone could account for the book you did find
|
||
appearing broken.
|
||
2. Open Library's cover URL was synthesized for every book whether or not art
|
||
existed, and a missing cover comes back as a **200 with a 43-byte 1×1
|
||
transparent GIF** — a "successful" load that paints nothing.
|
||
3. Because that synthesized URL was never blank, the merge rule could never fall
|
||
through to Google Books' thumbnail. The documented fallback was dead code for
|
||
covers.
|
||
|
||
What is left is a real coverage question, and there the measurements point at one
|
||
thing above all others: **the Google Books fallback is probably not answering at
|
||
all.** Every keyless request from this machine returned HTTP 429, and the app
|
||
turns any non-200 into `null`, which the UI presents as "No match found — enter
|
||
the details by hand." A rate-limited lookup and a book that genuinely exists
|
||
nowhere are, right now, indistinguishable to both you and me.
|
||
|
||
My recommendation is to fix the diagnosis before buying more data. Details in
|
||
[Recommendation](#recommendation).
|
||
|
||
---
|
||
|
||
## What I measured
|
||
|
||
**Sample.** 60 ISBNs drawn from the Harvard Library catalog — deliberately a
|
||
third party, so the sample doesn't presuppose the answer by coming from one of
|
||
the two sources under test. Ten publishers, weighted toward the small Catholic
|
||
and homeschool presses that a MODG family's shelf actually carries (Ignatius,
|
||
TAN, Sophia Institute, Bethlehem Books, Baronius) alongside mainstream trade
|
||
(Penguin, Random House, Scholastic, Crossway, Loyola).
|
||
|
||
**Method.** Direct HTTP against each API, one ISBN at a time, 1.2 s apart. A
|
||
source "hits" only if it returns a usable title.
|
||
|
||
### Results
|
||
|
||
| Source | Hit | Miss | Error | Hit rate |
|
||
|---|---|---|---|---|
|
||
| Open Library Books API (what the app calls today) | 53 | 4 | 3 | **88%** |
|
||
| Google Books, keyless (the app's fallback) | 0 | 0 | **60 × HTTP 429** | **0%** |
|
||
| Harvard LibraryCloud | 56 | 4 | 0 | 93% * |
|
||
| Open Library cover art exists for the ISBN | 47 | 13 | 0 | 78% |
|
||
|
||
\* Harvard is where the sample came from, so its number is inflated by
|
||
construction. It's here to show the API works and answers by ISBN-13, not as a
|
||
fair comparison.
|
||
|
||
Two further observations from the same runs:
|
||
|
||
- **Concurrency is punished.** The same 60 ISBNs run six-at-a-time dropped Open
|
||
Library from 88% to 70%, entirely through transport errors. The app makes one
|
||
request per scan, so this doesn't bite in normal use — but it does mean
|
||
"Open Library missed" in a log is not proof the book is absent. By the end of
|
||
this research my own IP was refused outright for a while.
|
||
- **17% of successful Open Library lookups have no cover art at all** (9 of 53).
|
||
Even with everything working, roughly one book in six will legitimately show
|
||
the placeholder. That is a data fact, not a bug, and it's worth knowing before
|
||
you read a placeholder as a failure.
|
||
|
||
### What this does not tell us
|
||
|
||
Worth saying plainly, because it bounds how much weight the numbers carry:
|
||
|
||
- n = 60, and the sample comes from a research library. It under-represents
|
||
recent mass-market paperbacks, reprints and print-on-demand editions — which
|
||
is exactly where Open Library is thinnest. Real shelf coverage is probably
|
||
*below* 88%.
|
||
- Every request came from a datacenter IP. The Google Books 429 may partly be
|
||
this host sharing a quota pool with other tenants; **your phone, on a
|
||
residential or mobile IP, may well get answers.** That's precisely why the
|
||
app needs to be able to tell us which it got.
|
||
- I don't know which three ISBNs you scanned. If you still have the books to
|
||
hand, those three numbers are worth more than another 60 sampled ones.
|
||
|
||
---
|
||
|
||
## The options
|
||
|
||
### A. Give Google Books an API key
|
||
Free, 1,000 requests/day, no billing account required. Turns the fallback from
|
||
"silently 429" into a working source. Roughly a dozen lines: a key in
|
||
`local.properties` → `BuildConfig` → `&key=` on the query.
|
||
|
||
The key ships inside the APK and can be extracted, so restrict it to the Books
|
||
API in the Google Cloud console. At our volume, someone stealing it costs us
|
||
nothing but the quota.
|
||
|
||
**Effort: hours. Cost: free. Likely the single biggest win.**
|
||
|
||
### B. Tell the difference between "not found" and "couldn't ask"
|
||
Both clients collapse every non-200, timeout and parse failure into `null`, and
|
||
`MetadataRepository` collapses that into "no match", and the UI writes "No match
|
||
found." A book that's offline, rate-limited, or hit a 500 is reported to you as
|
||
a book that does not exist.
|
||
|
||
Distinguishing these gets you a retry button instead of a manual-entry form, and
|
||
gets me a real answer next time you say "it missed."
|
||
|
||
**Effort: half a day. Cost: free. Do this regardless of what else we choose.**
|
||
|
||
### C. Retry with backoff
|
||
One retry on 429/5xx, a couple of seconds apart. Standing at a bookshelf, a
|
||
two-second retry is invisible; a manual-entry form is not.
|
||
|
||
**Effort: an hour. Cost: free.**
|
||
|
||
### D. Add a third free source
|
||
Only worth doing after A–C, when we can see what's actually still missing.
|
||
Ranked by what I'd try first:
|
||
|
||
| Source | Key? | Notes |
|
||
|---|---|---|
|
||
| **Open Library `search.json`** | No | Searches the whole OL index rather than the edition table the Books API reads. Cheapest possible fallback — same service, one more request, no new failure modes. |
|
||
| ~~**Harvard LibraryCloud**~~ | No | **Demoted.** Free, no registration, answers by ISBN-13 (`?identifier=<isbn>`), verified working — but it returned `numFound: 0` for *both* of the user's real failing books. It is a research library: strong on scholarly and older material, and it simply does not hold small-press children's historical fiction. The 93% in the table above is inflated by construction (the sample came from Harvard) and is misleading in exactly the direction that matters. Not a fit for this shelf. |
|
||
| **Library of Congress** | No | The SRU endpoint (port 210) is blocked from here; the `loc.gov` JSON API responds. Excellent for US imprints. Needs more probing before I'd commit. |
|
||
| **K10plus SRU** | No | Free German-led union catalogue, large and international. Cataloguing conventions differ enough that merging would need care. |
|
||
|
||
Dead ends, so nobody re-investigates them: **OCLC Classify** (retired 2021),
|
||
**Goodreads API** (retired 2020), **Amazon Product Advertising API** (requires an
|
||
affiliate account with qualifying sales), **WorldCat Search** (requires OCLC
|
||
membership — institutional pricing).
|
||
|
||
### E. Pay for ISBNdb
|
||
~$15–50/month depending on tier. Genuinely better coverage than anything free,
|
||
including cover art, and a single clean API. It is also a subscription for a
|
||
two-person home library, and I'd want proof that A–D leave a real gap before
|
||
recommending it.
|
||
|
||
**Effort: hours. Cost: $180–600/year.**
|
||
|
||
---
|
||
|
||
## What actually failed
|
||
|
||
Both failing ISBNs resolve cleanly against the source the app already uses:
|
||
|
||
| Check | 9781883937386 | 9781883937676 |
|
||
|---|---|---|
|
||
| Open Library Books API | *Hittite warrior*, Joanne S. Williamson | *Shadow hawk*, Andre Norton |
|
||
| Cover art | yes | yes |
|
||
| App's own parser (unit test) | parses | parses |
|
||
| ISBN-13 checksum | valid | valid |
|
||
| Harvard | not held | not held |
|
||
|
||
So the request either never went out, or went out and failed in a way the app
|
||
reported as "not found". Candidates, in the order I'd chase them:
|
||
|
||
1. **The barcode never decoded into a valid ISBN-13.** `ScanCodeFilter` drops
|
||
anything that fails the checksum, silently and with no UI feedback at all —
|
||
no sheet, no message, nothing. A book whose barcode carries a price add-on,
|
||
or is worn, or is a UPC-A rather than a Bookland EAN, looks to the user
|
||
exactly like a camera that isn't working. This is my leading theory, and it
|
||
fits "wasn't able to scan" better than "wasn't found".
|
||
2. **The HTTP request failed.** Both clients turn every non-200, timeout and
|
||
parse error into `null`, which reaches the user as "No match found". The
|
||
metadata `OkHttpClient` is constructed with no call timeout, so a stalled
|
||
connection hangs on default socket timeouts.
|
||
3. **The lookup ran and the sheet was dismissed before it landed.** While
|
||
Loading, the sheet passes an empty `onDismissRequest`, so it can't be
|
||
swiped away — but `onScanned` early-returns whenever a sheet is already
|
||
showing, so a stuck sheet blocks every subsequent scan.
|
||
|
||
Nothing here is a data-source problem. Note the sting in (1) and (2): both
|
||
failure modes are invisible or actively misleading, which is why three scans
|
||
produced no usable diagnosis.
|
||
|
||
## Recommendation
|
||
|
||
**Revised after the two real ISBNs came in.** Adding sources is now the *wrong*
|
||
next move: the books that failed are already in the source we query.
|
||
|
||
Do **B** first and on its own — make the app say what happened. A scan that
|
||
decodes nothing should say so on the camera screen; a lookup that fails should
|
||
offer retry, not a manual-entry form captioned "No match found". Add a call
|
||
timeout while in there.
|
||
|
||
Then rescan those two books. The app will tell us which of the three candidates
|
||
above it is, and that determines everything after it. **A** (the free Google
|
||
Books key) and **C** (retry/backoff) are still worth doing — cheap, and the 429
|
||
result is real — but they are no longer the leading theory.
|
||
|
||
**D** and **E** are on hold. Harvard specifically is off the list for this
|
||
shelf. Paying ISBNdb for coverage we demonstrably already have would be the
|
||
wrong order.
|
||
|
||
One thing worth deciding separately: 17% of books legitimately have no cover art
|
||
anywhere. The placeholder now looks deliberate rather than broken, but if you
|
||
want covers on everything, that's a different feature — photograph the book,
|
||
store it as the cover — and not a metadata-source problem at all.
|