The two books that failed are in Open Library after all
The user supplied the failing ISBNs: 9781883937386 (The Hittite Warrior) and 9781883937676 (Shadow Hawk), both Bethlehem Books. Both resolve against the source the app already queries — title, author, publisher, page count, cover art — and both real responses parse correctly through the app's own OpenLibraryClient. Captured as fixtures with a regression test, because these are the specific books that motivated the research. That kills the coverage hypothesis for these two and demotes Harvard, which holds neither: its 93% in the sample table is inflated by construction (the sample was drawn from Harvard) and misleading in exactly the direction that matters — it is a research library and does not carry small-press children's historical fiction. The failure is upstream of the metadata sources. Documented the three candidates; the leading one is that the barcode never decoded into a valid ISBN-13, which ScanCodeFilter drops silently with no UI feedback whatsoever. Revised the recommendation accordingly: make the app say what happened before adding any source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDcPottghJXEvfYKqFM7zf
This commit is contained in:
Sprite
@@ -61,6 +61,38 @@ class OpenLibraryClientTest {
|
||||
assertEquals("https://covers.openlibrary.org/b/id/675832-M.jpg", result.coverUrl)
|
||||
}
|
||||
|
||||
/**
|
||||
* Two real books off the user's shelf that the app failed to identify on its
|
||||
* first on-device run, captured verbatim from the live API on 2026-09-09. Both
|
||||
* are small-press (Bethlehem Books) children's historical fiction — the exact
|
||||
* profile we assumed Open Library would be thin on. It isn't: both carry title,
|
||||
* author, publisher and cover art, and neither exposes an `isbn_13` identifier,
|
||||
* which is why the lookup ISBN has to survive as the fallback. Whatever went
|
||||
* wrong on the phone, it was NOT this parser and NOT Open Library's coverage.
|
||||
*/
|
||||
@Test
|
||||
fun `parses the real responses for the two books the app failed to identify`() {
|
||||
val hittite = checkNotNull(
|
||||
client.parseResponse(fixture("openlibrary_hittite_warrior.json"), "9781883937386"),
|
||||
)
|
||||
assertEquals("Hittite warrior", hittite.title)
|
||||
assertEquals(listOf("Joanne S. Williamson"), hittite.authors)
|
||||
assertEquals("Bethlehem Books", hittite.publisher)
|
||||
assertEquals(237, hittite.pageCount)
|
||||
assertEquals("1883937388", hittite.isbn10)
|
||||
// No isbn_13 in the record — the ISBN we looked up has to carry through.
|
||||
assertEquals("9781883937386", hittite.isbn13)
|
||||
assertEquals("https://covers.openlibrary.org/b/id/930599-L.jpg", hittite.coverUrl)
|
||||
|
||||
val shadowHawk = checkNotNull(
|
||||
client.parseResponse(fixture("openlibrary_shadow_hawk.json"), "9781883937676"),
|
||||
)
|
||||
assertEquals("Shadow hawk", shadowHawk.title)
|
||||
assertEquals(listOf("Andre Norton"), shadowHawk.authors)
|
||||
assertEquals("9781883937676", shadowHawk.isbn13)
|
||||
checkNotNull(shadowHawk.coverUrl)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `returns null when the isbn key is absent from the response`() {
|
||||
val body = fixture("openlibrary_not_found.json")
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"ISBN:9781883937386": {"url": "http://openlibrary.org/books/OL395004M/Hittite_warrior", "key": "/books/OL395004M", "title": "Hittite warrior", "authors": [{"url": "http://openlibrary.org/authors/OL244960A/Joanne_S._Williamson", "name": "Joanne S. Williamson"}], "number_of_pages": 237, "pagination": "xvii, 237 p. ;", "by_statement": "Joanne Williamson.", "identifiers": {"goodreads": ["613497"], "librarything": ["33463"], "isbn_10": ["1883937388"], "lccn": ["98073485"], "openlibrary": ["OL395004M"]}, "classifications": {"lc_classifications": ["PZ7.W672 Hi 1999"], "dewey_decimal_class": ["[Fic]"]}, "publishers": [{"name": "Bethlehem Books"}], "publish_places": [{"name": "Warsaw, ND"}], "publish_date": "1999", "subjects": [{"name": "Bible", "url": "https://openlibrary.org/subjects/bible"}, {"name": "Canaanites", "url": "https://openlibrary.org/subjects/canaanites"}, {"name": "Fiction", "url": "https://openlibrary.org/subjects/fiction"}, {"name": "History", "url": "https://openlibrary.org/subjects/history"}, {"name": "History of Biblical events", "url": "https://openlibrary.org/subjects/history_of_biblical_events"}, {"name": "Jews", "url": "https://openlibrary.org/subjects/jews"}, {"name": "Juvenile fiction", "url": "https://openlibrary.org/subjects/juvenile_fiction"}, {"name": "Hittites", "url": "https://openlibrary.org/subjects/hittites"}, {"name": "Middle east, history", "url": "https://openlibrary.org/subjects/middle_east,_history"}, {"name": "Juvenile Fiction", "url": "https://openlibrary.org/subjects/juvenile_fiction"}], "subject_places": [{"name": "Palestine", "url": "https://openlibrary.org/subjects/place:palestine"}], "subject_people": [{"name": "Barak (Biblical figure)", "url": "https://openlibrary.org/subjects/person:barak_(biblical_figure)"}, {"name": "Deborah (Biblical judge)", "url": "https://openlibrary.org/subjects/person:deborah_(biblical_judge)"}, {"name": "Sisera (Biblical figure)", "url": "https://openlibrary.org/subjects/person:sisera_(biblical_figure)"}], "subject_times": [{"name": "To 70 A.D.", "url": "https://openlibrary.org/subjects/time:to_70_a.d."}], "notes": "\"Ages 10-up\"--P. 4 of cover.", "ebooks": [{"preview_url": "https://archive.org/details/hittitewarrior00will", "availability": "borrow", "formats": {}, "borrow_url": "https://openlibrary.org/books/OL395004M/Hittite_warrior/borrow", "checkedout": true}], "cover": {"small": "https://covers.openlibrary.org/b/id/930599-S.jpg", "medium": "https://covers.openlibrary.org/b/id/930599-M.jpg", "large": "https://covers.openlibrary.org/b/id/930599-L.jpg"}}}
|
||||
@@ -0,0 +1 @@
|
||||
{"ISBN:9781883937676": {"url": "http://openlibrary.org/books/OL3958151M/Shadow_hawk", "key": "/books/OL3958151M", "title": "Shadow hawk", "authors": [{"url": "http://openlibrary.org/authors/OL27951A/Andre_Norton", "name": "Andre Norton"}], "number_of_pages": 246, "pagination": "246 p. ;", "by_statement": "Andre Norton.", "identifiers": {"librarything": ["34114"], "goodreads": ["83108"], "isbn_10": ["1883937671"], "lccn": ["2001092576"], "oclc": ["52047589"], "openlibrary": ["OL3958151M"]}, "classifications": {"lc_classifications": ["PZ7.N82 Sh 2001"], "dewey_decimal_class": ["[Fic]"]}, "publishers": [{"name": "Bethlehem Books"}, {"name": "Ignatius Press"}], "publish_places": [{"name": "Bathgate, N.D"}, {"name": "San Francisco"}], "publish_date": "2001", "subjects": [{"name": "History", "url": "https://openlibrary.org/subjects/history"}, {"name": "Juvenile fiction", "url": "https://openlibrary.org/subjects/juvenile_fiction"}, {"name": "Egypt in fiction", "url": "https://openlibrary.org/subjects/egypt_in_fiction"}, {"name": "Fiction", "url": "https://openlibrary.org/subjects/fiction"}, {"name": "Children's stories", "url": "https://openlibrary.org/subjects/children's_stories"}, {"name": "Egypt -- History -- To 332 B.C. -- Juvenile fiction.", "url": "https://openlibrary.org/subjects/egypt_--_history_--_to_332_b.c._--_juvenile_fiction."}, {"name": "Fiction, science fiction, general", "url": "https://openlibrary.org/subjects/fiction,_science_fiction,_general"}], "subject_places": [{"name": "Egypt", "url": "https://openlibrary.org/subjects/place:egypt"}], "subject_times": [{"name": "To 332 B.C.", "url": "https://openlibrary.org/subjects/time:to_332_b.c."}], "ebooks": [{"preview_url": "https://archive.org/details/shadowhawk0000nort", "availability": "borrow", "formats": {}, "borrow_url": "https://openlibrary.org/books/OL3958151M/Shadow_hawk/borrow", "checkedout": false}], "cover": {"small": "https://covers.openlibrary.org/b/id/930617-S.jpg", "medium": "https://covers.openlibrary.org/b/id/930617-M.jpg", "large": "https://covers.openlibrary.org/b/id/930617-L.jpg"}}}
|
||||
@@ -7,6 +7,17 @@ scanned, only 1 has been discovered properly."
|
||||
primary, Google Books fallback) is unchanged. This is the evidence for deciding
|
||||
whether to change it.
|
||||
|
||||
> **Update, same day — the coverage hypothesis is dead.** The user supplied the
|
||||
> two ISBNs that failed: 9781883937386 (*The Hittite Warrior*) and 9781883937676
|
||||
> (*Shadow Hawk*), both Bethlehem Books. **Both are fully present in Open
|
||||
> Library** — title, author, publisher, page count and cover art — and the app's
|
||||
> own parser handles both real responses correctly (regression test:
|
||||
> `OpenLibraryClientTest.parses the real responses for the two books the app
|
||||
> failed to identify`). Whatever went wrong on the phone was upstream of the
|
||||
> metadata sources. Everything below still holds as background, but do not act
|
||||
> on "add a third source" until we know why a book Open Library *has* did not
|
||||
> reach the lookup. See [What actually failed](#what-actually-failed).
|
||||
|
||||
---
|
||||
|
||||
## Short version
|
||||
@@ -128,7 +139,7 @@ Ranked by what I'd try first:
|
||||
| Source | Key? | Notes |
|
||||
|---|---|---|
|
||||
| **Open Library `search.json`** | No | Searches the whole OL index rather than the edition table the Books API reads. Cheapest possible fallback — same service, one more request, no new failure modes. |
|
||||
| **Harvard LibraryCloud** | No | Free, no registration, answers by ISBN-13, verified working. Strong on older, scholarly and small-press books — the shape of gap we'd expect. Returns MODS; no cover art, and no ISBN-13 for pre-EAN books unless we convert. |
|
||||
| ~~**Harvard LibraryCloud**~~ | No | **Demoted.** Free, no registration, answers by ISBN-13 (`?identifier=<isbn>`), verified working — but it returned `numFound: 0` for *both* of the user's real failing books. It is a research library: strong on scholarly and older material, and it simply does not hold small-press children's historical fiction. The 93% in the table above is inflated by construction (the sample came from Harvard) and is misleading in exactly the direction that matters. Not a fit for this shelf. |
|
||||
| **Library of Congress** | No | The SRU endpoint (port 210) is blocked from here; the `loc.gov` JSON API responds. Excellent for US imprints. Needs more probing before I'd commit. |
|
||||
| **K10plus SRU** | No | Free German-led union catalogue, large and international. Cataloguing conventions differ enough that merging would need care. |
|
||||
|
||||
@@ -147,18 +158,58 @@ recommending it.
|
||||
|
||||
---
|
||||
|
||||
## What actually failed
|
||||
|
||||
Both failing ISBNs resolve cleanly against the source the app already uses:
|
||||
|
||||
| Check | 9781883937386 | 9781883937676 |
|
||||
|---|---|---|
|
||||
| Open Library Books API | *Hittite warrior*, Joanne S. Williamson | *Shadow hawk*, Andre Norton |
|
||||
| Cover art | yes | yes |
|
||||
| App's own parser (unit test) | parses | parses |
|
||||
| ISBN-13 checksum | valid | valid |
|
||||
| Harvard | not held | not held |
|
||||
|
||||
So the request either never went out, or went out and failed in a way the app
|
||||
reported as "not found". Candidates, in the order I'd chase them:
|
||||
|
||||
1. **The barcode never decoded into a valid ISBN-13.** `ScanCodeFilter` drops
|
||||
anything that fails the checksum, silently and with no UI feedback at all —
|
||||
no sheet, no message, nothing. A book whose barcode carries a price add-on,
|
||||
or is worn, or is a UPC-A rather than a Bookland EAN, looks to the user
|
||||
exactly like a camera that isn't working. This is my leading theory, and it
|
||||
fits "wasn't able to scan" better than "wasn't found".
|
||||
2. **The HTTP request failed.** Both clients turn every non-200, timeout and
|
||||
parse error into `null`, which reaches the user as "No match found". The
|
||||
metadata `OkHttpClient` is constructed with no call timeout, so a stalled
|
||||
connection hangs on default socket timeouts.
|
||||
3. **The lookup ran and the sheet was dismissed before it landed.** While
|
||||
Loading, the sheet passes an empty `onDismissRequest`, so it can't be
|
||||
swiped away — but `onScanned` early-returns whenever a sheet is already
|
||||
showing, so a stuck sheet blocks every subsequent scan.
|
||||
|
||||
Nothing here is a data-source problem. Note the sting in (1) and (2): both
|
||||
failure modes are invisible or actively misleading, which is why three scans
|
||||
produced no usable diagnosis.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Do **A + B + C** together — they're cheap, they're independent of any decision
|
||||
about new sources, and between them they cover the most likely explanation for
|
||||
1-in-3. Then rescan the same books and let the app tell us what it actually got.
|
||||
**Revised after the two real ISBNs came in.** Adding sources is now the *wrong*
|
||||
next move: the books that failed are already in the source we query.
|
||||
|
||||
If a real gap survives that, add **Open Library `search.json`** (D) as an
|
||||
in-family fallback before reaching for a new organisation's API, and Harvard
|
||||
after that.
|
||||
Do **B** first and on its own — make the app say what happened. A scan that
|
||||
decodes nothing should say so on the camera screen; a lookup that fails should
|
||||
offer retry, not a manual-entry form captioned "No match found". Add a call
|
||||
timeout while in there.
|
||||
|
||||
I'd hold off on **E** entirely until we have numbers from your own shelf. Paying
|
||||
for coverage we might already have would be the wrong order.
|
||||
Then rescan those two books. The app will tell us which of the three candidates
|
||||
above it is, and that determines everything after it. **A** (the free Google
|
||||
Books key) and **C** (retry/backoff) are still worth doing — cheap, and the 429
|
||||
result is real — but they are no longer the leading theory.
|
||||
|
||||
**D** and **E** are on hold. Harvard specifically is off the list for this
|
||||
shelf. Paying ISBNdb for coverage we demonstrably already have would be the
|
||||
wrong order.
|
||||
|
||||
One thing worth deciding separately: 17% of books legitimately have no cover art
|
||||
anywhere. The placeholder now looks deliberate rather than broken, but if you
|
||||
|
||||
Reference in New Issue
Block a user