From 1ba22a9f361cc95dc52861e7a3688ec9cbe161bf Mon Sep 17 00:00:00 2001 From: Sprite Date: Wed, 9 Sep 2026 10:36:33 +0000 Subject: [PATCH] The two books that failed are in Open Library after all MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The user supplied the failing ISBNs: 9781883937386 (The Hittite Warrior) and 9781883937676 (Shadow Hawk), both Bethlehem Books. Both resolve against the source the app already queries — title, author, publisher, page count, cover art — and both real responses parse correctly through the app's own OpenLibraryClient. Captured as fixtures with a regression test, because these are the specific books that motivated the research. That kills the coverage hypothesis for these two and demotes Harvard, which holds neither: its 93% in the sample table is inflated by construction (the sample was drawn from Harvard) and misleading in exactly the direction that matters — it is a research library and does not carry small-press children's historical fiction. The failure is upstream of the metadata sources. Documented the three candidates; the leading one is that the barcode never decoded into a valid ISBN-13, which ScanCodeFilter drops silently with no UI feedback whatsoever. Revised the recommendation accordingly: make the app say what happened before adding any source. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CDcPottghJXEvfYKqFM7zf --- .../data/metadata/OpenLibraryClientTest.kt | 32 +++++++++ .../fixtures/openlibrary_hittite_warrior.json | 1 + .../fixtures/openlibrary_shadow_hawk.json | 1 + docs/METADATA-SOURCES.md | 69 ++++++++++++++++--- 4 files changed, 94 insertions(+), 9 deletions(-) create mode 100644 app/app/src/test/resources/fixtures/openlibrary_hittite_warrior.json create mode 100644 app/app/src/test/resources/fixtures/openlibrary_shadow_hawk.json diff --git a/app/app/src/test/java/org/modg/bookshelf/data/metadata/OpenLibraryClientTest.kt b/app/app/src/test/java/org/modg/bookshelf/data/metadata/OpenLibraryClientTest.kt index cd9f70f..8251309 100644 --- a/app/app/src/test/java/org/modg/bookshelf/data/metadata/OpenLibraryClientTest.kt +++ b/app/app/src/test/java/org/modg/bookshelf/data/metadata/OpenLibraryClientTest.kt @@ -61,6 +61,38 @@ class OpenLibraryClientTest { assertEquals("https://covers.openlibrary.org/b/id/675832-M.jpg", result.coverUrl) } + /** + * Two real books off the user's shelf that the app failed to identify on its + * first on-device run, captured verbatim from the live API on 2026-09-09. Both + * are small-press (Bethlehem Books) children's historical fiction — the exact + * profile we assumed Open Library would be thin on. It isn't: both carry title, + * author, publisher and cover art, and neither exposes an `isbn_13` identifier, + * which is why the lookup ISBN has to survive as the fallback. Whatever went + * wrong on the phone, it was NOT this parser and NOT Open Library's coverage. + */ + @Test + fun `parses the real responses for the two books the app failed to identify`() { + val hittite = checkNotNull( + client.parseResponse(fixture("openlibrary_hittite_warrior.json"), "9781883937386"), + ) + assertEquals("Hittite warrior", hittite.title) + assertEquals(listOf("Joanne S. Williamson"), hittite.authors) + assertEquals("Bethlehem Books", hittite.publisher) + assertEquals(237, hittite.pageCount) + assertEquals("1883937388", hittite.isbn10) + // No isbn_13 in the record — the ISBN we looked up has to carry through. + assertEquals("9781883937386", hittite.isbn13) + assertEquals("https://covers.openlibrary.org/b/id/930599-L.jpg", hittite.coverUrl) + + val shadowHawk = checkNotNull( + client.parseResponse(fixture("openlibrary_shadow_hawk.json"), "9781883937676"), + ) + assertEquals("Shadow hawk", shadowHawk.title) + assertEquals(listOf("Andre Norton"), shadowHawk.authors) + assertEquals("9781883937676", shadowHawk.isbn13) + checkNotNull(shadowHawk.coverUrl) + } + @Test fun `returns null when the isbn key is absent from the response`() { val body = fixture("openlibrary_not_found.json") diff --git a/app/app/src/test/resources/fixtures/openlibrary_hittite_warrior.json b/app/app/src/test/resources/fixtures/openlibrary_hittite_warrior.json new file mode 100644 index 0000000..f9af5e1 --- /dev/null +++ b/app/app/src/test/resources/fixtures/openlibrary_hittite_warrior.json @@ -0,0 +1 @@ +{"ISBN:9781883937386": {"url": "http://openlibrary.org/books/OL395004M/Hittite_warrior", "key": "/books/OL395004M", "title": "Hittite warrior", "authors": [{"url": "http://openlibrary.org/authors/OL244960A/Joanne_S._Williamson", "name": "Joanne S. Williamson"}], "number_of_pages": 237, "pagination": "xvii, 237 p. ;", "by_statement": "Joanne Williamson.", "identifiers": {"goodreads": ["613497"], "librarything": ["33463"], "isbn_10": ["1883937388"], "lccn": ["98073485"], "openlibrary": ["OL395004M"]}, "classifications": {"lc_classifications": ["PZ7.W672 Hi 1999"], "dewey_decimal_class": ["[Fic]"]}, "publishers": [{"name": "Bethlehem Books"}], "publish_places": [{"name": "Warsaw, ND"}], "publish_date": "1999", "subjects": [{"name": "Bible", "url": "https://openlibrary.org/subjects/bible"}, {"name": "Canaanites", "url": "https://openlibrary.org/subjects/canaanites"}, {"name": "Fiction", "url": "https://openlibrary.org/subjects/fiction"}, {"name": "History", "url": "https://openlibrary.org/subjects/history"}, {"name": "History of Biblical events", "url": "https://openlibrary.org/subjects/history_of_biblical_events"}, {"name": "Jews", "url": "https://openlibrary.org/subjects/jews"}, {"name": "Juvenile fiction", "url": "https://openlibrary.org/subjects/juvenile_fiction"}, {"name": "Hittites", "url": "https://openlibrary.org/subjects/hittites"}, {"name": "Middle east, history", "url": "https://openlibrary.org/subjects/middle_east,_history"}, {"name": "Juvenile Fiction", "url": "https://openlibrary.org/subjects/juvenile_fiction"}], "subject_places": [{"name": "Palestine", "url": "https://openlibrary.org/subjects/place:palestine"}], "subject_people": [{"name": "Barak (Biblical figure)", "url": "https://openlibrary.org/subjects/person:barak_(biblical_figure)"}, {"name": "Deborah (Biblical judge)", "url": "https://openlibrary.org/subjects/person:deborah_(biblical_judge)"}, {"name": "Sisera (Biblical figure)", "url": "https://openlibrary.org/subjects/person:sisera_(biblical_figure)"}], "subject_times": [{"name": "To 70 A.D.", "url": "https://openlibrary.org/subjects/time:to_70_a.d."}], "notes": "\"Ages 10-up\"--P. 4 of cover.", "ebooks": [{"preview_url": "https://archive.org/details/hittitewarrior00will", "availability": "borrow", "formats": {}, "borrow_url": "https://openlibrary.org/books/OL395004M/Hittite_warrior/borrow", "checkedout": true}], "cover": {"small": "https://covers.openlibrary.org/b/id/930599-S.jpg", "medium": "https://covers.openlibrary.org/b/id/930599-M.jpg", "large": "https://covers.openlibrary.org/b/id/930599-L.jpg"}}} \ No newline at end of file diff --git a/app/app/src/test/resources/fixtures/openlibrary_shadow_hawk.json b/app/app/src/test/resources/fixtures/openlibrary_shadow_hawk.json new file mode 100644 index 0000000..422eeda --- /dev/null +++ b/app/app/src/test/resources/fixtures/openlibrary_shadow_hawk.json @@ -0,0 +1 @@ +{"ISBN:9781883937676": {"url": "http://openlibrary.org/books/OL3958151M/Shadow_hawk", "key": "/books/OL3958151M", "title": "Shadow hawk", "authors": [{"url": "http://openlibrary.org/authors/OL27951A/Andre_Norton", "name": "Andre Norton"}], "number_of_pages": 246, "pagination": "246 p. ;", "by_statement": "Andre Norton.", "identifiers": {"librarything": ["34114"], "goodreads": ["83108"], "isbn_10": ["1883937671"], "lccn": ["2001092576"], "oclc": ["52047589"], "openlibrary": ["OL3958151M"]}, "classifications": {"lc_classifications": ["PZ7.N82 Sh 2001"], "dewey_decimal_class": ["[Fic]"]}, "publishers": [{"name": "Bethlehem Books"}, {"name": "Ignatius Press"}], "publish_places": [{"name": "Bathgate, N.D"}, {"name": "San Francisco"}], "publish_date": "2001", "subjects": [{"name": "History", "url": "https://openlibrary.org/subjects/history"}, {"name": "Juvenile fiction", "url": "https://openlibrary.org/subjects/juvenile_fiction"}, {"name": "Egypt in fiction", "url": "https://openlibrary.org/subjects/egypt_in_fiction"}, {"name": "Fiction", "url": "https://openlibrary.org/subjects/fiction"}, {"name": "Children's stories", "url": "https://openlibrary.org/subjects/children's_stories"}, {"name": "Egypt -- History -- To 332 B.C. -- Juvenile fiction.", "url": "https://openlibrary.org/subjects/egypt_--_history_--_to_332_b.c._--_juvenile_fiction."}, {"name": "Fiction, science fiction, general", "url": "https://openlibrary.org/subjects/fiction,_science_fiction,_general"}], "subject_places": [{"name": "Egypt", "url": "https://openlibrary.org/subjects/place:egypt"}], "subject_times": [{"name": "To 332 B.C.", "url": "https://openlibrary.org/subjects/time:to_332_b.c."}], "ebooks": [{"preview_url": "https://archive.org/details/shadowhawk0000nort", "availability": "borrow", "formats": {}, "borrow_url": "https://openlibrary.org/books/OL3958151M/Shadow_hawk/borrow", "checkedout": false}], "cover": {"small": "https://covers.openlibrary.org/b/id/930617-S.jpg", "medium": "https://covers.openlibrary.org/b/id/930617-M.jpg", "large": "https://covers.openlibrary.org/b/id/930617-L.jpg"}}} \ No newline at end of file diff --git a/docs/METADATA-SOURCES.md b/docs/METADATA-SOURCES.md index 243a150..5042f85 100644 --- a/docs/METADATA-SOURCES.md +++ b/docs/METADATA-SOURCES.md @@ -7,6 +7,17 @@ scanned, only 1 has been discovered properly." primary, Google Books fallback) is unchanged. This is the evidence for deciding whether to change it. +> **Update, same day — the coverage hypothesis is dead.** The user supplied the +> two ISBNs that failed: 9781883937386 (*The Hittite Warrior*) and 9781883937676 +> (*Shadow Hawk*), both Bethlehem Books. **Both are fully present in Open +> Library** — title, author, publisher, page count and cover art — and the app's +> own parser handles both real responses correctly (regression test: +> `OpenLibraryClientTest.parses the real responses for the two books the app +> failed to identify`). Whatever went wrong on the phone was upstream of the +> metadata sources. Everything below still holds as background, but do not act +> on "add a third source" until we know why a book Open Library *has* did not +> reach the lookup. See [What actually failed](#what-actually-failed). + --- ## Short version @@ -128,7 +139,7 @@ Ranked by what I'd try first: | Source | Key? | Notes | |---|---|---| | **Open Library `search.json`** | No | Searches the whole OL index rather than the edition table the Books API reads. Cheapest possible fallback — same service, one more request, no new failure modes. | -| **Harvard LibraryCloud** | No | Free, no registration, answers by ISBN-13, verified working. Strong on older, scholarly and small-press books — the shape of gap we'd expect. Returns MODS; no cover art, and no ISBN-13 for pre-EAN books unless we convert. | +| ~~**Harvard LibraryCloud**~~ | No | **Demoted.** Free, no registration, answers by ISBN-13 (`?identifier=`), verified working — but it returned `numFound: 0` for *both* of the user's real failing books. It is a research library: strong on scholarly and older material, and it simply does not hold small-press children's historical fiction. The 93% in the table above is inflated by construction (the sample came from Harvard) and is misleading in exactly the direction that matters. Not a fit for this shelf. | | **Library of Congress** | No | The SRU endpoint (port 210) is blocked from here; the `loc.gov` JSON API responds. Excellent for US imprints. Needs more probing before I'd commit. | | **K10plus SRU** | No | Free German-led union catalogue, large and international. Cataloguing conventions differ enough that merging would need care. | @@ -147,18 +158,58 @@ recommending it. --- +## What actually failed + +Both failing ISBNs resolve cleanly against the source the app already uses: + +| Check | 9781883937386 | 9781883937676 | +|---|---|---| +| Open Library Books API | *Hittite warrior*, Joanne S. Williamson | *Shadow hawk*, Andre Norton | +| Cover art | yes | yes | +| App's own parser (unit test) | parses | parses | +| ISBN-13 checksum | valid | valid | +| Harvard | not held | not held | + +So the request either never went out, or went out and failed in a way the app +reported as "not found". Candidates, in the order I'd chase them: + +1. **The barcode never decoded into a valid ISBN-13.** `ScanCodeFilter` drops + anything that fails the checksum, silently and with no UI feedback at all — + no sheet, no message, nothing. A book whose barcode carries a price add-on, + or is worn, or is a UPC-A rather than a Bookland EAN, looks to the user + exactly like a camera that isn't working. This is my leading theory, and it + fits "wasn't able to scan" better than "wasn't found". +2. **The HTTP request failed.** Both clients turn every non-200, timeout and + parse error into `null`, which reaches the user as "No match found". The + metadata `OkHttpClient` is constructed with no call timeout, so a stalled + connection hangs on default socket timeouts. +3. **The lookup ran and the sheet was dismissed before it landed.** While + Loading, the sheet passes an empty `onDismissRequest`, so it can't be + swiped away — but `onScanned` early-returns whenever a sheet is already + showing, so a stuck sheet blocks every subsequent scan. + +Nothing here is a data-source problem. Note the sting in (1) and (2): both +failure modes are invisible or actively misleading, which is why three scans +produced no usable diagnosis. + ## Recommendation -Do **A + B + C** together — they're cheap, they're independent of any decision -about new sources, and between them they cover the most likely explanation for -1-in-3. Then rescan the same books and let the app tell us what it actually got. +**Revised after the two real ISBNs came in.** Adding sources is now the *wrong* +next move: the books that failed are already in the source we query. -If a real gap survives that, add **Open Library `search.json`** (D) as an -in-family fallback before reaching for a new organisation's API, and Harvard -after that. +Do **B** first and on its own — make the app say what happened. A scan that +decodes nothing should say so on the camera screen; a lookup that fails should +offer retry, not a manual-entry form captioned "No match found". Add a call +timeout while in there. -I'd hold off on **E** entirely until we have numbers from your own shelf. Paying -for coverage we might already have would be the wrong order. +Then rescan those two books. The app will tell us which of the three candidates +above it is, and that determines everything after it. **A** (the free Google +Books key) and **C** (retry/backoff) are still worth doing — cheap, and the 429 +result is real — but they are no longer the leading theory. + +**D** and **E** are on hold. Harvard specifically is off the list for this +shelf. Paying ISBNdb for coverage we demonstrably already have would be the +wrong order. One thing worth deciding separately: 17% of books legitimately have no cover art anywhere. The placeholder now looks deliberate rather than broken, but if you