Macroscope Collaboratory Librarian: Unified Catalog and Document Ingestion Pipeline Specification

@manual{hamilton2026macroscope, author = {Hamilton, Michael P., Ph.D.}, title = {Macroscope Collaboratory Librarian: Unified Catalog and Document Ingestion Pipeline Specification}, institution = {Canemah Nature Laboratory}, year = {2026}, number = {CNL-SP-2026-050}, month = {april}, url = {https://canemah.org/archive/document.php?id=CNL-SP-2026-050}, abstract = {The Macroscope Collaboratory Librarian is the fourth panel of the I3 shell (Instruments, Investigations, Intelligence, Librarian) and serves as the research surface for a unified personal holdings catalog and scientific document archive. This specification describes an extensible catalog model that supports four item types — documents, books, videos, and TV shows — unified under a single `catalog\_items` supertable with per-type extension tables and a shared asset layer, plus the worker-agent pipeline that ingests PDFs from a raw filesystem into the catalog. The initial corpus comprises approximately 5,300 legacy PDFs (post-deduplication from 8,515 files), 1,311 books cataloged in BookBuddy, and 652 movies and TV shows cataloged in MovieBuddy, totaling approximately 7,264 items at launch. The document ingestion pipeline comprises three stages: a filesystem preprocessing layer that flattens, deduplicates on SHA-256, and rename-normalizes incoming PDFs; a two-worker ingestion stage in which `read\_document\_basic` converts PDF to Markdown using PyMuPDF and pdfplumber while `classify\_document` runs a three-tier escalation ladder to generate bibliographic metadata; and a persistence stage that writes normalized records to the catalog with content-hash deduplication and archive-layout file placement. The classification ladder escalates from deterministic regex (Tier 1, zero cost) through Ollama gemma4:31b-cloud (Tier 2, local-daemon cloud inference) to Anthropic Claude Haiku with optional Sonnet escalation (Tier 3, paid API) before flagging unresolved documents for manual review (Tier 4). A dedicated `ingestion\_batch\_jobs` table tracks per-document state to provide crash resumability and a progress surface for the Librarian UI. Book and video records are imported from BookBuddy and MovieBuddy CSV exports via a parallel CSV import pipeline. The catalog model is designed to accommodate future item types (audio recordings, photographs, field specimens) without schema changes to the core supertable. This specification defines the catalog data model, worker contracts, the v2.0 metadata schema, database schema, escalation policy, cost envelope, CSV import pipeline, and the downstream interface to the Librarian web panel. --- **Keywords:** unified catalog; document ingestion pipeline; extensible schema; pdf text extraction; llm classification; metadata generation; three-tier escalation; ollama gemma; anthropic claude; content-hash deduplication; worker agent architecture; macroscope collaboratory; librarian ui; csv import; bookbuddy; moviebuddy} }