Macroscope Collaboratory Librarian: Document Ingestion Pipeline Specification

Document ID: CNL-SP-2026-XXX
Version: 0.1
Date: April 11, 2026
Author: Michael P. Hamilton, Ph.D.
Status: Draft for review
Visibility: Public

AI Assistance Disclosure: This specification was developed with assistance from Anthropic Claude (Sonnet 4.6) in a Cowork session on April 11, 2026. The AI contributed to architecture formalization, worker contract documentation, schema enumeration, and manuscript drafting. The author takes full responsibility for the content, accuracy, and conclusions.

Abstract

The Macroscope Collaboratory Librarian is the fourth panel of the I3 shell (Instruments, Investigations, Intelligence, Librarian) and serves as the research surface for a unified scientific document archive. This specification describes the worker-agent pipeline that ingests PDFs from a raw filesystem into the macroscope.documents table and the unified archive layout, transforming approximately 5,300 legacy PDFs (post-deduplication from an initial corpus of 8,515 files) into a searchable, metadata-rich, human-AI collaborative resource. The pipeline comprises three stages: a filesystem preprocessing layer that flattens, deduplicates on SHA-256, and rename-normalizes incoming PDFs into a stable Input directory; a two-worker ingestion stage in which read_document_basic converts PDF to Markdown and extracts figures using PyMuPDF and pdfplumber while classify_document runs a three-tier escalation ladder to generate bibliographic metadata; and a persistence stage that writes normalized records to the documents table with content-hash deduplication and archive-layout file placement. The classification ladder escalates from deterministic regex (Tier 1, zero cost) through Ollama gemma4:31b-cloud (Tier 2, local-daemon cloud inference) to Anthropic Claude Haiku with optional Sonnet escalation (Tier 3, paid API) before flagging unresolved documents for manual review (Tier 4). A dedicated ingestion_batch_jobs table tracks per-document state to provide crash resumability and a progress surface for the Librarian UI. This specification defines worker contracts, the v2.0 metadata schema, database extensions, escalation policy, cost envelope, and the downstream interface to the Librarian web panel.

Keywords: document ingestion pipeline; pdf text extraction; llm classification; metadata generation; three-tier escalation; ollama gemma; anthropic claude; content-hash deduplication; worker agent architecture; macroscope collaboratory; librarian ui; batch reprocessing

1. Introduction

1.1 Context

The Macroscope Collaboratory is a four-panel research application hosted inside the Claude Desktop app via a dedicated Model Context Protocol (MCP) server. The application is organized around the I3 shell: Instruments (live sensor dashboards across EARTH, LIFE, HOME, SELF domains), Investigations (structured human-AI research workflows), Intelligence (the STRATA micro-agent layer and narrative synthesis), and Librarian (the document archive and research-grade reading surface).

The Librarian panel is the newest of the four and subsumes a document corpus that has accumulated over approximately two decades across research projects at the James San Jacinto Mountains Reserve, Blue Oak Ranch Reserve, the Center for Embedded Networked Sensing (CENS), the Canemah Nature Laboratory, and personal research activity. The corpus includes peer-reviewed journal articles, conference papers, arXiv preprints, government reports, agency technical documents, field manuals, books, and internal working notes. As of April 2026 the archive contains 5,298 unique PDFs totaling 32.27 GB after dedup, having been reduced from an initial flat-directory count of 8,515 files through content-hash deduplication. The archive will continue to grow as new documents are collected.

1.2 Objectives

This specification defines the end-to-end document ingestion pipeline that transforms raw PDFs into structured, searchable records in the macroscope.documents table and the unified Archive/documents/ filesystem layout. Specifically, the pipeline must:

1. Normalize filenames to ASCII-safe identifiers while preserving original names as recoverable data.
2. Deduplicate content byte-for-byte via SHA-256 before any expensive processing.
3. Convert each PDF to a Markdown representation with figure extraction.
4. Generate bibliographic metadata (title, authors, DOI, journal, publication date, abstract, keywords, document type) using the cheapest reliable method available per document.
5. Persist all artifacts (source PDF, Markdown, metadata JSON, figures, visuals sidecar) under a stable archive path.
6. Track batch-reprocessing state in a database table that supports resumability after crashes and provides a real-time progress view to the Librarian UI.
7. Reuse the same worker contracts for single-document streaming ingestion as the archive grows post-migration.

1.3 Scope

This specification covers the preprocessing tools (Section 3), the worker-agent ingestion stage (Sections 4–6), the metadata schema and database extensions (Sections 7–8), the archive filesystem layout (Section 9), the batch reingest workflow (Section 10), and the downstream interface contract to the Librarian UI (Section 11). It does not cover the Librarian UI implementation itself, the STRATA Intelligence integration, or the Claude Desktop MCP server glue — each of which is documented in separate notes referenced in Section 13.

1.4 Non-Goals

The following are explicitly out of scope for this specification:

• OCR of scanned or image-only PDFs. The current pipeline assumes extractable text. A dedicated tier for OCR-required documents (historically using olmOCR-2, retired on April 11, 2026) is deferred to a future extension.
• Multi-language support. The pipeline targets English-language scientific documents.
• Full-text search backend selection. The documents table is the authoritative store; the search index choice (MySQL FULLTEXT versus an external engine) is a downstream decision.
• Access control beyond the existing Collaboratory three-tier auth (anonymous, user, admin).

2. System Architecture

2.1 High-Level Pipeline

 Legacy archive              Preprocessing                  Worker ingestion             Persistence              Librarian UI
 (nested PDFs in        →    (flatten, dedup,        →      (read_document_basic,  →     (documents table,  →    (search,
  Archive/Documents/         rename) produces               classify_document                Archive/                filters,
  PDF/)                      stable Input/)                 with three-tier ladder)         documents/)             viewer)

Each stage produces durable artifacts that survive crashes in the next stage: preprocessing writes JSONL manifests, worker ingestion writes per-document Markdown and metadata files and database rows, and persistence writes to the unified archive layout. The pipeline is reentrant at every boundary.

2.2 Component Inventory

2.3 Worker Framework Context

The two ingestion workers execute inside the Macroscope worker framework, a filesystem job queue at ~/Macroscope/Projects/Live/workers/ managed by launchd. Jobs are dispatched by an MCP tool (dispatch_worker_job) that writes a JSON payload to the worker inbox, and the worker writes its result as {job}.result.json on completion. The queue provides isolation between the MCP layer and long-running ingestion work, allows streaming jobs to be triggered from Claude Desktop conversations without blocking the UI, and gives launchd a uniform supervision target.

3. Preprocessing Pipeline

The preprocessing tools are filesystem-only utilities that prepare a clean, stable Input directory before any worker-based ingestion begins. Each tool is invoked manually, is dry-run by default, accepts --execute to commit changes, and writes a JSONL manifest that the downstream batch reingest tool consumes. All operations are reversible; nothing is deleted.

3.1 flatten_pdfs.py

Walks ~/Macroscope/Archive/Documents/PDF/ recursively, moves every .pdf file (case-insensitive) to ~/Macroscope/Archive/Documents/Input/ as a flat directory, and writes flatten_manifest.jsonl recording every move. Filename collisions are resolved by appending _1, _2, etc. to the stem. Non-PDF files are left in place. The original nested directory is preserved until the operator manually removes it.

Empirical result on the April 11, 2026 run: 8,515 PDFs flattened, 788 name collisions auto-suffixed, 50.97 GB total.

3.2 dedup_pdfs.py

Hashes every PDF in Input/ with SHA-256 (1 MB streaming chunks), groups files by hash, selects a canonical member per group (shortest filename with lexicographic tiebreak), and moves non-canonical aliases to ~/Macroscope/Archive/Documents/Input_duplicates/. Writes dedup_manifest.jsonl in Input/ with one record per unique document containing the canonical path, file size, alias count, and full list of alias filenames. Only catches byte-for-byte duplicates; near-duplicate detection via normalized-text SHA-256 is performed later during worker ingestion (see Section 6.6).

Empirical result: 5,298 unique documents, 3,217 aliases moved, 18.70 GB reclaimed.

3.3 rename_pdfs.py

Assigns sequential ASCII-safe filenames of the form input_NNNNN.pdf (five-digit zero-padded, lowercase) to every PDF in Input/ in alphabetic order and writes rename_manifest.jsonl in Input/ with one record per file containing sequence, new_name, original_name, and size_bytes. Uses ensure_ascii=False to preserve original filenames with Unicode punctuation, emoji, embedded newlines, or leading whitespace. Refuses to run if the manifest already exists or if any existing filename matches the target pattern, enforcing single-shot idempotency.

Empirical result: 5,299 files renamed (includes one file manually restored from Input_duplicates).

3.4 Manifest Contract

The three manifests together form a complete audit trail for every byte in the Input directory. The batch reingest tool reads all three at startup and constructs the mapping from sequential input filename to original filename, alias list, and pre-flatten source path. The original filename becomes the value of documents.original_filename and is preserved verbatim regardless of what characters it contained.

4. Worker Agent Architecture

4.1 Worker Framework Overview

Each worker is a Python module at workers/<worker_name>/worker.py with a standard entry point that takes a single argument: the path to a JSON job file in the worker inbox. The worker reads the job, performs its task, and writes a sibling .result.json file. On success the job file is moved to done/; on failure it is moved to failed/ with a .error.json sibling containing stderr and traceback. The worker framework supports concurrent execution via launchd agent cloning but is run single-threaded for ingestion to avoid rate-limit contention on external APIs.

4.2 Job Contract

Every worker job carries the following minimum envelope:

{
  "job_id": "uuid-v4",
  "job_type": "read_document_basic",
  "payload": { "...": "..." },
  "meta": {
    "caller": "batch_reingest",
    "batch_job_id": 4271,
    "created_at": "2026-04-11T18:00:00Z"
  }
}

The payload structure is worker-specific. The meta.batch_job_id references the row in ingestion_batch_jobs so the orchestrator can update per-document state on job completion.

4.3 Two-Worker Split Rationale

Document extraction (PDF to Markdown, figure extraction, page counting) and document classification (bibliographic metadata generation) are split into separate workers for three reasons:

1. Cost isolation. Extraction is local-CPU work with no external API cost. Classification may involve Anthropic API calls charged per token. Splitting the workers lets the orchestrator retry classification without repaying extraction.
2. Resumability. If classification fails midway through the batch, the orchestrator can re-run classify_document against the already-extracted Markdown without re-reading the source PDF.
3. Model substitution. A future worker variant (e.g., read_document_ocr for image-only PDFs) can be slotted in upstream of classification without changes to the classifier.

The read_document_olmocr worker, which provided heavier LLM-based extraction for scanned and image-only PDFs, is retired in this specification and its source is preserved under Projects/Reference/ for historical consultation.

5. Document Extraction (read_document_basic)

5.1 Extraction Engine

The worker wraps EnhancedPDFExtractor located at ~/Macroscope/Projects/Live/collectors/scripts/enhanced_pdf_extractor.py with the use_llm=False flag. The extractor uses PyMuPDF (fitz) as the primary engine for text and layout, falling back to pdfplumber for table detection when PyMuPDF's layout heuristics indicate tabular content. No LLM is invoked at this stage; all classification is delegated to the downstream classify_document worker.

5.2 Payload Contract

{
  "pdf_path": "/Users/.../Input/input_00042.pdf",
  "output_markdown_path": "/tmp/.../input_00042.md",
  "output_metadata_path": "/tmp/.../input_00042.meta.json",
  "output_figures_dir": "/tmp/.../input_00042.figures/",
  "workspace_dir": "/tmp/.../workspace/"
}

5.3 Output Artifacts

• Markdown file — the extracted text, with headings inferred from font-size hierarchy, embedded image placeholders (![figure_N](figure_N.png)), and table blocks in pipe syntax where detected.
• Metadata JSON — shallow extraction-stage metadata: num_pages, num_figures, num_tables, page_size, producer, pdf_version, has_toc, has_embedded_text, plus any bibliographic hints discovered in the PDF's own /Info dictionary (title, author, subject, keywords).
• Figures directory — one file per extracted figure, named figure_NN.png (or original format where preserved).

5.4 Limitations

The basic extractor does not perform OCR, will return empty Markdown for image-only PDFs, and does not attempt to reconstruct column order in complex multi-column layouts. Documents flagged as "no extractable text" by PyMuPDF are written to ingestion_batch_jobs with state = 'extraction_failed' and classification_status = 'manual_review', bypassing the classifier entirely.

6. Three-Tier Classification Ladder (classify_document)

6.1 Escalation Policy

The classify_document worker attempts bibliographic metadata generation through a sequence of methods ordered by cost and reliability, advancing to the next tier only when the current tier returns insufficient confidence or fails entirely. The policy is deterministic: given the same input Markdown, a document takes the same path through the ladder on every run.

Confidence is expressed as an enumeration (low, medium, high) and the worker advances if the current tier returns low. Each tier has the opportunity to supplement fields already filled by earlier tiers rather than replacing them wholesale (see Section 6.6).

6.2 Tier 1: Deterministic Regex

A pure-Python module inspects the first 4 kilobytes of extracted Markdown and the original filename for deterministic bibliographic patterns: arXiv identifiers (arXiv:YYMM.NNNNN), DOI prefixes (10.NNNN/...), standard journal-article headers (title on line N, authors on line N+1, affiliations indented), USGS and NOAA report numbers, and a small set of common preprint server URLs. When a pattern is found, the module resolves the bibliographic record by local lookup (no network call) and returns a metadata dict with extraction_confidence = 'high' and extraction_method = 'regex'.

Approximately 40–60% of well-formatted post-2005 journal articles resolve at Tier 1 with zero LLM cost. The tier is always attempted first; it is free and fast.

6.3 Tier 2: Ollama gemma4:31b-cloud

When Tier 1 returns low confidence, the worker sends the first 8,000 characters of extracted Markdown (plus the filename and any Tier 1 partial metadata) to a local Ollama daemon running on localhost:11434/api/chat with model = "gemma4:31b-cloud". The model is served by Ollama Turbo as a cloud-hosted inference endpoint; authentication is handled transparently by the local daemon using a device key obtained via ollama signin, so no Bearer token appears in the Python code path.

Request parameters: format = "json", num_ctx = 8192, num_predict = 1024, keep_alive = "30m". Transport is urllib HTTPS POST with a 60-second timeout. The system prompt instructs the model to return a strict JSON object matching the v2.0 metadata schema, with empty strings for fields it cannot determine.

Measured latency (April 11, 2026 validation): approximately 0.7 seconds for trivial inputs, 1–3 seconds for typical journal articles. Cost is zero at the point of call, absorbed by the Ollama Turbo subscription already paid for at the infrastructure level.

The returned metadata is merged with Tier 1 output, the extraction method is recorded as llm with llm_model = "gemma4:31b-cloud", and confidence is set from the model's own self-assessment clamped to the valid enumeration range.

6.4 Tier 3: Anthropic Claude Haiku / Sonnet

When Tier 2 returns low confidence or produces malformed JSON, the worker invokes the Anthropic API with Claude Haiku 4.5 as the default model. The same first 8,000 characters of Markdown are sent with a strict system prompt, and the response is parsed as JSON. If Haiku also returns low confidence or the document exceeds an empirically-determined complexity threshold (length, number of authors, presence of non-English fragments), the worker escalates a second time to Claude Sonnet 4.6 for a final attempt.

API credentials are loaded from an environment variable at worker start; the key never appears in job payloads or result files. Token usage is captured from the response and recorded in ingestion_batch_jobs for cost accounting.

Cost envelope (estimate, April 2026 pricing): Haiku at approximately $0.001 per document, Sonnet at approximately $0.01 per document. For a 5,300-document backfill with an expected 10% Tier 3 rate and 20% of those escalating to Sonnet, total Tier 3 cost is on the order of $10–$15 for the full batch.

6.5 Tier 4: Manual Review Flag

If Tier 3 Sonnet also fails to return adequate metadata, the worker sets classification_status = 'manual_review' on the documents row and writes a human-readable explanation to classification_notes describing why each tier was insufficient. The document is still written to the archive with partial metadata (whatever Tier 1 and Tier 2 produced), and the Librarian UI surfaces it in an admin queue for human curation.

6.6 Metadata Merge Semantics

Because each tier may run even after a partial Tier 1 resolution, the worker uses additive merge semantics: later-tier values supplement but do not overwrite earlier-tier values unless the earlier value was an empty string. The exception is extraction_confidence, which uses a max-rank semantics (low < medium < high) so that any high-confidence field elevates the document's overall confidence even if other fields remain uncertain. The merge logic is implemented in classify_document/worker.py::merge_metadata().

6.7 Normalized-Text Deduplication

After extraction but before tier dispatch, the worker computes sha256_text, a SHA-256 over the normalized Markdown (lowercase, whitespace collapsed, headings stripped). If this hash matches an existing row in documents, the worker short-circuits: no classification is performed, and the new ingestion is recorded as an alias of the existing document in document_aliases. This catches near-duplicates that differ only in PDF rendering but carry identical text content — a common case with preprint-to-published transitions and multiple journal reprints.

7. Metadata Schema v2.0

The v2.0 metadata schema is the canonical format for the per-document metadata.json file stored alongside each PDF in the archive. It is structured as six top-level sections, each populated by a specific pipeline stage.

7.1 source

Populated by the preprocessing and extraction stages. Fields: original_filename, source_type (one of legacy_archive, streaming_upload, api_import), source_path (pre-flatten filesystem path where the file was first discovered), source_url (empty for legacy archive, set for API-imported documents), ingested_by (operator identifier), ingested_at (ISO timestamp), archive_path (target path under Archive/documents/).

7.2 bibliography

Populated by the classification stage. Fields: title (string), authors (ordered array of strings), doi (nullable string), journal (nullable string), publication_date (ISO date, nullable), abstract (string), keywords (array of strings), document_type (enum: journal_article, conference_paper, preprint, book, book_chapter, report, thesis, working_paper, field_note, other).

7.3 classification

Populated by the classification stage. Fields: extraction_method (regex | llm | hybrid | manual), extraction_confidence (low | medium | high), llm_model (nullable string naming the model that produced the final metadata), classification_status (pending | complete | manual_review), classification_notes (free text), tier1_attempted, tier2_attempted, tier3_attempted (booleans), token_usage_input, token_usage_output (integers, for Tier 3 only).

7.4 extraction

Populated by read_document_basic. Fields: num_pages, num_figures, num_tables, num_references, page_size, producer, pdf_version, has_toc, has_embedded_text, extraction_duration_ms.

7.5 visuals

Populated by read_document_basic. Fields: figures_path (relative path to figures directory under archive), visuals_sidecar_path (path to visuals.json sidecar listing each figure with its filename, page number, and placeholder caption). Detailed per-figure metadata (captions, bounding boxes, OCR text) is deferred to a future pipeline extension.

7.6 content_hashes

Populated by read_document_basic and stored alongside the documents row. Fields: sha256_pdf (hash of source bytes, used for exact-duplicate detection), sha256_text (hash of normalized Markdown, used for near-duplicate detection), size_bytes.

8. Database Schema

8.1 documents Table Extensions

The existing macroscope.documents table (27 columns, 278 legacy rows to be dropped) is altered to accommodate v2.0 metadata. The migration widens the extraction_method enum to a VARCHAR(64) (to allow future extraction method names without schema changes), extends the document_type enum to the nine values listed in Section 7.2, and adds the following columns:

• sha256_pdf CHAR(64) NOT NULL — exact-duplicate key
• sha256_text CHAR(64) NULL — near-duplicate key
• arc_id VARCHAR(20) NOT NULL — canonical archive identifier ARC-PDF-NNNNNN, derived from documents.id
• schema_version SMALLINT NOT NULL DEFAULT 2 — metadata schema version
• source_type VARCHAR(32) NOT NULL
• source_path TEXT NULL
• source_url TEXT NULL
• ingested_by VARCHAR(64) NOT NULL
• llm_model VARCHAR(64) NULL
• thumbnail_path TEXT NULL

Three of the four "April 10" columns discussed in prior handoffs (metadata_path, llm_summary, extraction_timestamp) already exist on the legacy table and are retained.

Indices: unique on sha256_pdf, non-unique on sha256_text, non-unique on document_type, non-unique on classification_status.

8.2 document_aliases Table

id                 BIGINT AUTO_INCREMENT PRIMARY KEY
document_id        BIGINT NOT NULL   -- FK to documents.id
alias_filename     TEXT NOT NULL     -- original filename of the duplicate
alias_source_path  TEXT NULL         -- pre-flatten path if known
discovered_at      DATETIME NOT NULL
discovery_method   VARCHAR(32) NOT NULL  -- 'sha256_pdf' or 'sha256_text'

One row per alias preserved during dedup or near-dup detection. The Librarian UI surfaces the alias list on each document's detail page.

8.3 ingestion_batch_jobs Table

id                 BIGINT AUTO_INCREMENT PRIMARY KEY
input_filename     VARCHAR(32) NOT NULL   -- input_NNNNN.pdf
original_filename  TEXT NOT NULL
sha256_pdf         CHAR(64) NOT NULL
size_bytes         BIGINT NOT NULL
state              ENUM('pending','extracting','extracted',
                       'classifying','classified','written',
                       'failed','skipped_duplicate') NOT NULL
document_id        BIGINT NULL            -- set when state='written'
attempt_count      INT NOT NULL DEFAULT 0
last_error         TEXT NULL
tier_reached       TINYINT NULL           -- 1..4
extract_started_at DATETIME NULL
extract_ended_at   DATETIME NULL
classify_started_at DATETIME NULL
classify_ended_at  DATETIME NULL
written_at         DATETIME NULL
created_at         DATETIME NOT NULL
updated_at         DATETIME NOT NULL

One row per PDF in the batch. Provides crash resumability (any row in extracting or classifying on startup is re-dispatched), real-time progress for the Librarian UI, and cost accounting.

9. Archive Filesystem Layout

Per-document artifacts are stored under a unified archive layout keyed by arc_id:

~/Macroscope/Archive/documents/
    <year>/                              # derived from publication_date, or 'undated/'
        ARC-PDF-NNNNNN/
            source.pdf                   # original bytes, post-rename
            extracted.md                 # output of read_document_basic
            metadata.json                # v2.0 metadata (six sections)
            figures/
                figure_01.png
                figure_02.png
                ...
            visuals.json                 # sidecar with per-figure metadata
            thumbnail.png                # first-page thumbnail (optional)

The documents.archive_path column stores the path to the ARC-PDF-NNNNNN/ directory relative to ~/Macroscope/Archive/documents/. The Librarian UI joins on archive_path to locate artifacts for display and download.

Sibling buckets (audio/, video/, images/, icons/) under ~/Macroscope/Archive/ are reserved for future expansion of the archive to non-PDF asset types.

10. Batch Reingest Workflow

The batch reingest tool batch_reingest.py orchestrates the full corpus ingestion. On startup it reads the three preprocessing manifests, populates ingestion_batch_jobs with one row per input_NNNNN.pdf file in state pending, and then enters a dispatch loop.

For each pending row, the orchestrator:

1. Transitions state to extracting, dispatches a read_document_basic job, waits for the result.
2. Checks sha256_text against existing documents rows. If match, transitions state to skipped_duplicate, records the alias in document_aliases, and continues.
3. Transitions state to classifying, dispatches a classify_document job, waits for the result.
4. Assembles the full v2.0 metadata record by merging extraction output and classification output.
5. Writes the documents row, assigns arc_id = 'ARC-PDF-' + zero_pad(documents.id, 6).
6. Moves the source PDF, extracted.md, metadata.json, figures, and visuals sidecar into the archive layout.
7. Moves the source PDF in Input/ to ~/Macroscope/Archive/Documents/Input_ingested/.
8. Transitions state to written, sets document_id, and continues to the next pending row.

On crash or interrupt, the orchestrator can be restarted cleanly: any row in extracting or classifying is reset to pending on the next run, and all intermediate artifacts in the workspace directory are discarded. Rows in written are untouched. This gives full resumability with no risk of double-inserting a document.

The orchestrator logs every tier decision, every cost-bearing API call, and every state transition to ~/Macroscope/Projects/Live/collectors/batch_reingest/logs/batch_reingest.log.

11. Librarian UI Integration

The Librarian UI is a web panel rendered under ~/Macroscope/Projects/Workbench/Collaboratory/librarian/ and served at http://localhost/Projects/Workbench/Collaboratory/librarian/. It reads from the documents, document_aliases, and ingestion_batch_jobs tables and from the Archive/documents/ filesystem layout. This specification defines only the read contract; the UI implementation is documented separately.

11.1 Browse and Search

The default view lists all documents rows with classification_status = 'complete', ordered by publication_date DESC, ingested_at DESC. Filters include document type, publication year range, keyword, journal, and author substring. Full-text search is performed against the extracted.md content via an index to be specified.

11.2 Document Detail View

Each document has a detail page showing bibliographic metadata, the rendered Markdown content, an inline thumbnail, a figures gallery, the alias list (if any), and the download link to the original PDF. The detail page also shows the classification provenance: which tier resolved the document, what model was used, and the confidence rating.

11.3 Pipeline Progress Dashboard

An admin-only view reads ingestion_batch_jobs and shows the real-time state of the batch reprocessing run: total count, count per state, tier distribution, cumulative Tier 3 token cost, average latency per tier, and a live-updating progress bar. Rows in failed state are clickable and surface the error detail for debugging.

11.4 Manual Review Queue

A second admin-only view lists all documents with classification_status = 'manual_review'. Each row is editable: the curator can fill in missing bibliographic fields directly, promote the document to classification_status = 'complete', or delete the row if the document is judged unsuitable for the archive.

12. Open Questions

The following decisions are deferred to a subsequent specification revision and are not blocking the initial batch reprocessing run:

1. Visuals storage depth. Whether to create a dedicated document_visuals table for per-figure metadata or defer to a visuals.json sidecar.
2. Thumbnail generation tool. Candidates include macOS sips, ImageMagick, and the Python pdf2image library. Selection depends on available dependencies on Data and Galatea.
3. Pre-existing PDFs/ versus preprocessed/ directory handling. Legacy archive trees may contain already-preprocessed Markdown and metadata files from earlier ingestion attempts; the pipeline must decide whether to trust or reprocess them.
4. Non-PDF content in the legacy archive. EPUB, HTML, and plain-text documents currently sit alongside PDFs in the archive source tree. A separate reader worker family will be needed.
5. OCR tier reintroduction. Whether and when to add an OCR-based extraction path (formerly olmOCR-2) for image-only PDFs.

These are tracked as open items and will be resolved in v0.2 of this specification.

13. References

[1] Hamilton, M. P. (2026). "Macroscope Collaboratory: MCP-Hosted Research Application." Canemah Nature Laboratory Technical Note CNL-TN-2026-047.

[2] Hamilton, M. P. (2026). "Canemah Nature Laboratory Technical Note Style Guide." CNL-SG-2025-002 v1.1.

[3] Artifex Software (2024). "PyMuPDF: Python bindings for MuPDF." https://pymupdf.readthedocs.io (accessed April 11, 2026).

[4] Singer-Vine, J. et al. (2024). "pdfplumber: Plumb a PDF for detailed information." https://github.com/jsvine/pdfplumber (accessed April 11, 2026).

[5] Ollama (2025). "Ollama: Run Large Language Models Locally." https://ollama.ai (accessed April 11, 2026).

[6] Gemma Team (2025). "Gemma 3 Technical Report." Google DeepMind. arXiv:2503.19786.

[7] Anthropic (2026). "Claude API Reference." https://docs.claude.com (accessed April 11, 2026).

[8] Hamilton, M. P. (2026). "STRATA 2.0 Intelligence Architecture." Canemah Nature Laboratory Technical Note CNL-TN-2026-043.

Document History

End of Specification