Research systems · AI · Case study
Making the path from research question to cited claim inspectable.
RAGdoll is a terminal workspace for planning scholarly searches, curating paper collections, acquiring approved open evidence, and producing cited dossiers without hiding the research trail.
The problem
What had to become visible.
AI research assistants often collapse query planning, retrieval, source selection, evidence acquisition, and synthesis into one opaque response.
That makes it difficult to distinguish metadata from full text, recover the passage behind a claim, or understand which human decisions shaped the result.
Constraints
Rules before implementation.
- The operator must approve the research plan, paper collection, and full-text acquisition separately.
- Paper content is untrusted; remote acquisition cannot bypass paywalls, private addresses, redirects, or source terms.
- Model output must cross Pydantic validation before entering domain logic, and the model never writes files or contacts scholarly sources directly.
- A dossier is a bounded, reproducible result—not a claim of exhaustive literature coverage or automatic truth.
Engineering decisions
The choices that shaped the result.
Keep reasoning behind a narrow provider boundary
OpenAI and Ollama adapters return validated domain contracts. Retrieval, permissions, persistence, and export remain deterministic application responsibilities.
Persist the decisions, not only the answer
SQLite records exact queries, source identifiers, retrieval times, score components, approval fingerprints, evidence provenance, and page-aware chunks indexed with FTS5.
Invalidate derived work when its evidence changes
Changing the staged corpus deletes stale dossiers and grounded answers immediately and requires new evidence consent before synthesis resumes.
Treat extraction as a hostile boundary
Full-text acquisition prefers declared open-access HTTPS sources, rejects unsafe destinations, isolates PDF extraction, and labels abstract fallback rather than presenting it as full text.
What changed
Turning points, not a clean-room success story.
- The terminal interface moved from sequential command output to a resumable workspace where plans, papers, sources, dossier sections, and exact passages remain visible together.
- The acceptance audit found two claims whose citations resolved but did not directly support the wording. Those failures stayed in the published result and sharpened the distinction between citation integrity and entailment verification.
- A minimal live GPT-5.6 request failed for insufficient quota. The OpenAI Responses adapter and model configuration remain implemented, but the submission does not simulate or claim a successful paid request.
Measured result
What the evidence supports.
- 24 discovery candidates and six human-curated papers.
- Five open full-text documents plus one explicitly labeled abstract fallback.
- 307 page-aware evidence chunks and a checkpointed seven-section dossier.
- All 25 citation identifiers resolved; manual inspection found 23 of 25 claims directly supported by their cited passage.
- 98 offline tests, strict typing, security-focused adapters, more than 90% coverage, and a deterministic no-key judge demo.
Limitations
What remains unproven.
- Coverage varies by field, language, venue, date, and scholarly API availability; the retrieved collection is not the literature itself.
- Resolvable citations prove provenance, not that every generated claim is semantically entailed by its passage.
- PDF extraction does not perform OCR, and some open documents will remain unavailable or poorly structured.
- No successful paid GPT-5.6 execution is represented in the recorded demo.
Production perspective
What I would carry forward.
- Complete blinded retrieval-quality evaluation against the documented recall, ranking, coverage, and duplicate-rate gates.
- Strengthen semantic citation-support checks without replacing human review.
- Expand source coverage only through narrow adapters that preserve provenance and source terms.