RAG benchmark — JDF vs PDF

The question: if the same document enters a RAG pipeline as a PDF or as a JDF, which one lets the retriever find the right passage, and what does each cost to index and query? Everything else is held equal: same content, same embedding models, same lexical retriever, same questions, same hit rule. Everything on this page is generated from bench/results/*.json; nothing is typed in by hand.

git clone https://github.com/uurtech/jdf && cd jdf/bench
pip install -r requirements.txt
python rag_bench.py                      # accuracy: BM25 + a local embedding model
python rag_bench.py --embedder ollama:nomic-embed-text,st:BAAI/bge-base-en-v1.5
python cost_bench.py                     # RAG cost: 1,000 PDF files vs 1,000 JDF files
python rag_bench.py --verify             # re-run and compare with the published numbers

In 42 seconds

Every number on screen is read from docs/bench.json. Source: demos/rag-benchmark.

Retrieval accuracy

PDF side: pdf → text → LangChain-style fixed chunks (1000/200 and 2000/200 chars) → retriever. JDF side: jdf → jdf chunk --strategy section → retriever — section-aware chunks, tables serialised as Header: value rows, the heading breadcrumb prefixed to the text that gets embedded. Third pipeline, PDF → jdf convert → jdf chunk: the same PDFs converted with the shipped importer (tables rebuilt as real table elements, headings detected from font size), then chunked like native JDF — the path for anyone who only has PDFs today. Same retrievers for all. A retrieved chunk is a hit when it comes from the right document and contains both the answer and its row/subject key.

BM25 — lexical, offline, deterministic

Embeddings — nomic-embed-text

Embeddings — BAAI/bge-small-en-v1.5

Embeddings — sentence-transformers/all-MiniLM-L6-v2

Embeddings — BAAI/bge-base-en-v1.5

R@1k tokens = the answer was inside the first 1,000 tokens of retrieved context (chunk-size neutral: a 2,000-character PDF chunk “hits” more often at top-1 simply because it is a quarter of the document, and then costs 3× the tokens). Hit = right document and the chunk contains both the answer and its row/subject key. 24 documents, 120 pages, 192 questions (120 table cells, 48 prose, 24 list). Embeddings run locally (nomic-embed-text via Ollama 0.32.6; BAAI/bge-small-en-v1.5 via sentence-transformers 6.0.1; sentence-transformers/all-MiniLM-L6-v2 via sentence-transformers 6.0.1; BAAI/bge-base-en-v1.5 via sentence-transformers 6.0.1). JDF chunk hashes verified; editing one paragraph re-embeds 1 of 192 chunks. Apple M5, 2026-09-13.

RAG cost — 1,000 PDF files vs 1,000 JDF files

Same pipeline on both sides, only the input format differs: chunks → embeddings → vector store → top-5 context → LLM. The 24-document corpus is cycled to 1,000 files per format; embedding tokens are counted from the chunks each pipeline produces; embedding time is measured throughput; dollar figures are tokens × the published prices in bench/prices.json. The benchmark never calls a paid API.

1,000 files per format = the 24-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on Apple M5 (2026-09-13) applied to the totals. Prices from bench/prices.json (as of 2026-09-12); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. Scaling: figures are measured at 1,000 documents and 1,000,000 queries; anything at other volumes (10,000 documents, 10M queries) is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (jdf embed --incremental); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.

Method

Corpus

24 “Quarterly Operations Report” documents, 4–5 pages each, generated deterministically from a fixed seed. Each has eight sections: two prose sections, five tables (regional performance, headcount, pricing, service levels, vendor spend) and a risk list, plus a repeating page header and footer. The JDF files are the originals; the PDFs were printed from them by the real jdf.js renderer in Google Chrome, so the PDF text layer is clean and complete — the PDF side is not handicapped by scans or exotic fonts. Every document, every question and every retrieved rank is in bench/.

Questions

192, generated with the corpus and stored with ground truth: 120 table-cell lookups (“What was Acme Logistics's churn rate in EMEA for Q3 2025?”), 48 prose facts (on-call lead, capex budget), 24 list items (risk owner). Every question names the document the way a user would.

Why a generated corpus

A retrieval benchmark needs to know, for every question, which passage answers it. Public PDF collections don't ship that, and hand-labelling hundreds of answers is where benchmark bias usually creeps in. Generating the documents makes the ground truth exact and the whole thing auditable, and --verify proves nobody edited a document or a question after the fact by checking SHA-256 hashes against the manifest. The obvious limitation: synthetic reports are cleaner than real ones — real PDFs are usually worse for the PDF side (scans, multi-column layouts, broken tables), so treat these numbers as a floor for the gap. If you run the pipelines on your own corpus, open an issue with the numbers.

Reading the accuracy numbers honestly

Top-1 depends on chunk size: a 2,000-character PDF chunk is a quarter of the document, so it “hits” more often at rank 1 — and then costs three times the tokens per query. With the two smallest embedding models the PDF 2,000-char configuration edges out JDF at top-1; on the chunk-size-neutral metric (answer within the first 1,000 tokens of context) JDF leads with every retriever tested. Both views are in the tables above.