JDFJDF/ docs

CLI — @uurtech/jdf-cli

Command-line tool for the two unattended workflows the desktop reader can't cover: PDF → JDF for RAG / CI ingestion of legacy documents, and JSON → JDF for wrapping the output of LLMs and code generators into a validated, renderable document. Markdown import is a convenience.

Why this CLI matters: the same algorithm that powers the desktop reader's PDF import also runs from the command line, in CI, in a Lambda, in your RAG pipeline. Output is bit-identical between desktop and CLI for the same input. The CLI is the bridge between the world's existing PDFs and an AI workflow that wants structured data.

Install

# run on demand (no install)
npx @uurtech/jdf-cli validate doc.jdf
npx @uurtech/jdf-cli convert paper.pdf

# or install globally
npm install -g @uurtech/jdf-cli
jdf --help

Commands

jdf validate <file.jdf|file.jdfx>

Runs Ajv against spec/jdf-schema.json and reports path-level errors with warnings vs hard failures separated. For .jdfx bundles the validator opens the zip, validates document.json, and prints the asset count from the manifest.

$ jdf validate spec/examples/hello-world.jdf

✓ Valid: hello-world.jdf
  Format:    1.0.0
  Title:     Hello, JDF
  Pages:     2
  Elements:  14

Failures look like:

$ jdf validate broken.jdf

✗ Invalid: broken.jdf
  /pages/0/elements/3/heading — must be one of [true, 1, 2, 3, 4, 5, 6]
  /pages/0/elements/5 — required property "type" missing

Exit code: 0 if valid, 1 if not. Drop into a CI step:

# .github/workflows/ci.yml
- run: npx @uurtech/jdf-cli validate docs/whitepaper.jdf

jdf convert <file.pdf> [-o output.{jdf,jdfx}] [--json] [--password PW] [--drop-invisible-text]

Reads the page the way a person does. Multi-column pages come out in reading order (column by column between full-width blocks), body lines are folded into paragraphs (the PDF's line breaks are kept, so rendering does not move — but a sentence that wraps is one sentence for jdf chunk, search and the LLM), tables are rebuilt from geometry, and subscripts / small caps / footnote marks stay on their line. Ground truth for both: pnpm --filter @jdf/pdf-import verify:order and verify:tables; real-document numbers in the benchmark.

Convert a PDF to a JDF document. Same algorithm the desktop reader uses, packaged as a node CLI — both surfaces import @jdf/pdf-import from packages/jdf-pdf-import/, so output is bit-identical for the same PDF. The reader uses the browser entry point (DOM canvas, real Web Worker); the CLI uses the node entry point (@napi-rs/canvas, in-process).

jdf convert paper.pdf                       # pure-text PDF → paper.jdf
jdf convert contract.pdf                    # with images → contract.jdfx (auto)
jdf convert contract.pdf -o contract.jdf --json  # force pure JSON
                                           # (RAG / CI consumers prefer one text file)

The --json flag forces a single .jdf file even when the PDF embeds images — assets are base64-inlined. Pipelines that consume one text file per document (RAG ingestion, CI gates, agent context windows) should turn this on.

Per text run the importer extracts position (mm), font family, font size (pt), bold / italic, color, opacity, and link annotations (external URLs and internal #page-N destinations). Form XObjects are placed through their /Matrix; vector shapes (rect, line, path with Bezier curves, gradient fills as their average colour) and raster images (XObjects, repeated XObjects, inline images, stencil masks) are preserved at their original transforms. PDF bookmarks become headings with tocEntry, and document info lands in meta (author, created, modified, keywords, language).

Encrypted PDFs open with --password. Scanned PDFs keep their page image and their invisible OCR text layer — emitted with opacity: 0 so jdf chunk / jdf embed and the reader's search still see the words; pass --drop-invisible-text to omit it.

jdf convert <file.json> [-o output.{jdf,jdfx}] [--json]

Wrap raw JSON into a validated JDF document. Three input shapes are accepted:

  1. Full JDF document — has $jdf and pages. The CLI validates and re-emits (no transformation).
  2. Bare element array — wrapped into a single-page A4 document. The CLI fills in $jdf, meta, and a default page.
  3. Partial — { "elements": [...] } or { "pages": [...] } with optional meta, styles, resources. Filled in with sensible defaults (1.0.0, A4, no margins).
jdf convert response.json -o report.jdf     # LLM output → JDF
jdf convert elements.json                   # bare array → 1-page A4 doc
jdf convert partial.json                    # partial → filled-in doc

Validation runs after every JSON import. If the resulting document doesn't match the schema, the CLI exits with code 1 and prints the path-level errors. This is what makes it safe to gate CI on model output — a malformed JSON response from an LLM never silently ships as a broken document.

jdf convert <file.md> [-o output.{jdf,jdfx}]

Convenience: convert Markdown to JDF using marked-style parsing with full GFM (tables, blockquotes, fenced code, links, images, task lists, hr, strikethrough). Image references resolve against the source file's directory and are read from disk during import.

Auto file-shape selection

When you don't pass -o, the importer picks the right output:

Pass --json or specify -o foo.jdf explicitly to keep assets base64-inlined in resources.images. The schema is identical either way.

jdf chunk <file.{jdf,jdfx}> [--strategy section|element|fixed] [--format jsonl|json|inline] [--max-tokens N] [-o out]

Turn a document into retrieval-ready chunks — offline, deterministic, zero network. Because JDF already carries a heading hierarchy (heading / tocLevel) and typed elements, chunking reads structure instead of guessing at it. Same input + same options → byte-identical chunks (and stable content hashes), which is what makes incremental re-embedding possible.

jdf chunk report.jdf                        # → report.chunks.jsonl (section strategy)
jdf chunk report.jdf --strategy element     # one chunk per element
jdf chunk report.jdf --format inline        # write an "index" block back into the .jdf
jdf chunk report.jdf --max-tokens 256 -o out.jsonl

Each chunk line carries everything a vector store needs — no re-derivation:

{"id":"p3e7", "text":"…", "path":["Report","Pricing"],
 "page":3, "types":["text","table"], "tokens":142, "hash":"ab12cd…"}

Tables serialize as Header: value | Header: value rows — the column semantics stay attached to each value, so a retriever matches "Price: $4,800" instead of a bare number floating away from its column. This is the single biggest RAG win over PDF, where table structure is lost during extraction.

--format inline writes the chunks into the document itself under a top-level index block. Renderers ignore it (data-only); a pipeline reads it instead of recomputing. The .jdf stays schema-valid.

jdf embed <file.{jdf,jdfx}> [--provider ollama|openai] [--model NAME] [--strategy …] [--incremental] [--cache prev.embeddings.json] [-o out]

Compute embeddings for the chunks. Builds on jdf chunk — same deterministic boundaries, same hashes — and writes an embeddings sidecar (file.embeddings.json). Unlike convert/chunk, this is the one step that can touch a model, so it's opt-in and never part of conversion.

jdf embed report.jdf                        # local via Ollama (default) — no data leaves the machine
jdf embed report.jdf --incremental          # only re-embed chunks whose hash changed
jdf embed report.jdf --provider openai      # remote API (prompts for endpoint/key, or reads env)

--incremental is the ingestion speed win. It reads the previous sidecar and skips every chunk whose content hash is unchanged. Edit one paragraph in a 500-page document and you re-embed one chunk, not five hundred — because JDF is diffable JSON and the chunker is deterministic. Embeddings are cache: delete them and regenerate any time; they are never the source of truth. The text sent to the model is the heading breadcrumb followed by the chunk text (Report title > 7. Vendor Spend + rows), so each vector knows which document and section it came from; the chunk hash still covers the text only, so renaming a heading does not invalidate every chunk under it. Pass --cache to reuse a sidecar written to a different path.

jdf transcribe <file.{jdf,jdfx}> [--from subs.srt|.vtt|.json] [--provider whisper-cli|openai] [--model M] [--language tag] [--prompt text] [--element id|n] [--chapters file] [-o out]

Attaches time-stamped text to a video element. The transcript is stored inside document.json as transcript.segments[{t0,t1,text}] — it is text, not an asset — so a .jdf with a hosted src stays a single JSON file and a .jdfx keeps only the clip under assets/. Afterwards jdf chunk cuts the transcript into time windows that carry media: { element, t0, t1 }, and both viewers show it as a captions track.

Three ways in, same result:

About --prompt. Whisper's prompt is not an instruction — it is a vocabulary hint. Whisper conditions on it as if it were the preceding transcript, so spellings and proper nouns in it are reproduced: --prompt "JDF, jdfx, Ollama, nomic-embed-text, Uğur Kazdal" keeps product names, acronyms and non-English names intact. Things like “summarise” or “translate to Turkish” do nothing; use --language for the language and post-process the segments for anything else. Same flag works for whisper-cli (--prompt) and the API (prompt field).

Where the clip comes from is resolved automatically: a bundled asset (resources.videos), a data: URL, a local path next to the document, or an http(s) URL (downloaded to a temp file). Chapters (--chapters, JSON [{t,title}] or mm:ss Title lines) become an extra breadcrumb level for the chunks under them.

jdf transcribe talk.jdfx --from talk.srt --chapters chapters.txt --language en
jdf transcribe talk.jdfx --provider openai --language en --prompt "JDF, jdfx, Ollama, nomic-embed-text"
jdf transcribe talk.jdfx --provider whisper-cli --model ~/models/ggml-large-v3.bin
jdf chunk talk.jdfx --window 45     # → chunks like {"id":"clip@90","media":{"element":"clip","t0":90,"t1":135},"text":"[01:30–02:15] …"}

25-second walkthrough: transcribe → chunk with media.t0/t1 → question hits the second → jdf rag over a folder. Source: demos/video-rag.

In a viewer, a retrieval hit maps straight to playback: viewer.seek("clip", 90) in jdf.js scrolls to the video and starts it at 01:30. Search in the desktop reader indexes transcript text too.

jdf describe <file.{jdf,jdfx}> [--ocr tesseract|openai|none] [--caption ollama|openai|none] [--caption-model M] [--ocr-language eng] [--element id|n] [--force] [-o out]

The image-side twin of jdf transcribe. Every image element gets two kinds of text, both stored in document.json (the picture stays an asset): ocr.blocks[{text, bbox, confidence}] — what is written in the image — and caption — what it shows, one or two factual sentences from a vision model (chart type, trend, labels, numbers). jdf chunk indexes alt + caption + OCR, so a chart, a screenshot or a scanned page is retrievable instead of being a silent [image].

Images that already carry text are skipped unless --force. Scanned PDFs: jdf convert scan.pdf --ocr tesseract detects pages with no text layer and OCRs the page image during import; without --ocr it warns that those pages will be skipped by RAG.

jdf describe report.jdfx                                   # OCR + caption, all local
jdf describe report.jdfx --caption none --ocr-language tur  # OCR only, Turkish
jdf convert scan.pdf --ocr tesseract                       # scanned pages become searchable text

jdf rag <dir> [--provider ollama|openai] [--model NAME] [--strategy …] [--window sec] [--transcribe none|whisper-cli|openai] [--language tag] [--prompt text] [--no-embed] [--dry-run] [--out DIR]

One command for a whole folder. It walks <dir> recursively for .jdf / .jdfx files and, for each one: transcribes videos that have no transcript yet (--transcribe), OCRs and captions images that have no text yet (--ocr, --caption) — with the default none providers it only counts them — chunks it, embeds it with jdf embed --incremental semantics (one <file>.embeddings.json sidecar per document, only changed chunks re-embedded), and finally writes <dir>/.jdf-rag/index.jsonl — every chunk with its file, breadcrumb, hash and, for video chunks, media — plus manifest.json describing what was done and what was skipped. Re-run it whenever the folder changes; unchanged chunks cost nothing.

Nothing is skipped silently. The run ends with a media-coverage line — videos with a transcript, images with caption/OCR — and lists every media element that still has no text (also in manifest.json → mediaWithoutText). --strict turns that list into a non-zero exit, so a CI job cannot publish an index with blind spots. jdf chunk prints the same warning per file.

Configuration is a file in the folder, not a training recipe. Put a jdf.rag.json next to your documents with the same keys as the flags (provider, model, strategy, maxTokens, transcriptWindowSec, transcribe, language, prompt, noEmbed, out); flags override it. It is JSON rather than YAML so it needs no extra parser and validates like everything else in JDF. Nothing is “trained” — the folder is chunked and embedded so any retriever can search it.

jdf rag ./knowledge-base --dry-run                 # what would happen, nothing written
jdf rag ./knowledge-base                            # local Ollama embeddings; videos without transcripts are counted
jdf rag ./knowledge-base --transcribe openai --ocr tesseract --caption ollama --strict
# knowledge-base/jdf.rag.json
{ "provider": "ollama", "model": "nomic-embed-text", "strategy": "section", "transcriptWindowSec": 45,
  "transcribe": "whisper-cli", "ocr": "tesseract", "caption": "ollama", "strict": true }

Schema validation in CI

If your project generates .jdf files programmatically, validate them on every PR. Example workflow:

name: Validate JDF docs

on: [pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npx @uurtech/jdf-cli validate docs/*.jdf

RAG ingestion pipeline

Drop a PDF into a corpus loader, get a structured JDF tree out — your retriever chunks by element type instead of token windows that slice tables in half:

# Bash — PDF corpus → validated JDF → chunks → embeddings, end to end
for pdf in corpus/*.pdf; do
  out="${pdf%.pdf}.jdf"
  npx @uurtech/jdf-cli convert "$pdf" -o "$out" --json || exit 1
  npx @uurtech/jdf-cli validate "$out" || exit 1
  npx @uurtech/jdf-cli chunk "$out"                    # → *.chunks.jsonl
  npx @uurtech/jdf-cli embed "$out" --incremental      # local, skips unchanged
done

Re-run the loop after editing a few documents and --incremental re-embeds only the chunks that actually changed — the diffable-JSON payoff for ingestion cost.

LLM output gate

Models emit JSON. The CLI accepts that JSON and either passes it through (if it's already a valid JDF document) or wraps it (if it's a bare element array / partial). Validation runs at the end — non-zero exit fails the build:

# inside a workflow that calls a model
- run: |
    node generate-doc.js > dist/output.json
    npx @uurtech/jdf-cli convert dist/output.json -o dist/output.jdf
    # exit 1 here means the model violated the JDF schema

PDF AcroForm → JDF Forms

The CLI's PDF importer also walks AcroForm widget annotations and emits matching JDF form elements. Filled values in the source PDF survive the conversion — the resulting .jdf renders in jdf.js with the same fields pre-populated, and the user can keep editing in any browser without an Acrobat install.

# Convert an existing fillable PDF to JDF (values + structure preserved)
$ jdf convert w9-form.pdf -o w9-form.jdf
$ jdf validate w9-form.jdf

# Now host it; users fill the rest in their browser:
<jdf src="w9-form.jdf" save-button="Save"></jdf>

Full mapping table (single-line vs multi-line text, checkbox, combo / list dropdowns, signatures) is on the JDF Forms page.