When 95% Accuracy Isn't Good Enough: The PDF-to-Markdown Reliability Gap

Enterprise teams wire up retrieval pipelines, confident their PDFs will feed clean Markdown to language models. They discover their tables have become unreadable strings of floating numbers. Multi-column layouts have merged into gibberish. Nested structures have broken the parsing logic entirely.

Standard libraries extract text as giant structureless strings with scrambled reading order, replacing images in scanned documents with placeholders like <!-- image --> and discarding the bounding boxes, confidence scores, and layout metadata that make accurate extraction possible. The fundamental promise of PDF-to-Markdown conversion rests on a premise that the format transition itself undermines: that you can transform legacy documents into LLM-friendly formats without losing information.

And yet, HURIDOCS announced in June 2026 that their VGT model achieved 95.4% accuracy in benchmark tests that compared PDF-to-Markdown conversion against competitors including Marker, Docling, and MinerU. Three major open-source parsers now dominate production systems.

Marker from Datalab processes up to 120 pages per second on H100 GPUs. Docling from IBM Research supports CPU-only deployment across PDF, DOCX, PPTX, and LaTeX files. MinerU from Shanghai AI Lab excels at complex CJK layouts where Western-trained models struggle.

When teams measure performance on born-digital documents with clean typography and simple structures, the numbers look compelling. Marker scores 83.5, MinerU scores 83.3, and Docling achieves 97.9% table extraction accuracy on DocLayNet benchmarks. The tools ship fast, the benchmarks validate them, and the conversion pipelines get deployed to production.

What breaks is the assumption that benchmark scores translate to reliable extraction in the wild. OlmOCR-bench audit revealed that benchmark and annotation issues compromised evaluation methodology. The benchmarks saturate with label noise faster than models improve at true conversion quality, which means the 95.4% accuracy claim measures conformity to noisy annotations as much as it measures extraction fidelity. Beyond that, no tool exceeded 86% accuracy on low-quality scanned documents, the kind enterprises inherit from decades of digitization projects that prioritized storage over semantic structure.

Markdown itself is the constraint.

The format was designed for human-readable text with lightweight formatting: headings, lists, emphasis. Not for preserving the spatial relationships and confidence metadata that make PDF extraction tractable for downstream AI processing. When you convert a financial statement to Markdown, you discard the column alignment that distinguishes line items from subtotals, the font weights that indicate hierarchy, and the bounding box coordinates that let extraction models verify whether a figure belongs to revenue or expenses.

The format fundamentally discards critical layout metadata: reading order, table cell boundaries, image placement context. These elements are embedded in complex enterprise documents' structure. What you gain in LLM compatibility, you lose in extraction reliability.

The technical choice to target Markdown as the intermediate representation reflects a broader tension: optimizing for language model consumption versus preserving document semantics. Over 70% of enterprise generative AI initiatives require structured retrieval pipelines by 2026, and those pipelines depend on chunking strategies that work with semantic boundaries. Section breaks, paragraph transitions, table rows. Boundaries that Markdown either obscures or eliminates.

Production systems in 2026 use semantic chunking with 200-1,000 token windows and 10-20% overlap, but determining where those boundaries fall requires the layout information that Markdown conversion throws away. You end up chunking mid-sentence because the parser lost track of reading order, or splitting tables across multiple chunks because the format can't represent cell spans.

In practice, teams patch around Markdown's limitations by building custom post-processors that attempt to reconstruct the lost structure. They run table detection on the original PDF, extract bounding boxes separately, then try to align those coordinates with the Markdown output. Effectively undoing the conversion to recover the metadata they just discarded.

Marker includes an optional --use_llm flag for near-perfect output on messy layouts, which means feeding the document to a vision-language model that doesn't need Markdown in the first place. The workaround exposes the mismatch: if the ultimate solution involves calling a multimodal LLM to handle layout complexity, the intermediate Markdown representation becomes overhead rather than enabler.

The performance ceiling appears in the scanned document results. Across all tools, accuracy drops below 86% when input quality degrades. Blurry scans, handwritten annotations, low-contrast text. These are precisely the documents enterprises need to process at scale.

Historical archives. Legal filings. Medical records. Regulatory submissions.

The documents that matter most for compliance and institutional knowledge are the ones least likely to have clean typography and simple layouts. What works for born-digital academic papers and software documentation breaks down on the messy, real-world corpus that justifies building the extraction pipeline.

The cost structure reinforces the sustainability question. Marker's 120 pages/second throughput requires H100 GPU infrastructure, which runs around $2.50/hour on cloud providers before factoring in data transfer and storage. Processing a 10,000-page document corpus costs $35-50 in compute alone, not counting the engineering time to debug conversion failures, rebuild lost table structures, and validate that extraction didn't silently corrupt the data.

Docling's CPU-only deployment looks cheaper on the surface, but its lower benchmark scores mean more manual correction downstream, which shifts compute costs to labor costs without eliminating the burden.

The hallucination mitigation story depends on accurate retrieval. GPT-3.5 hallucinates in 39.6% of systematic research tasks, GPT-4 in 28.6%, and well-implemented RAG pipelines reduce those rates by 70-90%. But "well-implemented" assumes the retrieval layer surfaces correct information, which in turn assumes the document conversion didn't scramble tables, lose footnotes, or merge unrelated paragraphs.

When conversion errors introduce false relationships between data points, the RAG pipeline amplifies those errors rather than correcting them. Revenue figures associated with the wrong fiscal quarter. Citations attributed to the wrong author. The LLM generates confident, well-formatted answers anchored to corrupted extractions.

What emerges from the benchmark noise is not a solved problem but a shifting constraint. The tools improved rapidly between 2024 and 2026. Marker, Docling, and MinerU all crossed five-figure GitHub star counts, signal that developer adoption is real. But the improvement trajectory flattened as benchmarks saturated with annotation noise and the easy gains from better OCR ran out.

The remaining gap between 83-95% benchmark accuracy and the 100% reliability enterprises need for production deployment isn't a few more model iterations. It's the information loss inherent in targeting Markdown as the output format.

The architectural alternative that's gaining traction in 2026: skip Markdown entirely and feed PDF structure directly to multimodal LLMs that can consume layout, images, and text in a unified representation. Vision-language models like GPT-4V and Gemini Pro already handle document analysis without requiring an intermediate text-only format, and their error modes differ fundamentally from those of conversion-then-extraction pipelines.

They hallucinate on ambiguous visual elements, but they don't lose table structure or scramble reading order. For use cases where extraction accuracy matters more than chunk-level retrieval efficiency, the direct-to-LLM approach sidesteps the Markdown bottleneck at the cost of higher inference latency and compute spend. Legal document review. Financial auditing. Medical record analysis.

The sustainability calculus turns on whether teams can afford to re-process their document corpus every time a better model ships. Conversion-to-Markdown creates a static artifact that you can chunk, embed, and retrieve without touching the original PDF again. Direct LLM inference requires reprocessing on every query, which scales poorly when document sets exceed millions of pages or when query volume hits thousands per day.

The tradeoff between upfront conversion cost and ongoing inference cost depends on workload characteristics. Archival search favors conversion, interactive analysis favors direct inference. Neither approach fully solves the problem of extracting reliable structured data from complex layouts at scale.

Beyond the technical constraints, there's a coordination failure in how benchmarks define success. When benchmark failures stem from annotation issues rather than model errors, the problem is that evaluation methodology hasn't caught up with task complexity. The ground truth for "correct" Markdown representation of a multi-column PDF with nested tables and rotated text doesn't exist in any objective sense.

Different human annotators produce different outputs, and the benchmark scoring penalizes deviations from one arbitrary interpretation. What gets measured is consistency with noisy labels, not fidelity to document semantics.

The tooling ecosystem reflects this lack of consensus. Marker prioritizes speed and supports batch processing. Docling emphasizes broad format coverage and CPU deployment. MinerU optimizes for CJK layouts and complex document structures. Each tool makes different tradeoffs between accuracy, throughput, and computational requirements, and none of them claims to handle all document types reliably.

The fragmentation means teams end up maintaining multiple conversion pipelines for different document classes, which multiplies the engineering overhead and creates new failure modes at the boundaries where one tool hands off to another.

The enterprise adoption pattern reveals where the real friction lives. Teams don't choose a PDF-to-Markdown tool based on benchmark scores alone. They run pilot tests on their actual document corpus, measure error rates on documents that matter to their business, and calculate the fully-loaded cost of conversion including manual correction labor and downstream data quality issues.

What they discover: born-digital product documentation and technical manuals convert reliably with any of the leading tools, but legal contracts with dense multi-column layouts require human review on 30-40% of pages, and scanned forms with handwritten annotations fail conversion entirely and need separate OCR pipelines. The advertised 95% accuracy becomes 70% on the documents that drove the investment in automation, and the workaround involves hiring contractors to fix conversion outputs rather than eliminating manual processing.

The developer experience gap compounds the production reliability problem. When conversion fails, debugging why requires understanding PDF structure at the byte level. Character encoding quirks. Font embedding issues. Transparency group nesting. Knowledge that most application developers don't have and shouldn't need.

The tools abstract away the complexity until something breaks, at which point the abstraction becomes a barrier to diagnosis. Teams ship workarounds that handle specific failure modes on their document set without understanding whether those workarounds will hold when document formats evolve or when new document types enter the corpus. The technical debt accumulates silently until a regulatory change or business expansion introduces documents the pipeline can't handle.

What the 2026 state of PDF-to-Markdown conversion reveals: the format mismatch between how documents encode structure and how LLMs consume text creates a persistent gap that better OCR and smarter parsing can narrow but not close. The teams shipping production systems in 2026 build pipelines that acknowledge this gap explicitly.

They validate conversion outputs against known-good samples. They flag low-confidence extractions for human review. They maintain separate code paths for simple born-digital PDFs versus complex scanned documents. They budget for the reality that no automated solution will achieve 100% accuracy on real-world corpora.

The benchmark numbers keep climbing, but the practical ceiling remains the information loss inherent in flattening spatial documents to linear text.

The innovation here isn't in the conversion models themselves. It's in recognizing that Markdown was never the right target format and building architectures that preserve document structure through the extraction pipeline. Whether that means extending Markdown with layout annotations, adopting intermediate representations like JSON-LD or HTML with semantic markup, or bypassing text formats entirely and feeding structured PDFs to vision-language models, the next generation of document processing systems will treat layout as first-class data rather than metadata to be discarded.

Until then, the 95.4% benchmark accuracy remains a number divorced from extraction reliability in production.