RAG that survives contact with real enterprise documents

Enterprise RAG fails on real documents because retrieval, not generation, is the weak link: scanned PDFs, tables, and versioned policy documents defeat naive chunking. A production pipeline needs document-type-aware ingestion, retrieval evaluated separately from generation, and a recall metric measured against a labelled question set before anyone judges answer quality.

Key takeaways

• In enterprise RAG, the failure is almost always retrieval, not generation.
• Scanned PDFs, tables and versioned policy defeat fixed-size chunking.
• Evaluate retrieval separately from generation; measure recall first.
• Changing the embedding model forces a full re-index: plan it as a migration.

This is a page we can write from having built it. Chokmah's retrieval work is real: production RAG over document corpora, vector search, and the unglamorous ingestion that decides whether any of it works. The engagement scenarios on this site are illustrative because the practice is new, but the technical claims below are not; they come from pipelines we have shipped.

Why does RAG work in the demo and fail on your documents?

Because the demo used clean documents and your organisation does not have any. In a demo the corpus is a handful of well-structured, machine-readable pages, and retrieval looks solved. Point the same pipeline at the real drive (scanned contracts, spreadsheets exported to PDF, three versions of the same policy with no version field), and quality collapses. Nothing changed about the model. Everything changed about the input.

The important reframe is where the failure lives. In enterprise retrieval-augmented generation, generation is rarely the weak stage. The model is perfectly capable of writing a good answer from good passages. It cannot write a good answer from passages your retriever never found, and on a messy corpus the retriever misses constantly. The generation step then does exactly what it is designed to do (produce fluent text), which is why a broken pipeline fails so convincingly: the answers read well and are wrong.

Gartner names poor architecture among the causes behind its forecast that over 40% of agentic AI projects will be cancelled by the end of 2027 (Gartner, 25 June 2025). In RAG systems, "poor architecture" almost always means the ingestion and retrieval layers were treated as plumbing rather than as the product: a specific case of the broader pattern in why 95% of GenAI pilots fail.

Ingestion: the part everyone underestimates

Ingestion is where the quality ceiling is set, and it is the part demos skip entirely. A page that reaches the index badly represented can never be retrieved well, no matter how good the embedding model or the reranker downstream.

Real ingestion is document-type-aware. A scanned PDF needs OCR, and OCR errors propagate silently into every embedding built from that text. A table loses all meaning when flattened into a token stream: the relationship between a row label and its value simply disappears, so tables need structure-preserving extraction. A long policy document has sections and clauses that carry the semantics, and chunking across those boundaries shreds the meaning. Each document type gets its own handling. There is no single ingestion path that serves them all, and pretending there is one is the first architectural mistake.

How do you evaluate retrieval separately from generation?

You build a labelled question set and measure the retriever alone, before the model is allowed to speak. This is the single highest-leverage practice in the whole discipline and the one most often skipped.

Take real questions the system will face. For each, mark which passages in the corpus should be retrieved to answer it. Now run retrieval and compute recall (did the right passages come back), and precision (how much of what came back was relevant) at a fixed cut-off. Only when retrieval clears a threshold do you judge answer quality. Grade answers on top of unmeasured retrieval and you learn nothing, because a bad answer could be a retrieval miss or a generation fault and you cannot tell which. This is the same separation-of-stages discipline as any evaluation harness: measure each stage against fixed cases so a regression has an address.

Chunking strategies and when each breaks

Chunking is where naive pipelines die. The strategy has to follow the document, not a global token count.

| Strategy | Works on | Breaks on |
|---|---|---|
| Fixed-size (N tokens) | Uniform prose, quick prototypes | Tables, clause boundaries, anything structured |
| Structure-aware (by section/heading) | Policies, contracts, manuals | Documents with no reliable structure markers |
| Recursive with overlap | Mixed prose where context spans chunks | Cost and index size grow; duplication in results |
| Table-aware extraction | Spreadsheets, financial documents | Requires per-format parsers, more engineering |

The practical rule: never let a chunk straddle a boundary that carries meaning. A chunk that ends halfway through a table row, or splits a clause from its condition, produces an embedding that represents neither half well and retrieves for neither.

Versioned and conflicting documents

Enterprises do not hold one truth per topic. They hold the current policy, last year's policy, a regional variant, and a draft, often with no field that says which is which. A retriever that treats all four as equally valid will return whichever embeds closest to the question and the model will answer from it confidently.

The fix is metadata, applied at ingestion, and explicit logic in retrieval: effective date, version, jurisdiction, status. Retrieval then applies recency and precedence so the authoritative version wins, and where a genuine conflict remains, the system surfaces both passages and routes the decision to a person rather than silently choosing. This is also a governance boundary: a RAG system that cannot say which version of a policy it answered from is not auditable, and auditability is not optional in a regulated GCC.

An operational trap worth stating plainly

Changing the embedding model is not a configuration change. Because every stored vector was produced by the old model, a new embedding model means every document must be re-embedded and the entire index rebuilt before retrieval is coherent again. Teams discover this the hard way when a provider deprecates a model. Treat an embedding-model change as a migration with a cutover plan, not a flag you flip. The mechanics sit inside embeddings and vector search.

What we build in a workflow sprint

A RAG-heavy workflow sprint leaves behind the pipeline and the thing that proves it works: document-aware ingestion for your actual document types, retrieval instrumented and measured on a labelled set, chunking chosen per document type, versioning and conflict logic, and a harness your team can re-run after every change. The client owns the code. The point is not a demo that answers one question well; it is a system you can trust at volume and characterise at any moment.

What this means for a GCC transformation owner

If a vendor demos RAG on your documents and the answers look great, ask one question: what is the retrieval recall on a labelled set drawn from our real corpus? If they cannot answer, they measured the wrong stage, and the demo is telling you nothing about production. Retrieval quality on your messy documents (not generation quality on clean ones) is the number that predicts whether this survives contact with your drive. Insist on seeing it before anyone talks about the model.

Sources

1. Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, 25 June 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
2. MIT NANDA, The GenAI Divide: State of AI in Business 2025, July 2025. https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf

Related reading: what a RAG pipeline is · how vector search works · for logistics and freight teams: the EDI and RAG domain wedge