- 91%answer accuracy
- -70%time spent searching
- 120k+documents indexed
- 1.8sP95 time to first token
Background
A mid-sized manufacturer with many product lines and fast iteration. Pre-sales, after-sales and production each maintained their own material: manuals on a shared drive, troubleshooting experience in the ticketing system, process specifications in the OA, and a great deal of knowledge living only in a few veteran employees’ heads.
New hires took three months to become productive; veterans were interrupted seven or eight times a day with repeat questions. The client’s initial ask was simple: “can we get a search box we can actually ask?”
What we did first: build the eval set
Before writing any code we spent two weeks on something that doesn’t look like development — collecting 220 real questions with reference answers, drawn from ticket history, internal chat and interviews with the veterans.
That eval set became the most valuable artefact of the project. Every change to chunking, retrieval or prompting was scored against it, showing immediately whether things got better or worse. Without it, all optimization is guesswork.
The hard parts
One: chaotic document formats
Among the 120,000 documents were scanned PDFs, Word files, spreadsheet process tables, exported CAD annotations and a lot of screenshots.
- Scans go through OCR, with low-confidence passages flagged separately
- Tables get their own handling: row and column structure is preserved rather than flattened, otherwise “what is the rated voltage of model A” is guaranteed to be wrong
- Drawing annotations are stored linked to their images, so answers can surface the picture
Two: chunking sets the ceiling
Fixed-length chunking gave only 63% accuracy on the eval set. Switching to structure-aware chunking (chapter → section → paragraph, retaining the heading hierarchy as context) and prefixing each chunk with document name and section path pushed accuracy to 81%.
It was the highest-return change in the project, at essentially no added cost.
Three: vector search alone misses things
Pure vector retrieval is poor at exact terms like model number “XG-2400.” We moved to hybrid retrieval: vector plus BM25 recall, merged and then reranked, taking the top K into the context. Accuracy went from 81% to 88%.
The remaining three points came from query rewriting — turning colloquial questions into retrieval-friendly phrasing before searching.
Four: permissions cannot leak
Manufacturing process files carry classification levels. We filter permissions at retrieval time rather than after generation — chunks the user may not see never enter the context at all, which removes the leak by construction.
Five: making it trustworthy
Every answer lists its source chunks, clickable through to the original with the passage highlighted. Three cases are distinguished explicitly: a grounded answer, nothing relevant in the knowledge base, and conflicting sources found — the third lists both documents and leaves the judgement to a person.
Better to say “I couldn’t find it” than to invent something plausible. That rule went into the prompt and into the acceptance criteria.
Cost control
Pre-launch projections showed that routing every question to the largest model would exceed the client’s budget threefold. Three measures fixed it:
- Semantic caching for frequent questions — about 34% hit rate
- Tiered routing: a small model judges complexity and answers simple questions directly; only complex ones escalate
- Context trimming: only genuinely relevant chunks survive reranking, cutting average context length 45%
Monthly cost landed at 60% of budget.
Outcomes
- 91% accuracy on the eval set, with explicit “not found” instead of fabrication
- Average time to find information down from 8 minutes to 2.4
- Far fewer interruptions for senior staff; onboarding time roughly halved
- Ingestion pipeline runs automatically — changed documents are re-indexed within 15 minutes
Related
The method behind this project is now part of our AI Application Development service, and the same retrieval and evaluation thinking went into the AI Operations Copilot.