Production RAG Assistant — From Demo to Dependable
Taking a production RAG assistant from demo to dependable.
- •Hardened a production 10-agent RAG assistant for a Top-5 global CPG client
- •Compliance answers over a 200+ page marketing code, across 5 user segments
- •Diagnosed the prompt-only ceiling, then architected the agent that removed 96% of tone violations
- •Security assessment: surfaced a live auth bypass and exposed credentials
- •Retrieval & prompt improvements; ReAct agent optimization (~2.9x faster)
- •Went from external assessor to first merged contributor PR
Tap to enlargeIt’s easy to build a RAG assistant that works in a demo. Making one dependable in production is the hard part. Working with a Top-5 global CPG client inside a top-tier services company, I moved a production RAG assistant toward reliability — through security assessment, retrieval and prompt improvements, and architecture optimization — and earned the trust to ship changes into the client codebase myself. The screens below are anonymised illustrations of that work — the client’s own system and data are not shown.
The Question That Used to Take Weeks
A verdict with the rule cited, on the spot — instead of a queue in front of a central team
Every agency and partner running a campaign for a large consumer brand has to answer one question before anything ships: does this comply with the brand's marketing code? The code is long, it is specific, and getting it wrong is a public problem rather than a private one.
So the partner did two things, both slow. They read the code themselves, hunting for the clause that covers their case among rules written for every market and every product line. And they emailed the brand's central team and waited — sometimes weeks — for a verdict, while the campaign sat still. The same handful of people answered the same families of questions for every partner, all year.
The assistant turns that queue into a conversation: the partner asks in plain language and gets a verdict with the specific rule cited, on the spot. That is where the value sits. The security work, the retrieval fixes and the latency work all exist to make that answer one you can actually act on.
Security First
Reliability starts with security you can prove, not assume
Before improving answers, I pressure-tested the system’s safety. The assessment surfaced critical gaps in a pre-production environment — a live authentication bypass and exposed credentials — escalated with clear, reproducible evidence so they could be closed before reaching production.
Better Retrieval, Faster Reasoning
Fix retrieval and orchestration — where the real failures hide
The real defects lived in retrieval and orchestration, not just prompts. Targeted retrieval and prompt improvements recovered previously-failing cases, and moving the assistant to a ReAct-style agent loop cut latency substantially (~2.9x faster) with no loss of answer quality.
Assessor to Contributor
Forward-deployed: embedded, trusted, accountable for the outcome
The engagement started as an outside assessment and ended with the first contributor pull request merged into the client’s own codebase — a shift from “telling them what’s wrong” to “shipping the fix with them.” That’s the forward-deployed mode: embedded, trusted, and accountable for outcomes.
Interested in building something similar?
Let's Connect