The retrieval system that survives contact with real users
Every retrieval-augmented system looks excellent in a demo, because the person demonstrating it asks questions they already know the answer to. The first week of real traffic is where the design assumptions get tested, and the failures are rarely about the model.
The three things that break first are chunking, permissions and staleness. Chunking fails when a policy document’s meaning lives in the heading two pages up. Permissions fail when the index flattens access control that the source system enforced carefully. Staleness fails quietly, which makes it the worst of the three.
A demo answers the questions you thought of. Production answers the ones you did not. Here is the evaluation practice that closes the gap.
Our answer is an evaluation harness written before the first retrieval call. It holds a golden set of real questions from the people who will use the system, graded by those people, and it runs in CI on every change to the pipeline. When someone proposes a new embedding model, the argument is settled by a number rather than a preference.
None of this is exotic. It is the same discipline any other production system gets, applied to a component that the industry has been unusually willing to ship on vibes.
Leave a Reply