Some questions need a second search
A customer support assistant can often answer a simple policy question from one well-chosen passage. A question spanning a contract, an amendment and an exception may require several searches. Agentic retrieval-augmented generation gives the model tools to inspect evidence, identify a gap and retrieve again. Its value depends on whether those extra steps improve the answer users actually need.
Consider an insurance colleague asking which conditions apply to a revised policy. The assistant may need to identify the product, retrieve the applicable version, follow a reference to an endorsement and ask for a missing date. That workflow needs explicit sources and version handling. A larger context window alone does not establish which document governs the case.
Give the agent useful retrieval tools
Start with clean ingestion, stable document identifiers and metadata that reflects the business. Preserve headings, table context, dates and relationships between documents. Provide complementary tools where they help: keyword search for exact terms, semantic search for varied phrasing, filters for document versions and structured queries for authoritative records. Keep calculations in deterministic code.
Self-RAG is an important research example of learning when to retrieve and how to critique retrieved evidence. It is a trained framework, rather than a promise that any prompted reflection loop will improve quality. For an enterprise application, begin with the simplest retrieval baseline and introduce an adaptive step only when your failure analysis supports it.
Make the loop accountable
Each tool should have a narrow purpose, an explicit input contract and results that expose their origin. Set limits for searches, elapsed time and context. Require the assistant to stop, ask for clarification or hand over when evidence remains insufficient. Anthropic’s guidance on effective agents highlights environmental feedback and stopping conditions; these controls belong in the application, alongside the prompt.
Apply document access rules at retrieval time. Treat retrieved instructions as untrusted content and keep the agent’s authority separate from the documents it reads. Preserve the evidence chain through to the answer. A citation is useful only when the cited passage supports the associated claim, including its scope, exceptions and date.
Evaluate retrieval and the final answer separately
Use real questions, including multi-document cases and questions the corpus cannot answer. Score whether the system found the necessary evidence, represented it faithfully and avoided unsupported conclusions. FRAMES was designed to examine factuality, retrieval and reasoning together on multi-hop questions. Its benchmark is a useful research reference; your own documents and users still need their own evaluation set.
Compare the adaptive system with the baseline on answer quality, cost per accepted answer and latency, including slow cases. Inspect the traces of failures: an incorrect answer may come from a missing document, an outdated source or poor synthesis. Those require different fixes. A successful agentic RAG rollout delivers a searchable evidence trail and a repeatable improvement process, with additional autonomy justified by measured gains.
Sources & further reading
Written by Gilles Jacobs, AI & data engineering consultant at Exverbai.
