
AI Search · Retrieval-Augmented Generation · LLM Engineering
1. Introduction
Retrieval-augmented generation (RAG) combines a language model with information retrieved from a specified knowledge source. Its value is not simply that a model can produce longer answers. It is that responses can be anchored to documents the system is authorized to use.
A proof of concept may chunk a few PDFs, create embeddings, and send the top results to a language model. Production RAG demands more: document ingestion, permission enforcement, freshness, quality measurement, citation integrity, and resilience to untrusted content.
2. Separate Retrieval From Generation
A simple architecture looks like this:
Each stage has independent failure modes. An excellent language model cannot reliably cite a policy that was never retrieved. A good retriever cannot prevent an answer generator from making unsupported claims without suitable constraints and evaluation.
3. Start With Document Quality
RAG begins before vector search. Documents may contain tables, section hierarchy, scanned pages, duplicated headers, outdated versions, and access restrictions.
Parsing should preserve meaningful structure and source references. Chunk boundaries should not split a critical condition from its exception or detach a table from its heading. Choose chunk sizes using retrieval tests rather than assuming one universal number of tokens.
Maintain provenance: original document ID, version, source location, update time, and access-control labels.
4. Choose Retrieval Strategies Deliberately
Dense vector retrieval is useful for semantic similarity. Lexical methods such as BM25 can be stronger for exact product codes, names, or error messages. Hybrid retrieval combines complementary signals.
Reranking can improve ordering after candidate retrieval. Query rewriting may help ambiguous questions, but it can also distort intent; retain the original query and validate the rewritten search against it.
Measure retrieval quality using annotated question-to-evidence pairs before tuning prompts. Otherwise, a generated answer may conceal retrieval misses.
5. Keep Permissions in the Retrieval Path
An enterprise RAG service must not retrieve private content and hope the language model avoids revealing it. Authorization belongs in query planning and retrieval filtering, backed by verifiable server-side checks.
Document permissions may change after indexing. Synchronize access-control changes and deletions promptly. Test cross-tenant isolation, revoked users, stale indexes, and citations that refer to content the requester can no longer access.
6. Design the Answer Policy
The model should distinguish supported findings, uncertainty, and missing information. A well-designed system can answer “the available documents do not establish that” rather than inventing a plausible statement.
Require source references for material claims where feasible. Citations should resolve to the actual retrieved passage, not merely a related document title. Treat retrieved instructions as untrusted content to reduce prompt-injection risks.
7. Evaluate Two Layers
Retrieval evaluation: recall@K, nDCG, permission correctness, freshness, and evidence coverage.
Answer evaluation: factual support, citation correctness, completeness, abstention behavior, latency, and usefulness to the intended user.
Human-reviewed evaluation sets remain valuable, especially for complex enterprise workflows. Automatic graders can help triage results but need calibration because they may share biases with the models being evaluated.
8. Control Cost and Latency
RAG costs include parsing and ingestion, embeddings, vector infrastructure, reranking, model tokens, observability, and maintenance. Retrieve only the context needed for the question; excessively large prompts can increase latency and distract the model.
Cache appropriately where permissions and freshness allow. Instrument per-stage latency so teams can distinguish a slow vector query from model generation delay or document parsing backlog.
9. Plan for Continuous Updates
A knowledge base changes. Define indexing schedules, incremental updates, version retention, and deletion semantics. Monitor broken citations, retrieval drift, and answer changes when the document corpus evolves.
Document sources should remain the authority for business policy. RAG is a retrieval and explanation layer, not an uncontrolled replacement for governance or human approval.
10. Conclusion
Reliable RAG is primarily a knowledge-system engineering challenge. The strongest designs combine structured ingestion, effective retrieval, authorized access, grounded answer policies, and continuous evaluation. These are the foundations for useful AI Search products that enterprises can maintain over time.




