RAG — the little library that teaches an LLM what it didn't know.
Don't memorize RAG buzzwords. Understand the journey: knowledge → retrieval → evidence → generation → evaluation → production learning.
Why RAG?
LLM knowledge
Knowledge stored inside model parameters is useful, but private or frequently changing enterprise information isn't automatically there.
External knowledge
PDFs, policies, tickets, Git, databases and internal wikis live outside the model and change continuously.
The entire RAG story — one flow
PDF • Wiki • Git • DB"] B["Parse / OCR
Layout"] C["Chunk"] D["Embed"] E["Search Index
Vector + BM25"] A --> B B --> C C --> D D --> E end %% ================================ %% QUERY %% ================================ Q["๐ค USER
QUESTION"] R["Query
Understanding"] Q --> R %% ================================ %% RETRIEVAL %% ================================ subgraph RET["๐ RETRIEVAL PLANE"] BM["BM25
Exact terms"] VS["Vector Search
Meaning"] HY["Hybrid
Fusion"] RR["Reranker
Top evidence"] BM --> HY VS --> HY HY --> RR end R --> BM R --> VS E --> BM E --> VS %% ================================ %% SECURITY %% ================================ SEC["๐ Auth / RBAC
Tenant Filter"] R --> SEC SEC --> BM SEC --> VS %% ================================ %% GENERATION %% ================================ RR --> CTX["๐งฉ Context Builder
chunks + metadata"] CTX --> LLM["๐ค LLM
generate from evidence"] LLM --> OUT["๐ฌ Grounded Answer
+ citations"] %% ================================ %% ADDITIONAL KNOWLEDGE %% ================================ LLM -. "need another source?" .-> G["GraphRAG"] LLM -.-> SQL["SQL"] LLM -.-> API["Live API"] G --> CTX SQL --> CTX API --> CTX %% ================================ %% PRODUCTION LEARNING %% ================================ OUT --> TRACE["๐ฌ Production Trace"] TRACE --> SAMPLE["Real Query Sample"] SAMPLE --> FIX["๐ง Error Analysis"] FIX --> EVAL["๐ Regression / Golden Eval"] EVAL -. "release gate" .-> R %% ================================ %% STYLES %% ================================ classDef source fill:#eaf2ff,stroke:#155dcc,color:#152033,stroke-width:2px classDef retrieve fill:#e8f8fa,stroke:#007f91,color:#152033,stroke-width:2px classDef generate fill:#f1ecff,stroke:#6842b8,color:#152033,stroke-width:2px classDef security fill:#fdebf4,stroke:#ad2864,color:#152033,stroke-width:2px classDef ops fill:#e8f8f1,stroke:#087b55,color:#152033,stroke-width:2px class A,B,C,D,E source class BM,VS,HY,RR retrieve class CTX,LLM,OUT,G,SQL,API generate class SEC security class TRACE,SAMPLE,FIX,EVAL ops
First understand the building blocks
Text → tokens
LLMs don't directly read words. Text is split into tokens using methods such as BPE, WordPiece or SentencePiece.
Meaning → vector
Text becomes numbers in a vector space. Similar meanings tend to occupy nearby regions.
Document → evidence
Large documents are split into retrievable units while preserving enough local context.
Evidence → control
Store document ID, version, page, tenant, permissions, timestamp and embedding version.
Evidence → searchable
Store chunks so the system can quickly find candidates using lexical and semantic search.
Evidence → prompt
Retrieved chunks are assembled into a controlled context window before generation.
Retrieval is the real game
BM25
Finds important words and exact terms.
INC-48291
SAP-PRD-001
Best for: identifiers, names and exact terminology.
Vector Search
Finds semantic similarity.
≈
"remote work"
Best for: meaning, paraphrases and concepts.
Two-stage retrieval: cheap retrieval creates a broad candidate set; the reranker spends more computation choosing the strongest evidence.
When basic RAG isn't enough
Query rewriting
Transform vague questions into queries that are easier for the retrieval system to understand.
Multi-query
Search the same question from multiple perspectives, merge results and rerank them.
HyDE
Create a hypothetical answer, embed it, then search for real supporting documents.
Parent-child retrieval
Find a precise child chunk but return its larger parent context to the model.
Metadata filtering
Filter tenant, RBAC, version, department and language before generation.
Graph / SQL / API
Graph for relationships, SQL for structured facts and APIs for live information.
RAG → Agentic RAG
Basic RAG
Fixed retrieval path.
↓
Search
↓
Context
↓
Answer
Agentic RAG
The system decides which tool or retrieval path is needed.
↓
Decide
↓
Vector / Graph / SQL / API
↓
Enough evidence?
↓
Answer
Golden data = the answer key
Golden data is a trusted benchmark . It tells you whether a new RAG version still works.
The production learning loop
PRODUCTION FAILURES BECOME FUTURE TEST CASES
Golden data ≠ production tracing
๐ Golden data
Controlled benchmark.
+
Known truth
Answers:
“How good is this version?”
๐ฌ Production tracing
Runtime microscope.
→ Rewrite
→ Retrieve
→ Rerank
→ LLM
→ Answer
Answers:
“What actually happened?”
Three production rules people forget
Authorize before retrieval
Apply RBAC / ABAC / tenant filters before evidence reaches the model.
Retrieved text is untrusted
Documents can contain prompt injection. Retrieved text is evidence, not automatically instructions.
Version everything
Track document, index, chunking and embedding-model versions.
Production checklist
The answer to remember
“RAG separates knowledge from the model. Instead of expecting an LLM to memorize private or frequently changing information, we ingest external sources, parse and chunk them, create embeddings, and index them for retrieval. At query time we can combine semantic and lexical retrieval, apply metadata and security filters, rerank candidates, and give the most relevant evidence to the LLM for grounded generation. In production I’d also handle incremental ingestion, document and embedding versioning, authorization, prompt-injection risks, observability, golden-data evaluation, regression testing, latency and cost.”
No comments:
Post a Comment