How Retrieval-Augmented Generation Works

Large language models are phenomenal pattern recognizers, but they possess two fatal weaknesses in production: their knowledge is frozen at training cutoff, and when asked about facts they do not know, they hallucinate plausible-sounding nonsense with absolute confidence.

Retrieval-Augmented Generation (RAG) eliminates hallucination by turning the model into an open-book test taker. Instead of relying solely on parametric memory (weights), the system retrieves verifiable evidence from an external vector index and injects it directly into the prompt context.

The Three Stages of the RAG Loop

• 1. Ingestion & Embedding: Long documents are split into overlapping chunks (e.g. 500 characters with 100-character overlap) and converted into continuous mathematical vectors v ∈ ℝᵈ using an embedding model.
• 2. Dense Semantic Retrieval: When a user poses a question, the query is mapped into the same vector space. The system calculates the distance (typically cosine similarity) between the query vector and millions of stored chunk vectors to retrieve the top-k nearest neighbors.
• 3. Augmented Synthesis: The retrieved chunks are formatted into a prompt template alongside a strict grounding directive: 'Answer the question strictly based on the provided context.'

The Mathematics of Vector Similarity

Why does comparing floating-point arrays identify semantic meaning? Embedding models map synonymy and semantic intent into geometric direction. A query about 'slow SQL queries' and a document discussing 'database index bottlenecks' point in almost identical directions in 768-dimensional space, even without sharing words in common.

By dividing out the Euclidean norm ‖A‖ ‖B‖, cosine similarity normalizes for document length: a 2,000-word chapter and a 10-word query can be matched purely on conceptual alignment rather than word count magnitude.

An End-to-End RAG Loop in Python

Here is a complete, minimal implementation of semantic search and context injection using NumPy and OpenAI embeddings:

import numpy as np
from openai import OpenAI

client = OpenAI()

# 1. Corpus of documents
knowledge_base = [
    "The Space blog is built with Next.js 16 and Tailwind CSS v4 on Turso SQLite.",
    "Database transactions use libSQL over HTTP with drizzle-orm for type-safe queries.",
    "The newsletter delivers articles via Resend and supports RFC 8058 one-click unsubscribe.",
    "Postgres connection pools should be sized based on available CPU cores and RAM."
]

def get_embedding(text: str) -> np.ndarray:
    res = client.embeddings.create(input=text, model="text-embedding-3-small")
    return np.array(res.data[0].embedding, dtype=np.float32)

# 2. Ingest corpus vectors
doc_vectors = np.array([get_embedding(doc) for doc in knowledge_base])

# 3. Query retrieval
query = "How are emails sent from the blog?"
query_vector = get_embedding(query)

# Compute cosine similarities
dot_products = np.dot(doc_vectors, query_vector)
norms = np.linalg.norm(doc_vectors, axis=1) * np.linalg.norm(query_vector)
similarities = dot_products / norms

top_index = np.argmax(similarities)
best_context = knowledge_base[top_index]
print(f"Top Match (Score: {similarities[top_index]:.4f}): {best_context}")

# 4. Grounded Synthesis
prompt = f"""Answer the question using ONLY the provided context.

Context:\n{best_context}\n\nQuestion: {query}"""

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.0,
)
print("\nAnswer:", response.choices[0].message.content)

Where Naive RAG Fails in Production

Moving from a weekend prototype to production RAG reveals real architectural challenges:

• Embedding Drift: You must never query an index with an embedding model different from the one that generated the index. Dimensions and semantic projections are not transferable.
• Chunk Truncation: When crucial context spans across a chunk boundary, both chunks score low on similarity. Implement sliding window chunking with 20% overlap or hierarchical parent-child chunking.
• Needle-in-a-Haystack Dilution: Injecting 20 retrieved chunks causes the LLM to overlook the key fact if it is buried in the middle of the context window. Always apply a cross-encoder re-ranking pass (e.g. Cohere Rerank) before prompt injection.