CAG Over RAG, When Speed Is the Constraint

Retrieval-Augmented Generation (RAG) has been the default enterprise AI pattern for the past two years. Need to query an internal knowledge base? Chunk the documents, embed them into high-dimensional vectors, store them in a vector database, and perform cosine similarity search on every user prompt. It works, and for terabyte-scale corpuses, it remains indispensable.

However, in latency-critical and high-frequency user interactions, RAG carries a heavy hidden tax. Enter Cache-Augmented Generation (CAG): a paradigm shift enabled by modern long-context models and provider-level KV cache prefix retention (Prompt Caching).

The Latency Tax of the Traditional RAG Pipeline

When a user asks a question in a RAG-backed application, the request must traverse a multi-stage pipeline before the LLM can generate a single character:

• Embedding Generation: Sending the user query to an embedding model (~40–80 ms).
• Vector Search: Querying an approximate nearest neighbors index (HNSW / IVF-PQ) in Pinecone, Qdrant, or pgvector (~30–100 ms).
• Re-ranking Pass: Passing top-k chunks through a cross-encoder model to filter false positives (~100–250 ms).
• Prompt Assembly & Network Transit: Concatenating retrieved chunks into the prompt and transmitting them across the wire to the LLM (~100 ms).
• Cold Attention Prefill: The LLM must compute Key-Value (KV) matrices for the entire newly assembled prompt before outputting the first token (~400–1,200 ms).

Summed together, Time to First Token (TTFT) frequently ranges between 1.5 and 3.5 seconds. In conversational voice interfaces, interactive coding completions, or real-time simulation loops, a three-second latency is perceived as broken.

How CAG Operates Under the Hood

Instead of searching for snippets on demand, Cache-Augmented Generation pre-loads entire knowledge bases—technical documentation, full codebase indexes, legal contracts, or customer histories (up to hundreds of thousands of tokens)—directly into the LLM's system prompt.

Because the prefix remains identical across queries, model serving infrastructures (such as Anthropic, Google Cloud Vertex, or vLLM) cache the precomputed Key-Value (KV) activation tensors directly in GPU VRAM or host memory. When a query arrives, the LLM skips the attention prefill pass over the preloaded knowledge base entirely.

RAG Latency Pipeline:
[Query] ──► [Embed (60ms)] ──► [Vector DB (80ms)] ──► [Re-rank (150ms)] ──► [LLM Prefill (800ms)] ──► [TTFT: ~1,100ms+]

CAG Latency Pipeline (Prompt Caching):
[Query] ──► [KV Cache Hit (Instant)] ──────────────────────────────────► [LLM Decode (50ms)]   ──► [TTFT: ~150ms]

Prompt Caching in Practice (Python Example)

Modern APIs make CAG trivially easy to implement. With Anthropic's Claude API, for example, marking a large reference text block with cache_control tells the engine to snapshot the KV activations:

import anthropic

client = anthropic.Anthropic()

# Pre-load full documentation corpus (e.g. 80,000 tokens)
with open("entire_api_documentation.md", "r") as f:
    documentation_corpus = f.read()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a specialized technical assistant. Answer questions using the reference documentation below.",
        },
        {
            "type": "text",
            "text": documentation_corpus,
            # Mark the massive corpus as a persistent cached prefix
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[
        {"role": "user", "content": "How do I handle connection timeouts in the auth service?"}
    ],
)

# Verification of cache usage
usage = response.usage
print(f"Cache Read Tokens: {usage.cache_read_input_tokens}")
print(f"Cache Creation Tokens: {usage.cache_creation_input_tokens}")

The Architectural Decision Matrix

CAG is not a silver bullet, and choosing between RAG and CAG is a matter of strict systems engineering constraints:

• Corpus Size: If your knowledge base fits within 100k to 1M tokens (roughly 300 to 3,000 pages of text), CAG is almost always superior. If your data is 50 gigabytes across millions of customer files, RAG is required.
• Mutation Frequency: CAG thrives when knowledge is read frequently and updated infrequently (e.g., product docs, API specs, policy manuals). If records change every 5 seconds, cache invalidation negates the performance gain.
• Economic Dynamics: Cached tokens on Claude and Gemini receive a 50% to 80% discount compared to base input pricing, often making CAG cheaper than paying vector database hosting fees alongside cold LLM tokens.