Inference
Topic

Inference

Latency, throughput, and hardware acceleration: LPUs, GPUs, and high-performance inference microservices.

3 posts · updated weekly
SystemsEngineeringFindingsResearchInference
Most recent
CAG Over RAG, When Speed Is the Constraint
Inference

Feb 12, 2025 · 4 min read

CAG Over RAG, When Speed Is the Constraint

RAG searches for information on every turn. CAG pre-computes the KV cache. Where Time-to-First-Token and conversational latency matter, caching flips the architecture.

LPUs and What They Do That GPUs Do Not
Inference

Jan 30, 2025 · 3 min read

LPUs and What They Do That GPUs Do Not

Language Processing Units are purpose-built for language models rather than general compute. The specialization shows up in where the weights live.

Deploying AI Models with NVIDIA NIM: Production LLMs as Microservices
Inference

Dec 10, 2024 · 4 min read

Deploying AI Models with NVIDIA NIM: Production LLMs as Microservices

Running self-hosted LLMs used to mean stitching vLLM, Triton, and CUDA drivers by hand. NVIDIA NIM packages TensorRT-LLM in standardized, production-ready OCI containers.

Sorted by most recent.