Evals & retrieval¶

40 hours · what the job actually is

This pillar doesn't appear on conventional ML interview roadmaps, which is precisely why it's here. Those roadmaps were written for a job — build and train models — that most AI engineering roles no longer involve. The 2026 version of the work is less about gradient descent and more about chunking strategy, retrieval quality, and faithfulness evaluation.

It's also the pillar that separates two candidates who both say "I built a RAG system." One can tell you their retrieval precision, how they measured it, and which change moved it. The other has a demo.

Because it's closest to work I already do, it compounds immediately — which is why it starts in Phase 1 rather than waiting for Phase 2.

Read this first

What We've Learned From A Year of Building with LLMs — Yan, Husain, Shankar and co-authors · work through. Free, and probably the single best document in this entire roadmap for this target: tactical (prompting, RAG, evals), operational (shipping, teams), and strategic ("no GPU before PMF", "focus on the system not the model"). Two hours, and it reframes everything below.

Pair it with Patterns for Building LLM-based Systems — Eugene Yan · skim. Evals, RAG, fine-tuning, caching, guardrails, defensive UX, as a coherent set rather than a list of techniques.

Modules¶

1 · Evaluation as an engineering discipline · ~12h¶

The core shift: evals are not a report you produce at the end, they're a gate that runs continuously. A mature setup runs at three points — offline against curated datasets, online against production traffic, and pre-merge in CI before any prompt or model change ships.

AI Engineering — Chip Huyen, O'Reilly 2025 · work through the evaluation chapters. The best book-length treatment for foundation-model applications, deliberately light on code so it doesn't rot. Also the backbone of AI system design — one book, two pillars.
OpenAI · Evaluation best practices — skim. Primary source, practical, short.

Concepts that must be automatic: task-specific metrics vs. generic ones; LLM-as-judge and its failure modes; why judges need validating against human labels before you trust them; sampling production traffic for continuous scoring; regression gates on prompt and model changes.

Done when: given an unfamiliar LLM feature, you can produce — in 20 minutes, written — a metric list, a dataset-construction plan, the specific way your judge could mislead you, and the CI gate. Then hand it to someone and have them find the hole you left.

2 · Error analysis · ~14h¶

The highest-leverage module in this pillar, and the one I'd originally left out entirely. Most people write evals before knowing what their system gets wrong, which produces metrics that move without quality moving.

The discipline runs the other way round: look at your data, label failures, cluster them into categories, and only then build evals that target the categories that actually hurt.

LLM Evals: Everything You Need to Know — Hamel Husain & Shreya Shankar · work through. The densest practitioner resource that exists on this, from two people who've taught it to 700+ engineers. Covers error analysis, building judges that correlate with humans, synthetic data for finding failures, and evaluating agentic systems with tool calls and retrieval.

What to actually do: take a system you own, sample 50–100 real traces, read every one, and write down what went wrong in your own words. Cluster the notes. The categories that emerge are your eval suite. This is unglamorous, nobody does it, and it's the thing that makes the rest work.

Done when: you've done exactly that on a real system, and can name your top three failure categories with rough frequencies — plus one eval you built because of it and one you deleted because it was measuring nothing.

3 · Retrieval as an engineering problem · ~14h¶

RAG has stopped being a model problem. Retrieval determines what the model sees; chunking determines how it sees it. Nearly all quality movement lives here rather than in generation.

Introducing Contextual Retrieval — Anthropic · work through. Primary source with numbers: prepending chunk-specific context before embedding cuts failed retrievals by 49%, and 67% with reranking. Implement it — it's a weekend, and it's the clearest demonstration that chunking is a design decision rather than a default.
Effective context engineering for AI agents — Anthropic · skim. Context as a finite budget to be curated, which is the framing that makes long-context systems behave.

Work the tradeoffs concretely: fixed-size vs. semantic vs. hierarchical chunking; overlap; hybrid dense + BM25; reranking; late chunking. Then measure — hold everything constant except the chunking strategy and watch precision and recall move independently. A change that lifts recall while sinking precision is dumping noise into the context window, and it will look like an improvement on any metric that doesn't separate the two.

Done when: you've built an evaluation set for a retrieval system you own, changed one variable, and can state what it did to precision and recall with numbers rather than impressions.

Staying current without a reading habit that eats the budget¶

The 2025 AI Engineering Reading List — Latent Space · reference. One paper, blog or model family per week, curated for engineers rather than researchers. Use it as a bounded queue when you want to stay current — which is the entire reason broad paper-reading is otherwise cut from this roadmap.

What I'm skipping¶

Vendor tool tours. The eval-tooling landscape churns constantly and the concepts don't.
Benchmark leaderboards. MMLU scores tell you nothing about whether your feature works.
Building a judge before doing error analysis. This is the actual failure mode, and it's the one that feels most like progress.