AI system design¶

45 hours · the differentiating round

This pillar assumes the framework, building blocks and estimation habits from the general pillar. Every answer here is a general system design answer with a model in the middle of it, and attempting it in the other order produces architectures that describe an embedding pipeline in loving detail and never mention how requests reach it.

The round where senior candidates separate from mid-level ones, and the one people under-prepare most — partly because it's hard to practise alone, and partly because reading about architectures feels like preparing for it. It isn't. This is a live conversation under time pressure, and the only preparation that transfers is producing designs out loud, on a clock.

The 2026 change: loops that used to ask you to design a recommender now include a generative AI component — systems built on LLMs, retrieval, and agents acting on a user's behalf. Both still appear. Prepare for both, weighted toward the generative side for AI engineering roles.

Modules¶

1 · The framework, adapted · ~8h¶

The general framework still runs — requirements, estimation, API, data, high-level design, deep dive, bottlenecks. What changes is where the weight falls.

Roughly: clarify the problem and constrain scope → define what success means and how it's measured → data → the model or system choice, with alternatives named → serving and infrastructure → evaluation and monitoring → failure modes and cost.

Note the shape: evaluation appears twice and modelling appears once. That's the correct weighting for AI systems, and inverting it is the most common way this round goes wrong.

AI Engineering — Chip Huyen, O'Reilly 2025 · work through. End-to-end treatment of building on foundation models: evaluation, the prompt-vs-finetune-vs-API decision, RAG, inference optimisation, and product alignment. Deliberately minimal code so the frameworks outlive the tooling. If you read one book in this roadmap, read this one.
Designing Machine Learning Systems — Chip Huyen, O'Reilly 2022 · reference. The predecessor, still the best treatment of data pipelines, training-serving skew, and drift. Read the chapters the newer book doesn't replace, not the whole thing.

Done when: you can drive the framework end to end on an unseen prompt in 45 minutes, out loud, without notes.

2 · Case practice — written, not read · ~30h¶

The core work of this pillar. One case written out per week, on a timer. A case you wrote is worth ten you skimmed, because the gaps only surface when you have to commit to a design.

Generative AI System Design Interview — Ali Aminian & Hao Sheng, ByteByteGo · work through. Ten worked GenAI cases with a consistent seven-step framework and heavy diagramming — RAG, ChatGPT, image generation, text-to-video. The closest thing to a canonical source for the generative half of this round. Attempt each case cold before reading the solution. Available as a book and as a course; the content is the same.
Machine Learning System Design Interview — Ali Aminian & Alex Xu, ByteByteGo · reference. The classical half: recommendation, search, ranking, ads. Still asked, still worth having.

Cases worth writing regardless of source: a RAG system over a large private corpus · an LLM-powered support agent with escalation · semantic search at scale · a content moderation pipeline · an eval harness gating a prompt deploy · serving a fine-tuned model under a latency budget.

Done when: eight cases written under time, and you can identify your own recurring weakness across them — most people have exactly one, and it's usually skipping straight to architecture without pinning down what "good" means.

3 · Where existing experience does the work · ~7h¶

Anyone arriving from production LLM work has a real advantage in this round and usually under-uses it. The right move is to convert shipped systems into design narratives before the interview: the constraint you hit, the alternative you rejected, the number that made the decision.

Interviewers consistently reward "we tried X, it failed because Y, so we did Z" over a clean whiteboard diagram with no scars on it. That story cannot be invented under pressure — it has to be prepared. See interviews.

Done when: three shipped systems are written up as design stories with concrete constraints, rejected alternatives, and measured outcomes.

What I'm skipping¶

Classical distributed systems design. Load balancers and sharding are worth knowing and are not the differentiator for these roles. If a general system design round is on the schedule, prepare it separately.
Interactive prep platforms. Expensive, and the same practice happens for free with a peer and a timer. Spend the money on mock interviews with humans instead — see interviews.
Reading more cases. Past a point this is pure procrastination. Write, don't read.