LLM internals

60 hours · the credibility gap

I use these systems every day and have never built one. That's the exact shape interviewers find: fluent about agents, retrieval and prompting, then hesitant one layer down. Attention mechanics, tokenization, sampling, KV caching — the vocabulary is familiar, the mechanics aren't.

This pillar is deliberately instrumental. The goal is not to become a pretraining engineer; it's to be unbluffable one layer below the resume, and to make better engineering decisions because I understand what the API is doing. Sixty hours buys that. It does not buy a research education, and pretending otherwise is how this pillar quietly eats the whole budget.

Modules

1 · Build a transformer from scratch · ~30h

The single highest-value item in this roadmap for an applied practitioner. It converts orchestration-layer knowledge into internals knowledge faster than anything else.

• Neural Networks: Zero to Hero — Andrej Karpathy · work through. Builds backprop → makemore → GPT in code, ~10 hours of video and considerably more of coding. The nanoGPT lecture is the payoff, but the earlier ones are the reason it lands. Code along; don't watch. Also covers the deep learning foundations pillar — one course, two pillars.
• Build a Large Language Model (From Scratch) — Sebastian Raschka, Manning 2025 · work through. What to do after Karpathy: the same journey at book pace and greater depth, through pretraining, fine-tuning for classification, and instruction tuning. Free code repo and video companion. Pick this or Karpathy as your one "work through" — doing both in sequence is a legitimate choice, doing both in parallel is not.
• Attention Is All You Need — reference. Read it after you've implemented attention, not before. It's short, and it reads completely differently once the code is in your fingers.

Done when: you can implement scaled dot-product attention and a full multi-head attention block from memory in numpy, and explain — without notes — why attention is scaled by √d_k, why layer norm rather than batch norm, and what the residual stream is for.

2 · Inference and serving mechanics · ~15h

Where applied engineers are weakest, and where the questions get uncomfortable fast. This is also the material that most directly improves day-job decisions: latency, cost, and throughput are all downstream of it.

• Efficient Memory Management for LLM Serving with PagedAttention — the vLLM paper, SOSP 2023 · work through. The clearest single document on why naive KV cache allocation wastes most of your GPU memory, and how an idea borrowed from OS virtual memory fixes it. Read it alongside the vLLM source.
• vLLM documentation — reference. Continuous batching, prefix caching, speculative decoding. Serve a model locally and watch throughput change as you turn knobs.

Get concrete: self-host a small model, measure tokens/sec, then measure again with batching and with quantization. Numbers you produced yourself are the ones you'll recall under pressure.

Done when: you can explain, on a whiteboard, why the KV cache dominates memory at long context, what prefill and decode do differently, how continuous batching improves throughput, and what quantization actually trades away — and you have your own measured numbers to quote.

3 · Adaptation: when to fine-tune, and how · ~10h

The question is asked constantly and answered badly. Most of the value is in knowing when not to.

• Hands-On Large Language Models — Jay Alammar & Maarten Grootendorst, O'Reilly 2024 · skim, code repo free. Strongest on visual intuition for embeddings, attention and tokenization. Use it to fill gaps, not as a primary path.
• LoRA: Low-Rank Adaptation of Large Language Models — reference. The primary source. QLoRA is the 4-bit variant you'll actually run.
• Direct Preference Optimization — reference. Preference tuning without a separate reward model; the default starting point for alignment work now, with IPO/KTO/SimPO as variants when its assumptions break.

Run one small QLoRA fine-tune end to end on a task you care about. One is sufficient; the point is to have touched the pipeline, not to become a fine-tuning specialist.

Done when: you can lay out the decision — prompt, few-shot, RAG, fine-tune, or a different model — with the cost, latency, data and maintenance implications of each, and defend a recommendation for a concrete scenario.

4 · Tokenization and sampling · ~5h

Small module, disproportionate interview return, because it's where surprising behaviour comes from.

• BPE from scratch — Karpathy's minbpe lecture in Zero to Hero covers it in one sitting.
• Sampling: temperature, top-k, top-p, repetition penalty. Know what each does to the distribution and when each fails.

Done when: you can explain why token counts differ across languages, why a model can't reliably count characters in a word, and what temperature is actually doing to the logits.

Depth on tap, if this becomes the thing

Stanford CS336 · Language Modeling from Scratch — reference. Percy Liang and Tatsunori Hashimoto take students through building a language model end to end: data, tokenizer, architecture, training, evaluation. Spring 2026 lectures are on YouTube and the assignments are public on GitHub.

Be honest about the cost. It's a full graduate course — Stanford quotes 20–25 hrs/week over 10 weeks, which is roughly the entire 300-hour budget. It is the right answer for an ML Engineer or Research Engineer target, and the wrong answer for this one. I'm treating individual lectures as reference material, not enrolling.

What I'm skipping

• Pretraining anything meaningful. Compute I don't have, for a job I'm not applying to.
• CUDA and custom kernels. Genuinely valuable, entirely out of budget, and only tested for infrastructure roles.
• Distributed training internals. FSDP, tensor and pipeline parallelism — know the words and the tradeoffs, skip the implementation.
• Paper-of-the-week reading. Unbounded, low retention, feels productive. Read papers when a specific question needs answering.