Andrej Karpathy's AI From-Scratch Curriculum: From micrograd to nn-zero-to-hero
A step-by-step developer guide to Andrej Karpathy's open-source repositories, covering the scalar autograd engine micrograd and the comprehensive nn-zero-to-her
As modern machine learning frameworks mature, most practitioners interact with neural networks through high-level APIs that completely abstract away tensor computations and backpropagation. Open-source educational projects created by Andrej Karpathy, such as micrograd and nn-zero-to-hero, strip away these abstractions to expose the computational graphs and gradient flows operating underneath.

Image source: https://x.com/F5Adon/status/2116475548549669005
First created in 2020 (micrograd) and expanded in 2022 (nn-zero-to-hero), these repositories lay out a coherent progression from basic scalar-valued automatic differentiation to modern GPT architectures and tokenizers. Both projects are distributed under the permissive MIT license, allowing developers to inspect, run, and modify the source code freely.
The Architecture of micrograd: A Scalar Autograd Engine
The core of karpathy/micrograd consists of an autograd engine implemented in approximately 100 lines of Python code, paired with a lightweight neural network library spanning roughly 50 lines.
While providing an API style familiar to PyTorch users, micrograd operates strictly over scalar values rather than multidimensional tensor blocks. It decomposes every mathematical operation within a neuron—individual additions, multiplications, and activation functions like ReLU—into nodes within a dynamically constructed directed acyclic graph (DAG). Forward passes evaluate node values, while backward passes compute partial derivatives using the chain rule.
The repository can be installed directly via pip:
pip install micrograd
Inside the repository, demo.ipynb demonstrates training a 2-layer multi-layer perceptron (MLP) binary classifier built using the micrograd.nn module. Featuring two 16-node hidden layers, the network is trained on the synthetic moon dataset using an SVM max-margin loss and stochastic gradient descent (SGD), visualizing the learned decision boundary.
Additionally, trace_graph.ipynb leverages Graphviz to render visual computational graphs showing both forward numerical values and backward gradients for each node. To verify mathematical precision, the repository includes unit tests (python -m pytest) that validate computed gradients against PyTorch as a baseline reference.
The 'Zero to Hero' Curriculum: From Single Neurons to GPT and Tokenizers
The karpathy/nn-zero-to-hero repository houses the lecture notebooks (lectures/ directory) paired with Karpathy's video lectures, guiding learners through building deep learning models from foundational principles.
Starting with the manual implementation of micrograd, the curriculum progressively scales in complexity:
- Building micrograd (Lecture 1): Introduces backpropagation and gradient descent mechanics at the scalar level.
- The makemore Series (Lectures 2–6): Covers language modeling fundamentals, transitioning from character-level Bigram models to MLPs, batch normalization statistics and internal activations, manual backpropagation passes without PyTorch autograd, and hierarchical dilated convolutions inspired by WaveNet.
- Building GPT from Scratch (Lecture 7): Follows the "Attention is All You Need" paper and OpenAI's GPT-2/GPT-3 designs to construct self-attention heads, multi-head attention blocks, and transformer decoders directly in code.
- Building the GPT Tokenizer (Lecture 8): Explores byte pair encoding (BPE) algorithms and text tokenization through the standalone
minBPErepository.
Rather than relying on opaque library functions, the lectures focus on tensor manipulations, matrix multiplications, and explicit indexing to demystify language model pipelines.
Practical Scope and Inherent Limitations
These repositories serve primarily as educational instruments designed to cultivate practical intuition for deep learning mechanics.
Because micrograd tracks computations across individual Python scalar objects, it does not incorporate vectorized SIMD operations or GPU tensor cores. As a result, it is not engineered for training large-scale models in production environments. Production engineering workloads require production-grade frameworks such as PyTorch or JAX.
Furthermore, running the unit tests in micrograd or executing advanced exercises in the nn-zero-to-hero course requires a local Python environment with PyTorch installed as a reference dependency. For developers seeking to understand the mechanics hidden behind framework black boxes, this codebase remains an essential technical reference.
Sources
- GitHub Repository: karpathy/micrograd
- GitHub Repository: karpathy/nn-zero-to-hero