I had heard about backpropagation many times, but I never really understood how it actually worked. When someone asked me what backpropagation was, all I could say was, "It's how neural networks learn… I guess?"
I'd seen the formulas, I knew the chain rule, but it always felt like there was a wall between me and true understanding.
So I spent two days reading Karpathy's micrograd project source code. It's honestly the best explanation of the core principles of neural networks I've ever come across!
👉 https://github.com/karpathy/micrograd
This post documents what I learned and how I understand it now.
Why This Post
I'd watched plenty of YouTube videos explaining backpropagation, but most of them just said:
Backpropagation is repeatedly applying the chain rule.
That's true, but for a beginner, it's basically saying nothing.
I was used to frameworks where a single line — loss.backward() — got the job done, with no idea what was actually happening under the hood.
The Key Code & What It Does
class Value:
def __init__(self, data, _child = (), _op = "", label = ""):
self.data = data
self._prev = set(_child)
self._backward = lambda: None
self.grad = 0.0
self._op = _op
self.label = label
def __repr__(self):
return f"Value(data = {self.data})"
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad += out.grad * 1.0
other.grad += out.grad * 1.0
out._backward = _backward
return out
def backward(self):
topo = []
visited = set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build_topo(child)
topo.append(v)
self.grad = 1.0
build_topo(self)
for node in reversed(topo):
node._backward()
micrograd also defines __mul__, tanh, __pow__ and other methods — they all work the same way, each attaching a different _backward note. So I'm only showing the core code here.
Try this:
a = Value(1.0)
b = Value(2.0)
n = a + b
L = n.tanh()
You'll get this result:
How It Works
When you write n = a + b, here's what happens:
- A new
Value instance n is created.
- A
_backward function is attached to n._backward() but not executed yet.
- When
backward() is called, it traverses the graph in reverse topological order, executing each node's _backward() one by one.
Things You Should Know
Chain Rule:
∂e/∂a = (∂e/∂c)(∂c/∂a) + (∂e/∂d)(∂d/∂a)
At first I didn't accumulate gradients in _backward() — just assigned them. That led to some serious bugs.
Imagine you write:
a = Value(2.0)
b = a + a
What do you expect?
a.grad = 2, b.grad = 1? Or a.grad = 1, b.grad = 1?
If you run it, you'll find a.grad = 1 — which is completely wrong!
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad = out.grad * 1.0 # self = a, other = a;
other.grad = out.grad * 1.0 # both set a.grad = 1
out._backward = _backward
return out
Here's another example:
Try this:
a = Value(2.0)
b = Value(3.0)
c = a + b
d = a b
e = c d
What are a.grad and b.grad this time?
Clearly, the gradients of a and b got overwritten!
So you MUST accumulate in _backward()!
My Confusion
For a long time I couldn't understand what def _backward() and out._backward = _backward were really doing. Sounds silly, I know.
I kept wondering: when Value c is created, out.grad defaults to 0 — how can it possibly backpropagate to find self.grad and other.grad? (Of course, back then I thought out._backward = _backward would immediately execute the _backward function 😅)
Now I finally get it:
After creating a new Value c, c._backward just stores the _backward function without running it. It only executes when backward() is called, traversing in reverse topological order.
It's a naive question that came from a naive assumption, but I hope it helps other developers who've been stuck on the same thing.
A Thought That Followed
Imagine this:
a = Value(2.0)
b = Value(3.0)
c = Value(2.0)
d = a + b
e = c + d
Do a, b, and c have their _backward executed?
Answer: All three have _backward set to lambda: None (an empty function). When backpropagation reaches them, nothing happens — their gradients have already been accumulated through the upstream path.
What I Gained
After carefully reading through Karpathy's micrograd source code, I was genuinely impressed. I'd never understood the core of backpropagation so intuitively before. Here's what I took away:
- I now understand the core mechanism of backpropagation —
.backward() is no longer a black box to me.
- I finally truly understand what "backpropagation is just repeatedly applying the chain rule" means.
- This has kicked off my deep dive into deep learning, and now I can better compare micrograd with PyTorch and modern neural network frameworks.
Closing Thoughts
Thanks for taking the time to read this. I wrote it right after going through the source code, so there may be mistakes in my understanding — corrections are very welcome.
If you're also a beginner in deep learning, feel free to reach out. I'd love to learn together with you.
If you enjoyed this post, don't forget to like, bookmark, comment, and follow! I'll be sharing my learning journey in deep learning, machine learning, backend development, and more from time to time.
Check out my Dev.to: https://dev.to/elysianx138
My GitHub will also track my learning progress: https://github.com/elysianx138
