A calm, sequential guide for anyone entering the field. The article is divided into smaller chapters to make it easier to understand.
Part 1.2 left us with a working method but an unexamined assumption. We had a loss, the mean squared error, and we had turned it into a training procedure through the derivative and gradient descent. We watched the loss fall from 18.67 to 0.297 in a single step and then converge to the true relationship in the data. Everything worked. But we never asked where the mean squared error itself came from. We wrote it down as a reasonable way to measure error — square the gap, average it — and moved on. Because it worked, the question never surfaced.
Look again at how it entered. In Part 1.2, the model gave us a form for the line, ŷ = wx + b, but no way to judge whether a particular guess of w and b was any good. The equation described the model and contained no built-in measure of its quality. Faced with that gap, the article said, in as many words: “So we invented one.” We compared the prediction to the truth across all the data and summarized the gap into a single number. Squaring came along because it was reasonable — it makes every error positive and punishes large misses — not because anything forced it.
Chapter 7: What a Cost Function Really Measures
A Note on Three Names
At the end of Part 1.2 we had a complete loop: predict, measure the loss, compute the gradient, take a step. The loss at the centre of that loop was
A quick word on names first, because this article introduces a third one and I do not want it to look like a third thing. In Part 1.1 we called this the error function. In Part 1.2 we called it the loss. In this part, and in most textbooks, you will also see it called the cost function, written J(θ). These are not three different objects. They are three names for the same idea — a single number that measures how wrong the model is and that training tries to make small. “Cost function” is simply the more general term the literature settles on once the measure is no longer tied to one particular formula, which is exactly the situation this article is about to put us in. Wherever you read “cost” below, you may read “error” or “loss” and lose nothing.
The Assumption We Never Examined
With the naming settled: as the introduction recalled, this measure entered the story as an invention — the reasonable thing we reached for when the model gave us no measure of its own quality. Squaring makes every error positive, so overshooting and undershooting both count as wrong, and it punishes a large miss far more than a small one. These are good properties, and they were enough to justify the formula for a first encounter.
But “good properties” is not the same as “forced.” An invention fills a gap; it does not prove it was the only thing that could have filled it. A reader who accepts squared error should be able to answer the harder question that the invention quietly left open: out of all the ways to measure a gap, why this one? Some of that question we have in fact already answered. In Part 1.1 we asked the most natural competing question — why not the absolute gap |y − ŷ|, which also removes the sign — and we ruled it out for a concrete reason: the absolute value has a sharp corner at zero where its slope jumps from −1 to +1 with no value in between, and gradient descent, which lives or dies by reading a clean slope, receives contradictory signals at exactly the point it is trying to settle into. So the absolute value is not an open door; Part 1.1 already closed it, on grounds of the slope.
What Part 1.1 did not close is the deeper question. Ruling out the absolute value tells us squaring is better-behaved than one rival. It does not tell us that squaring is right — that out of every smooth, well-behaved measure we could have invented, this is the one the problem actually calls for. For that we need to stop justifying squared error from the outside and find the principle that produces it from the inside. This chapter goes looking for that principle.
From a Point Guess to a Distribution
In Part 1.2, the model produced a single number: ŷ = wx + b. Given an input, it committed to one answer. This is the natural picture, and it is also the picture that hides the principle we are looking for.
The more honest picture is this. A model is uncertain. Given an input x, it does not truly believe the answer is exactly one number. It believes the answer is probably near some number, with smaller and smaller chances the further you move away. In other words, the model does not output a point. It outputs a distribution — a spread of belief over all the answers that could occur. We write this distribution as
This part is about where that invention actually came from. Why squared error, and not the plain distance, or the cube? The honest answer is that the mean squared error is not a starting point at all. It is a consequence. It is what you are forced to when you begin from a deeper and more general principle about what it means for a model to be good. In this part we start from that principle, watch it produce a single general recipe for building cost functions, and then watch the mean squared error fall out of that recipe on its own.
Readers of Part 1.1 were promised this. Its closing note mentioned, almost in passing, that minimizing the mean squared error is equivalent to maximum likelihood estimation under the assumption of Gaussian noise. That sentence was a debt. This part pays it: everything below is the unpacking of that one line. Part 1.4 then asks the next question — whether the loss we just derived is the one we should actually use.
....
....
Read the full article at the official link: https://husseinmahdi.xyz/writing/math-behind-nn-1-3/