Part 1.4 — Which Loss to Choose, and Why. A calm, sequential guide for anyone entering the field. The article is divided into smaller chapters to make it easier to understand.
Part 1.3 ended on a quiet victory. The mean squared error, which we had simply assumed back in Part 1.1 and optimized in Part 1.2, was no longer an assumption. We had derived it — from a single principle, maximum likelihood, and a single modelling choice, that the model’s output is a Gaussian centred on its prediction. The loss we had trusted for two parts was finally standing on a foundation we could see through. The recipe was built: choose the output distribution, and the cost follows.
But a recipe that produces one dish invites an obvious question about all the others. We chose the Gaussian, and squared error came out. That was a choice — the introduction to Chapter 9 said as much and then set the point aside. This part picks it back up. If a different output distribution is chosen, a different cost comes out, and it predicts a different summary of the data. Squared error was never the only item on the menu; it was the first one we happened to order.
There is also a problem hiding inside the loss we derived, one that Part 1.3 had no reason to expose. In the very common case of a yes-or-no prediction, squared error can go quiet at exactly the wrong moment — its gradient falling nearly to zero while the model is still confidently wrong, leaving gradient descent with nothing to follow. The fix is not to patch squared error but to derive a different cost, cross-entropy, from a different distribution, and watch the problem dissolve on its own. This is where the slope idea from Part 1 returns for the last time, and where the choice of loss and the choice of output finally reveal themselves as one decision, not two.
So this part is about reading the menu that Part 1.3 built. We look at what other distributions give, why the loss you pick quietly decides which statistic your model chases, and why practitioners so often reach past the squared error we worked so hard to justify.
Chapter 10: When the Gradient Lies
We built the general recipe, and we chose the Gaussian, and squared error came out. So we could stop. But the practitioners’ literature contains a quiet recommendation that seems, at first, to contradict everything we just did: for many problems, do not use squared error. Use cross-entropy instead. This chapter explains why, and the explanation reaches all the way back to the slope idea from Part 1.
What Is Actually Being Compared
Putting squared error next to cross-entropy can look like a category mistake, so it is worth being clear about what is being compared. They are the same kind of object — both are cost functions, both are numbers measuring how wrong the model is, both are handed to us by the recipe once we pick an output distribution (Gaussian gives squared error; the binary distribution we will meet in Part 2 gives cross-entropy). So comparing them is fair. The reason they come into conflict is not the costs themselves. It is what sits in front of them.
The conflict appears in a specific and very common situation: a binary classification task, where the model must output a yes-or-no probability. To produce a number between 0 and 1, the output is almost always passed through a sigmoid first. And it is the sigmoid, wedged between the network and the cost, that causes the trouble. As we will see, the sigmoid has flat regions, and when squared error is stacked on top of a sigmoid, those flat regions leak into the gradient and flatten it too. During back-propagation — the backward pass we built in Part 1.2 and will extend across layers later — a flat gradient means the update is negligibly small, so the model barely changes even when it is badly wrong. Cross-entropy is the cost whose shape cancels that flattening. That is the whole subject of this chapter: not “which cost is better in the abstract,” but “which cost survives contact with the sigmoid.”
One note before the mechanics. Below, a term written σ′(z) will appear — the slope of the sigmoid. Do not worry about where it comes from in detail here; it is simply the sigmoid’s own steepness at a point, and it enters through the chain rule exactly as the slope did in Part 1.2. We are using it only to see the effect. The full story of the sigmoid, its slope, and the binary (Bernoulli) distribution that makes cross-entropy the derived cost for this task belongs to Part 2.1, where output units are the main subject. Here we only need to watch what σ′(z) does to the gradient.
Two Reasons a Gradient Can Be Small
Recall from Part 1.2 that training lives or dies by the gradient. If the slope of the loss goes flat, gradient descent has nothing to follow and learning stalls, like standing on a flat plateau in fog with no sense of which way is downhill.
Now here is the subtlety. A gradient can go small for two completely different reasons, and only one of them is benign.
The first is the good kind. In your worked example in Part 1.2, as w approached its true value of 2, the errors shrank and the gradient shrank with them. A small gradient there means you have essentially arrived — take smaller steps. Nothing is wrong.
The second is the disaster. The gradient can go nearly to zero while the model is still badly wrong. When this happens, the small gradient is lying: it whispers “nothing to fix here” at exactly the moment the model is confidently mistaken and most needs a shove. This second case has a name. It is called saturation, and avoiding it is the reason cross-entropy is preferred.
Where Saturation Comes From, With Numbers
To see saturation appear, we need an output that can be confidently wrong. Binary outputs are the clean example, and this is exactly the setting flagged above: the model must answer yes-or-no, so it produces a probability ....... .
To read the full article, please visit the link : https://husseinmahdi.xyz/writing/math-behind-nn-1-4/
or this : https://blog.gopenai.com/the-math-behind-neural-networks-explained-like-nobody-did-for-me-87f299008ffb
Refrences
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Relevant chapter: Chapter 6 (Deep Feedforward Networks), Section 6.2.1 — Cost Functions. Available free at deeplearningbook.org.