Where do squared error and cross-entropy actually come from?

Question

Short answer: maximum likelihood. Choose a distribution for the output, then:

1. Read its parameter as the network's output → output function
2. Take −log of it → error function

Gaussian (regression)

• Output: ŷ = Wᵀh + b (the mean)
• Error: −log N(y; ŷ, σ²) → squared error (constants drop out)

Bernoulli (binary classification)

• Output: exponentiate + normalize → ŷ = σ(z)
• Error: −log P(y) → cross-entropy = softplus((1 − 2y)z)

Why the pairing matters: with cross-entropy, the log cancels the sigmoid's exponential, so ∂J/∂z = σ(z) − y. With squared error, σ′(z) stays in the gradient and goes to ~0 when the model is confidently wrong, so learning stalls.

Full derivations: https://husseinmahdi.xyz/writing/series/math-behind-neural-networks/

Next up: softmax + cross-entropy for k classes.