Most of us here ship code against model APIs long before we ever open up the layer that produced the output. This is the short version of what that layer is doing, written for developers who want the mechanics without a semester of notation.
One Neuron Is Three Operations
A neuron takes a vector of inputs, multiplies each input by a learned weight, sums the results, and adds a bias. That sum goes through a nonlinear activation function, and the output becomes an input for the next layer. In code that is a dot product, an addition, and a function call.
The weights are the only part that changes during training. Everything else about the layer, its width, its activation, its position in the stack, is a decision you make before the first batch ever runs.
Why The Nonlinearity Is Not Optional
Stack two linear layers and you get another linear layer. The composition collapses, and the network can only draw straight boundaries no matter how deep you make it.
The activation function is what prevents that collapse. ReLU, which is just max(0, x), is enough to do it. That single bend per layer is what lets a stack of trivial operations approximate curved, interacting relationships in real data.
Training Is Error Attribution
A forward pass produces an output and a loss. Backpropagation walks that loss backwards through the graph and works out, for every weight, how much it contributed to the error. Each weight then takes a small step in the direction that reduces the loss.
Repeat across millions of examples and the weights settle into values that encode the patterns. No rule is written anywhere, there is only a very long sequence of small corrections, which is also why a model can be confidently wrong in ways no line of code explains. This complete neural networks guide walks through the network types and the full training loop if you want the longer version.
Where The Brain Comparison Ends
The name is a metaphor and it leaks badly. Biological neurons fire in time, adjust their own connectivity, and can learn a new category from two examples. A trained network needs thousands of labeled examples for that same category and cannot rewire itself at all.
Worth keeping in mind whenever someone says a model works like a brain. The resemblance is at the level of a rough cartoon, not the mechanism.
Takeaway
If you can hold three operations in your head, weight, sum, bend, you can reason about what any layer is capable of and where it will fail. Most of the confusion around model behaviour comes from imagining something more deliberate happening in there than actually is.
