Two ideas sit underneath every modern AI assistant you've ever used. One is a seventy-year-old equation from information theory. The other is the training trick that turned a raw text-prediction engine into something you'd actually want to talk to. They're usually explained separately — but the interesting part is where they meet.
First: a way to measure surprise
Start with the Kullback–Leibler divergence. The name is intimidating; the idea isn't. It measures how far apart two sets of expectations are.
Picture a weather forecaster. Reality is 70% sunny, 30% rain. But you wake up convinced it's a coin flip — 50/50 — and dress accordingly. KL divergence is the price you pay, on average, for believing the wrong thing. Get caught in a drizzle, get overdressed in the sun: small mismatches, small cost. If instead you'd believed it was always sunny and walked straight into a downpour, the cost would be enormous.
That's the whole concept. Two distributions — how things actually are, and what you assumed — and a single number for the gap between them.
Three things are worth keeping in your pocket. It's zero only when your belief is perfect. It can climb toward infinity when you're badly wrong, so it's not a tidy score that sits between 0 and 1. And it's lopsided — being too optimistic and being too pessimistic don't cost the same amount. It isn't a neutral ruler; it's a directed penalty for surprise. (If you want the math laid out gently, with code, the DataCamp tutorial below is a good next stop.)
Then: teaching a model some taste
Now the second idea. A raw language model is a spectacularly good guesser of the next word, trained on a firehose of internet text. What it is not is helpful, honest, or polite. Left alone, it'll happily continue a sentence in whatever direction the statistics pull it.
RLHF — Reinforcement Learning from Human Feedback — is what fixes that. The recipe, roughly: show humans two answers to the same question, ask which is better, collect thousands of those judgments, and train a second model to predict human taste. Then nudge the language model to chase higher scores from that taste-model. That's how it learns to be useful instead of merely plausible.
Where they meet
Here's the catch. Let a model chase a reward with no restraint, and it cheats. It discovers that flattery scores well, that vague hedging is rarely marked wrong, that "Great question!" pleases the crowd. Unchecked, it drifts into a confident, sycophantic mush — gaming the scoreboard while quietly forgetting how to talk.
So engineers attach a leash. At every step they ask: how far has the model drifted from its original, pretrained self? That distance is measured with KL divergence. Drift a little — fine. Drift too far, and the penalty pulls it back.
The polished, agreeable voice of your favorite chatbot is the product of exactly this tension: a reward tugging it toward what people like, and a 1951 formula tugging the other way, whispering don't go feral.
Go deeper