Airbnb published how it evaluates GenAI at scale, and it’s excellent. It’s also, underneath the vocabulary, the funniest engineering document of the year: a nine-figure AI operation whose most ad
Airbnb’s engineering team just published how they run evals for their GenAI features, the review highlights, the AI customer support, all of it, and LinkedIn is passing it around like prasad. Deservedly. It’s one of the most honest “here’s how we actually do it” documents a big company has shipped in a while, and if you’re building anything LLM-powered, you should read the original.
So I’m not here to argue with the practice. The practice is superb.
I’m here to point out what the practice quietly admits, because buried inside the layered architecture and the calibration loops and the Greek-letter agreement statistics is a confession so wholesome it made me laugh out loud, and then, about four seconds later, made me sit up straight.
Let me show you the confession first.
The one rule cost them a fortune to learn, and your school teacher knew it
Airbnb’s document, after establishing that GenAI breaks every assumption of traditional software testing, non-deterministic outputs, subjective correctness, chains of retrieval and tool calls that can each fail independently, delivers its single most important recommendation. The one rule. The thing they tell every team before frameworks, before tooling, before methodology.
”When in doubt, look at your data.”
That’s it. Run a hundred examples through your prototype and read the outputs. With your eyes. Categorize the mistakes. Airbnb states, flatly, that this one habit does more for product quality than any framework or methodology in the entire document.
Screenshot 2026-09-06 at 10.56.05 AM.png
Sit with the comedy of this for a second. We built machines that write like humans, and the frontier technique for managing them is to read like humans. The most sophisticated AI companies on Earth, armed with virtual judges and golden datasets and Cohen’s kappa, all converge on a quality practice that is, at its core, check your work. Your maths teacher ran this methodology in 2003 with a red pen and zero venture funding.
But here’s the four-seconds-later part, the reason this isn’t a dunk: everything expensive in AI quality happens because teams skip this step. Airbnb names the three ghosts precisely. A generic “helpfulness” score looks great, you ship, and it turns out helpfulness wasn’t the thing failing. A prompt tweak silently breaks a dimension nobody was measuring. Or, my favourite, a team builds a gorgeous eval pipeline for metrics that don’t correlate with anything users care about, which is the engineering equivalent of installing a very precise speedometer in a parked car.
The one rule isn’t primitive. It’s load-bearing. Everything else in the document is scaffolding built so that “read your outputs” can survive contact with millions of users.
The courtroom, or: it’s turtles all the way up
Once reading everything by hand stops scaling, Airbnb’s answer is a three-layer system, and the best way to understand it is as a judicial hierarchy.
Screenshot 2026-09-06 at 10.56.16 AM.png
Layer one is the bouncer. Cheap, deterministic, code-based checks. Is this valid JSON? Is it a suspicious length? Does it contain forbidden words? No LLM involved, no nuance required, just a big guy at the door turning away outputs that showed up without a schema. You’d be amazed how much garbage never needs a sophisticated opinion, it just needs to be told it isn’t even wearing the right file format.
Layer two is the judge. For the nuanced stuff, tone, faithfulness, coherence, you use a stronger LLM to grade the output LLM against a written rubric. Yes: an AI marking another AI’s homework. And the rubrics are gloriously specific. Airbnb’s sample readability rubric for listing descriptions has to explicitly instruct the judge to fail any output that uses pompous jargon instead of plain words, the canonical example being a model that would rather say “aquatic recreation area” than “pool.” Somewhere in a datacenter, a machine trained on the sum of human writing had to be formally told, in writing, to stop talking like a hotel brochure that went to law school.
Layer three is the supreme court, and it’s people. Ground truth, high-stakes calls, and, crucially, resolving cases where the automated judges disagree with each other. Humans sit at the top of the appeals process. Which means the whole gleaming stack terminates, inevitably, at a person.
It’s turtles all the way up. The top turtle has a Jira board.
Who judges the judge? A PM named ground truth
Now the part I find genuinely profound, hiding inside a percentage.
An uncalibrated LLM judge, Airbnb warns, is worse than no judge, because it manufactures false confidence at scale. So before you trust the robot, you calibrate it: build a golden set of 50 to 100 examples, and, this is the detail everyone skips, it must include bad examples. You cannot test discernment on a dataset of only good answers, the same way you cannot test a smoke detector by never lighting anything on fire.
Then you run the judge and measure how often it agrees with your human expert, typically a PM or subject-matter expert. In Airbnb’s own walkthrough, the faithfulness judge started at 78% agreement, which wasn’t good enough, and analysis showed the robot was penalizing perfectly accurate paraphrases as lies, a judge so strict it would fail you for saying “car” when the source said “automobile.” Fix the rubric, add examples of the mistake, re-run. 88%. Now you may trust it. Provisionally. Recalibrate as failure modes drift.
Screenshot 2026-09-06 at 10.56.26 AM.png
Read what that loop actually says about the chain of authority. The model is graded by a judge. The judge is graded by its agreement with a human. Which means the ground truth of one of the world’s most advanced AI operations is not a benchmark, not a leaderboard, not an emergent property of scale. It is a specific person’s judgment, written down carefully enough that a machine can imitate it 88% of the time.
An eval stack is not a trust machine. It is a distrust machine, layer upon layer of institutionalized suspicion, and that suspicion, operationalized, is the only way trust has ever actually been manufactured.
There are two more details worth stealing verbatim. Airbnb says 3 to 5 sharp, single-dimension judges beat 20 to 30 noisy ones, no “God evaluators” trying to score everything at once, because a judge asked to assess tone, accuracy, format, and vibes simultaneously assesses none of them. And the instruction I’d frame on the wall: if your human experts disagree on a label, stop. Do not automate. You cannot teach a machine a standard your own team hasn’t agreed on; you’ll just be scaling the argument, at GPU prices.
Oh, and for agents, they check the path, not just the answer, step level, trajectory level, session level, because an agent can reach the right answer via a broken reasoning route, and we all went to school with that guy. He peaked in the exam hall. He does not do well in production.
Steal this, the not-Airbnb edition
Architect hat on. You do not have Airbnb’s infrastructure team, and the wonderful news is that the highest-leverage 80% of this document costs approximately one sprint and a little humility.
4 (1).jpg
Read a hundred of your own outputs, one painful afternoon that replaces months of false confidence. Build a golden set of fifty examples over a weekend, deliberately including failures, which replaces the vibe check you’re currently calling QA. Write three to five small judges, one dimension each, instead of one confused omniscient grader. And when your humans disagree about what “good” means, hold the awkward meeting before writing the automation, because the meeting is cheap and the automated argument is not.
Notice what’s absent from that list: any particular model, any particular framework, any particular vendor. Airbnb’s own closing note says the teams that win with AI aren’t the ones with the best models, they’re the ones with the clearest shared definition of what good looks like. Which, if you read my last piece on where AI’s premium is migrating, should sound familiar. The models are becoming everyone’s. The judgment isn’t.
So yes, laugh at the surface of it, the trillion-dollar industry whose deepest secret is read your work, and get a grown-up to check it. I did. Then notice that this is precisely how every high-trust discipline in history was built. Aviation checklists. Surgical timeouts. Double-entry bookkeeping. Every one of them is somebody’s expensive lesson, written down as a boring rule, followed forever.
AI just got its first few boring rules. That’s not the industry getting less magical.
That’s the industry growing up.
-Hardik
qr-code-buy-me-a-coffee.png
I’ve spent about two decades building data and AI systems, long enough to have sat in the meeting where two experts couldn’t agree on what “good” meant and we automated it anyway. (It went how you think.) I write about AI in production, the parts that are genuinely new, and the parts your school teacher would recognize instantly. If this made you want to go read a hundred of your own model’s outputs and subscribe :)
Sources / further reading: “Eval-driven development: Lessons from evaluating GenAI at scale,” The Airbnb Tech Blog (Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly; July 2026), on Medium at medium.com/airbnb-engineering — read the original for the full rubric example, the evaluation-scenarios table, and the agentic trace-traversal details. Related: Airbnb Engineering on making LLM evaluation fast enough to iterate on (airbnb.tech); Hamel Husain’s writing on error analysis and LLM-as-judge practice; the Pragmatic Engineer’s practical guide to LLM evals (2025). This piece is commentary and summary in my own words; all credit for the underlying practice belongs to the Airbnb teams who did the unglamorous work of writing it down.
