Anthropic just published research that should make every developer uncomfortable. When AI produces code, documents, or working apps, users stop checking the work. They get more precise about what they ask for up front — but less rigorous about evaluating what comes back.
The company analyzed 9,830 Claude conversations from January 2026 and built what it calls the AI Fluency Index: a baseline measurement of 11 observable behaviors that indicate how effectively people collaborate with AI. The findings are practical and specific. And if you write code with AI — which, at this point, most of us do — the data points directly at habits worth changing.
What They Measured
Anthropic worked with Professors Rick Dakan and Joseph Feller to develop a framework of 24 behaviors representing effective human-AI collaboration. Eleven are directly observable in chat conversations — including iteration, clarifying goals, specifying formats, questioning reasoning, identifying missing context, and fact-checking. Each conversation was scored for the presence or absence of each behavior.
The Iteration Effect
The strongest signal in the data: iteration predicts everything else.
85.7% of conversations showed iteration and refinement — users building on previous responses rather than accepting the first answer. These conversations exhibited 2.67 additional fluency behaviors on average, roughly double the 1.33 in non-iterative conversations.
The effect is most pronounced for critical evaluation. Iterative conversations were 5.6 times more likely to include users questioning the AI's reasoning and 4 times more likely to surface missing context.
This quantifies something experienced engineers already sense. The first response from an AI coding tool is a draft. The developers who get the most value are the ones who push back, ask follow-up questions, and refine through multiple rounds.
The Polished Output Trap
Here's where the data gets uncomfortable. 12.3% of conversations involved artifacts — code, documents, interactive tools, and other concrete outputs. In these conversations, users were significantly more directive up front. They clarified goals (+14.7 percentage points compared to non-artifact conversations), specified formats (+14.5pp), and provided examples (+13.4pp).
But that care didn't carry through to evaluation. Users were less likely to identify missing context (-5.2pp), check facts (-3.7pp), or question the reasoning behind the output (-3.1pp).
Anthropic calls this "more directive but less evaluative." For developers: when Claude hands you a working component or a clean function, your instinct is to ship it. The output looks finished, so you treat it as finished. But Anthropic's own Economic Index found that the most complex tasks are where AI struggles most.
Developers might be evaluating artifacts outside the conversation — running tests, doing code review. Anthropic can't observe that. But a 5.2 percentage point drop in identifying missing context is worth paying attention to.
Three Habits the Data Supports
Anthropic's report identifies three practical behaviors where most users have room to improve. All three are directly relevant to developers.
Stay in the conversation. Don't accept the first response. Ask follow-up questions. Push back on parts that don't look right. The data are clear: iteration is the strongest predictor of all other fluency behaviors. Treat every AI response as a draft you're reviewing, not an answer you're accepting.
Question polished outputs specifically. When AI gives you code that runs, that's when to pause and ask: Is this handling edge cases? What assumptions is it making? What's missing? The research shows this is exactly the moment people stop asking those questions. Build the habit of treating functional output as the starting point for review, not the endpoint.
Set the terms of the collaboration. Only 30% of conversations included users telling the AI how they wanted it to interact with them. Add instructions like: "Push back if my assumptions are wrong." "Walk me through your reasoning before giving me the answer." "Tell me what you're uncertain about." These meta-instructions change the dynamic of the entire conversation. They turn a passive code generator into an active collaborator that surfaces its own limitations.
What This Means Going Forward
Anthropic is treating this as a baseline and plans cohort analyses comparing new users with experienced ones, plus studies exploring whether encouraging iteration actually leads to better evaluation.
For developers, the takeaway is concrete. The gap between using AI tools and using them well isn't about prompt engineering tricks. It's about collaboration habits. Iterate. Question the output that looks most finished. Tell the AI what kind of collaborator you want it to be.
The AI Fluency Index gives us a number for what the best developers already practice: the first response is never the final answer.
