Amid our excitement over AI, we’ve fallen back into old habits, and this should prompt us to reset our approach.
It’s complicated. While the industry faces crucial debates over intellectual property and the environmental impacts of AI, there is a more fundamental issue: How do we use these tools to deliver high-quality work that benefits our organization and the community it serves?
This article doesn’t address ethical or moral concerns but instead focuses on the growing pressure from leaders who want to introduce AI tools to increase productivity.
We can use the 2024 Accelerate State of DevOps report to guide us. In the report, the research team has taken a deep dive into the use of AI to get a baseline on its adoption and impact. We’ll use these insights to explain how we’ve fallen back into old habits, and this should prompt us to reset our approach and find a more impactful path forward.
AI Adoption
The majority of organizations are increasing AI adoption. More than 80% of organizations are keen to add AI in some form, with code assistance, information summaries and documentation presenting the most sought-after opportunities.
Code assistance is highly desired for tasks like writing code, understanding existing code and optimizing code. Code editors have long utilized algorithmic refactoring, machine learning and AI assistance, so AI adoption for coding purposes is a natural evolution of existing use cases.
Using AI to analyze information and data is less well-established, though it has a precedent in the ubiquitous TL;DR (too long; didn’t read) summary added by kind folks to the start of their long-form writing.
Productivity Increases with AI
Most survey respondents (over 70%) reported productivity gains when using AI. People feel like they can get work done faster when they have its assistance. My long-running problem with developer productivity is that it’s often not a useful measure.
You can assess your personal productivity because it’s a feeling rather than a number. You don’t feel productive when dealing with busy work or handling constant interruptions. When you get a solid chunk of time to complete a task, you feel great. If an organization is interested in this kind of productivity, it should check in on employee satisfaction because people tend to be more satisfied when they can get things done.
The State of DevOps report confirms this problem, as the high ratings for AI-driven productivity aren’t reducing toil work or improving software delivery performance, which we’ve long held to be a solid way for development teams to contribute to the organization’s goals.
A 25% increase in AI adoption brings modest increases in flow, satisfaction and productivity. It also decreases burnout. However, it doesn’t yet bring a meaningful reduction in toil work and doesn’t allow more time to be spent on valuable work.
AI Lowers Software Delivery Performance
The same 25% increase in AI adoption that made folks feel more productive hurts software delivery performance.
AI reduced throughput by 1.5% and stability by 7.2%. If perception of code quality is increasing with AI assistance but stability is dropping, it indicates a mismatch between how well we think it performs a task and how well it actually does.
Given the intense focus on increasing the speed of coding, we’re likely seeing suboptimization on a massive scale. Writing code is rarely the bottleneck for feature development. Speeding up the code itself is less valuable if you aren’t catching the bugs it introduces with automated tests. It also fails to address the broader software delivery system or guarantee your features are useful to users.
If you aren’t working at the constraint, your optimizations don’t improve throughput. In many cases, optimizing away from the constraint harms the end-to-end system. Speeding up coding tasks has no good outcome unless you have smoothed the flow of everything that comes after.
So, how do we use all these new tools well?
Getting Things in the Right Order
The rush to adopt AI has led us back to a troublesome old pattern of buying a solution before we understand which problem it will solve. This means searching for a problem retrospectively, often in vain.
A more successful approach to AI adoption would be to start with your problems, decide which one is holding you back the most and look for ways to solve it. This would leave all possible solutions open to you.
Starting with the problem to be solved means you can choose the most impactful solution. You can eliminate unnecessary work or find a way to automate it using an existing part of your toolchain. If you need to bring in a new tool, you can select from all available options whether they use AI or not.
If the solution turns out to be AI-based, I’ll bet it will be more successful if the problem leads you to the AI solution rather than the other way around.