An end-to-end test fails.
The report is red.
Someone asks, “Is it a bug?”
Maybe.
But at that moment, all we really know is that the test didn't observe the expected behavior.
That distinction sounds small, but it changed the way I investigate failures.
When I started working more deeply with automation, I gradually learned that a failed test is not a diagnosis. It's the beginning of an investigation.
The same failure can have very different causes
Imagine an automated checkout test that expects an order to be created.
The test fails because the order confirmation never appears.
The cause could be an application defect. But it could also be a failed API request, unavailable service, incorrect test data, expired credentials, a deployment issue, an environment configuration problem, or a timing issue in the test.
All of those can produce a similar result in the report.
If we immediately label the failure as a product bug, we may send developers looking in the wrong place.
If we immediately label it as a flaky test, we may hide a real defect.
The first job is to find out which one it is.
Start with the failure, not the assumption
My first step is to understand exactly what failed.
Was it an assertion? A timeout? A missing element? An HTTP error? Did the test fail before the application was even reached?
The details matter.
A test that receives a 500 response is different from a test that cannot connect to the server. A timeout waiting for a button is different from a timeout waiting for an API response.
I also want to know whether the failure is reproducible.
Does it happen every time? Only in CI? Only in one environment? Only when tests run in parallel? Did it start after a particular deployment?
Those questions help narrow the problem before anyone starts changing code.
Check whether the application is actually healthy
One of the most useful habits I developed was to stop looking only at the test report.
If an API test fails, I want to know whether the API is responding outside the test.
If an end-to-end test fails, I want to know whether the underlying service is healthy.
A simple request, a health endpoint, or a look at the service logs can reveal that the test is only reporting a symptom of a larger problem.
For example, a login test might fail because the authentication service is unavailable.
Changing the Playwright locator won't fix that.
Neither will adding another ten seconds to the timeout.
Environment differences matter
I remember how confusing DEV, QA, STG, and PROD were when I first entered QA.
Today, environment differences are one of the first things I consider during an investigation.
The same code can behave differently because of configuration, data, dependencies, feature flags, external integrations, or infrastructure.
If a test passes locally but fails in CI, I want to compare the conditions.
Are the same versions running? Is the test using the same configuration? Are required services available? Is the data equivalent? Are there resource constraints? Is there a difference in browser or runtime versions?
“Works on my machine” isn't a diagnosis either.
It's another clue.
Logs connect the test to the system
This is where my long-standing interest in Linux and Bash became especially useful.
I don't need to understand every internal detail of a service to start investigating it.
But I do need to know how to find relevant information.
Application logs, container logs, system services, network responses, and CI output can help connect the failure in the test to what actually happened in the system.
The important skill isn't memorizing a hundred commands.
It's knowing what question you're trying to answer.
Did the request reach the service? Did the service reject it? Did a dependency fail? Was the process restarted? Was there an error at the same time as the test failure?
When the timestamps and evidence line up, the investigation becomes much more useful.
Don't fix flakiness by hiding it
Flaky tests are frustrating, especially when they slow down a team.
But adding retries or longer waits without understanding the cause can make the situation worse.
Sometimes a retry is a reasonable mitigation. But it should not become a substitute for investigating why the test is unreliable.
Is the application genuinely slow? Is the test waiting for the wrong condition? Is shared test data causing interference? Are tests running in parallel without proper isolation? Is the environment overloaded?
A test that passes on the third attempt may still be telling you something important.
The question is whether you understand what that something is.
Test data deserves more attention than it gets
Test data is another common source of misleading failures.
An account might already exist. A record may have been modified by another test. An external system may have changed state. A cleanup step may not have run.
A test can be technically correct and still fail because its assumptions about data are no longer true.
This is why I increasingly value predictable setup, isolation, cleanup, and clear ownership of test data.
Reliable automation isn't only about good assertions.
It's also about controlling the conditions under which those assertions run.
Look at the change that happened before the failure
When a previously stable test starts failing, I want to know what changed.
A new commit? A dependency update? A deployment? A configuration change? A database migration? A new feature flag?
The recent change isn't automatically the cause, but it provides useful context.
This is also where understanding CI/CD becomes valuable for QA.
A pipeline isn't just a place where tests run. It contains information about the build, dependencies, environment, and changes that produced the result.
The more of that context you can read, the less you have to guess.
A useful investigation produces more than “fixed”
Eventually, the failure is resolved.
Maybe it was an application bug. Maybe the test needed a better wait condition. Maybe a service was unavailable. Maybe the data setup was unreliable.
But I think a good investigation should leave behind something more useful than a green rerun.
What was the actual cause? How did we confirm it? Could the same failure happen again? Can we improve the test, environment, monitoring, or process so it's easier to diagnose next time?
A failure is an opportunity to improve the system's ability to explain itself.
That matters to QA, DevOps, and everyone responsible for reliable delivery.
The broader lesson
The deeper I moved into automation, the more I realized that debugging requires an expanding field of view.
Sometimes the answer is in the test.
Sometimes it's in the application.
Sometimes it's in the environment, CI, infrastructure, or deployment history.
You don't need to be an expert in every layer to investigate effectively.
But you do need to be willing to look beyond the layer where the failure first appeared.
A red test is useful evidence.
It tells us something didn't go as expected.
The real engineering work is finding out why.
This is the second article in my From QA to DevOps / Release Engineering series.
When an automated test fails in your team, what's the first thing you check outside the test report?
