From Data Engineering to AI Systems: Hardik Goel’s Developer Journey in the Age of AI
“A prompt is a single turn of hope. A loop is a system.”
Software development has changed dramatically over the past two decades.
The tools have changed. Infrastructure has changed. Development workflows have changed. And now AI is changing not only how developers write software, but also how they think about building and operating software systems.
For Hardik Goel, however, one thing has remained constant: understanding what happens underneath the technology.
His journey began with a fascination for code, machines, and especially the way data is structured and moves through a system. He started his career as a backend and QA engineer before moving into Big Data, data engineering, and machine learning.
Today, his focus increasingly sits at the intersection of software engineering and AI — particularly the systems, guardrails, evaluation techniques, and workflows needed to make AI useful in the real world.
This is Hardik Goel’s Developer Journey.
Where It Started: Understanding What Happens Underneath
Hardik says he has always enjoyed writing code and understanding the dynamics behind machines.
Even today, the underlying data structure remains one of the things that fascinates him most.
“Once you know how data is laid out and how it moves, everything sitting above it starts to make sense.”
He began his career as a backend and QA engineer. Then Big Data and Hadoop started emerging as major areas of interest.
The idea of working with enormous volumes of data immediately caught his attention.
One particular observation helped connect the pieces for him.
Hardik was fascinated by how Netflix and other streaming platforms could work with huge amounts of data while still providing users with an uninterrupted experience.
The machines themselves had not fundamentally changed.
The difference was in how the system around them had been designed.
That realization pushed him deeper into Big Data, eventually leading him into data engineering and machine learning — areas he has continued exploring ever since.
Two Decades of Technology: What Changed and What Didn't
After spending around two decades in technology, Hardik has watched many technologies rise, fall, and evolve.
But he believes many of the things that matter most in software engineering have remained remarkably consistent.
Requirements are still unclear on day one.
The difficult part of a project is often the final ten percent.
Production systems still break because of seemingly ordinary problems — an unexpected schema change, a retry storm, a timezone issue, or a disk running full.
And debugging still involves reading logs and forming a hypothesis.
For Hardik, a good engineer is still the person willing to ask the uncomfortable question during a design review rather than simply being the person who types the fastest.
At the same time, he identifies several technologies and practices that genuinely transformed the daily life of developers.
Version control, code review culture, CI/CD, cloud infrastructure with elastic storage and compute, containers, observability, and iterative development all made a lasting difference.
More recently, AI assistance inside the development environment has become another significant change.
Hardik also sees a recurring pattern in technology adoption.
Technologies that remove toil, shorten feedback loops, and make failure visible tend to survive.
Technologies that promise to remove thinking are much less likely to do so.
The Other Side of Technology Hype
Hardik has also seen plenty of technologies become overextended beyond the problems they were actually suited to solve.
He points to Big Data being adopted as a buzzword before companies knew which questions they were trying to answer.
Data lakes could become data swamps when nobody owned governance.
NoSQL could be presented as a replacement for relational databases rather than as a complement.
He has seen blockchain proposed for almost everything, microservices treated as the default architecture even for very small teams, and serverless presented as an answer to almost every problem.
Now, he sees similar enthusiasm around autonomous agents shipping software end to end without human oversight.
For Hardik, the pattern is straightforward:
“Technology that removes toil, shortens the feedback loop and makes failure visible tends to survive. Technology that promises to remove thinking does not.”
From Writing Code to Thinking About Systems
One of the biggest changes in Hardik's own approach to software has been moving beyond the individual piece of code.
He believes writing individual pieces of code is no longer the primary moat.
Instead, developers need to think about everything surrounding the code.
That includes proper guardrails, Responsible AI, and evaluation techniques such as using an LLM as a judge.
The developer increasingly needs to understand the complete path through the system and define how AI should operate within it.
Hardik describes the evolution in his thinking through a simple shift.
Instead of asking:
“How do I write this function?”
he now thinks about:
“What does this function do to the system at three in the morning when it behaves differently than it did in testing?”
That changes the unit of design.
The contract between components matters.
So do failure modes, fallbacks, and the signals that tell engineers something has gone wrong before the user discovers it.
Despite the rapid progress of AI, Hardik believes we are still some distance away from fully autonomous AI.
When AI Changed the Way He Worked
Hardik has always tried to explore new technology and keep up with advancements in the field.
When AI coding tools became increasingly capable, one immediate benefit stood out: they could reduce the pain of searching across Stack Overflow and countless browser tabs for a solution.
The tools could provide results that were comparable to — and sometimes better than — what he might find through traditional searching.
But the bigger realization came later.
It wasn't simply that AI could generate code.
It was that AI began changing the order of his work.
Hardik noticed that he was reviewing more than he was writing.
More of his time was going into framing the problem and checking the output rather than producing the first draft himself.
That was the point when AI became more than another tool in the toolbox.
It had started changing the developer workflow itself.
Beyond Prompt Engineering: The Loop Is the System
One of Hardik's strongest ideas is that developers can become too focused on writing better prompts.
The goal shouldn't simply be to find the perfect wording that convinces an AI model to produce the correct answer.
Instead, developers should design systems where mistakes are cheap, visible, measurable, and recoverable.
Hardik puts it this way:
“A prompt is a single turn of hope. A loop is a system.”
A reliable AI workflow needs much more than a prompt.
It needs a definition of what good output actually means for the task.
It needs an evaluation set based on real failure cases from the domain.
It needs automated checks that run against the output.
It needs a retry path that is genuinely different from the first attempt rather than simply sending the same prompt again.
It needs logging so developers can identify drift before users experience it.
And it needs a clear point where the loop stops and a human signs off.
Once these elements are in place, Hardik argues, the prompt itself becomes less important than many people assume.
A mediocre prompt inside a well-designed loop can be more useful than a carefully crafted prompt with nothing around it.
The loop also has another advantage: it can continue working even when the underlying model changes.
What Happens to the Developer?
If AI increasingly handles parts of the inner coding loop — writing, testing, debugging, and iterating — the role of the developer begins to expand.
Hardik agrees that developers may increasingly take on responsibilities associated with architecture, product thinking, reviewing, and system design.
He also points to roles such as the Forward Deployed Engineer (FDE), where an engineer works directly with clients, understands the business problem, contributes to productization, and architects and builds the technology around it.
The important change, however, is broader than a new job title.
Hardik believes the set of responsibilities carried by developers is becoming wider rather than narrower.
The developer may need to decide what should not be built.
They may need to own the quality bar for output they did not personally type.
And they may be accountable for a system whose individual components they did not write line by line.
For Hardik, that last responsibility may become particularly important when thinking about seniority in software engineering.
The Human in the Loop
As AI systems become more capable, Hardik believes the human role becomes even more important in areas involving direction, context, priorities, and consequences.
He is concerned about a future in which people place so much trust in AI systems that they lose their own identity and judgment in the process.
That is where he sees Responsible AI becoming important.
Human knowledge about the direction a project should take — based on experience and independent thinking — remains valuable.
For Hardik, there are several questions he does not expect to simply hand over to AI:
- Which problem deserves to be solved?
- What trade-offs are acceptable?
- What is acceptable to get wrong?
- What is not acceptable to get wrong?
- What did the customer actually mean rather than simply write in a ticket?
At the same time, Hardik doesn't believe all expert judgment will remain exclusively human.
Some of what people call judgment is really pattern recognition developed through repeated experience.
Machines can become very good at that kind of pattern recall.
The distinction, for him, is judgment that carries consequences.
When a decision goes wrong, someone still needs to stand behind it.
Building Reliable AI Systems
For Hardik, one of the most important lessons of working with AI systems is that something looking impressive in a demo doesn't necessarily mean it is ready for production.
A production AI system needs checks and balances.
He places particular importance on observability, governance, security, and proper monitoring.
The system should provide enough visibility for engineers to understand what it is doing instead of relying on users to discover problems first.
For Hardik, reliability also means understanding the kinds of responses a model can produce and the degree of deviation it can take.
That deviation needs to be contained and understood.
Moving from an AI prototype to a real system also introduces a large amount of unglamorous engineering work.
Data changes.
Models don't automatically understand those changes.
Evaluation sets may not represent real production traffic.
And assumptions that seemed reasonable during a prototype can become significant problems once the system is exposed to real users.
For Hardik, building reliable AI therefore requires many of the engineering disciplines that have always mattered in software — with additional attention to model behavior, evaluation, monitoring, governance, and security.
The Developer's Job Is Changing, Not Disappearing
Hardik's journey offers an interesting perspective on the current AI transition.
He doesn't see the future simply as humans writing less code while machines write more.
Instead, he sees the developer's responsibility expanding.
The developer increasingly becomes the person who understands the problem, designs the system, defines the quality bar, creates the feedback loops, evaluates AI output, and takes responsibility for what ultimately reaches users.
The syntax may become easier.
The first draft may become faster.
But the surrounding engineering problem remains.
And perhaps that is the part of software development that has changed the least over the past two decades.
Lessons From Twenty Years in Technology
Looking across Hardik's career, there is a clear thread connecting his early interest in data structures with his current interest in AI systems.
The technology keeps changing.
The underlying questions remain remarkably similar.
How does data move?
What happens when a component fails?
How do you know whether something is working?
How do you detect problems before users do?
What should the system do when reality differs from the assumptions made during development?
AI adds a new layer to those questions, but it doesn't make them disappear.
If anything, it makes understanding the system around the technology even more important.
Hardik Goel's Perspective on the AI Era
Hardik's Developer Journey sits at an interesting point between two generations of software development.
He began in an era where developers wrote and debugged code largely by hand.
He moved through the rise of Big Data, data engineering, machine learning, cloud infrastructure, CI/CD, containers, and observability.
Now he is working through another major shift: AI-assisted and agentic software development.
His central message is not that developers should stop using AI.
He is already using it himself.
The more important question is how developers use it.
For Hardik, the future isn't simply about generating more code.
It is about building better systems around the intelligence we now have access to — systems that can evaluate, observe, recover, and involve humans when judgment and responsibility matter.
And that brings the conversation back to where his own journey began: understanding what happens underneath the surface.
About Hardik Goel
Hardik Goel is a technology professional with a background spanning backend engineering, QA, Big Data, data engineering, machine learning, and AI systems.
His work and writing explore software engineering, AI-assisted development, agentic workflows, Responsible AI, and the changing role of developers as AI becomes part of the software development process.
His perspective is shaped by years of working with systems where understanding the underlying data, architecture, failure modes, and production behavior matters as much as writing the code itself.
CoderLegion Developer Stories
This story is part of CoderLegion Developer Stories, a series featuring developers, engineers, founders, and builders sharing their experiences, lessons, projects, and perspectives on technology.
Interviewed and edited by Mehadi Hasan, Community & Editorial Team at CoderLegion.
