Prompt Engineering Without A Test Set Is Just Rewriting

Leader 1 8 78
calendar_today agoschedule1 min read

Most of the prompt discussion I see around here is about wording, so I want to share the part that actually changed my results, which is the measurement around the prompt. If you are shipping anything LLM backed, this is the piece that saves you from silent regressions.

A prompt is code with no tests. That is the whole problem in one line. You write it, it works on the four inputs you tried, it ships, and three weeks later the output drifts on a class of input you never tried. There is no exception to catch, because the model returns something that looks perfectly valid.

What Actually Fixed It

A fixed test set. Twenty real inputs in a file, paired with the outputs I would accept. Run it before and after every prompt edit. It is boring, and it is the only thing that turns prompt work into engineering instead of opinion.

Structured output. Ask for a schema instead of prose. A quality failure you cannot see becomes a parse error you can catch in CI.

Pinned model versions. A prompt hand tuned against one model version is fitted to that version's quirks. When the provider updates it under you, behavior moves and none of your code changed.

The Counterintuitive One

Shorter prompts are not automatically better. Two or three examples of the exact output shape usually beat a paragraph describing that shape, and the extra tokens cost far less than the debugging time you spend without them. The same goes for splitting one long prompt into two chained ones, since the intermediate result gives you somewhere to put a check.

The long version, with the full technique list, the agent specific parts and the token budget math, is in this prompt engineering guide.

What is the smallest evaluation setup that has actually held up for you in production?

🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

I’m a Senior Dev and I’ve Forgotten How to Think Without a Prompt

Karol Modelski - Mar 19

Why Prompt Engineering Is Just an Expensive Way to Be Incompetent

Karol Modelski - May 21

Sovereign Intelligence: The Complete 25,000 Word Blueprint (Download)

Pocket Portfolio - Apr 1

Your AI Doesn't Just Write Tests. It Runs Them Too.

Kevin Martinez - May 12

Architecting a Local-First Hybrid RAG for Finance

Pocket Portfolio - Feb 25
chevron_left
3.4k Points87 Badges
United Statest.co/5LlztlB5C5
82Posts
12Comments
16Connections
Our AI Apps are a self expanding AI SaaS ecosystem used to create the custom web application of your... Show more

Related Jobs

View all jobs →

Commenters (This Week)

2 comments
1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!