You have eleven versions of the same prompt in a doc, three of them named "final," and no real idea which one scores best on last week's failures. So you tweak it again by hand, read ten outputs, decide it seems better, and ship. That is not optimization. It is guessing with extra steps, and it stops scaling the moment you have more than a couple of prompts and a real eval set.
A prompt optimization tool exists to do that search for you: rewrite the prompt, score it against your data, and keep the version that measurably wins. The trap I see teams fall into is that most tools sold near this problem only manage prompts, they do not optimize them. Versioning, a playground, and A/B testing track the prompts a human writes. Optimization is a separate capability: an algorithm that proposes candidates, scores them against your data, and converges on the best one. On a feature grid both say "prompt testing," and only one of them changes the prompt for you.
Here are the eight questions I would use to tell a real optimizer from a prompt manager with a playground. Score each pass or fail and aim for six.
1. Does it optimize the prompt, or only store and version it?
Versioning tells you which prompt you shipped. It does not give you a better one. Look for a run that takes a starting prompt and returns a measurably improved one, not a diff view of versions you wrote yourself. If "optimization" means you edit and compare manually, it is a registry, not an optimizer.
2. Does it optimize against your own eval, not a generic metric?
A prompt that improves on a vendor's generic "quality" score can still fail your task. The only objective worth optimizing is the one you actually grade on. You should be able to hand it your evaluator, a metric or an LLM judge, and have the optimizer use that score as its objective. Avoid a fixed "improve my prompt" button with no way to plug in your own scoring.
3. Does it use a real search algorithm, or a single rewrite?
Asking a model once to "make this better" is a guess, not a search. Real improvement comes from proposing many candidates and selecting on score across your data. There are a few named families worth knowing so you can compare two tools on the same axis:
- Gradient-based search: compute a "prompt gradient," a critique of why a prompt failed, and edit toward it.
- Genetic or evolutionary search: mutate and select over a population of prompts.
- DSPy-style optimizers: search instructions and few-shot examples together.
- Meta-prompt rewriting: a model that reflects on the prompt and improves it.
If the whole "optimizer" is a single LLM call that rewrites once, it is not searching anything.
4. Can you trust the result?
"The new prompt seems better" is the problem you came to solve, not a result. You need a number on data the optimizer did not train on, plus a record of how it got there: the final score on a held-out set, the per-iteration history, and how many evaluations it ran. Without a score and a trajectory, you cannot tell improvement from a lucky edit or an overfit.
5. Is it open source, so you can inspect and self-host it?
A prompt optimizer rewrites the instructions at the center of your product. A closed "improve my prompt" API asks you to trust a black box with your most important text. Open source lets you read the search strategy, run it in your own environment, and fork it when your needs outgrow the defaults. Look for a permissively licensed optimizer with the algorithms in the open, not hidden behind an API.
6. Is it code-first, running on any model and your own data?
A prompt that optimizes in a playground but cannot be called programmatically is a demo. An optimizer locked to one provider strands you when you switch models. It has to run as code, on whatever you serve, against your dataset, not a fixed benchmark. A GUI-only flow tied to one provider is the thing to avoid.
7. Can you run it in CI/CD, not just once in a session?
You re-optimize when the base model changes, a prompt regresses, or the data shifts. That has to be a pipeline step, not a session someone remembers to run. An optimizer you can only drive by hand drifts out of date the moment a model upgrades. Look for a scriptable entry point you can wire into CI and trigger on a change, so the run is reproducible.
8. Does it also cover the prompt-management workflow your team needs?
Optimization produces a better prompt, but your team still has to version it, review it, and let non-engineers edit safely. This is the honest tradeoff: an optimization engine is not the same thing as a no-code registry for a non-technical team, and a versioning UI is not optimization. If your primary need is a shared registry and a collaboration UI, that is a different category of tool, and the optimizer is what you point at the prompts it holds. Ideally the two hand off cleanly, so a winning prompt lands in versioning and you can compare two versions on your eval before you promote one.
How I would weigh them
Not every question carries the same weight. It depends on what you are actually buying:
- You have more than a few prompts and manual tuning has stopped scaling: questions 1, 2, and 3 decide it. This is the core.
- You need to prove a prompt got better, not just claim it: question 4.
- You need to inspect the search or keep it in your own environment: question 5.
- You re-optimize across model upgrades and want it in CI/CD: questions 6 and 7.
- Your main need is versioning and collaboration: question 8, which points you at a management tool, not an optimizer.
The thing I would not lose sight of: optimization and prompt management are two different jobs, and a tool that is strong at one is usually weak at the other. Most teams need both. Buy them as two decisions, not one, and make sure the optimizer is actually searching against your eval and not just rewriting your prompt once.