Most teams optimize prompts by hand: change a line, rerun, eyeball the output, repeat. It works until you have more than a handful of cases, then it stops scaling. A prompt optimizer replaces that with a scored search you run on demand, and you do not need a vendor SDK to build one. The core loop is four repeatable steps: generate candidate prompts, score them against a fixed set, select and mutate the winners, and check for convergence. There is a longer build guide with the full code if you want it.
The reason this beats manual editing is not cleverness, it is memory. A human can eyeball five outputs, not five hundred, and keeps no record of which past edit caused which regression. An optimizer stores every candidate and its score, so it never reintroduces a change it already disproved. That is the whole advantage: a search with perfect recall. One caveat worth stating up front: a research method like textual gradients or genetic search only describes the candidate-generation step. The golden set, the scorer, and the stopping rule are the engineering that decides whether any of it works, and they are the parts most guides skip.
Step 1: build the golden set first
The golden set is what the optimizer runs against every round, so it has to be real inputs your prompt will actually see, pulled from production logs or support tickets, not synthetic edge cases invented in isolation.
- Size it at 30 to 80 examples to start. Fewer and the optimizer cannot tell a better prompt from noise; more than a few hundred and each round gets expensive without adding signal.
- Split it 70/30. The optimizer searches on 70 percent and never sees the 30 percent held out, which you check after a round to confirm the gain generalizes instead of fitting the examples it stared at.
- Label each example with what a correct output is: an exact answer, a required set of facts, or a rubric a judge can apply. "Sounds good" produces a scorer that cannot discriminate.
- Cover the messy inputs once you clear the 30-example floor: short queries, ambiguous phrasing, edge-case formatting. A set built from one easy failure teaches the optimizer to fix that one thing.
- Refresh it. Add new failures the week you find them, before you forget the conditions that caused them.
The scorer is the piece everything depends on
You need a function that turns an output into a number, consistently, across hundreds of runs. Two approaches cover most tasks.
Deterministic scoring works when the task has a checkable answer: exact match, JSON schema validity, a regex, or a reference metric like ROUGE or embedding similarity. Fast, cheap, same score every time, which makes debugging the optimizer itself much easier. LLM-as-judge scoring covers what deterministic cannot: open-ended writing, summarization quality, tone, faithfulness. Write a rubric, hand it to a judge model with the candidate output, and run the judge at low temperature with explicit criteria or the scores drift between runs.
Whichever you pick, sanity-check it against a handful of outputs you have already judged by hand. If the scorer disagrees with you on obvious cases, fix the scorer before you touch the optimizer, because every downstream decision depends on that number.
Step 2: generate candidates
With a set and a scorer in place, produce new variants. Three approaches cover most builds, and you can mix them in one loop.
- Instruction rewriting. Feed a stronger model your current prompt plus a sample of its failing outputs, and ask for a revision that would have avoided them. This is the meta-prompt idea behind ProTeGi: the failures act as a textual gradient the rewriter edits against, so it gets concrete evidence instead of guessing blind.
- Few-shot selection. Instead of rewriting the instruction, search over which examples to include as demonstrations. This is DSPy's MIPRO mechanism, and it matters more than people expect: swapping which three examples you show can shift accuracy as much as rewording the instruction.
- Mutation-based search. Take the current best prompt and apply small structural edits (reorder sections, adjust constraints, vary output-format instructions), testing each independently. This is closer to GEPA's evolutionary search and works once you already have a strong starting prompt and want incremental gains.
Generate three to eight candidates per round, not one. A single new prompt is one data point; a batch lets selection compare and separates real improvement from lucky variance. Record why each candidate was generated, not just its score, so you know later which generation strategy to lean on.
Step 3: select, keep the champion, loop
Keep the top one or two performers, discard the rest, feed the survivors back into generation. The loop: score the seed on the golden set, generate a batch, score every candidate on the same set, keep the best if it beats the current champion, repeat with the new champion as seed, and stop when the score plateaus across two consecutive rounds. Always carry the best-so-far forward as a fallback so you never lose a prior gain, and log every candidate, score, and round, because that log is the only way to tell whether a later regression came from the set, the scorer, or a genuinely worse prompt.
For a single metric, keeping the top scorer each round is fine. For competing objectives (faithfulness and brevity), keep several candidates that each lead on a different dimension instead of collapsing to one average too early, the same logic as Pareto-style search.
A worked example: a seed prompt scores 0.61 on a summarization set with an embedding metric. Round one produces five candidates, three from rewriting and two from few-shot selection, and the best hits 0.68. That becomes the seed, round two mutates it and the best hits 0.70, round three produces nothing above 0.70, so you have converged.
Step 4: know when to stop
Convergence is a signal, not a round count. Track the champion's score and stop once it plateaus for two or three rounds. Running past that does not help: additional rounds mostly search noise, and the risk of drifting toward a prompt that games your specific examples goes up, not down. Before promoting a converged prompt, run it against the held-out split. If validation tracks training, the gain is real. If validation lags well behind, the optimizer overfit, so grow the set or tighten the scorer before trusting it.
The failure modes nobody documents
- Overfitting a small set. A narrow 30-example set lets the optimizer tune to those exact phrasings. The fix is the held-out split, checked every round. If validation and training diverge by more than a few points, stop and grow the set.
- Reward hacking the metric. If the scorer rewards length, the optimizer pads with filler; if it rewards keywords, it stuffs them. Spot-check the top scorer by hand every round rather than trusting the number, and watch for score spikes with no visible quality gain.
- Drift across model versions. A prompt tuned to one checkpoint quietly underperforms after a silent model update. Re-run the golden set against the live prompt on a schedule, the way you would a regression suite, so a silent regression shows up before a customer finds it.
The parts that separate a working optimizer from a fragile one are not the search algorithm. They are the discipline around it: a held-out split you actually check, a scorer you have sanity-tested against your own judgment, and a habit of re-running the set after model updates. Get those three right and the rest of the loop is mechanical, and building it once teaches you more about your prompt's failure modes than another week of hand-editing.
