Most AI engineering roadmaps are lists of topics: prompting, RAG, agents, evals. You read the list, study something about each item, and three months later you still can't tell whether you know retrieval-augmented generation or have only seen it explained. A topic list has no exit condition.
This plan replaces topics with checkpoints. Each one is a small Python program plus a check that fails until you understand the idea behind it. You move on when the check passes, not when the calendar says so. The six checkpoints follow the order in which problems show up when you add an LLM feature to a real product: one call, then output your code can trust, then search over your own data, then answers grounded in that data, then measurement, then a model that takes actions.
The examples use Python and the Anthropic Python SDK (pip install anthropic). The concepts carry over to any provider; only the field names change.
Before you start
You need Python 3.10 or newer, an API key in the ANTHROPIC_API_KEY environment variable, and a folder of your own text: internal docs, notes, the README files of a project you work on. Thirty to a hundred pages is enough. Pick material you know well, because you will be judging whether answers are right, and you can only do that on content you understand.
Decide up front how much you are willing to spend on the whole plan, and track it with the cost function from the first checkpoint. All the code goes into one file, in the order shown, so later checkpoints reuse earlier functions. Every test in it runs without an API key.
Each checkpoint has three parts: what you build, the check that tells you you're done, and a question you should be able to answer afterwards without looking anything up.
Checkpoint 1: one call, and what it really costs
The first program sends one message and returns two things: the text and its cost in dollars. Cost belongs in the very first program because every later decision, from chunk size to how many agent steps you allow, is also a cost decision.
import anthropic
MODEL = "claude-opus-5-5"
# Base prices in USD per million tokens, from the Claude pricing page on 19 Nov 2026.
PRICE_PER_MTOK = {"input": 4.00, "output": 20.00}
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environment
def cost_usd(usage) -> float:
return (usage.input_tokens * PRICE_PER_MTOK["input"]
+ usage.output_tokens * PRICE_PER_MTOK["output"]) / 1_000_000
def text_of(response) -> str:
if response.stop_reason == "refusal":
raise RuntimeError("the model declined this request")
if response.stop_reason == "max_tokens":
raise RuntimeError("the answer was cut off; raise max_tokens")
return "".join(b.text for b in response.content if b.type == "text")
def ask(prompt: str, max_tokens: int = 16000) -> tuple[str, float]:
response = client.messages.create(
model=MODEL,
max_tokens=max_tokens,
messages=[{"role": "user", "content": prompt}],
)
return text_of(response), cost_usd(response.usage)
if __name__ == "__main__":
answer_text, cost = ask("In two sentences: what is a context window?")
print(answer_text)
print(f"${cost:.5f}")
Three details in this code teach more than the call itself.
response.content is a list of blocks, not a string. Current models can return thinking blocks before the text, so the code keeps only blocks of type text instead of reading content[0]. Code that assumes the first block is the answer breaks on the first response that starts with something else.
stop_reason says why the response ended. end_turn is the normal case. max_tokens means the answer was cut off, and passing a truncated answer downstream as if it were complete is one of the most common silent bugs in LLM features. refusal means the model declined, and the text is not an answer either. The Messages API reference lists the other values.
The cost comes from usage, not from the length of what you print. Claude Opus 5.5 runs with adaptive thinking that can't be switched off, and thinking is billed even when the response doesn't show it, so a short visible answer can still have a long bill. The prices in the code are the base rates on the pricing page on 19 November 2026; check them again before relying on them, and remember that cached input is priced differently once you start using prompt caching. To estimate the input side before sending anything, the API also has a token counting endpoint.
The check. Run the same prompt twenty times and log input tokens, output tokens and cost. The input is identical on every run; the output is not. You pass when you can explain the spread and predict, within a factor of two, what a thousand calls of your real prompt would cost.
The question. Why does a longer system prompt cost money on every call, even when the answer is short?
Checkpoint 2: output your code can trust
Next, ask for data instead of prose: fields your program reads. Structured outputs constrain the response to a JSON schema, so you no longer parse half-broken JSON. That solves syntax. It does not solve meaning, because a response can match the schema perfectly and still be impossible.
import json
from datetime import date
INVOICE_SCHEMA = {
"type": "object",
"properties": {
"supplier": {"type": "string"},
"issue_date": {"type": "string", "description": "YYYY-MM-DD"},
"due_date": {"type": "string", "description": "YYYY-MM-DD"},
"total": {"type": "number"},
"currency": {"type": "string", "enum": ["EUR", "USD", "RON"]},
},
"required": ["supplier", "issue_date", "due_date", "total", "currency"],
"additionalProperties": False,
}
def extract(document: str) -> dict:
response = client.messages.create(
model=MODEL,
max_tokens=16000,
messages=[{"role": "user", "content": f"Extract the invoice fields.\n\n{document}"}],
output_config={"format": {"type": "json_schema", "schema": INVOICE_SCHEMA}},
)
return json.loads(text_of(response))
def check_invoice(inv: dict) -> list[str]:
"""Rules the schema cannot express. Returns the problems found; empty means usable."""
try:
issued = date.fromisoformat(inv["issue_date"])
due = date.fromisoformat(inv["due_date"])
except ValueError:
return ["dates are not in YYYY-MM-DD format"]
problems = []
if due < issued:
problems.append("due date is before the issue date")
if inv["total"] <= 0:
problems.append("total is not positive (credit note?)")
if not inv["supplier"].strip():
problems.append("supplier is empty")
return problems
def test_schema_valid_but_impossible():
inv = {"supplier": "Acme", "issue_date": "2026-11-10",
"due_date": "2026-10-01", "total": 120.0, "currency": "EUR"}
assert check_invoice(inv) == ["due date is before the issue date"]
The schema guarantees that total is a number and due_date is a string. It cannot know that a due date before the issue date is impossible, or that a negative total is a credit note your pipeline doesn't handle. Those rules live in your code, and every response passes through them before anything downstream sees it.
The check. The test passes without an API call: it feeds a schema-valid but impossible invoice to the validator and expects exactly one problem. Then run ten real documents through extract and write down every field the model got wrong. You pass when each failure is either caught by a rule in check_invoice or recorded as a known limitation.
The question. Which of your fields can be checked by a rule, and which only by a person?
Checkpoint 3: retrieval you can measure
Retrieval-augmented generation, answering from your documents instead of from what the model remembers, starts as a search problem, and search can be measured before any model is involved. Build the simplest serious baseline first: BM25, the ranking function from classic information retrieval described in Robertson and Zaragoza's review. It needs no embeddings, no vector database and no API key.
import math
import re
from collections import Counter
def tokenize(text: str) -> list[str]:
return re.findall(r"\w+", text.lower())
def chunk(doc_id: str, text: str, max_chars: int = 1200) -> list[dict]:
"""Split on blank lines, then pack paragraphs into chunks of at most max_chars."""
chunks, current = [], ""
for para in re.split(r"\n\s*\n", text):
if current and len(current) + len(para) > max_chars:
chunks.append(current)
current = ""
current = f"{current}\n\n{para}".strip()
if current:
chunks.append(current)
return [{"id": f"{doc_id}#{i}", "text": c} for i, c in enumerate(chunks)]
class BM25:
def __init__(self, chunks: list[dict], k1: float = 1.5, b: float = 0.75):
self.chunks, self.k1, self.b = chunks, k1, b
self.tf = [Counter(tokenize(c["text"])) for c in chunks]
self.lengths = [sum(tf.values()) for tf in self.tf]
self.avg_len = sum(self.lengths) / len(self.lengths)
n = len(chunks)
df = Counter(term for tf in self.tf for term in tf)
self.idf = {t: math.log(1 + (n - d + 0.5) / (d + 0.5)) for t, d in df.items()}
def search(self, query: str, k: int = 5) -> list[dict]:
scored = []
for i, tf in enumerate(self.tf):
norm = self.k1 * (1 - self.b + self.b * self.lengths[i] / self.avg_len)
score = sum(self.idf[t] * tf[t] * (self.k1 + 1) / (tf[t] + norm)
for t in tokenize(query) if t in tf)
scored.append((score, i))
top = sorted(scored, reverse=True)[:k]
return [self.chunks[i] for score, i in top if score > 0]
def recall_at_k(index: BM25, labelled: list[tuple[str, str]], k: int = 3) -> float:
"""labelled = [(question, a short phrase from the passage that answers it), ...]
Labelling by phrase instead of chunk id keeps the test valid when chunking changes."""
hits = sum(1 for q, phrase in labelled
if any(phrase.lower() in c["text"].lower() for c in index.search(q, k)))
return hits / len(labelled)
Then write the part most tutorials skip: twenty questions about your documents, each paired with a short phrase from the passage that answers it. That list is the most valuable artifact in the whole plan, and you will reuse it in the next two checkpoints. Labelling by phrase rather than by chunk id matters more than it looks: the first thing you'll want to tune is chunk size, and labels tied to chunk ids would silently stop matching.
The check. Compute recall_at_k with k = 3 on your twenty questions. Whatever the number is, keep it: it's the baseline every later change has to beat. Then add five questions phrased with words that don't appear in the documents, the way a customer would ask. BM25 matches exact words (this version doesn't even reduce words to their stem, so a query for "refund" won't match "Refunds"), so you'll watch it miss, and you'll know exactly what embeddings or hybrid search are supposed to fix, with a test that shows whether they did.
The question. When a question fails, is the answer in a chunk ranked too low, split across two chunks, or missing from your documents altogether?
Checkpoint 4: answers you can verify
Now give the retrieved chunks to the model and ask for an answer. The usual instruction is "answer only from the context", and the problem is that you can't see whether it did. So make the answer checkable: require the id of each source the model relied on and the exact sentence it used, then verify both in code.
ANSWER_SCHEMA = {
"type": "object",
"properties": {
"answer": {"type": "string"},
"found": {"type": "boolean"},
"citations": {
"type": "array",
"items": {
"type": "object",
"properties": {"chunk_id": {"type": "string"}, "quote": {"type": "string"}},
"required": ["chunk_id", "quote"],
"additionalProperties": False,
},
},
},
"required": ["answer", "found", "citations"],
"additionalProperties": False,
}
SYSTEM = ("Answer only from the sources. For every claim, give the source id and copy the "
"exact sentence you relied on into 'quote'. If the sources do not contain the "
"answer, set found to false and leave citations empty.")
def answer(question: str, index: BM25, k: int = 5) -> tuple[dict, list[dict]]:
sources = index.search(question, k)
context = "\n\n".join(f"[{c['id']}]\n{c['text']}" for c in sources)
response = client.messages.create(
model=MODEL,
max_tokens=16000,
system=SYSTEM,
messages=[{"role": "user", "content": f"Sources:\n{context}\n\nQuestion: {question}"}],
output_config={"format": {"type": "json_schema", "schema": ANSWER_SCHEMA}},
)
return json.loads(text_of(response)), sources
def verify_citations(result: dict, sources: list[dict]) -> list[str]:
if result["found"] and not result["citations"]:
return ["claims an answer but cites nothing"]
by_id = {c["id"]: " ".join(c["text"].split()) for c in sources}
problems = []
for cit in result["citations"]:
text = by_id.get(cit["chunk_id"])
if text is None:
problems.append(f"cites {cit['chunk_id']}, which was not sent")
elif " ".join(cit["quote"].split()) not in text:
problems.append(f"quote not found in {cit['chunk_id']}")
return problems
def test_invented_quote_is_caught():
sources = [{"id": "faq#0", "text": "Refunds are possible within 14 days of purchase."}]
result = {"answer": "Refunds within 30 days.", "found": True,
"citations": [{"chunk_id": "faq#0", "quote": "Refunds are possible within 30 days."}]}
assert verify_citations(result, sources) == ["quote not found in faq#0"]
The check is mechanical: the cited chunk must be one you actually sent, and the quote must appear in it word for word, ignoring whitespace. A quote that isn't there is the clearest signal of an invented source you will get. It won't catch every wrong answer, since the model can quote a real sentence and still draw the wrong conclusion from it, but it turns "the model sometimes makes things up" into a number you can count. The idea of pairing a retriever with a generator comes from the original RAG paper by Lewis et al.; the citation check is the cheapest way to see whether your version holds.
The check. Run your twenty questions and count how many answers pass verify_citations. For questions whose answer isn't in your documents, confirm the model returns found: false instead of citing something nearby. You pass when you can name the most common failure in your own data.
The question. Should an answer that fails the citation check be hidden, retried or shown with a warning, and who decides?
Checkpoint 5: an eval that catches a bad change
So far everything ran by hand. The next change you make, whether a new prompt, a different chunk size or another model, will improve some answers and break others, and you won't notice the broken ones by reading a few outputs. An eval is a script that runs the same cases every time and tells you what changed.
CASES = [
# (question, phrase the answer must contain or None if the docs lack it, phrase it must not contain)
("How long is the refund window?", "14 days", "30 days"),
("Who approves expenses over the limit?", "finance lead", None),
("What is the office wifi password?", None, None),
]
def run_eval(index: BM25, k: int = 5) -> dict[str, bool]:
results = {}
for question, must, must_not in CASES:
result, sources = answer(question, index, k)
text = result["answer"].lower()
if must is None:
ok = not result["found"]
else:
ok = (result["found"] and must.lower() in text
and not verify_citations(result, sources))
if must_not and must_not.lower() in text:
ok = False
results[question] = ok
return results
def regressions(before: dict[str, bool], after: dict[str, bool]) -> list[str]:
return [q for q, ok in before.items() if ok and not after.get(q, False)]
def test_regression_is_reported():
before = {"refund window": True, "expenses": True}
after = {"refund window": True, "expenses": False}
assert regressions(before, after) == ["expenses"]
Replace the three sample cases with your twenty questions from checkpoint 3. Start with deterministic checks like these: does the answer contain the expected phrase, does it pass the citation check, does it avoid a phrase you know is wrong. They cost nothing beyond the calls themselves and never disagree with themselves. Model outputs vary between runs, so a case that flips once may be noise; run the suite twice before you trust a single regression. For open-ended answers that no phrase can judge, Anthropic's guide to building evaluations covers grading with a model and with people.
The check. Change one thing on purpose, for example run_eval(index, k=1) against a baseline with k=5, and print regressions(baseline, candidate). You pass when the script names the specific cases that broke, and you would trust it enough to run it before every change.
The question. What is the smallest set of cases that would have caught your last bug?
An agent is a model in a loop with tools: it asks for a function call, your code runs it, the result goes back, and the loop continues until the model answers. The loop is where cost and risk grow, because the model now decides how many steps to take. Build one with your search from checkpoint 3 as the only tool, and give it limits your code enforces. The tool use overview explains the message format.
from types import SimpleNamespace as NS
SEARCH_TOOL = {
"name": "search_docs",
"description": "Search the internal documents. Returns the most relevant passages with their ids.",
"input_schema": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
"additionalProperties": False,
},
}
def run_tool(name: str, args: dict, index: BM25) -> str:
if name != "search_docs":
raise ValueError(f"unknown tool {name}")
hits = index.search(args["query"], k=3)
return "\n\n".join(f"[{c['id']}]\n{c['text']}" for c in hits) or "No results."
def agent(question: str, index: BM25, llm=client, max_steps: int = 4) -> str:
messages = [{"role": "user", "content": question}]
for _ in range(max_steps):
response = llm.messages.create(
model=MODEL, max_tokens=16000, tools=[SEARCH_TOOL], messages=messages,
)
if response.stop_reason != "tool_use":
return text_of(response)
messages.append({"role": "assistant", "content": response.content})
results = []
for block in response.content:
if block.type != "tool_use":
continue
try:
output, is_error = run_tool(block.name, block.input, index), False
except Exception as exc: # the model sees the error and can correct its call
output, is_error = f"Error: {exc}", True
results.append({"type": "tool_result", "tool_use_id": block.id,
"content": output, "is_error": is_error})
messages.append({"role": "user", "content": results})
raise RuntimeError(f"no answer after {max_steps} steps")
class AlwaysSearches:
"""A fake client whose model asks for a tool on every turn and never answers."""
def __init__(self):
self.messages = self
self.calls = 0
def create(self, **kwargs):
self.calls += 1
call = NS(type="tool_use", id=f"t{self.calls}", name="search_docs",
input={"query": "refunds"})
return NS(stop_reason="tool_use", content=[call])
def test_loop_stops_at_the_limit():
index = BM25(chunk("faq", "Refunds are possible within 14 days of purchase."))
fake = AlwaysSearches()
try:
agent("refund window?", index, llm=fake, max_steps=3)
except RuntimeError:
assert fake.calls == 3
else:
raise AssertionError("the loop did not stop")
Three rules carry most of the safety here. The step limit is enforced by your code, not requested in a prompt. A failing tool returns a tool_result with is_error: true instead of crashing the loop, so the model can see what went wrong and correct its call. And every tool the model can reach should be one you're comfortable running with whatever input a stranger could type; search is a good first tool because it only reads. Passing the client in as a parameter is what makes the test possible: the fake asks for a tool forever, and the test passes only if your loop gives up at the limit.
The check. Run the test, then run the real loop on your twenty questions and log the number of steps for each. You pass when you can say which questions needed more than one search, and why.
The question. Which action would you never let this loop take without a person approving it?
What the plan leaves out on purpose
Training models, fine-tuning, GPU infrastructure and the mathematics of neural networks are real skills, but they come later, if your product needs them at all; the six checkpoints cover the problems you meet first when you put a hosted model behind a feature. Embeddings and vector databases are also missing as a step of their own. That's deliberate: checkpoint 3 gives you the baseline and the labelled questions that tell you whether you need them and whether they helped.
How long it takes
There's no honest fixed number. It depends on how much Python you already write and how messy your documents are. A workable rule is to time-box each checkpoint to a week of evenings, and if the check still fails at the end, write down why before you move on. A failing check you understand teaches more than a passing one you don't.
Conclusion
The checkpoints build on each other. The cost function from the first runs under every later call, the twenty labelled questions from the third feed the fourth and fifth, and the search becomes the agent's tool in the sixth. At the end you have one small codebase that calls a model, extracts data, searches your documents, answers with citations you can check, measures itself and runs a bounded tool loop, plus tests that fail when a change makes it worse.
That last part outlasts the rest. Models, prices and SDK fields will change within months. Building every feature with an exit condition, a known cost and a test is the habit that carries over to each of those changes.
I'm Dragoș; I build and teach LLM systems at [Cursuri AI](https://courses-ai.com / https://cursuri-ai.ro/).