Stop judging the same job twice: the dedup layer my LLM agent needed more than a better prompt
My Job Copilot is supposed to run over job postings at night and hand me a scored shortlist in the morning. For a while, it did — and then I checked the logs. The same postings, byte for byte identical, were being sent to the LLM again. And again. Each run re-paid for a judgment it had already made, and because LLM verdicts are not deterministic, duplicates sometimes scored differently from themselves.
So before I improved a single prompt, I needed the boring thing: dedup. "Have I already judged this?" This post is that layer — fingerprints instead of URLs, a sqlite store instead of infrastructure, and one graph node that never calls the model.
Note to self, verify before publish: opening anecdote: confirm from real logs that duplicate re-judging actually happened (which run, roughly how many duplicate calls). If the incident is different, rewrite the paragraph to match reality.
"Did I see this URL already?" is the wrong question
My first instinct was the naive one: keep a set of URLs, skip anything already in it. It fails in two directions at once.
The first failure is the same job under a different URL. Recruiters repost through different boards and the same posting comes back with tracking junk appended — ?utm_source=linkedin&gh_jid=777. A URL-keyed set sees a new job. Your wallet disagrees.
The second is the same URL with a different job. Postings get edited: salary bump, deadline extended, one requirement dropped. If the URL is the key, an edited posting is stuck with its first verdict forever — even though it's now a genuinely different deal.
So the identity of a job posting is the body, not the address. That leads to the classic move: a content fingerprint. But hash the raw text and the same posting hashed from two sources will differ anyway — one has CRLF line endings, or the listing was rendered with an extra <br> that became \n\n. Whitespace is noise, so noise must be normalized before hashing:
# fingerprints.py
import hashlib
import re
WHITESPACE = re.compile(r"\s+")
def normalize(text: str) -> str:
"""Lowercase, collapse whitespace.
Two postings that differ only in case or line breaks must produce
the same fingerprint.
"""
text = text.lower().strip()
return WHITESPACE.sub(" ", text)
def url_key(url: str) -> str:
"""Strip tracking params so reposts with UTM junk collide correctly."""
return url.split("?")[0]
def fingerprint(text: str, length: int = 16) -> str:
return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()[:16]
Two small decisions hiding in there:
url_key keeps the path and drops the query string, because tracking parameters are the single most common source of fake-new URLs, and I've never met one that carried signal.
- The fingerprint is truncated to 16 hex chars. A short posting gives sha256 no meaningful extra collision resistance past that, and 16 hex chars is nice in logs.
Note to self, verify before publish: url_key: confirm how JobCopilot identifies a posting (does it canonicalize URLs at fetch time already, or not at all?), and whether sha256/16-char truncation matches the repo style.
The store: one sqlite table, a primary key, and an expiry nobody asked for but everybody needs
I did not want a vector DB and I do not want a cache service. The whole dedup state is one sqlite table:
# seen_store.py
import sqlite3
import time
SCHEMA = """
CREATE TABLE IF NOT EXISTS seen_jobs (
fingerprint TEXT PRIMARY KEY,
url_key TEXT NOT NULL,
verdict TEXT NOT NULL,
score REAL NOT NULL,
judged_at REAL NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_seen_url ON seen_jobs (url_key);
"""
class SeenStore:
def __init__(self, path: str = "seen_jobs.db", window_days: int = 14):
self.conn = sqlite3.connect(path)
self.conn.executescript(SCHEMA)
self.window = window_days * 86400
def by_fingerprint(self, fp: str) -> dict | None:
row = self.conn.execute(
"SELECT verdict, score, judged_at FROM seen_jobs"
" WHERE fingerprint = ?",
(fp,),
).fetchone()
if row is None:
return None
return {"verdict": row[0], "score": row[1], "judged_at": row[2]}
def by_url(self, url_key: str) -> list[dict]:
rows = self.conn.execute(
"SELECT fingerprint, verdict, score, judged_at FROM seen_jobs"
" WHERE url_key = ? ORDER BY judged_at DESC",
(url_key,),
).fetchall()
return [
{"fingerprint": r[0], "verdict": r[1], "score": r[2], "judged_at": r[3]}
for r in rows
]
def record(self, fp: str, url_key: str, verdict: str, score: float) -> None:
self.conn.execute(
"INSERT INTO seen_jobs"
" (fingerprint, url_key, verdict, score, judged_at)"
" VALUES (?, ?, ?, ?, ?)"
" ON CONFLICT(fingerprint) DO UPDATE SET"
" url_key = excluded.url_key,"
" verdict = excluded.verdict,"
" score = excluded.score,"
" judged_at = excluded.judged_at",
(fp, url_key, verdict, score, time.time()),
)
self.conn.commit()
def expired(self, judged_at: float) -> bool:
return (time.time() - judged_at) > self.window or (time.time() < judged_at - 60)
Three things in this class earned their place:
The store uses ON CONFLICT ... DO UPDATE instead of read-then-insert. The copilot is a batch process; the first version of this code did SELECT then INSERT and I got a UNIQUE constraint failed the first time the same posting appeared twice in one batch. The upsert makes record() idempotent, which is exactly what a thing named "record that I saw this" should be.
The fingerprint is the primary key and the URL is just an index, which is the data-model version of the section above: body is identity, address is lookup. Having by_url return a list of fingerprints is what makes the "this posting was edited" logic possible in the next section — I can see the whole history of judgments for one URL.
Finally, expired(): postings go stale, so the cache must too. A verdict from two months ago is not a cache entry, it's a libel suit against the employer (they might have added the budget; my resume also changed). Two weeks felt right for a job market, and it's an argument, not a constant — tune it. The second half of that boolean is the paranoid one: if judged_at is somehow in the future (clock moved backwards between runs, or a copy-pasted row), treat the record as expired rather than trusting it. It has fired exactly once, on data I had mangled myself.
Note to self, verify before publish: persistence: does JobCopilot already keep run state (sqlite? json? nothing)? If there's an existing store, this table should slot into it rather than being a parallel db file. window_days=14 and path are stand-ins — set real values.
The graph part: two dumb nodes that never call a model
Part 3 of this series established the rule I still like best about my LangGraph graph: a node never calls another node; nodes return state patches, and routing decides what runs next. Dedup fits that shape perfectly, because a cache check is the ideal node — three branches, zero tokens:
# dedup_node.py
from fingerprints import fingerprint, url_key
from seen_store import SeenStore
from state import AgentState
store = SeenStore("seen_jobs.db", window_days=14)
SKIP_BUDGET = 70.0 # reusing a stored score below this is never worth it
def dedup_node(state: AgentState) -> dict:
fp = fingerprint(state.job_description)
hit = store.by_fingerprint(fp)
if hit is not None and not store.expired(hit["judged_at"]):
# exact body seen before -> reuse the verdict, no LLM call
if hit["verdict"] == "match" or hit["score"] < SKIP_BUDGET:
return {
"verdict": hit["verdict"],
"score": hit["score"],
"skip_llm": True,
}
# same job URL with a different fingerprint -> the posting changed,
# re-judge and keep the new fingerprint
return {"skip_llm": False}
def record_node(state: AgentState) -> dict:
"""Runs after the judge node, writes the verdict back."""
store.record(
fingerprint(state.job_description),
url_key(state.job_url),
state.verdict,
state.score,
)
return {}
For the demo, AgentState is the same shape as the state model from part 3 plus two fields:
# state.py
from pydantic import BaseModel
class AgentState(BaseModel):
"""Same shape as the demo graph from part 3, plus dedup fields."""
job_url: str = ""
job_description: str = ""
score: float = 0.0
verdict: str = "" # "match" / "no_match" / "skip"
skip_llm: bool = False
The dedup_node → judge → record_node chain is exactly the part-3 graph with one decision added to the router: skip_llm routes to the end instead of to judge. No new machinery. The graph got a new edge, not a new idea.
SKIP_BUDGET is the sneaky half. Reusing a cached match verdict is obviously fine. But a cached rejection near my threshold is worth re-judging, because the model is noisy near the boundary anyway (that's the lesson from the eval-bands post) and a fresh opinion on a 68-score posting is cheap. So the cache keeps confident rejections, and re-judges everything in the gray zone.
Note to self, verify before publish: AgentState fields and node names ("judge"/"fail" etc.) must match the real repo; SKIP_BUDGET=70.0 and the re-judge-below-budget rule are my stand-in for a policy — confirm what a sensible band is for the real scoring.
The proof, then the honest part
I ran a stub judge (a dict of (verdict, score) keyed by fingerprint — same trick as the eval fixtures) over three postings twice. Pass 2 contained: the same job unchanged, the same job reposted with UTM junk and different line breaks, and the same URL with an updated body.
pass 1: https://boards.example.com/acme/eng/123 -> match score= 82.0 JUDGED
pass 1: https://boards.example.com/acme/eng/777 -> no_match score= 0.3 JUDGED
pass 1: https://boards.example.com/acme/eng/444 -> match score= 62.0 JUDGED
pass 2: https://boards.example.com/acme/eng/123 -> match score= 82.0 SKIP (cache hit)
pass 2: https://boards.example.com/acme/eng/777 -> no_match score= 0.3 SKIP (cache hit)
pass 2: https://boards.example.com/acme/eng/444 -> match score= 68.0 JUDGED
LLM calls: pass 1 = 3, pass 2 = 1
rows in store: 4
That's the whole thesis in ten lines: the repost with tracking junk was free, the unchanged posting was free, and the edited posting was judged again — which is the point, because score= 68.0 is not a cache miss, it's news.
The honest part, since this is a learning-in-public series:
- The first fingerprint version normalized only case, not whitespace. My demo passed; the real scrape failed because the two copies of a posting had different line endings. Test data must include the whitespace case, not just the "identical string" case.
- I initially keyed the table on URL with the query string — see above, that's what
url_key exists to forget.
- A near-duplicate that isn't byte-identical (posting reworded slightly) still gets re-judged today. Embedding cosine on the normalized body would catch that (the exact trick from part 2 of this series), but I haven't needed it yet, so it stays an idea, not code.
Note to self, verify before publish: if JobCopilot's real dedup/scrape layer differs (e.g. it dedups at fetch time, or on a different key), reconcile the "what broke" list with the real history before publishing.
Wrap-up
The unglamorous takeaway: my agent's biggest cost bug wasn't a bad prompt or a bad model — it was memory. Three small pieces fixed it: a whitespace-normalized body fingerprint, a sqlite table with judged_at expiry, and a graph node that routes on cache hits before the expensive node ever runs. The expensive model got faster the day I stopped asking it questions it had already answered.
This is part 7 of Building an AI Job Copilot in Python. Previous post: unit-testing LangGraph nodes and checkpoints. Earlier parts covered structured JSON output, retrieval without a vector DB, the first graph, and the eval harness — all with the same recurring theme: the layer that keeps a small LLM pipeline predictable costs nothing to run.
If you're building something similar: follow here or on dev.to — the next part gets into making the nightly batch run resilient when the LLM is down (retries that don't multiply your spend). And if you've built dedup for an LLM pipeline, I genuinely want to know whether you used a content hash or something smarter.