Most agent projects that generate code evaluate it by asking another model whether the code looks correct. I did that for about a week in TerraformAgent before I got tired of it.
The failure mode is boring and predictable. The judge reads a .tf file, sees resource blocks with plausible attribute names, and says yes. The file has an undeclared variable, or a reference to a resource that does not exist, or an attribute that HashiCorp removed in provider v5. The judge has no way to know. It is pattern-matching on what Terraform usually looks like, which is exactly what the generator was doing when it produced the bug.
Terraform ships a checker. It is called terraform validate. It is deterministic, it takes about a second, and it is never wrong about syntax or references. The whole reason people reach for an LLM judge here is that running the real thing inside a Lambda is annoying. It is annoying. It is also not that hard, and this post is the parts that actually bit me.
TerraformAgent is a 6-node LangGraph pipeline that turns a natural-language infra request into Terraform. Orchestrator, researcher, parallel domain subagents, aggregator, reviewer, evaluator. The evaluator is the node I care about here.
The evaluator writes to disk and shells out
The generated code lives in state as a filename-to-content dict. To run real Terraform against it, you have to put it on a filesystem. Lambda gives you /tmp, so that is where it goes:
thread_id = state.get("thread_id", "default")
temp_dir = f"/tmp/tf_eval_{thread_id}"
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
os.makedirs(temp_dir, exist_ok=True)
aggregated_output = state.get("aggregated_output", {})
for filename, content in aggregated_output.items():
filepath = os.path.join(temp_dir, filename)
with open(filepath, "w") as f:
f.write(content)
The thread_id in the path matters. Lambda execution environments get reused across invocations, so two runs can land in the same container. Without the thread scoping, run B validates a directory containing run A's leftovers.
Then three subprocess calls, in order:
fmt_result = subprocess.run(
["terraform", "fmt", "-check", "-recursive"],
cwd=temp_dir, capture_output=True, text=True, timeout=30,
)
eval_results["terraform_fmt_pass"] = fmt_result.returncode == 0
init next, then validate only if init succeeded. Skipping that guard gives you a validate failure that is really an init failure, and the retry logic then blames the wrong thing.
if init_success:
val_result = subprocess.run(
["terraform", "validate", "-no-color"],
cwd=temp_dir, capture_output=True, text=True, timeout=60,
)
eval_results["terraform_validate_pass"] = val_result.returncode == 0
else:
eval_results["terraform_validate_pass"] = None
eval_results["terraform_validate_output"] = "skipped (init failed)"
Note the three-state result. True, False, and None. None means the check did not run. If you collapse that to False you get a retry loop chasing a validation error that was never evaluated. There is a fourth case in the fmt block for the same reason: FileNotFoundError sets the result to None with the message terraform binary not found (running locally without Docker?), because the binary only exists inside the container image and I run these tests on my laptop.
terraform init downloads the AWS provider plugin. It is about 60 MB. Doing that on every invocation is slow, costs egress, and breaks the moment HashiCorp's release endpoint has a bad minute.
So bake it into the image at build time:
RUN mkdir -p /tmp/tf-provider-cache /opt/tf-plugins && \
cd /tmp/tf-provider-cache && \
printf 'terraform {\n required_providers {\n aws = {\n source = "hashicorp/aws"\n version = "~> 5.0"\n }\n }\n}\n' > main.tf && \
TF_PLUGIN_CACHE_DIR=/opt/tf-plugins terraform init -backend=false && \
rm -rf /tmp/tf-provider-cache
ENV TF_PLUGIN_CACHE_DIR=/opt/tf-plugins
A throwaway main.tf that declares nothing but the provider requirement, one init, and the plugin lands in the cache directory.
The cache goes in /opt, not /tmp. This one cost me real time. Lambda mounts a fresh empty /tmp on every execution environment at runtime. Anything you write to /tmp during the Docker build is gone. Not stale, not partially there. Gone. /opt is part of the read-only image layers and survives to runtime.
The read-only cache breaks on version mismatch
Baking into /opt fixes the download. It introduces a second problem, and this is the part I did not see coming.
/opt is read-only at runtime. Terraform treats TF_PLUGIN_CACHE_DIR as a directory it can write to. As long as the generated code asks for a provider version matching what got baked in, Terraform reads from the cache and everyone is happy. The moment the generated code pins something else, Terraform tries to download that version into the cache dir, hits a read-only path, and fails outright. Not a slow path. A hard failure.
The fix is a copy, once per execution environment:
baked_cache = os.environ.get("TF_PLUGIN_CACHE_DIR", "/opt/tf-plugins")
writable_cache = "/tmp/tf-plugins"
if not os.path.isdir(writable_cache):
if os.path.isdir(baked_cache):
shutil.copytree(baked_cache, writable_cache)
else:
os.makedirs(writable_cache, exist_ok=True)
env = os.environ.copy()
env["TF_PLUGIN_CACHE_DIR"] = writable_cache
init_result = subprocess.run(
["terraform", "init", "-backend=false", "-input=false", "-no-color"],
cwd=temp_dir, capture_output=True, text=True, timeout=120, env=env,
)
The if not os.path.isdir(writable_cache) guard is what makes this cheap. Warm containers skip the copy entirely. Cold ones pay a local filesystem copy, which is nothing next to a 60 MB download. Matching versions stay fully offline. Non-matching versions still work, they just fall back to a normal network install into a directory Terraform is allowed to write to.
That is the tradeoff I would call out in a review: I could have refused non-matching versions and kept it strictly offline, which would be faster and more predictable. I chose to degrade to a network fetch instead, because an agent that generates version = "~> 4.0" for a good reason should not hard-fail on infrastructure trivia.
The LLM judge is still there. It just answers a different question.
I want to correct something, because I have seen this framed as replacing the LLM judge, and that is not what happened.
The evaluator runs both. Mechanical checks answer "is this valid Terraform." An LLM judge answers "does this do what the user asked":
judge_prompt = f"""Task: {state.get('task', '')}
Generated Terraform code:
{json.dumps(aggregated_output)}
Does the generated code satisfy the original task? Evaluate:
1. Are all requested resources present?
2. Are the configurations correct for the stated requirements?
3. Are there any obvious omissions?
Answer strictly with JSON: {{"task_success": true/false, "reason": "...", "missing_resources": [...]}}"""
Nothing deterministic can check whether "give me a VPC with a private subnet for the database" produced the right resources. That is a semantic question and it needs a model. The point is not that LLM judges are useless. The point is that they should not be answering questions a compiler already answers.
Routing the retry
The evaluator's results feed a function that decides which domains re-run. Only the flagged ones:
is_blocking = "BLOCKING_ERROR" in review_feedback
fmt_failed = eval_results.get("terraform_fmt_pass") is False
validate_failed = eval_results.get("terraform_validate_pass") is False
if not (is_blocking or fmt_failed or validate_failed):
return []
Two things here. is False and not falsy, so the None case does not trigger a retry. And when something failed but no specific file was named in the output, every active domain retries rather than silently returning an empty list. A retry that does nothing is worse than an expensive one.
The graph wires it as a conditional edge back to the subagents, capped at 3:
workflow.add_conditional_edges(
"evaluator",
should_retry_or_finish,
{"domain_subagents": "domain_subagents", "end": END},
)
What I would do differently
The fmt output is a filename list, which makes attributing a failure to a domain easy. validate output is prose from Terraform, and I am string-matching known filenames against it. That works until a filename appears in an error message for a reason unrelated to that file being broken. Parsing terraform validate -json would give me structured diagnostics with real file and line attribution. I have not done it yet.
The retry cap of 3 is a guess. I have no data on how often a third attempt fixes something a second one did not. That is a metric I should be emitting and am not.
And the judge sees json.dumps(aggregated_output) with the entire generated codebase inline. On a large request that is a lot of tokens for one boolean.
94 tests pass on this. None of them mock terraform, because the whole point was to stop trusting a stand-in for the real thing.
Takeaway
If a real tool can answer your evaluation question, run the real tool, and save the model for the questions that have no compiler.
Repo: github.com/akashpersetti/terraform-agent