Every time I added a benchmark suite to EvalBench, I used to touch the database. New suite, new columns, a migration, a change to the aggregation query, and usually a change to the frontend so it knew the new metric existed. By the third suite that p...

A migration agent spends real money and real minutes per run. It clones a repo, installs its deps, plans a file order, then rewrites files one at a time, running the target's test suite after each one. On a mid-sized repo that is a lot of Bedrock cal...

TerraformAgent turns a plain-English infrastructure request into validated Terraform. Six nodes: an orchestrator plans the domains, a researcher pulls AWS docs, a set of domain subagents write code in parallel, an aggregator stitches it together, a r...

terraform validate failed on nearly every multi-file run with the same error class: Error: Reference to undeclared input variable on iam.tf line 14, in resource "awsiampolicy" "appaccess": 14: Environment = var.environment An input variabl...

I was looking at the EvalBench dashboard a few weeks ago and one cell stopped me: Retries to valid 0.125 95% CI -0.035 – 0.285 n=40 A negative number of retries. You cannot retry a request negative-point-oh-three-five times. The mean was fine. The...

Most agent projects that generate code evaluate it by asking another model whether the code looks correct. I did that for about a week in TerraformAgent before I got tired of it. The failure mode is boring and predictable. The judge reads a .tf file...

When I moved Wingman onto Lambda, I assumed the thing I'd have to watch was compute duration. That's the serverless mental model everyone starts with: you pay per millisecond, so keep the function fast and cheap. That model was wrong for an agent. T...

Most eval setups have a quiet lie in them. You run a fixed question set through your agent, a judge model scores the answers, you get a number, and you feel good. But that number describes a sandbox. It says nothing about the question a stranger type...

Most agent demos stop at "the model called a tool and gave an answer." That answer is often wrong, and nothing in the loop notices. I wanted an agent that checks its own work against a success criteria before returning, and retries when it falls shor...