Twelve words
Same model, same weights, same 32 articles. A support agent went from 0.649 to 0.910 on tickets it never saw, and the only thing that changed was a text file. The first post of a series on measuring AI agents, where evaluation, benchmarking and training turn out to be one measurement used three ways.
This morning, a few hours before I went on stage at Red Hat Innovate Together, a training run finished on an OpenShift cluster. It took 19 minutes and 48 seconds. When it started, the support agent it was training had this as its entire set of instructions:
You are a helpful assistant for Red Hat customers. Answer their questions.Twelve words. On ten support tickets it never saw during training, that agent scored 0.649. The version the run ended with scored 0.910.
The model did not change. Its weights did not change. The 32 knowledge base
articles it answers from did not change. What changed is its instructions — the
same AGENTS.md, grown from twelve words into an answer contract — and nobody
wrote a line of it by hand. A second agent wrote it, from the reasons six
judges gave for every answer they marked down.
That number opened the talk. It is not the point of the talk. The point is the sentence the talk closes with: it is a better support engineer now, and we can prove it, because we measured it. This series is about the measuring.
Which agent do you ship?
Ask a team how they decided which agent to put in front of customers, and the answers sound like this:
- “It looks fine on the tickets I tried.” The honest one. How many tickets, and which ones?
- “The model scored 92% on the benchmark.” A benchmark number of the model tells you nothing about your agent, on your tickets, under your rules.
- “We changed the prompt and it got better.” Better by what?
Three answers, zero measurements. And an agent is a hard thing to measure, because it is not one thing. It is a harness — OpenCode, Claude Code, Codex — a model, a set of tools, and a pile of instructions, and any one of the four can be the reason it got better or worse.
What is being measured
I call the pile of instructions the brain: an AGENTS.md that the agent
loads on every ticket, and up to three skills in the Agent Skills format, each
loaded only when its description fits the ticket. Both are open formats.
OpenCode and Codex read AGENTS.md as it is, Claude Code reads it through a
CLAUDE.md that imports it, and all three read skills. So one brain runs in all
three agents. That turns out to matter: a brain trained in one agent can be
benchmarked in the other two, on any model.
The job is a support desk for Red Hat products. The product names are real — OpenShift, RHEL, Ansible Automation Platform, OpenShift AI. Everything else is invented: 32 knowledge base articles with fictional IDs, versions and fixes; 30 tickets from fictional customers; and two policies the agent is never allowed to read. The model under the agent is Qwen 3.8 27B on Cerebras, an open model that answers fast enough to score thirty tickets in about a minute. The data is a demo. The method is not.
One unit of measurement, three uses
The talk had one idea, and the console I built for it has the same three words in its menu.
One measurement. Is this brain good enough for my job? The answer is a release gate: pass or blocked, and why.
Many measurements, in a grid. Which agent, model or brain is better? Frozen tickets, every variant run several times: a mean and a spread.
Many measurements, in a loop. Make this brain better: score it, adjust it, score it again, keep or revert.
All three stand on the same measurement. If it is wrong, the gate lets a bad brain through, the leaderboard crowns the wrong winner, and training learns the wrong thing. So the order matters: build the evaluation first, then use it three ways.
It took me a week to see it that way. The project started as a training loop — the flywheel in its name — and the evaluation was only the step inside it that produced a number. A week in, I wrote the correction into the plan:
You need to understand the evaluation before you use it in the training.
A benchmark, in the same note, “is only a predefined set of tickets on a predefined set of agents and models: run multiple evaluations and show the result as metrics.” Once the evaluation was the unit, the other two stopped being separate systems. A benchmark is evaluations in a grid. Training is evaluations in a loop, with one extra step between them.
That step needed a better name too. For most of the week, the agent that rewrites the brain was the trainer. The evening before the talk I renamed it everywhere, down to the Kubernetes Job names, because the name was wrong:
The trainer is not doing any training … the adjuster is the correct terminology.
It trains no weights. It adjusts the brain. In machine-learning terms it plays the optimizer, the loop around it plays the trainer, and the score on the training tickets is the loss.
Nothing special underneath
Every step is a Kubernetes Job. Every version of the brain is a ConfigMap. Every score is an MLflow run, with one trace per ticket. One LiteLLM gateway is the only pod that holds a model key. That is the platform. There is no evaluation server to install first, and the same containers run on a laptop with Podman. So a plumbing test runs for free on a local model, and only a run whose number goes on a slide costs money.
What this is not
It is not fine-tuning. No weight moves. The “training” produces text files, which you can diff, review and roll back like any other code.
It is not a public benchmark. Thirty invented tickets cannot rank models for anyone else. They rank brains for this desk, and that is the only ranking a team shipping an agent needs.
It is not one number. The 0.910 above is one run. The posts after this one spend many words on why one run is not a measurement.
Update, 12 October: the rest of the series
The film at the top is a five-and-a-half-minute walkthrough of the console. It was recorded after the week’s later work, so its numbers come from a later run than the ones in this post. The five posts that followed this one take the pieces in the order the talk did:
- Thirty tickets nobody filed — the dataset: look-alike articles, answer keys, and why the rulebook had to leave the agent’s folder.
- The mean passes, two rules block — one evaluation: six judges, pass@k against pass^k, and a release gate that never trusts the average alone.
- Closer than the spread is a tie — a benchmark: what a frozen recipe freezes, 60 evaluations in 21 minutes, and a leaderboard whose top five are a tie.
- The best brain, not the last one — training: score, adjust, keep or revert, and an adjuster that learned to cite a policy it never read.
- Nothing here is special — the platform: Jobs, ConfigMaps, MLflow, one gateway with the keys, the harness, Harbor and EvalHub as switches, and the sandbox around the agent.