Agent Flywheel
measure an AI agent once, then use that to gate, rank and train it on OpenShift
2026
- Nothing here is specialEvery step is a Job, every brain is a ConfigMap, every score is an MLflow run, and one gateway holds every key. The platform under the agent flywheel, the libraries it is built from — the evaluation harness, Harbor, EvalHub, GEPA — what each one needed wrapped around it, and the sandbox that is built against the one that is only designed.
- The best brain, not the last oneTraining is evaluation in a loop: score a brain, let a second agent adjust it from the judges' reasons, score it again, keep or revert. Nobody touches a weight. What the adjuster may read, what it never sees, what it wrote in five steps, and three different engines for the adjust step.
- Closer than the spread is a tieA benchmark is an evaluation with three promises: frozen, shared, comparable. Sixty evaluations of three agents on four models ran in 21 minutes for 5.39 USD, and the top five variants were a tie. What has to be frozen, how many runs one variant needs, and why a benchmark must never pay twice for the same measurement.
- The mean passes, two rules blockOne evaluation is one brain, one set of tickets, six judges and one score. The score is the least useful part of it. Why five judges are code and one is a model, why one run is not a measurement, and a release gate that blocked a brain with a 0.910 average.
- Thirty tickets nobody filedAn evaluation is only as good as the tickets under it. The first version of this dataset hit the ceiling after one training step, because the agent could read the rulebook the judges graded by. What the second version hides, what it plants, and what every ticket carries for the judges.
- Twelve wordsSame model, same weights, same 32 articles. A support agent went from 0.649 to 0.910 on tickets it never saw, and the only thing that changed was a text file. The first post of a series on measuring AI agents, where evaluation, benchmarking and training turn out to be one measurement used three ways.