Writing
6 posts · tagged evaluation · oldest Mar 2025
- The best brain, not the last oneTraining is evaluation in a loop: score a brain, let a second agent adjust it from the judges' reasons, score it again, keep or revert. Nobody touches a weight. What the adjuster may read, what it never sees, what it wrote in five steps, and three different engines for the adjust step.agentsaievaluation15 min
- Closer than the spread is a tieA benchmark is an evaluation with three promises: frozen, shared, comparable. Sixty evaluations of three agents on four models ran in 21 minutes for 5.39 USD, and the top five variants were a tie. What has to be frozen, how many runs one variant needs, and why a benchmark must never pay twice for the same measurement.agentsevaluationmodels11 min
- The mean passes, two rules blockOne evaluation is one brain, one set of tickets, six judges and one score. The score is the least useful part of it. Why five judges are code and one is a model, why one run is not a measurement, and a release gate that blocked a brain with a 0.910 average.agentsevaluationtesting11 min
- Thirty tickets nobody filedAn evaluation is only as good as the tickets under it. The first version of this dataset hit the ceiling after one training step, because the agent could read the rulebook the judges graded by. What the second version hides, what it plants, and what every ticket carries for the judges.agentsevaluationtesting9 min
- Twelve wordsSame model, same weights, same 32 articles. A support agent went from 0.649 to 0.910 on tickets it never saw, and the only thing that changed was a text file. The first post of a series on measuring AI agents, where evaluation, benchmarking and training turn out to be one measurement used three ways.agentsaievaluation7 minvideo