Writing
32 posts · tagged agents · oldest Feb 2025
- Nothing here is specialEvery step is a Job, every brain is a ConfigMap, every score is an MLflow run, and one gateway holds every key. The platform under the agent flywheel, the libraries it is built from — the evaluation harness, Harbor, EvalHub, GEPA — what each one needed wrapped around it, and the sandbox that is built against the one that is only designed.agentskubernetesinfrasecurity14 min
- The best brain, not the last oneTraining is evaluation in a loop: score a brain, let a second agent adjust it from the judges' reasons, score it again, keep or revert. Nobody touches a weight. What the adjuster may read, what it never sees, what it wrote in five steps, and three different engines for the adjust step.agentsaievaluation15 min
- Closer than the spread is a tieA benchmark is an evaluation with three promises: frozen, shared, comparable. Sixty evaluations of three agents on four models ran in 21 minutes for 5.39 USD, and the top five variants were a tie. What has to be frozen, how many runs one variant needs, and why a benchmark must never pay twice for the same measurement.agentsevaluationmodels11 min
- The mean passes, two rules blockOne evaluation is one brain, one set of tickets, six judges and one score. The score is the least useful part of it. Why five judges are code and one is a model, why one run is not a measurement, and a release gate that blocked a brain with a 0.910 average.agentsevaluationtesting11 min
- Thirty tickets nobody filedAn evaluation is only as good as the tickets under it. The first version of this dataset hit the ceiling after one training step, because the agent could read the rulebook the judges graded by. What the second version hides, what it plants, and what every ticket carries for the judges.agentsevaluationtesting9 min
- Twelve wordsSame model, same weights, same 32 articles. A support agent went from 0.649 to 0.910 on tickets it never saw, and the only thing that changed was a text file. The first post of a series on measuring AI agents, where evaluation, benchmarking and training turn out to be one measurement used three ways.agentsaievaluation7 minvideo
- Point, don't pasteReviewing a document with an AI still means copying a paragraph into a chat, explaining where it came from, and carrying the answer back by hand. REX puts the conversation on the document instead — every comment is a thread, the thread stays attached to its passage, and nothing reaches your file until you approve it.agentsaiproduct9 minvideo
- A wrong place is worse than no placeA broken anchor does not fail. It lands somewhere, reports `ok`, and paints a highlight that looks exactly like a correct one. How REX keeps a comment on its passage while the document is edited, the three times it lied anyway, and the rule that came out of it: orphan before you guess.agentsarchitecturedebugging12 min
- A comment is a conversation that waits for youEvery REX comment is its own thread with its own agent session, stored on the document rather than in a chat window. What is stored and what is derived, how four SDKs resume a conversation four different ways, and the resume that was silently broken until a debug report asked the filesystem instead of the SDK.agentsaiarchitecture11 min
- A question cannot change your fileREX makes one safety promise: ASK cannot write, ACT edits a copy, and only Approve touches your file. Three times it broke without a sound — a line gone under "Applied to 0 file(s)", a committed file deleted under a notice saying it was put back, a copy removed while the agent was reading it — and what an allowlist, four SDKs and a repository's own settings file taught me about where a boundary actually is.agentsaisecurity11 min
- Binary files differThe ACT loop ends in a diff you approve, and on a .pptx that diff is two words. So the agent writes a plan, REX performs it on a copy, and the checker has to prove not only that the change happened but that nothing else did — plus the three numbers I measured wrong on the way.agentsarchitecturetesting11 min
- Six hundred lines or it does not shipA 2,313-line orchestrator with a 1,073-line function is maintainable by a human with an IDE and unmaintainable by an agent. The refactor whose acceptance criterion was an agent behaviour, the CI guard that enforced it — and the fact that the guard is red as I write this.agentsarchitecturerefactoring10 min
- Driving the Agent SDK: permissions, hooks and capsA stage is one query() call. The five options that define what an autonomous agent is allowed to do, the four hooks that make an unattended run watchable, and an honest account of what bypassPermissions actually means.agentsaiarchitecture8 min
- One authority, three counters: deriving state from git refsNaming a source of truth is the easy half. The hard half is naming the moment every other view gets rebuilt from it — and the run where a UI, a state file and a git history gave three different answers to "how much of this is built".agentsdebuggingarchitecture9 min
- Git as the databaseEvery stage commits, the run is reconstructable from refs alone, and the app owns exactly two branch families. Why an autonomous builder should store its work in the versioning system its users already have — and what that choice costs at review time.gitarchitectureagents9 min
- Your test suite bills by the tokenTwenty minutes and real dollars to verify a two-line UI change. A pluggable agent backend, a mock that refuses to improvise, and checkpoint replay took it to sixty seconds and zero — plus an honest account of what the mock does not buy you.agentstestingarchitecture9 min
- Structured handoffs, and what the model is still allowed to decideWhen one agent hands work to the next, prose is the wire protocol — and a catastrophically bad one. The manifest that made feature selection cost zero tokens, and the line between what a schema pins down and what genuinely needs judgement.agentsaiarchitecture8 min
- Nine stages, nine amnesiacs: context isolation as an architectureDex never lets a session get long enough to rot. Every stage is a brand new agent with an empty context window, state travels through the filesystem, and the unit of everything — execution, resume, storage, the UI — is the stage. Including what that costs.agentsaiarchitecture9 min
- Three pillars, and what each one is missingDex is not a new idea. It is three existing ideas — a bash loop, a spec workflow, and an SDK — each of which is most of an autonomous coding system and none of which is all of it. The interesting part is the shape of the holes.agentsaiarchitecture9 min
- Five months, two repos, one lesson: the model was the easy partA retrospective on OpenOperator. What I would keep, what I would redo, and why a commit called "remove hardcoded API keys" landing a year late is the most honest thing in the history.agentsinfraretrospective6 min
- MCP for mouse and keyboard: a desktop as a tool serverSix tools, one resource, and a design decision I still think is underrated — the screenshot is not a tool, it is a resource. Plus the reason we composite a cursor into every capture.mcpagentspython7 min
- "It looked like it worked": grading a non-deterministic agentThe least trustworthy signal in the system is the agent reporting success. Here is the evaluator we built instead — and the four-word commit that forced a whole second repository into existence.agentstestingevaluation7 min
- Tasks as config, not codeA JSON scenario format with lifecycle hooks, so an agent test starts from the same desktop every time — and so the person writing the test does not have to be the person who wrote the agent.agentstestingarchitecture7 min
- Four agents in four weeks: from AutoGen to a hand-rolled plan-and-solve loopThe old_agents/ directory still holds all the corpses. What each generation got wrong, and why most of a computer-use agent is not acting — it is checking whether the last action worked.agentsaiarchitecture8 min
- Reading the screen: OmniParser as a serviceTurning a screenshot into a numbered list of clickable things — YOLOv8 for detection, Florence-2 for captions, OCR for text. And why it had to be a server, not a library.aiagentspython6 min
- We tried every vision model we could get our hands onMagma, Llama 3.2 Vision, Gemma 3, Phi-4, GPT-4o, Claude — including feeding a screen recording instead of stills. What actually moved the needle was not the model.aiagentsmodels5 min
- You cannot debug an agent from logsThe day we put noVNC in front of the guest, our mean time to diagnosis dropped by an order of magnitude. Failures in computer use are visual.agentsdebugginginfra4 min
- The API is the screenWhy we built an agent that uses Microsoft Teams the way a person does — and why the model turned out to be the easy part.agentsaiinfra7 min