Tasks as config, not code

The first dozen agent tasks lived in Python. A function per task, a bit of setup, call the agent, assert something. It worked, and it had two problems that only show up once you take testing seriously.

Problem one: the starting state drifted. Run a task, and the desktop is now different — a window is open, a chat is marked read, a file exists. Run the next task and it starts from that. Your results are not measuring the agent, they are measuring the agent plus an unspecified amount of history.

Problem two: only the agent’s author could write a task. Which is exactly backwards. The people who know what Teams workflows matter are not the people who know the executor’s node graph.

So tasks became data.

The format

{
  "instruction": "Find a list of chats inside Microsoft Teams. Switch between the chats 5 times. Summarize the latest chat thread in 3 paragraphs with bullet points.",
  "workflow": {
    "params": {
      "max_plan_versions": 20,
      "max_plan_step_iterations": 3,
      "max_plan_step_actions": 5
    }
  },
  "environment": {
    "start": [
      { "func": "close_all_windows", "args": null },
      { "func": "delete_files", "args": { "paths": ["/data/logs/teams-telemetry/teams-telemetry-0.log"] } },
      { "func": "start_network_proxy", "args": null },
      { "func": "open_application", "args": { "app_name": "ms-teams" } }
    ],
    "end": [
      { "func": "stop_network_proxy", "args": null },
      { "func": "close_all_windows", "args": null },
      { "func": "get_evaluation_results", "args": {
          "evaluation": [{
            "evaluator": "teams_scenarios",
            "scenarios": ["chat_switch"],
            "telemetry_file": "/data/logs/teams-telemetry/teams-telemetry-0.log"
          }]
      }}
    ]
  }
}

Four sections, and each one is there because leaving it out hurt.

instruction — what the agent is told, and nothing else

The natural-language goal. Deliberately the only thing the agent receives about the task. No hints about which button to press, no pre-seeded plan.

That discipline matters. It is very tempting, when a task keeps failing, to enrich the instruction until it is effectively a script — “click the Chat icon in the left rail, then…”. At that point you are no longer testing whether an agent can accomplish a goal; you are testing whether it can follow directions. Both are valid tests. They are not the same test, and conflating them means your numbers mean nothing.

workflow.params — the budget, per task

The circuit breakers live here rather than in global config, because the right budget is a property of the task. “Open Notepad and type a word” needs three plan versions. “Scroll a long chat history and summarise it” needs twenty.

Keeping them per-scenario also makes budget an explicit, reviewable part of the test rather than a global constant somebody bumps at 2am to make a run pass.

environment.start — the part that makes results comparable

This is the section that solves the drift problem, and it is the one I would argue hardest for.

{ "func": "close_all_windows", "args": null },
{ "func": "delete_files", "args": { "paths": ["/data/logs/.../teams-telemetry-0.log"] } },
{ "func": "start_network_proxy", "args": null },
{ "func": "open_application", "args": { "app_name": "ms-teams" } }

In order: clear the desktop, delete the telemetry file from the last run, start traffic capture, launch the app under test.

Each of these came from a real bug.

• close_all_windows — an agent that started with a leftover Explorer window behaved differently from one that started clean, and the difference looked like model variance until we controlled for it.
• delete_files — the telemetry capture appends. Without a clean-out, run 3’s evaluation sees run 1 and 2’s events and cheerfully reports success. This is a real result we got, and it was wrong, and finding out was unpleasant. There is a commit called clean up file before we start a run and it fixed our numbers, not our code.
• start_network_proxy — start capture before the app, or you miss the startup events.
• open_application — the agent should not spend three plan steps finding Teams. Getting the app open is scaffolding, not the thing under test.

environment.end — teardown and grading

Symmetric with start, and it does two jobs. It tears down (stop the proxy, close windows) and it grades — get_evaluation_results runs the evaluator against artifacts the run produced.

Putting grading in teardown rather than in the agent is the important call. The agent finishes and reports what it thinks happened. Then, entirely separately, teardown inspects the world and decides what actually happened. The agent has no influence over its own grade, which given how confidently agents report success is a property you want structurally rather than by good intentions.

The function registry

The func strings resolve against a registry of small, single-purpose modules:

agents/functions/default/
├── close_all_windows.py
├── delete_files.py
├── get_evaluation_results.py
├── open_application.py
├── start_network_proxy.py
├── stop_network_proxy.py
└── teams_sign_in.py

Adding a capability is adding a file. A scenario author composes from what exists, and when they need something new the diff is one small module, reviewed on its own.

The same registry pattern shows up again on the platform side, where the test runner needs start_recording / stop_recording / get_telemetry / maximize_window / wait_time. Same idea, different deployment — which is itself an argument that the abstraction was the right size.

The road not taken: rewards

There is an unused file in the repo, prefixed with __ so nothing loads it, that sketches a richer grading model:

"rewards": [
  {
    "name": "reward1",
    "points": 10,
    "conditions": [
      { "func": "file_exists", "args": { "path": "C:\\test-folder\\test.txt" }, "expected_result": true }
    ]
  },
  {
    "name": "final_goal",
    "points": 100,
    "conditions": [
      { "func": "is_on_screen", "args": { "text": "Last message ... 'Hello my friend'" }, "expected_result": true }
    ]
  }
]

Partial credit. Instead of binary pass/fail, points for intermediate milestones — so “got 60% of the way and then got stuck at the summarise step” is distinguishable from “never opened the app,” which under binary grading look identical.

We did not ship it. The reason is honest and slightly boring: designing a good reward decomposition per scenario is real work, and we had a more urgent problem in getting binary grading to be trustworthy at all. It stayed in the repo as a marker of intent.

I still think it is the right direction. Binary pass/fail on a twelve-step task throws away almost all the diagnostic signal, and when you are trying to work out whether a model change helped, “3/5 → 4/5” is a much worse instrument than “average score 62 → 78.”

JSON versus a DSL

The obvious criticism: this is a programming language with extra steps. No loops, no conditionals, no variables, and a growing pile of scenario-N.json.

All true. The counter-argument, which held for us:

• A non-engineer can write one. That was the whole point, and it worked.
• It diffs. A scenario change is a reviewable JSON diff, not a code review of imperative setup buried in a test helper.
• It serialises. The scenario is stored with the run, so a result from six weeks ago carries the exact configuration that produced it. That is very hard when your test is a Python function that closed over the state of the repo.
• Complexity has somewhere to go. When a scenario needs logic, that logic becomes a named function in the registry — reviewed, tested, reusable — rather than an inline if nobody else will find.

That last point is the real defence. The format stays dumb on purpose, and every time it is not expressive enough the pressure pushes complexity into a place with better tooling. A config format that resists becoming a language is a feature.

Next: how you actually know the agent succeeded — and why the answer had to live inside the guest.