How to actually evaluate an agent swarm

Evaluating an agent swarm means measuring two systems, not one. A single-agent eval tests a prompt, its tools, and the model behind it. A swarm eval tests routing, handoffs, budget behaviour, and whether the coordination overhead bought anything. Report them together as one average and you have built a number that cannot tell you what broke.

I write this down because I got it wrong first. I had a suite that mixed both and reported a single success rate. It went from 82 percent to 74 percent one week. I spent two days on prompts before finding the real cause, which was a routing change sending a task class to the wrong layer. The prompts were fine. The average had hidden that.

Split the report

swarm eval reports single-agent and swarm results separately, and that separation is the most useful thing about it.

When single-agent scores hold and swarm scores drop, the problem is coordination. When single-agent scores drop, no amount of routing work will save you. That diagnostic is the entire return on keeping them apart.

Measure four things, not one

Success rate on its own rewards a system that succeeds expensively and slowly.

Task success. Did the output meet the criterion. Define the criterion before the run, in the task file, not after you have read the output.

Cost per task. Dollars, from the same ledger the runtime uses to enforce budgets. Not an estimate computed separately, or you are grading your own estimator.

Tokens per task. Correlates with cost but drifts from it as model prices change, so keep both. Token counts let you compare a run from March against a run from July.

Cap-hit rate. How often runs hit the budget ceiling or the agent cap. A rising cap-hit rate means your tasks are outgrowing their budgets, and it shows up here before it shows up as a quality regression.

Cap the eval itself

An eval suite is a fan-out. It will happily spend real money proving that your fan-out controls work.

Every eval run goes through the same reserve-and-commit budget ledger as production, with the same per-agent budget_limit_usd and per-layer caps. Two practices on top of that:

• Iterate against cheap tool-capable models through OpenRouter, and reserve premium models for the baseline runs you intend to publish. Free models tend to fail tool calling, which produces failures that are about the model rather than your system.
• Keep the suite small enough to run often. A 500-task suite nobody runs is worth less than a 40-task suite that runs on every change.

Baselines belong in version control

The rule I hold to: no published number without a committed artifact behind it.

Baselines live in evals/baselines/latest.json in the core repo, and published assets are rendered from that file rather than typed by hand. That gives you three properties.

Numbers cannot drift silently. If the site claims a figure, the file says the same figure, because the file generated it.

Regressions are diffable. A baseline change is a pull request with a visible delta, reviewed like code.

Claims have a date. Every number carries the run that produced it, so nobody has to guess whether a figure is from March or from a demo two years ago.

The latest baseline page is that render. Reproducing has the steps to run it yourself, which is the only real check on anyone's eval numbers, including mine.

Numbers I do not trust, including my own

Any success rate without a task catalog. If I cannot see the tasks, the number tells me the vendor chose the tasks well. The task catalog exists so you can judge the difficulty yourself.

Comparisons across different task sets. Framework A at 91 percent and framework B at 84 percent on different suites is not a comparison, it is two facts placed near each other.

Anything without cost attached. A system that hits 95 percent at ten dollars a task is not better than one that hits 88 percent at forty cents, unless your task is worth more than ten dollars.

Single-run results. Model outputs vary. One run is an anecdote. Report the spread or say you did not measure it.

When formal evals are not worth it

• Fewer than ten recurring tasks. Read the output. You will learn more, faster.
• The criterion is subjective. If you cannot write down what success means, an eval harness will measure something adjacent to what you care about and give you false confidence.
• The system is changing daily. Baselines are for detecting regression against a stable target. Against a moving one they generate noise and you will start ignoring them, which is worse than not having them.

Run it

pip install techtide-swarm
swarm init
swarm eval

swarm eval is Beta, tracked on the status page along with everything else. Read methodology for how tasks are scored, task catalog for what is in the suite, and cost control for LLM agent fleets for the budget primitives the eval harness reuses.