Durable checkpoints: resume, cancel, replay, and fork

A twelve-step agent run that dies at step nine has already spent the money for steps one through eight. If your only recovery is to start again, you pay twice and you wait twice. Durable checkpoints are the fix, and they are less about reliability theatre than about not repurchasing work you already own.

RunState with SQLite checkpoints under .swarm/ is how Swarm 357 handles it.

Four operations on the same state

swarm inspect <id>      # state, step, and anything pending
swarm resume <id>
swarm cancel <id>
swarm replay <id>
swarm fork <id>

All of these are Beta, tracked on the status page.

Cancel is the one people get wrong

The intuitive implementation holds a cancellation token in memory and checks it between steps. That works in one process on one machine, which is not where your runs live.

The problem: the process receiving the cancel is usually not the process executing the step. An operator hits cancel through the API, the API process sets a flag in its own memory, and the worker never hears about it. Meanwhile the run keeps spending.

So cancel persists. cancel_requested is written onto the checkpoint, the executing run reads it at its next safe boundary, and the flag survives a restart. Three properties follow:

• A restart does not resurrect a cancelled run. The flag is still there.
• Any process can observe the request, so swarm inspect tells you a cancel is pending even if the worker has not reached its boundary yet.
• The record of who cancelled and when outlives the process.

The tradeoff is that cancellation is not instantaneous. A run cancels at the next boundary, not mid-call. That is the correct trade, because tearing down mid-call leaves partial side effects with no record.

Resume needs a boundary you can trust

Checkpointing is only useful if the state at the boundary is coherent. Two rules make that true.

Checkpoint after side effects, not before. If a step writes a file and then the checkpoint fails, resuming re-runs the write. Idempotent steps survive that; most real steps are not idempotent. Record the effect, then checkpoint.

Keep the checkpoint boundary at the step level, not inside it. A half-finished model call is not a resumable state. Resume from the last completed step and repay for the one that was in flight. That is one step of waste, bounded and predictable.

This is also where approvals meet persistence. When a run pauses for human-in-the-loop approval, it may sit for hours. If pending approvals lived in memory, every deploy would strand every waiting run. Durable ApprovalRecord plus durable run state means the approval can arrive after a restart and the run continues.

Replay and fork are debugging tools

Replay re-executes a completed run. Its value is reproduction: a run behaved oddly, you fix a soul template, you replay to see whether the fix changes the outcome. Note that model outputs vary, so a replay that differs is evidence, not proof.

Fork branches from a checkpoint. Its value is comparison: you have a run that got to step six correctly and then went sideways. Fork at six, change the routing, and run both. You keep the original.

The habit worth building is forking rather than editing and re-running. Editing loses the artifact you were comparing against, and you will want it twenty minutes later.

What lives on disk

.swarm/
├── MEMORY.md          # pointer index
├── topics/            # flat-file knowledge
├── checkpoints/       # durable run state, SQLite
└── traces.jsonl       # structured events

Checkpoints hold run state. Topics hold knowledge. They are separate on purpose: you frequently want to discard a bad run without discarding what the agent learned, and occasionally the reverse. See portable agent memory with Memvid for the knowledge half.

Two operational notes. Back up .swarm/checkpoints/ if runs are long enough that losing one hurts, because a container without a persistent volume loses everything on redeploy. And retain traces longer than checkpoints, since traces are what you read during an incident.

When checkpoints are not worth it

• Sub-minute runs. Restarting costs less than the state management.
• Genuinely idempotent pipelines. If re-running from scratch is free and safe, that is simpler.
• Runs with no side effects and trivial cost. Checkpointing a cheap read-only summarisation is machinery for its own sake.

The threshold in practice is roughly: if a failed run would make you sigh, checkpoint it.

Try the control plane

pip install techtide-swarm
swarm init
swarm demo
swarm inspect

Then read checkpoints and resume, replay and fork, inspect, resume, cancel, and runtime files for what each path under .swarm/ holds.