My agent's output tests stayed green while it refunded a customer twice
A developer stress-tested agent testing with a synthetic support agent: the payments API committed a refund then timed out, the agent retried without an idempotency key, and the customer was refunded twice. Every output test passed because they only asserted on the agent's reply, which was correct and polite. The lesson: assert on the state after the run — ledger entries, idempotency keys, tool-call sequences — not on what the agent said.