Playbook · Incidents

When an AI agent gets something wrong

Sooner or later an agent will refund the wrong order, delete the wrong record or send the wrong email. What you do in the first hour decides whether it stays an anecdote or becomes an incident.

Updated November 2026

In short

Stop the agent first: pause it or revoke its keys, so it can’t repeat the mistake. Then read the log for everything it did, undo what can be undone, tell the people affected, write down what happened, and change the rule that let it happen, usually by making that action wait for a person’s approval next time.

Your first hour

Pick what happened. Tick each step as you go; the order matters.

What happened

The first hour, in order

The order matters: a running agent can repeat a mistake while you are still reading about the first one.

  • Stop: pause the agent or revoke its keys. Don’t debug a running agent.
  • Look: read the log for everything it did, not only the mistake you noticed.
  • Undo: reverse what can be reversed; some things, such as a refund or a sent email, can’t be.
  • Tell: the people affected, and your team, in plain words.
  • Record: what happened, when, what you did, and what it cost.
  • Fix the rule: decide which action should have waited for a person.

Why it happened, usually

Agent mistakes look different every time, but the causes repeat.

  • The key could do more than the job needed.
  • An instruction hidden in an email, ticket or web page steered the agent.
  • Nothing stood between the proposal and the change: no approval step.
  • Nobody read its log, so a small mistake repeated.

Prevent the next one

Most agent incidents share one cause: the agent could act alone. Separate proposing from doing, so risky changes wait for a person; give each agent only the access its job needs; and keep keys where agents can use them without holding copies.

Checklist

Ready before it happens

  • You know how to pause each agent in one step.
  • You can revoke each agent’s keys in one place.
  • Every agent action is logged.
  • Irreversible actions wait for a person.
  • Someone owns each agent.

How MoltenRock prevents the usual cause

In MoltenRock Connect, changes in Stripe, Cloudflare and GitHub wait for your approval, so an agent can’t make them alone. In MoltenRock, keys are released per agent, and each release is logged.

Questions

Should we switch the agent off for good?

Not necessarily. Most incidents come from one action that should have needed an approval. Fix that, and the agent is safer than before.

Do we have to report it?

It depends on what happened and where you are; personal data and payments can carry duties. Ask someone who knows your obligations.

Who should own an agent?

The person who owns the system it works in: billing, the website, the code. They set its rules and read its log.

More guides