When an automation fails: a recovery runbook for small teams

Define the stop rule before the next failure

Croissant recommends writing this plan while the workflow is healthy. Name a recovery owner and a backup. Record where an authorized operator can pause new actions, where incoming work will wait, and which business deadline triggers manual handling. A pause that discards incoming requests needs a different fallback from one that preserves them in a queue. Test that distinction with non-customer records.

Separate urgent business work from technical repair. An operations owner may reconcile records and assign tasks; a system owner may investigate the integration. Specify who can approve external messages, financial changes or deletions. Recovery should retain the same approval boundaries as normal operation. A broken workflow does not grant broader authority.

Contain the failure and build a recovery list

When the alert arrives, record the time and time zone, workflow name, last known successful item and observed symptom. Pause the smallest affected part using an approved control. Preserve incoming work and execution history. Avoid deleting failed runs to tidy the dashboard; they may contain the evidence needed to distinguish unfinished work from completed actions.

Create one recovery list in a tool your team already uses. For each business item, record its stable reference, current owner, deadline, evidence of completed steps and next permitted action. Use explicit statuses such as confirmed complete, confirmed incomplete and uncertain. Keep sensitive message bodies and credentials out of this list; link to access-controlled records when needed.

Check scheduled retries before anybody starts manual work. Stripe documents that manually resending a failed webhook event does not cancel its automatic retries, even after a successful response. That behavior is specific to Stripe, but it illustrates why a recovery owner must inspect the actual integration settings rather than assume a manual fix has stopped background activity. ^[1]

Check the destination before choosing a restart

A failed workflow run does not, by itself, tell you the outcome of every business step. Open the destination record and compare it with the execution history. If a task already exists, attach its identifier to the recovery list. If a message outcome is uncertain, ask an authorized operator to check available delivery evidence before considering another send.

Restart options differ by platform. AWS Step Functions documents redrive for eligible failed Standard Workflow executions, preserving successful steps while continuing from the unsuccessful step. Its documentation also lists exceptions for certain parallel and map failures. These rules do not establish how your own automation platform behaves. Have its owner confirm the exact restart scope before using a replay button. ^[2]

Choose an action for each item: close it with evidence, finish only the missing step, or hold it for investigation. Our guide to [preventing duplicate emails and tasks](/blog/prevent-duplicate-ai-actions/) explains stable identifiers and uncertain outcomes in more detail. Keep the recovery list focused on business completion and ownership.

Worked example: a report workflow stops after creating tasks

Consider a hypothetical weekly reporting workflow. It prepares an internal summary, creates follow-up tasks and then posts a team notification. On Monday, the run reports a failure at the notification step. This is an illustration, not a customer incident or a measured result.

The operations owner pauses the affected workflow through its approved control and inspects the destination systems. The summary exists. Four follow-up tasks exist, each linked to the original reporting period. The team notification is confirmed absent. The owner records those facts and links the existing summary and task identifiers in the recovery list.

The team needs the notification before its planning meeting. An authorized colleague posts it manually, records the message link and marks that step complete. The system owner then checks whether pending retries can repeat the notification, using the platform’s documented controls. The owner does not rerun the whole report merely because the run still displays a failure.

Before the next scheduled run, the team confirms that this reporting period is handled and that the next period will be processed normally. If the platform cannot exclude the completed notification safely, the affected automation stays paused while the owner agrees a supported recovery path. The manual fallback remains visible until that decision is resolved.

Run the manual fallback through the same queue

Give manual work a named owner and a completion record. A colleague should claim an item before acting, check the destination, complete the approved step and attach evidence. Use the same recovery list for every operator. Separate spreadsheets passed around in chat make it harder to establish which version reflects the latest work.

Limit the fallback to what your team can verify. For a report, that may mean delivering the checked summary without generating speculative analysis. For a customer-facing workflow, a person may need to review the actual message before sending. Use your existing approval process; our [human approval guide](/blog/ai-agent-human-approval/) can help define that boundary.

Resume in a small batch and reconcile the backlog

Croissant recommends a controlled restart after the cause is understood and an authorized owner confirms the repair. Test a representative item with harmless data where possible. Then process a small, reviewed portion of the backlog. Check destination outcomes before allowing the remaining items to proceed. Set a stop rule for recurring errors or unexplained actions.

Compare the original incoming list with confirmed completions, held items and manually completed work. Every item needs a disposition and an owner. Close the incident when business work is accounted for and the responsible owner accepts the restart evidence. Record what changed in the runbook while the details are fresh.

Know the limits of this runbook

This is an operating template, not a guarantee of recovery. Your ability to reconcile depends on retained source records, destination evidence and supported pause or replay controls. An irreversible action may need a separately approved correction. Missing evidence should stay marked uncertain rather than becoming an invented success.

If you want help mapping these steps to your existing tools, [bring Croissant one workflow to review](/#contact). Include its business deadline, current owner and a sanitized description of the failure. Keep passwords and private customer records out of the enquiry.

Implementation checklist

• Name the recovery owner, backup and authorized technical owner.
• Document the pause control and how incoming work is preserved.
• Set a business deadline for starting manual handling.
• Record each affected item and evidence of completed steps.
• Inspect pending retries and the documented restart scope.
• Assign manual items before acting and record completion evidence.
• Preserve approval requirements for messages and sensitive actions.
• Resume with a reviewed batch and a clear stop rule.
• Reconcile every backlog item and update the runbook.