Your AI Agent Crashed at Step 12: How to Resume Without Double-Charging Your Users

Picture this scenario:

You built an AI agent to handle customer refunds and travel booking adjustments. It executes a multi-step workflow:

1. Step 1 to 10: Read user emails, parse flight details, check cancellation policies, calculate penalty fees.
2. Step 11: Cancel the flight seat in the airline partner API.
3. Step 12: Charge the customer a $45 rebooking fee via Stripe.
4. Step 13 to 20: Issue new boarding pass, update CRM, send confirmation email.

The agent finishes Step 12. The credit card is debited.

And then... the server crashes.

Out of memory error. An AWS node failure. A container auto-restart. Whatever the reason, the process died instantly.

A few seconds later, the server restarts. What happens to that workflow?

• If the agent starts over from Step 1: It cancels an already cancelled flight, and charges the customer a second time.
• If the agent gives up and dies: The customer was charged $45, but never received their new ticket or confirmation email.

This is the dirty secret of building AI agents: getting an LLM to follow instructions is only 20% of the job. The remaining 80% is making sure the agent doesn't wreck your business when the server inevitably crashes.

Let's break down how to solve this, without drowning in academic jargon.

The Core Problem: "Side Effects"

In computer science, operations fall into two buckets:

┌─────────────────────────────────────────────────────────────┐
│                      20-Step Agent Flow                     │
│                                                             │
│  [Steps 1-11]  ──►  [Step 12: Charge $45]  ──►  💥 CRASH!   │
│     (Done)             (Side Effect Done)      (RAM wiped)  │
└─────────────────────────────────────────────────────────────┘
                               ▲
                      How do we safely resume
                       from Step 13 onwards?

When your agent runs purely in memory (in Python or Node.js memory), all knowledge of what just happened vanishes when the process crashes.

To survive a crash, we need three simple concepts:

1. Checkpointing (The Video Game Save State)
2. Idempotency Keys (The Order Receipt)
3. Smart Resumption (Fast-forwarding without re-executing)

1. Checkpointing: The Video Game Save State

If you play a 40-hour video game, you don't start from World 1-1 every time your console turns off. You save at checkpoints.

Your agent needs the exact same thing.

Never keep an agent's multi-step execution state solely in RAM. At every single step boundary, save a snapshot to a persistent database (PostgreSQL, Redis, DynamoDB):

{
  "workflow_id": "wf_order_8849",
  "current_step": 12,
  "status": "COMPLETED",
  "step_history": [
    { "step": 11, "action": "cancel_seat", "output": { "seat_id": "14B", "status": "released" } },
    { "step": 12, "action": "charge_customer", "output": { "charge_id": "ch_3Mv8", "amount": 4500 } }
  ],
  "agent_memory": { ... }
}

The Golden Rule of Checkpointing:

Write the checkpoint synchronously before and after any tool with side effects.

Before calling Step 12, set the status to IN_PROGRESS: step_12.
Once Step 12 succeeds, immediately commit COMPLETED: step_12 with its output.

If the machine explodes half a second later, the next worker that boots up queries the database and immediately knows: Step 12 is already done.

2. Idempotency Keys: The Elevator Button Principle

What if the server crashes right in the middle of Step 12?

Did the bank charge the card, or did the network die before the request reached them? You don't know.

If you retry, you risk double charging. If you don't retry, you might skip billing.

This is where Idempotency saves you.

Idempotent means: No matter how many times you perform an action, the end result is identical to performing it once.

Think of an elevator call button: pressing it once calls the elevator. Pressing it 15 times in rapid frustration does not send 15 elevators.

Request 1: POST /charges { amount: $45, key: "wf_8849_step_12" } ──► Card Charged ($45)
💥 Crash before response saved!
Request 2: POST /charges { amount: $45, key: "wf_8849_step_12" } ──► "Already processed. Here is the receipt."

How to use Idempotency Keys with AI Agents:

Whenever an agent prepares to run a tool with side effects, assign it a deterministic key derived from the workflow ID and step number:

idempotency_key = f"{workflow_id}_step_{step_number}"

# When calling Stripe, Shopify, or your internal API:
stripe.Charge.create(
    amount=4500,
    currency="usd",
    customer=customer_id,
    idempotency_key=idempotency_key # Stripe guarantees this is only billed once!
)

If your server crashes, restarts, and accidentally retries Step 12, the external service says: "I already processed that exact key 30 seconds ago. Here is the existing receipt." No double charge. No damage done.

3. How to Actually Resume the Agent

When the worker wakes up and sees a workflow stopped at Step 12, how does it continue?

There are two main ways to resume an AI agent:

Approach A: The State Machine / DAG (Best for predictable flows)

If your workflow is a pre-defined graph (like LangGraph, Temporal, or AWS Step Functions):

1. Load the state from database.
2. See that Step 12 is marked COMPLETED.
3. Pass Step 12's output directly into the input of Step 13.
4. Run Step 13.

The LLM is only called to think for Step 13. Earlier steps are never re-run.

Approach B: Trajectory Replay / Tool Memoization (Best for open-ended agents)

If your agent is an autonomous LLM dynamically choosing its own tools in a loop:

1. Rebuild the conversation history.
2. For steps 1 through 12, feed the model its own past thoughts and the cached tool outputs from the database.
3. Do NOT run the actual tool functions again. When the code sees the agent calling charge_customer, it intercepts the call, grabs the saved result from the database, and returns it immediately:

def execute_tool(tool_name, tool_args, workflow_history):
    # Check if this exact step was already performed
    if tool_name in workflow_history:
        print(f"[REPLAY] Returning cached result for {tool_name}")
        return workflow_history[tool_name].result
    
    # Otherwise, execute for real
    return run_real_tool(tool_name, tool_args)

The LLM sees that Step 12 already succeeded, inspects the receipt, and naturally decides: "Great, step 12 is done. Now I will proceed to Step 13: issue the boarding pass."

Summary Checklist: Making Agents Crash-Proof

If you are putting AI agents into production with access to real tools, ask yourself these 4 questions:

1. Are read actions separated from write actions?
   Mark every tool in your code as either read_only=True or has_side_effects=True.
2. Is state saved to disk after every action?
   If your server gets unplugged right now, can your database tell you the exact tool output of the last step?
3. Does every write tool support an idempotency key?
   If an action gets retried by accident, will it safely return the previous result without repeating the side effect?
4. Is your tool executor capable of replaying cached outputs?
   Can your agent "catch up" by reading past results from disk rather than calling external APIs again?

Final Thoughts

The magic of generative AI makes it easy to forget that an AI agent interacting with tools is fundamentally a distributed system.

The LLM is just the brain that decides what to do next. The plumbing around it—persisting state, guaranteeing idempotency, and handling recovery—is what turns a fragile prototype into software you can trust with real customers and real money.