Crash Recovery and Durable Execution in Always-On AI Agents

Moving an AI agent from an interactive CLI session to an autonomous, 24/7 background daemon introduces a severe distributed systems failure mode: interrupted multi-step execution. Quick solution: Implement a write-ahead log in SQLite alongside deterministic SHA-256 idempotency keys derived from task IDs and step indices before dispatching state-mutating tool calls.

import hashlib
import json

def generate_idempotency_key(task_id: str, step_index: int, arguments: dict) -> str:
    payload = f"{task_id}:{step_index}:{json.dumps(arguments, sort_keys=True)}"
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

Prerequisites for this architecture include a reliable local storage engine such as SQLite configured with write-ahead log mode, and external APIs that accept idempotency tokens in their request headers. If the runner crashes at step seven due to an out-of-memory error, a naive supervisor restarts the agent from step one. This causes runaway token waste by re-querying the language model for all prior steps, and triggers side-effect disasters such as duplicate git branches, duplicate pull requests, or multiple webhook notifications because the agent re-executes external mutations from scratch.

The Non-Determinism and Token Waste Dilemma

Because large language models are inherently non-deterministic, restarting a failed multi-step task from scratch may produce an entirely different reasoning path on the second run. This divergence often leaves orphaned resources scattered across external systems. Furthermore, re-running completed steps burns thousands of unnecessary input and reasoning tokens. In production environments, this operational cost scales linearly with the frequency of background daemon restarts.

To prevent these failures, agent engineers must borrow architectural patterns from database theory, specifically write-ahead logging and idempotent state machines. You can examine how similar reliability guarantees are implemented in distributed systems by reviewing how Temporal works. By logging every intent before execution and caching intermediate outputs, the agent runtime maintains strict state consistency without needing a heavy external orchestration cluster.

The Three-Stage Durability Protocol

To guarantee crash resilience and zero duplicate mutations, the agent daemon implements a strict three-stage durability protocol comprising intent logging, deterministic dispatch, and atomic state committing.

1. The Write-Ahead Journal Pattern

Before executing any tool that produces side effects, the daemon writes a pending execution journal entry to persistent disk storage or a local SQLite database with the status marked as pending, along with the step index, tool name, and arguments hash. Only after the tool returns successfully is the journal entry updated to committed alongside its serialized output.

2. Tool Classification: Read-Only Versus State-Mutating

The runtime classifies all available agent tools into two distinct operational tiers:

• Idempotent and Read-Only: Tools like reading files, performing grep searches, listing directories, or fetching web pages are safe to replay. If a crash occurs, these can be re-executed safely or satisfied immediately from the local step cache without interacting with external APIs.
• Transactional and State-Mutating: Tools like committing code, creating pull requests, sending webhooks, or updating databases must be protected. Every mutation tool call requires a deterministic idempotency key generated from the task identifier and step sequence. External APIs receive this key, guaranteeing that duplicate execution attempts resolve cleanly as safe no-ops.

3. Replaying Context Without Token Re-Spend

When a crashed agent daemon boots up after an unexpected termination, it inspects the local task journal rather than feeding the raw conversation history back to the model. It re-hydrates the working memory context using previously committed outputs. The language model is only called for the unresolved step, preserving call budgets and avoiding reasoning divergence.

Protecting External APIs with Idempotency Keys

When a mutation tool executes, it must pass a stable idempotency key that survives process restarts. If the daemon crashes immediately after the external API receives the request but before the local journal can record the success status, the supervisor will trigger a retry upon reboot. When the retry dispatches the same request with the identical idempotency key, the external provider recognizes the duplicate identifier and returns the original response instead of executing the mutation a second time.

For engineers automating background tasks reliably across infrastructure boundaries, reviewing implementation patterns like Count Publication Attempts Without Double Counting Retries offers valuable context on managing stateless background loops and handling idempotency safely. Always ensure that your cloud functions or worker scripts handle retries idempotently, as network partitions and timeouts can cause duplicate delivery attempts from upstream webhooks or message queues.

Production Decision Matrix

Dimension	Naive Loop Retries	In-Memory Snapshotter	Journaled Durable Execution	Recommendation
Crash Recovery	Starts over from Step 1	Recovers if snapshot intact; lost on OOM kill	Recovers exactly at last uncommitted step	Use journaled durable execution for production daemons
Token Cost on Failure	Full re-burn of all prior steps	Partial re-burn of active window	Zero re-burn for completed steps	Minimizes operational inference costs
Mutation Safety	High risk of duplicate PRs and commits	Moderate risk	Zero duplication via deterministic keys	Essential for external API integrations
Infrastructure Overhead	None	Low	Low (single SQLite WAL table)	Balances robustness with operational simplicity
Divergence Risk	High due to fresh LLM roll	Moderate	Zero as prior step outputs are pinned	Maintains deterministic execution flow

Verification and Execution Checklist

• Write-Ahead Log table created in SQLite with columns for task_id, step_index, tool_name, args_hash, status, output, and created_at.
• Simulated SIGKILL crash test executed by killing the agent process with kill -9 midway through step four of an eight-step workflow.
• Verified that upon daemon reboot, steps one through three load directly from the local disk cache with zero LLM API calls.
• Verified that step four presents the exact same idempotency key and completes successfully without duplicating external resources.
• Measured token consumption by comparing resume token count versus full restart token count to verify greater than seventy percent token savings on crash.