Polling Cursor Checkpoint Crash Halfway Through Page

Polling remote APIs often fails midway through a page of results, leaving distributed systems in an ambiguous state where some items have processed and others remain untouched. When a background worker crashes after executing only a subset of records in a batch, advancing the global pagination cursor immediately causes silent data skips or duplicate side effects on the subsequent execution run. Distributed architectures require distinct boundaries between fetching remote batches and executing downstream mutations.

Quick solution:

async function processBatch(batch: Item[], cursorState: Cursor) {
  for (const item of batch) {
    await db.prepare('INSERT OR IGNORE INTO processed_items (id) VALUES (?)').bind(item.id).run();
    if (!(await isProcessed(item.id))) {
      await executeSideEffect(item);
      await markProcessed(item.id);
    }
  }
  await advanceCursor(cursorState);
}

Prerequisites include tracking individual item state in a reliable storage layer like the Cloudflare D1 database API before triggering external actions. Cautions apply when upstream APIs alter sorting order or purge records before the next polling cycle begins.

Understanding the Anatomy of Mid-Batch Failures

When scaling background data ingestion, developers frequently encounter scenarios where a worker consumes a page of fifty records, processes twenty successfully, and encounters a network timeout or memory limit on the twenty-first item. If the pagination state machine ties retrieval pointers directly to batch entry points without individual persistence, the system forgets which records succeeded. On reboot, the pagination engine advances past the entire page, treating the unfinished thirty items as successfully handled. This architectural flaw creates silent data loss that is difficult to diagnose in production logs.

To build resilient background services, engineers must decouple retrieval loops from execution engines. As detailed in the Cloudflare rules of Workflows, state transitions must remain explicit and deterministic across async boundaries. By recording granular item states before dispatching side effects, systems establish a durable audit trail that survives worker restarts and infrastructure scaling events.

When Is a Cursor Safe to Advance?

A pagination cursor represents retrieval progress rather than execution completion. If you advance the global cursor immediately after fetching a page, any item within that page that fails to complete before a crash is permanently skipped because the next polling cycle starts beyond it.

A cursor is safe to advance only when all items within the current retrieval window have either completed their side effects or have their individual retry states safely persisted for subsequent recovery passes. When designing polling architectures, the cursor must be tied to the tail of the persistence layer rather than the head of the ingestion queue. If three items out of a batch of fifty are still pending or failed, the global offset must remain stationary until those specific items resolve or move into a dead-letter queue. Moving the cursor prematurely trades data integrity for throughput, violating core reliability guarantees.

How Are Successful Items Distinguished After a Mid-Page Crash?

Distinguishing completed work requires separating the fetch phase from the execution phase through per-item state tracking. When a worker processes items sequentially, each record must transition through a recorded database state before external side effects occur. If a crash occurs halfway through the batch, the system re-runs the same page during the next polling interval.

Because individual items check their persistence record in the database before running side effects, already processed items are safely bypassed. This prevents duplicate actions without requiring the global cursor to leapfrog unhandled records. Consider a scenario where item IDs A through J are fetched. Items A through E finish successfully and record their completion hashes in the database. Item F throws an uncaught exception, crashing the worker process. When the scheduler restarts, the worker fetches the exact same page containing IDs A through J. The execution loop inspects the database, sees that A through E are already marked as complete, skips their side effects entirely, and resumes execution immediately at item F. This mechanism also relates to patterns analyzed when debugging silent skips in poll based reply bots where state verification prevents logic bypassing.

What Happens When the Retention Horizon Is Exceeded?

Upstream APIs typically enforce strict retention limits, expiring older events, messages, or database records after a fixed time window or volume threshold. If your processing pipeline stalls due to prolonged downtime, network partitioning, or unexpected downstream congestion, the retention horizon may be exceeded before recovery completes.

In this scenario, idempotency reduces duplicate effects for items still present in the upstream window, but it cannot recover events that the upstream API has already dropped or purged. If an upstream service purges data older than 24 hours and your worker remains down for 36 hours, resuming the cursor or re-running pages will yield empty results or missing IDs for that stale window. Monitoring ingestion lag against the upstream retention window prevents silent data loss when backlogs grow larger than the pagination cap. Engineers must configure alerting thresholds when backlog lag approaches seventy percent of the upstream retention limit.

Trade-Offs in State Persistence Strategies

Implementing per-item tracking introduces architectural trade-offs that teams must evaluate against throughput requirements. Writing every item state to a relational database before executing external side effects increases database write I/O significantly. For high-volume polling systems processing thousands of items per second, synchronous individual writes can saturate database connection pools and increase end-to-end latency.

Alternatively, batching database writes reduces I/O overhead but reintroduces the risk of inconsistent state if a crash happens between batch persistence flushes. Teams must choose between strict durability, which guarantees zero data loss at the cost of lower throughput, and buffered execution, which maximizes speed while accepting a small window of potential duplicate processing upon failure recovery.

Verification and Execution Checklist

• Confirm that a crash occurring before a side effect replays safely without duplication by inspecting the database state before and after the simulated failure.
• Verify that a crash occurring after a side effect preserves the same idempotency identity without re-triggering external actions.
• Test backlogs larger than the page cap to ensure older items remain visible within the upstream retention horizon without expiring prematurely.
• Audit database connection pooling configurations to ensure per-item state writes do not saturate available connections during peak load intervals.