03 Sep 2026

The Durable Object Alarm That Called Itself

I wrote < where I meant <=. Over the last 30 days that character has cost me 8 billion database operations.

Some background. CartQL is a shopping cart API where every cart is its own Durable Object with its own SQLite database. Carts expire after 7 days of sitting untouched, and a Durable Object alarm handles the sweep: mark the cart abandoned, delete it, work out when to run again.

That sweep has been eating my account since March.

Reading the graphs backwards

A ratio gave it away. Over 30 days the worker served 191,000 requests and performed 8 billion storage operations. That is roughly 42,000 database operations per request, on an API whose busiest endpoint reads one cart row and its items.

Zooming into 24 hours said the same thing in smaller numbers: 7,000 requests, 336 million storage operations. Requests were trending down 12.64% and storage operations were climbing 7.47%, which is the shape of a problem that has nothing to do with traffic.

The error breakdown was the second clue. A 10.4% error rate, and the bulk of it sat in two buckets:

Exceeded memory limits: around 300
Client disconnected: around 346

A cart API that reads one row does not exhaust 128MB of memory. Something was looping.

Two functions, each calling the other

Here is the shape of the sweep, trimmed down.

alarm() finds the expired carts, deletes them, then asks for the next schedule:

const cutoffTimestamp = Math.floor((Date.now() - CART_EXPIRY_MS) / 1000);
const abandonedCarts = await this.storage.sql
  .exec(`SELECT id FROM carts WHERE updated_at < ? AND abandoned = 0`, [cutoffTimestamp])
  .toArray();

for (const cart of abandonedCarts) {
  await this._markCartAsAbandoned(cart.id);
  await this._deleteCartCompletely(cart.id);
}

await this._setNextAlarm();

_setNextAlarm() looks at the oldest cart and decides when the alarm should fire:

const alarmTime = updatedAt * 1000 + CartDurableObject.CART_EXPIRY_MS;

if (alarmTime > Date.now()) {
  await this.ctx.storage.setAlarm(alarmTime);
} else {
  console.log('Found expired carts, triggering immediate cleanup');
  await this.alarm();
}

Read those two together. alarm() calls _setNextAlarm() at the end. _setNextAlarm() calls alarm() when it finds something already expired. Two functions, each calling the other, with the cart table as the only thing that stops them.

That terminates fine as long as both agree on what expired means.

They disagreed by one second

The two conditions were written differently, and that is where the whole thing lives.

_setNextAlarm() recurses when updated_at * 1000 + EXPIRY <= now, which is updated_at <= (now - EXPIRY) / 1000. That right hand side is a float.

alarm() selects rows where updated_at < Math.floor((now - EXPIRY) / 1000). That right hand side is floored to a whole second first.

Call the float X, and the two tests become updated_at <= X for the reschedule and updated_at < floor(X) for the sweep. Since updated_at is stored as an integer number of seconds, there is exactly one value that satisfies the first and fails the second: updated_at == floor(X).

When a cart sits on that value, _setNextAlarm() sees an expired cart and calls alarm(). alarm() runs its query, matches nothing, deletes nothing, and calls _setNextAlarm(). Round it goes, awaiting all the way down, until the Durable Object exhausts its 128MB and it gets killed.

The window is one second of wall clock. Once the clock ticks past, floor(X) increments, the cart becomes eligible, the sweep deletes it, and everything reads as normal.

One second sounds survivable. It is not, and the reason is the constructor:

ctx.blockConcurrencyWhile(async () => {
  await this._migrate();
  await this._setNextAlarm();
});

Every instantiation runs that check before the DO will serve anything. So the object spins, dies on memory, and the next request wakes a fresh one that walks straight back into the same second. That accounts for the client disconnects too, because the DO never got far enough to answer.

The logs are unambiguous

Once I knew what to search for, the evidence was sitting there. The same line, the same durable object, the same millisecond, over and over:

17:19:20.230  cec904804b4a...  Found expired carts, triggering immediate cleanup
17:19:20.230  cec904804b4a...  Found expired carts, triggering immediate cleanup
17:19:20.230  cec904804b4a...  Found expired carts, triggering immediate cleanup

Then, eighteen seconds later, the same object finally getting on with it:

17:19:38.594  Marked cart 519B763D-71B4-4FB9-BA67-A9309F9D91BC as abandoned
17:19:38.594  Cart 519B763D-71B4-4FB9-BA67-A9309F9D91BC completely deleted
17:19:38.594  No active carts found, no alarm needed

Timestamps that identical are the tell. Nothing legitimate in a cart API runs thousands of times inside a millisecond.

Two lines

The first change closes the gap between the conditions:

- .exec(`SELECT id FROM carts WHERE updated_at < ? AND abandoned = 0`, [cutoffTimestamp])
+ .exec(`SELECT id FROM carts WHERE updated_at <= ? AND abandoned = 0`, [cutoffTimestamp])

Since updated_at is an integer, n <= X and n <= floor(X) are the same test for every value of n. The two conditions now agree everywhere and no cart can sit in the gap.

The second change stops the recursion:

- console.log('Found expired carts, triggering immediate cleanup');
- await this.alarm();
+ console.log('Found expired carts, scheduling immediate cleanup');
+ await this.ctx.storage.setAlarm(Date.now());

Both were needed. Fixing only the comparison leaves a function pair that can still call each other forever if I get a future condition wrong. Fixing only the recursion converts a stack overflow into an alarm storm, which is cheaper and still wrong.

The part I would pass on

Do not call your own alarm handler. Call setAlarm() and let the runtime invoke it.

The direct call builds a stack that never unwinds, so a scheduling mistake reaches you as an out of memory kill with no obvious cause and no useful trace. Going through setAlarm() gives the runtime a chance to pace the work, and the same bug arrives as a repeating log line you can read in ten seconds.

The alarm handler is a callback owned by the platform. Treat it as one.

There is a second habit worth stealing here, which is to watch ratios and let the absolute numbers alone. 8 billion looks alarming and tells you nothing. 42,000 operations per request against an endpoint you know reads one row is a number you can reason about immediately.

What else fell out

Debugging this meant reading the Durable Object properly for the first time in months, and a few other things turned up.

There were no indexes! Not a missing one on a hot column, none at all, on any table. Every lookup on cart_items(cart_id) was a full scan, and DO SQLite bills per row scanned.

At Turso, a customer would turn up with a mad query bill and my first question was always the same… have you got any indexes?

Six CREATE INDEX IF NOT EXISTS statements fixed it. Worth knowing: CREATE INDEX IF NOT EXISTS does apply to a table that already exists, so old carts pick the indexes up on next instantiation, which is not true of adding a column.

getCart is an N+1. It fetches the items in one query, then issues another query per item to get that item’s attributes.

Migrations do not belong in the app

The last one is the habit I want to break properly. _migrate() runs inside blockConcurrencyWhile in the constructor, so every instantiation executes nine CREATE TABLE IF NOT EXISTS statements, now with six CREATE INDEX IF NOT EXISTS on top, before the object will serve a byte. At a request rate low enough for these objects to hibernate between requests, almost every request pays for that.

I know how it got there. Durable Objects give every cart its own SQLite database, so there is no central database to point a migration tool at and no deploy step that can reach ten thousand of them. Putting the DDL in the constructor is the path the platform pushes you down, and it works, which is the dangerous part.

It has all the usual problems anyway. CREATE TABLE IF NOT EXISTS is a no-op on a table that exists, so a new column reaches new carts and silently never reaches old ones. There is no version tracking, no ordering, and no way to know which shape a given cart is in. The schema is also written twice, once as Drizzle definitions and once as raw SQL in the constructor, kept in step by hand.

The fix is a stored schema version: read it on construction, apply anything newer, write it back, and do nothing at all on the common path. That is a migration runner, and it belongs behind a version check whichever way the database is shaped. Every cold start currently pays for DDL that ran correctly months ago.

None of this caused the incident. It was all sitting behind it, and I would have carried on not knowing about any of it.

I have been handing out this advice for years. Put indexes on the columns you filter by, and keep migrations out of the application. Both were broken in my own cart API, and it took a month of $20 bills I finally got sick of looking at before I went and checked.