The Next BeaconWhen a train is cancelled, Perron tells everyone who follows that station. On a quiet Tuesday that's a handful of phones. On the evening a storm closes half the network, it's thousands, all at once, and all at the moment people need it most.
For a long time we sent those notifications inline, from the same scheduled job that noticed the disruption. It worked in testing and it was fragile in exactly the conditions that mattered: one slow answer from the push service held up everything behind it, and a big alert could eat the job's whole budget. In October we moved the fan-out to Cloudflare Queues.
The queue itself is the easy part. This post is mostly about the edge cases, because with push notifications the edge cases are the work.
Why the inline version broke
Perron's five-minute cron does a lot. It flushes statistics, checks quotas, polls the national disruption feed, works out who's owed a delay refund, and sends notifications for all of that. On the free plan, one invocation got 50 subrequests in total. Every push request to Expo was a subrequest, and so was everything else the cron did.
We'd already been bitten by that budget once: eleven jobs each sized against 50 on their own, together spending about a hundred, failing silently every five minutes. We wrote about that in the cron triggers post. Push fan-out was the part of the job most likely to blow the budget, because its size depends on how many people follow the affected stations, and that's exactly the number that spikes during a storm.
A queue fixes that structurally. The cron hands the work over and moves on. Delivery happens in a separate consumer invocation, with its own budget, and if Expo is slow, it's slow in the consumer, not in the job that also has to notice the next disruption.
At the time, Queues needed Workers Paid. That was the push to upgrade, and Perron moved to the paid plan on 2026-10-15. Cloudflare has since added a free allowance for Queues, so check the current pricing page if you're on the free plan.
The config
"queues": {
"producers": [{ "binding": "PUSH_QUEUE", "queue": "push" }],
"consumers": [{
"queue": "push",
"max_batch_size": 10,
"max_batch_timeout": 2,
"max_retries": 5,
"max_concurrency": 3,
"retry_delay": 30
}]
}
Small batches and limited concurrency are deliberate. In a storm, the queue fills up quickly, and what we want is a steady send rate that stays under Expo's limits, not a burst that gets us rate-limited and then a pile of retries. Three consumers, ten messages each, is plenty for our traffic and gentle on Expo.
max_retries only matters for a handler that throws. Normal delivery failures are handled differently, as you'll see.
We don't run a dead-letter queue. Anything that can't be delivered is acknowledged and recorded in analytics, and a dead-letter queue full of alerts about trains that left an hour ago wouldn't be something we'd ever replay.
One message, one push batch
We send through Expo's push service, not straight to Apple and Google. Expo accepts up to 100 notifications per request, so one queue message carries up to 100 targets, plus what the push is about and when it was first asked for:
interface PushJob {
v: 1;
kind: "disruption" | "delay_claim" | "lost_found";
station: string; // a station code, never a name
enqueuedAt: number; // first request time, kept across retries so the age stays honest
attempt: number; // 1 on the first send
targets: PushTarget[];
}
An Expo push token is the capability to notify that device. There's no separate credential: if you have the token, you can send to the phone. That shapes how we treat them. Tokens live in the queue message because the consumer needs them, and nowhere they don't have to be. They never go into analytics or logs.
A queue message can be at most 128 KB. A hundred targets with long notification texts can get close, so any chunk over 100,000 bytes is split in half, and halved again until it fits. The rest of the 128 KB is margin for the envelope.
If enqueuing fails part-way through a big fan-out, the remaining chunks are sent inline instead. We'd rather the cron take a little longer than drop an alert because the queue had a bad moment.
The staging Worker has no queue binding at all. When PUSH_QUEUE is missing, the same function sends inline. That keeps one code path for both, and it's what our tests use too.
The consumer, in order
For each message in a batch, the consumer asks four questions.
Can it be parsed? A malformed message is acknowledged and recorded, never retried. A poison message that retries forever is the classic queue bug, and there's no fixing a message by sending it again.
Is push switched off? We have an admin switch for push notifications, and it has to apply to messages already sitting in the queue, not only to new ones. If you flip a kill switch because something's sending garbage, you don't want the queue to keep draining garbage for ten minutes. If the switch can't be read, we deliver. Silently dropping alerts because of a config hiccup is the worse failure.
Is it too old? If so, it's acknowledged and counted as expired.
Only then do we send.
Every kind gets its own age limit
This is the part we're proudest of, because it's a product decision written as code.
A late notification can be worse than none at all. An alert that your train has moved to platform 7, arriving forty minutes late, describes a train that has already left. It doesn't just fail to help; it actively misleads. On the other hand, a message saying you're owed money for a delayed journey is worth telling you about all day. And a reply to your lost-and-found post sits somewhere in between.
const MAX_AGE_MS = {
disruption: 20 * 60_000, // 20 minutes
lost_found: 2 * 60 * 60_000, // 2 hours
delay_claim: 24 * 60 * 60_000, // 24 hours
};
The age is measured from enqueuedAt, the moment of the first request, and that field is carried through every retry. Otherwise a message that's been retried four times would look brand new each time, and the limit would never trigger.
Retry only what failed
Expo can partly succeed. Ninety of a hundred notifications accepted, ten not, because of rate limiting or a server error on their side. The tempting thing is to retry the message. That would send ninety people the same notification twice.
So the consumer sorts every target into one of three outcomes. Delivered: done. Refused for a reason a retry can't fix, like a device that has unregistered: counted as failed and dropped. Never reached Expo at all, because of a 429, a 5xx, a network error or a rate-limit error: those go back on the queue.
const requeueDelaySeconds = (attempt: number) => Math.min(300, 30 * 2 ** (attempt - 1));
// 30 s, 60 s, 120 s, 240 s, then capped at 5 minutes
if (out.unsent.length > 0 && attempt < 5 && ageMs + delay * 1000 <= MAX_AGE_MS[kind]) {
await env.PUSH_QUEUE.send(
{ ...job, attempt: attempt + 1, targets: out.unsent },
{ delaySeconds: delay },
);
}
msg.ack();
The original message is acknowledged, and a new, smaller message goes onto the queue with only the targets that never got through. We don't call msg.retry() for this, because that would resend every target in the message, including the ninety who already have it.
Before requeueing, the consumer checks that the retry will still land inside the age limit. There's no point scheduling a disruption alert for a retry at minute 22 when it expires at minute 20.
A handler that throws is a different case. That means a bug or something truly unexpected, not a delivery problem. The message is retried on its own with a 30-second delay, and after three crashes it's acknowledged and recorded. One bad message shouldn't take down the batch around it, and it shouldn't loop forever either.
We're honest with ourselves about one gap. If Expo accepted part of a request and then the connection dropped before we saw its answer, we can't know which part got through, and a retry may notify some people twice. It's rare. Getting a cancelled-train alert twice is annoying. Not getting it at all is worse, so we accept that trade.
Expo's HTTP 200 means "accepted", not "delivered". Each notification gets a ticket, and the real result arrives later as a receipt. We read the tickets for immediate refusals and treat them as failures. That's what lets us tell "this device is gone" apart from "Expo is busy".
The double-count bug
We caught this one before it shipped, but it's a nice example of how adding a queue changes what your numbers mean.
Before the queue, pushes were counted in two places: the caller counted what it asked for, and the sender counted what it sent. With everything inline, both happened in the same invocation and the totals lined up. With a queue in the middle, the caller doesn't know the outcome yet. If it kept counting, every queued push would be counted once when handed over and again when the consumer delivered it.
The rule we settled on: each push is counted once, by whoever actually learns its fate. The inline path returns delivered, and the caller counts that. Anything put on the queue comes back as queued, and the caller counts nothing for it. The consumer counts sent and failed once Expo has answered. Written down like that it sounds obvious. It wasn't, until we drew it.
Watching it
Every step writes one point to Workers Analytics Engine and one structured log line for Workers Logs. Enqueued, delivered, requeued, gave up, expired, suppressed, malformed, crashed, and one point per Expo error code. A point carries the kind, the station code, counts, the attempt number, how long the message waited and how old the job was. It never carries a push token or the notification text.
That turns questions like "are disruption alerts arriving in time during a storm?" into a query instead of a guess. We wrote about how we lay out those Analytics Engine points separately.
Recording never throws. Every analytics write is wrapped so a failure is swallowed. Losing an analytics point must never lose a push.
What it costs
On Workers Paid, Queues include a million operations a month, then $0.40 per million. A message usually costs three operations: a write, a read and a delete. Operations are counted per 64 KB, so a 100 KB message counts as two. There's no charge for egress. The free plan now includes 10,000 operations a day, with shorter message retention.
At a hundred targets per message, even a storm that notifies thousands of phones is a few dozen messages. The queue's cost is a rounding error next to everything else, which was never the point. The point was that the cron stopped failing at the one time it mattered.
If you're building this yourself
Decide how late is too late, per kind of notification, before writing any code. Carry the first-request time through every retry. Retry only the targets that never reached the push service, and acknowledge the original. Make malformed and kill-switched messages acknowledge, not retry. Decide who counts each push, once. And keep the tokens out of everything except the message that needs them.
None of these are Cloudflare-specific. Queues just made them easy to put in one place.
More from the same app: the Workers free plan limits we designed around for months, and the rate limiting binding.