The notification sender

The idea

People often say "notifications" to mean the whole feed-and-fan-out system. This page is about a narrower, grittier piece: the outbound delivery worker that actually hands each message to email, SMS, or push providers — and refuses to lose it.

The shape is always the same. A durable queue decouples whoever produces a notification from the worker that sends it. The worker dequeues a message, checks the recipient's preferences and quiet hours, dedups against an idempotency key, routes it to the right channel, and calls a third-party provider. Providers fail constantly in small, recoverable ways, so the worker retries with exponential backoff. When something can never succeed, the message goes to a dead-letter queue instead of spinning forever.

See it work

How it works

The worker is a loop. It pulls one message, refuses to send a duplicate (the idempotency key it already delivered), tries the provider, and on a transient error it sleeps for a growing delay before retrying. The backoff is min(cap, base · 2^attempt) plus a little random jitter so a thousand workers don't all retry on the same beat. When the attempts run out — or the error is permanent — the message is parked in the dead-letter queue rather than blocking the line.

BASE, CAP, MAX_ATTEMPTS = 1.0, 30.0, 5   # seconds

def worker_loop(queue, dlq, store):
    while (msg := queue.dequeue()) is not None:
        # 1. dedup: have we already delivered this exact key?
        if store.already_delivered(msg.idempotency_key):
            queue.ack(msg)                 # safe no-op, drop it
            continue

        # 2. respect the recipient before spending a provider call
        if not allowed_now(msg.recipient, msg.channel):   # opt-out / quiet hours
            queue.ack(msg)
            continue

        for attempt in range(MAX_ATTEMPTS):
            try:
                provider = route(msg.channel)              # email / sms / push
                provider.send(msg, idempotency_key=msg.idempotency_key)
                store.mark_delivered(msg.idempotency_key)  # record before ack
                queue.ack(msg)                             # remove from queue
                break
            except PermanentError:        # bad number, opted out, 400-class
                dlq.push(msg, reason="permanent")
                queue.ack(msg)
                break
            except TransientError:        # 429 / 503 / timeout
                if attempt == MAX_ATTEMPTS - 1:
                    dlq.push(msg, reason="retries_exhausted")
                    queue.ack(msg)
                    break
                delay = min(CAP, BASE * 2 ** attempt)
                delay += random.uniform(0, delay)          # full jitter
                sleep(delay)                               # then retry

Note the order: mark delivered, then ack. If the worker crashes after sending but before acking, the message reappears and is retried — but the dedup check (and the provider's own idempotency key) make that re-send a harmless no-op. Acking before the send would silently drop messages on a crash.

Cost / trade-offs

Watch out for

• No idempotency key. At-least-once delivery will re-deliver on retry or crash recovery. Without a key the recipient gets the same text twice. Generate the key when the notification is created, not when you send.
• Retrying permanent failures. A 400 invalid phone number or an opted-out recipient will never succeed. Retrying it five times wastes quota and delays everything behind it — classify errors and dead-letter permanent ones immediately.
• Backoff without jitter. If every worker uses the exact same base · 2^attempt, they all retry in lockstep and hammer a recovering provider in synchronized waves. Add randomness so the herd spreads out.
• Ignoring provider rate limits. Blasting past the provider's quota earns you 429s, then throttling, then a worse outage than you started with. Track tokens per provider and pace sends.
• Acking before the send. Remove the message from the queue only after delivery is recorded. Ack-then-send loses messages on any crash in between.
• Quiet-hours and timezone bugs. "10pm" is meaningless without the recipient's timezone. Compute quiet hours in their local time, or you'll wake people at 3am.
• Unbounded retries. A message that never succeeds and never gives up blocks the worker (or pins a partition). Always cap attempts and dead-letter on exhaustion.

Worked example

Message A — transient 503, then delivered. An order-shipped email is enqueued (key=ord-91:shipped). The worker dequeues it, sees the recipient hasn't opted out and isn't in quiet hours, routes to the email provider, and calls send.

Attempt 1 returns 503 service unavailable — transient. With base=1s, the delay is min(30, 1·2^0)=1s, plus jitter, so it waits roughly 1–2s and retries. Attempt 2 returns 503 again; delay is min(30, 1·2^1)=2s plus jitter (~2–4s). Attempt 3 returns 202 accepted — delivered. Because the same idempotency key rode along on every attempt, even if attempt 1 had actually sent before the connection dropped, the provider would have deduped it. Delivered count +1, message acked.

Message B — dead-lettered. An SMS to +1-555-0000 is enqueued. The worker routes to the SMS provider, which returns 400 invalid number — a permanent error. There is no point retrying a number that can't exist, so the worker pushes the message to the dead-letter queue with reason="permanent" and acks it. The DLQ count goes to 1; an operator (or a repair job) can inspect it later, but the live worker moves on instead of looping forever.

Check yourself