Design a notification service

In 2 minutes

I would build a thin API in front of queues. A calling service posts a request with an idempotency key, and the API checks the key in a small store, so a retried call is not accepted twice. It replies at once, and a router does the real work. The router reads the user's preferences and quiet hours, renders the message, and puts one job per channel on that channel's queue. Every channel has its own queue and its own worker pool, and each has an urgent lane and a bulk lane, so a 10-million-message campaign, which takes about 83 minutes at a provider limit of 2,000 a second, never delays a login code. Workers call providers through small adapters behind a rate limiter, and a circuit breaker lets us fail over. Queues are at-least-once, so each notification has a stable ID that the worker records before and after sending. Temporary errors retry with backoff and jitter, permanent ones do not, and jobs that keep failing go to a dead-letter list. Preferences are checked again at send time. For login codes I prefer a rare duplicate over a miss.

The question

Clarifying questions

• Which channels do we support now, and which might come later? Each channel has its own provider, speed limit and cost, so adding one should mean adding a lane, not rewriting the service.
• Are there different kinds of message, such as login codes, order updates and marketing? A login code must not wait behind a million marketing messages, so kinds decide priority and which user preferences apply.
• Is a missed message worse than a duplicate one? For a login code, a duplicate is better than nothing. For marketing, it is the other way round. That picks the delivery guarantee.
• Do users control what they receive, and when? Preferences, unsubscribes and quiet hours must be checked at send time, and the law may require quick handling of opt-outs.
• Do callers need to know whether a message was delivered? Delivery receipts come from providers by callback, so tracking them needs a status store and a webhook endpoint.

Requirements

Functional

• Accept a notification request from internal services, with a user, a kind and a template.
• Choose channels from the user's preferences and send through the right provider.
• Retry failed sends safely, and never send the same request twice.
• Record a status for every notification: queued, sent, delivered or failed.

Non-functional

• Latency: high-priority messages such as login codes leave the system within 5 seconds at p99.
• Throughput: 12,000 notifications a second at peak.
• Reliability: no accepted notification is lost, even if a provider is down for an hour.
• A large campaign must never delay a high-priority message.

Back-of-the-envelope estimates

Assumptions for this exercise:

• These numbers are assumptions for the exercise, not facts about any real service.
• 20 million users receive 5 notifications a day on average, and peak traffic is 10 times the average.
• A provider accepts about 2,000 messages a second per channel. One worker handles 200 jobs a second, since it mostly waits on the network.
• A delivery record takes about 1 KB, and we keep 30 days.

The arithmetic:

• Notifications per second: 20M × 5 = 100M a day ÷ 86,400 s ≈ 1,157 → ≈ 1,200/s on average, about 12,000/s at a 10x peak
• Workers at peak: 12,000 ÷ 200 jobs per worker → ≈ 60 workers, scaled per channel
• A 10-million-message campaign: 10M ÷ 2,000 a second = 5,000 s ÷ 60 → ≈ 83 minutes, so campaigns need their own lane
• Delivery log storage: 100M a day × 1 KB = 100 GB a day × 30 days → ≈ 3 TB

High-level design

A Calling service posts a request to the Notification API. The API checks the request, and uses an idempotency key held in the Dedupe store to make sure a retried request is not accepted twice. It then hands the request to the Router and replies that it was accepted. The caller never waits for delivery.

The Router reads the user's Preferences, picks the channels, renders the message from a template and puts one job per channel on that channel's queue. The picture shows a Push queue and an Email queue. SMS is a third lane built the same way, with tighter cost limits.

Channel workers take jobs from the queues and call the Providers, the push, email and SMS services run by other companies. A job leaves the queue only after the worker has acknowledged it, so a crash means the job comes back, not that it is lost.

How a request flows

Accept and route a request

1. A service posts 'order shipped' with a user ID, a template name and a unique request key.
2. The API records the key in the dedupe store. A key seen in the last day means a repeat, and it is dropped.
3. A new request goes to the router, and the API replies 'accepted' to the caller without waiting.
4. The router looks up the user's preferences: which channels are on, and whether this kind of message is allowed.
5. The preferences say push and email are on, and it is outside quiet hours.
6. The router renders the push text and puts a job on the push queue.
7. It renders the email too and puts a second job on the email queue.

Deliver a push

1. A push worker takes the next job from the queue. The job stays invisible, not deleted.
2. The worker calls the push provider with the device token and the text.
3. The provider accepts the message, and later sends a receipt by callback.
4. The worker acknowledges the job, so the queue deletes it and the status becomes sent.

A provider fails

1. An email worker takes a job from the queue.
2. It calls the email provider, which is having an outage.
3. The provider answers 503. This is a temporary error, so the job is worth retrying.
4. The worker puts the job back with a delay that doubles on each retry. After five tries it goes to a dead-letter list.

Deep dive: channel queues, workers and providers

Give each channel its own queue and its own pool of workers. Push, email and SMS differ in speed, cost and failure mode. If they share one queue, a slow email provider holds up a push that could have gone out in a moment. Separate pools also scale on their own: 60 workers for push at peak, perhaps 10 for SMS.

Split by priority as well. A login code, a payment alert and a marketing campaign are different traffic. Use at least two lanes per channel: urgent and bulk. Workers drain urgent first, and bulk gets what is left. A 10-million-message campaign then takes its 83 minutes without delaying a single login code.

Each provider has a speed limit, and exceeding it brings errors or a ban. Put a token bucket in front of every provider call, set a little below the limit. When bulk traffic hits the limit, jobs wait in the queue instead of failing. The queue acts as a shock absorber.

Wrap each provider behind a small adapter with one interface: send, parse the receipt, classify the error. A new provider then means a new adapter and nothing else. With two providers per channel you can fail over when one has an outage, using a circuit breaker that stops calls to a provider after repeated failures and tries again after a pause.

Providers report results later through callbacks, called webhooks: delivered, bounced, unsubscribed. A small endpoint receives them, checks their signature and updates the status store. Push cannot promise delivery to a phone that is off, so treat 'delivered' as best effort.

Deep dive: retries, idempotency and dedupe

Queues give at-least-once delivery. A job is handed out, and if the worker does not acknowledge it in time, it is handed out again. So the same job may run twice, and the system must make that harmless. That is idempotency.

There are two layers of dedupe.

1. At the door. The caller supplies a request key. The API stores it with a one-day expiry, using an atomic set-if-absent. A second request with the same key is acknowledged but not processed again. This stops a caller that retries after a timeout.
2. At the worker. Give each notification and channel a stable ID. Before calling the provider, the worker marks it 'sending' in the status store, and marks it 'sent' once the provider accepts. If the provider supports idempotency keys, pass that ID along as well.

A gap remains. A worker can call the provider, the provider accepts, and the worker crashes before recording it. On retry it sends again. You cannot close that gap, only choose its direction. For login codes and security alerts, prefer a rare duplicate to a missed message. For marketing, prefer to skip it.

Treat errors differently. Permanent errors, such as an unregistered device token or an unsubscribed number, are not retried. Remove the token or mark the address. Temporary errors, such as a timeout, a 5xx or a 429, are retried with exponential backoff and random jitter, honouring any Retry-After header. After a fixed number of attempts, such as five, move the job to a dead-letter queue and alert a person.

Also collapse noise. Ten likes in a minute are better as one notification. Merge them by a collapse key within a short window.

Deep dive: preferences, rate limits and quiet hours

Store preferences by user, kind and channel: for example, order updates by push on, marketing by email off. Record when and how consent was given. Opt-outs must take effect quickly, so keep the preferences in a fast store, with a short cache in the router.

Check preferences at send time, not only at accept time. A job can sit in a queue for minutes. If the user unsubscribes in that gap, the worker must read the current preference before it calls the provider and drop the job if it is now off.

Quiet hours depend on the user's local time zone. If the time is inside the window and the kind is not urgent, do not send. Re-schedule the job for the end of the window. Many queues support a delayed delivery time, or you can keep a separate scheduled-jobs store that a timer drains into the queue. Urgent kinds, such as security alerts, ignore quiet hours.

Frequency caps stop the service from becoming a nuisance. Keep a counter per user, kind and day. For instance, at most three marketing pushes a day. When the cap is reached, drop the notification or fold it into a digest. A digest groups many items into one message at a set time.

Keep rules in one place so they apply to every channel and every caller. The router is that place. Callers say what happened, and the router decides whether, how and when to tell the user.

Trade-offs

• Delivery guarantee: At-least-once with idempotent workers over At-most-once, with no retries. No accepted message is silently lost. The cost is rare duplicates, which dedupe keys keep small.
• Queue layout: A queue per channel, with urgent and bulk lanes over One shared queue for everything. A slow provider or a big campaign cannot block urgent messages. The cost is more queues and pools to run and watch.
• Preference checks: At accept time and again at send time over Only when the request is accepted. A late opt-out is honoured even for queued jobs. The cost is one more lookup per job.
• Providers: Adapters with a fallback provider per channel over Call one provider directly. An outage at one company does not stop delivery. The cost is an adapter layer and two vendor contracts.

Common follow-up questions

How do you send a huge campaign without delaying password resets?
Put the campaign on the bulk lane, which has its own workers and a lower priority. Throttle it to a fraction of the provider's limit, and spread it over time. Urgent traffic is always drained first, so it never waits behind the campaign.

How do you know a notification was delivered?
Providers send receipts by webhook, and the service updates the status from queued to sent to delivered or failed. Push can only be best effort, since a phone may be off. For email, bounces and complaints come back later, and they feed the suppression list.

How do you handle a user with several devices?
Store one push token per device. The router creates one job per token, so each device is retried on its own. When a provider reports a token as invalid, delete that token and nothing else.

How would you add a new channel, such as in-app messages?
Add a queue, a worker pool and a provider adapter, and add the channel to the preference model. The API and router logic stay the same, apart from a rule for choosing the channel.