Design a chat app

In 2 minutes

Each phone keeps a WebSocket open to a gateway. A gateway only holds sockets: it registers the user in a session registry that maps user to gateway, and marks presence. At about 100,000 sockets per machine, 20 million connections need around 200 gateways. A message goes to the chat service, which gives it the next sequence number in its conversation, writes it to a store keyed by conversation and sequence, and only then counts it as accepted. The app attaches an ID of its own, so a retry returns the same record and never duplicates. To deliver, the service looks up the recipient in the registry and sends to their gateway. If there is no entry, it queues a push notification. Each device remembers the highest sequence number per conversation, so on reconnect it asks for everything newer, and a gap in the numbers means a message was missed. Groups store one copy and push to each online member, while very large groups send a signal and let members pull. For presence I use change events with a grace period, not 30-second heartbeats, which would cost ten times the message rate.

The question

Clarifying questions

• One-to-one chat only, or groups too? How large can a group be? Group size decides how a message is delivered: a loop over a few dozen members, or a signal that members pull from.
• Do we keep history on our servers, and for how long? Stored history drives the storage design. End-to-end encryption would mean the servers hold only ciphertext.
• Do people use several devices at once? Each device needs its own connection and its own record of what it has already received.
• Do we show delivery ticks, read receipts and online status? Each of these is an extra stream of small events, and presence can easily cost more than the messages themselves.
• Is media included, such as photos and voice notes? Media should go to object storage and travel as a link, so the chat path stays small and fast.

Requirements

Functional

• Send and receive one-to-one and group text messages in real time.
• Deliver messages that were sent while a user was offline, as soon as they reconnect.
• Keep messages in order within each conversation.
• Show sent, delivered and read states, and online status.

Non-functional

• Delivery to an online recipient: p99 under 500 ms.
• No accepted message is ever lost.
• No duplicates and no reordering within a conversation, even when the client retries.
• Handles 70,000 messages a second at peak and 20 million open connections.

Back-of-the-envelope estimates

Assumptions for this exercise:

• These numbers are assumptions for the exercise, not facts about any real product.
• 50 million people use the app daily and send 40 messages each. Peak traffic is 3 times the average.
• 20 million connections are open at the busiest time, and one gateway machine holds 100,000 of them.
• A stored message takes about 200 bytes with its metadata. A naive design sends a presence heartbeat every 30 seconds.

The arithmetic:

• Messages per second: 50M × 40 = 2B a day ÷ 86,400 s ≈ 23,150 → ≈ 23,000/s on average, about 70,000/s at a 3x peak
• Gateway machines: 20M connections ÷ 100,000 per machine → 200, plus spares
• Message storage: 2B × 200 bytes = 400 GB a day × 365 days → ≈ 146 TB a year before replication
• Presence updates with 30-second heartbeats: 20M connections ÷ 30 s → ≈ 667,000/s, ten times the message rate
• Presence updates with change events and a 5-minute refresh: 20M ÷ 300 s → ≈ 67,000/s

High-level design

Each Mobile app keeps one long-lived WebSocket to a WebSocket gateway. A gateway only holds connections and forwards frames. It writes the user's location to the Session registry, which maps a user to the gateway that holds their connection, and it updates the Presence service.

A message goes from the gateway to the Chat service. The service gives it the next sequence number in its conversation, writes it to the Message store, and only then treats it as accepted. The store is the source of truth.

To deliver, the Chat service looks up the recipient in the Session registry. If they are online, it hands the message to their gateway. If not, it queues a job for the Push provider, which sends a notification to the phone. When the recipient reconnects, the app asks for everything after the last sequence number it saw.

How a request flows

Send to an online friend

1. The sender's app sends the message over its WebSocket, with an ID the app made up so a retry can be recognised.
2. The gateway forwards the frame to the Chat service.
3. The service gives the message the next sequence number in the conversation and writes it. Only then is it accepted.
4. It asks the session registry where the recipient is connected.
5. The registry answers that the recipient's connection is on gateway 7.
6. The service sends the message to that gateway, which can be a different machine from the sender's.
7. The gateway writes the message to the recipient's open socket, and the recipient's app acknowledges it.

Send to an offline friend

1. The sender's app sends the message over its WebSocket.
2. The gateway forwards it to the Chat service.
3. The service sequences the message and stores it, so it is safe whatever happens next.
4. It looks up the recipient in the session registry.
5. There is no entry, so the recipient is offline.
6. The service queues a push job holding the recipient, the conversation and a short preview.
7. A worker passes the job to the push provider, which wakes the phone with a notification.

Reconnect and catch up

1. After a tunnel or a flight, the app reconnects and sends the last sequence number it has for each conversation.
2. The gateway records that this user is now reachable on this gateway.
3. It marks the user online. The mark expires on its own if the gateway dies.
4. It forwards the app's catch-up request to the Chat service.
5. The service reads every message after those sequence numbers, conversation by conversation.
6. The store returns the missed messages in order.
7. The service sends them to the gateway in batches.
8. The app shows them and acknowledges the highest sequence number it now holds.

Deep dive: gateways, the session registry and presence

Polling a server for new messages wastes effort and adds delay. A WebSocket is a connection that stays open and lets either side send at any time, so a message can arrive within milliseconds. The cost is that the server must hold the connection. A gateway machine can keep about 100,000 sockets in our estimate, so 20 million connections need around 200 of them.

Gateways should keep nothing but their sockets. All real state lives elsewhere, so a gateway can fail and be replaced. Put them behind a load balancer that works at the connection level, and drain a gateway before a deploy by asking its clients to reconnect elsewhere.

The session registry is a fast in-memory map from user to the gateway (and the device) that holds their socket. A gateway adds an entry when a socket opens and removes it on close. Entries also carry a short TTL as a safety net. When a gateway crashes, its entries expire and its 100,000 clients reconnect. To keep that from becoming a stampede, clients wait a random time that grows with each retry.

Presence is where designs get expensive. Heartbeats every 30 seconds from 20 million users make 667,000 writes a second, ten times the message rate. Do this instead:

• Mark a user online when the socket opens, and offline when it closes plus a grace period of about 30 seconds, so a flaky network does not make people flicker.
• Refresh a TTL only every few minutes as a safety net, which gives about 67,000 writes a second.
• Do not broadcast every change to every contact. Send presence only to people who currently have that contact's chat or list on screen, and fetch it when a screen opens.

Deep dive: ordering, acknowledgements and offline sync

Client clocks cannot order messages, because they are wrong and they disagree. Order comes from the server. Each conversation has a sequence number: 1, 2, 3 and so on. Route all writes for one conversation to one owner, for instance by hashing the conversation ID to a Chat service partition. That owner increments a counter, so no two messages share a number and there are no gaps.

Duplicates. The app tags each message with an ID it creates. Networks drop acknowledgements, so the app may resend. The store has a unique key on the conversation and that ID. A resend finds the existing record and returns its original sequence number.

Acknowledgements come in layers:

1. The server confirms it has stored the message and returns the sequence number. The sender sees one tick.
2. The recipient's device confirms receipt. The sender sees two ticks.
3. The recipient reads the conversation. Do not store a receipt for every message. Store one number per user per conversation, the highest sequence number read.

Offline sync. Each device remembers the highest sequence number it has received for every conversation. On reconnect, it sends that map. The server returns everything newer. If a device receives sequence 12 when it holds 10, it knows 11 is missing and asks for it. The same rule repairs most lost frames, so delivery to a live socket can be best effort. Because each device keeps its own cursor, several devices per user work without extra machinery.

A message counts as accepted only after the store write succeeds. Delivering before the write would risk showing a message that later disappears.

Deep dive: groups and message storage

For a small group, such as 50 members, the Chat service stores the message once, under the group's conversation ID and a sequence number. It then loops over the members, looks up each in the session registry and pushes to their gateways. Offline members get a push job. Storing one copy per conversation, not one per member, keeps writes and storage low. Each member's progress is just a cursor: the highest sequence number they have seen.

For very large groups or channels, with tens of thousands of members, a loop is too slow and the registry lookups too heavy. Switch to a pull signal. The service stores the message, then sends a light 'conversation has news' signal. Members' apps fetch what is new. Only members who have the chat open need a real-time push.

Storage layout. Reads are nearly always 'the latest messages in this conversation' or 'messages after sequence N'. So key the data by conversation ID, and sort within it by sequence number, newest first. A wide-column or key-value store fits well, and it spreads conversations across nodes by hashing the conversation ID.

One busy group would grow a single partition without limit. Add a time bucket to the key, such as the conversation ID plus the month, and read buckets in order. It also helps with retention, since an old bucket can be dropped or moved to cheaper storage as a whole.

Keep media out of this path. Upload a photo to object storage, and send a message that carries only its link and a thumbnail. If you add end-to-end encryption, the server stores ciphertext and each device holds its own keys. The design stays the same, but server-side search and moderation become hard.

Trade-offs

• Transport: WebSocket connections over Polling or long polling over HTTP. Messages arrive in milliseconds with little overhead per message. The cost is stateful gateways and a reconnect strategy for when they fail.
• Ordering: A server-assigned sequence per conversation over Sort by client timestamps. The order is the same for everyone and gaps are detectable. The cost is that each conversation needs a single owner to assign numbers.
• Group storage: One stored copy, with a cursor per member over A copy of each message in every member's inbox. Writes and storage do not grow with group size. The cost is that reading a member's view means reading from the group, not from a private inbox.
• Presence: Change events with a grace period, shown on demand over Constant heartbeats broadcast to all contacts. It cuts presence traffic from about 667,000 to about 67,000 writes a second. The cost is that status can lag by up to the grace period.

Common follow-up questions

What happens if a gateway crashes?
Its clients notice the closed socket and reconnect to other gateways after a random delay. Their registry entries expire or are overwritten, and each app asks for messages after its last sequence number. Nothing is lost, because the store holds every accepted message.

How do you support several devices for one user?
Register each device separately in the session registry, and deliver to all of them. Every device keeps its own cursor, so each catches up on its own. Read state is stored per user, so reading on one device clears the unread count on the others.

How would you add end-to-end encryption?
Devices create key pairs and exchange public keys. The sender encrypts for each recipient device, and the server stores and routes ciphertext. Routing and sequencing work as before, but the server can no longer search or moderate message text.

How do you handle a group with a hundred thousand members?
Do not push to every member. Store the message once, and send a lightweight signal to members who have the chat open. Everyone else sees the news when they next open the app and ask for messages after their cursor.