Design live comments for a video stream

I would hold viewers on WebSocket gateways, about 50,000 connections each, so a stream of a million viewers uses around 20 gateways. A comment goes up to a comment service, which limits each user to one comment every two seconds and sends it through a fast moderation filter. Clean comments are published on a topic for the stream in a light pub/sub broker. Each gateway with a viewer of that stream subscribes to the topic, so a comment crosses the backend once per gateway: 20 MB/s rather than a million sends.

The broker also copies each comment to a durable store and to a recent buffer holding the last 10 seconds. Nobody can read 5,000 comments a second, so each gateway batches for 500 ms and picks at most 5 comments per window, which is 10 a second, with the creator, moderators and followed accounts always favoured. A late joiner subscribes to the topic, reads the recent buffer, and drops duplicates by comment id. Ordering is loose, with time-ordered ids and no coordination between gateways.

Slower toxicity checks run after publishing and send remove events on the same topic. The main trade-off is that most viewers see only a sample, in exchange for a system that stays cheap and fast at any size.

These numbers are assumptions for the exercise, not facts about a real system:

• One top stream has 1 million viewers connected, and viewers post 5,000 comments a second at the peak.
• A comment takes about 200 bytes on the wire, including the id, the author and the text.
• One gateway server holds 50,000 open connections.
• A viewer can read about 10 comments a second. A gateway batches for 500 ms before each send, so each batch carries at most 5.

The arithmetic:

Gateway servers for one stream

1,000,000 viewers ÷ 50,000 connections each

20 gateway servers

Deliveries if everyone got everything

5,000 comments/s × 1,000,000 viewers

5 billion deliveries a second, not possible

Bandwidth if everyone got everything

5,000 × 200 B = 1 MB/s per viewer; × 1M viewers

1 TB/s of outbound traffic

Backend to gateways

5,000/s × 200 B = 1 MB/s per gateway; × 20 gateways

20 MB/s in total, once per gateway

Outbound after sampling to 10 a second

10 × 200 B = 2 KB/s per viewer; × 1M viewers = 2 GB/s; ÷ 20 gateways

≈ 100 MB/s per gateway, 0.8 Gbit/s on a 10 Gbit card

Recent buffer for one hot stream

5,000 comments/s × 10 s × 200 B

≈ 10 MB, 50,000 comments, easy to keep in memory

The key is to put a middle layer between the backend and the viewers. The comment service never knows who is watching. It publishes once to a topic, a named channel for the stream, and the broker hands the message to every subscriber of that topic.

The subscribers are gateways, not viewers. A gateway is a server that holds tens of thousands of open WebSocket connections. When the first viewer of a stream connects to a gateway, the gateway subscribes to that stream's topic. When its last viewer leaves, it unsubscribes. A comment is then sent from the broker to each gateway that has a viewer, once, and the gateway writes it to its local sockets. For our stream, that is 20 messages from the broker instead of a million, and 20 MB/s on the backend network.

For a very large stream, a flat set of thousands of gateways subscribed to one topic is itself heavy on the broker. Add a relay tier: a few relays subscribe to the topic, and each gateway subscribes to a relay. The tree keeps every broker's fan-out in the dozens.

Which broker. Delivery here is best effort. A comment that one gateway misses for a second is not a disaster, and the durable copy is saved separately. So use a light in-memory publish and subscribe system, not a heavy durable log. Spread topics over broker nodes by hashing the stream id. Give a huge stream its own node, or split it into sub-topics.

Batching. A gateway should not write to a socket for every comment. Collect comments for 500 milliseconds, then send one frame with the whole batch. That is fewer system calls and packets, and it adds at most half a second, which fits inside the 2-second target. At the cap of 10 a second, each frame carries at most 5 comments.

Reconnects. Connections drop. The client sends the id of the last comment it saw when it reconnects, and the gateway fills any gap from the recent buffer. Add random delay to client retries, so a gateway that fails does not trigger 50,000 simultaneous reconnects.

Delivery is limited by what people can read, not by what the network can carry. A reader can follow about 10 comments a second, so the target is at most 10 per viewer, or 5 per 500 ms window. At 5,000 arriving, each window holds about 2,500 comments, and 5 of them is a sample of 0.2%. The rule scales with traffic: a quiet stream with 8 comments a second has 4 in a window and shows all of them, and the sampling rate only drops as the arrival rate climbs above the limit.

Where to sample. Do it at each gateway, per batch window. The gateway has the whole stream's comments in memory for that window, picks at most 5 of them, and sends the same pick to all of its viewers. That is cheap, since there is one decision per window rather than one per viewer. Viewers on different gateways see different samples, which nobody notices.

What to prefer. Plain random sampling is fair. A reservoir sample lets you pick k from a stream of unknown length in one pass. Improve on it with priority.

• Comments from the creator and moderators always go through.
• Comments from accounts the viewer follows get one or two reserved slots. The gateway loads the viewer's short follow list when the viewer connects.
• Identical text collapses into one comment with a count, so three hundred copies of the same cheer become a single line with a number.
• Quality signals, such as account age and a history of reports, can raise or lower a comment's chance.

Throttling at the source. Sampling hides the volume from viewers, and posting limits reduce it at the source. Allow one comment per user every two seconds. Offer slow mode, where the creator raises that gap, and followers-only mode. Those protect the moderation filter and the broker as well.

Local echo. A person's own comment should always appear for them straight away, even if the sample would drop it for others. Without that, posting feels broken.

Catching up. Keep a recent buffer for each stream: the last 10 seconds of comments, which is about 50,000 at the peak. A joining viewer's gateway reads the buffer, samples 50 of them with the same priority rules, which is five seconds of chat at the shown rate, and sends them. It then continues with the live topic. To avoid a gap or a repeat between the two, subscribe to the live topic first, then read the buffer, and let the client drop any comment id it has already shown.

Ordering. Chat is not a ledger, so loose ordering is fine. The comment service gives each comment a time-ordered id, built from a timestamp and a counter. Clients sort by id inside the small window they hold. Two gateways may deliver two near-simultaneous comments in a different order, and that is acceptable. There is no global sequence and no coordination across gateways, which is what keeps the path fast.

Moderation before delivery. A comment that is shown to a million people cannot be taken back from them, so check it before it enters the topic. The filter runs quick checks in a few milliseconds: banned words and patterns, links from new accounts, floods of repeated text and users muted on this stream. A comment that fails is dropped, and its author may still see it, so that spammers get no signal.

Slower checks, such as a machine learning toxicity score, run after publishing. If one flags a comment, publish a remove event on the same topic. Gateways and clients hide that id at once, and the buffer and store mark it removed. Creators and moderators use the same event to remove a comment or mute a user. The mute list is checked by the filter and by gateways.

The price of pre-delivery filtering is some latency on each comment, about 10 ms for the fast checks. That fits well inside the 2-second target, and avoiding even one abusive comment on a huge stream is worth it.