System design interview
Design a content moderation system
Hash matching, thresholds, reviewer queues, decision logs and appeals.
Press Play to follow a request through the design, or step through it.
- Client and edge
- Service
- Data store
- Cache and queue
- A user posts text and a photo. Before the post goes live, the platform sends the photo to Auto triage for a hash check.
- Triage computes an exact hash and a perceptual hash of the photo, and looks them up in the known-bad lists. There is no match.
- Triage answers 'clear', and the platform shows the post. A match would have blocked it and gone to the decision service.
- The platform queues a classifier job with the post ID and version.
- Auto triage takes the job.
- It sends the text and the photo to the classifier, which returns one score for each policy. The highest is 0.62 for harassment.
- 0.62 is between the thresholds of 0.30 and 0.95, so the item goes to the review queue with its policy, its score and its reach so far.
- For a clear case, such as a hash match or a score of 0.95, triage sends an automatic decision to the decision service, which logs and applies it.
- Content platform: posts, feeds, caches
- Decision log: append only
- Scan queue: classifier jobs
- Auto triage: hash, score, route
- Decision service: one writer of outcomes
- Classifier: scores per policy
- Known-bad hashes: exact and perceptual
- Review service: priority, leases
- Reviewer console: people
On this page
In 2 minutes
I would check each post in two stages. Before it goes live, the platform sends its image to auto triage, and an exact hash and a perceptual hash catch known bad images, with the perceptual hash indexed in 7 chunks so that any hash within 6 bits is found by looking up whole chunks. A match is blocked. Otherwise the platform shows the post and queues a classifier job, with a 5-second target, and the classifier scores each policy.
Two thresholds, set per policy, decide the route: under 0.30 allow, at 0.95 or more act, and in between send to a person. With 20 million posts a day, that means about 80,000 automatic removals, of which about 800 are mistakes, and about 420,000 reviews a day, which needs around 280 reviewers. The review queue is a table with severity lanes, priority by reach with deadlines winning near the end, language groups and 5-minute leases taken with one atomic statement. A sample is double reviewed to measure agreement.
Every outcome goes through one decision service into an append-only log with the actor and version. It then sends an idempotent action to the platform to hide the content everywhere within 60 seconds, and a reconciler repairs any miss. An appeal goes to a different reviewer and is a new entry in the log, so the full history stays.
Step 1 of 6: Clarify the scope
Interviewer asks
Design the system that screens what users post on a social app, both text and images, for policy violations. Machines should handle the clear cases, human reviewers the unclear ones, and a user who thinks a decision was wrong should be able to appeal.
Questions to ask first
Ask these before you draw anything. Under each one is why its answer changes the design.
- Is content checked before it goes live, or just after?
Checking first adds delay to every post. Checking after lets harm be seen for a few seconds, so the choice depends on the type of harm.
- How many posts a day, how many have images, and what share break the rules?
The share of violations sets how much work reaches humans, which is the real cost driver, not the scan volume.
- How costly is a wrong removal compared with a miss?
That ratio sets the thresholds. A mistaken takedown angers a user, while a miss can hurt a person, and the two costs differ by policy.
- What are the response-time promises for reviewers, and which languages must they cover?
Service targets and languages decide how the queue is ordered and split between reviewer groups.
- Do users get to appeal, and who must we be able to show the history to?
Appeals and audits mean every decision is a permanent record, not a flag on the post that gets overwritten.
Requirements
Functional
- Scan every new post and image automatically, and match them against known-bad content.
- Allow clear cases, act on clear violations, and send the rest to human review.
- Give reviewers a queue ordered by severity, reach and age, with no item shown to two reviewers at once.
- Apply each decision everywhere the content appears: feeds, search, caches and links.
- Let users appeal, with a different reviewer, and keep an audit trail of every decision.
Non-functional
- A known-bad image is blocked before its post goes live, and the hash check adds under 100 ms.
- The classifier scan finishes within 5 seconds for 99% of posts.
- Urgent items are reviewed within 15 minutes, and all others within 24 hours.
- A removed post disappears from every surface within 60 seconds of the decision.
- Decisions are never overwritten or lost, and each one records who or what made it.
- Scan capacity of 700 posts a second at the peak.
Step 2 of 6: Size it
Scanning is cheap compared with people. The thresholds decide whether 280 reviewers is enough, so they are a business decision, and the decision log and appeals must be built as a record, not a status field.
Assumed for the exercise, not facts about a real system:
- 20 million posts a day, 6 million of them with an image. The peak is 3 times the average.
- 1% of posts break the rules, so 200,000 a day. 10 million known-bad image hashes of 8 bytes each.
- From a validation set: a classifier score of 0.95 or more catches 40% of violations at 99% precision. The band from 0.30 to 0.95 holds a further 50% of violations, at 25% precision.
- A reviewer handles 250 items an hour and has 6 productive hours a day. 5% extra volume, 20,000 items, is for quality checks.
- Scan jobs per second
- 20M ÷ 86,400 s ≈ 230/s; × 3
- ≈ 230/s average, ≈ 700/s at the peak
- Automatic removals
- 200,000 × 40% = 80,000 correct; 1% of the removals are wrong
- ≈ 80,000 a day, about 800 of them mistakes
- Review queue
- Band holds 200,000 × 50% = 100,000 violations at 25% precision = 400,000 items; + 20,000 for quality checks
- ≈ 420,000 items a day, 2% of all posts
- Reviewers needed
- 420,000 ÷ 250 per hour = 1,680 hours ÷ 6 hours each
- ≈ 280 reviewers
- Missed by machines
- 200,000 × (100% − 40% − 50%)
- ≈ 20,000 a day, found later by user reports
- Hash lookup memory
- 10M × 8 bytes = 80 MB, plus 7 chunk indexes of about 84 bytes a hash in all
- ≈ 0.9 GB, held in memory
Step 3 of 6: Sketch the path of one request
Step through the diagram at the top of the page to watch a request move through these parts.
The Content platform sends each new image to Auto triage before the post goes live. Triage looks the image up in the Known-bad hashes, and a match blocks the post. Otherwise the platform shows the post and puts a scan job on the Scan queue. Triage takes the job and asks the Classifier for a score per policy. Then it applies two thresholds: allow the post, act on it straight away, or send it to a person.
Items for people go to the Review service, which owns a priority queue with leases. A reviewer works in the Reviewer console, which asks for the next item. Their choice goes back through the review service to the Decision service.
The Decision service is the only writer of outcomes. It appends every decision, from a machine or a person, to the Decision log, and then tells the platform to apply it. An appeal goes the same way in reverse, to a different reviewer, and becomes a new entry in the log.
Other paths through the system
Review and decide
- A reviewer who speaks the post's language asks for the next item. The review service picks the top open item for that group.
- It leases the item to this reviewer for 5 minutes and returns the post with its context.
- The reviewer chooses to remove it, citing the harassment policy.
- The review service sends the decision to the decision service, with the item, the reviewer and the policy.
- The decision service appends the decision to the log. It does not change any earlier entry.
- It tells the platform to hide the post. The platform removes it from feeds and search, clears caches and records a strike.
Appeal a removal
- The author appeals the removal from the app. The platform sends the appeal to the decision service.
- The decision service opens an appeal task and hands it to the review service, barring the reviewer who made the first decision.
- The review service offers the task to a different, more senior reviewer.
- That reviewer decides the post did not break the rules and reverses the removal.
- The review service sends the new decision to the decision service.
- The decision service appends it to the log, linked to the first decision, so both stay on record.
- It tells the platform to restore the post and take back the strike.
Step 4 of 6: Go deep on the hard part
Automated triage
Interviewer asks
How do you decide automatically what to allow, remove or send to a person, and how do you pick the thresholds?
Triage has three layers, from cheapest and most certain to costliest and least certain.
Read moreShow less: Automated triage
The two hash layers run before the post goes live: they are in-memory lookups that take milliseconds, and they catch known illegal images, which must never be shown. The classifier runs after.
- Exact hash. Hash the file's bytes. An identical copy of known-bad content matches at once. It misses any copy that has been resized or compressed again.
- Perceptual hash. A 64-bit fingerprint of what the image looks like, so a resized or re-encoded copy gets a nearly identical hash. Two images match if their hashes differ in at most 6 bits. Comparing against 10 million hashes one by one is too slow, so index each hash in 7 chunks of about 9 bits. If two hashes differ in at most 6 bits, at least one of the 7 chunks must be identical. Fetch the candidates that share a chunk with the query, then compare the full hashes. That is about 0.9 GB, in memory.
- Classifier. A model returns a score from 0 to 1 for each policy, such as harassment or violence. This catches content that nobody has seen before.
Content a person confirms as a violation adds its hashes to the known-bad list, so the next copy is caught by the cheap layer.
Two thresholds. Below 0.30, allow. At 0.95 or more, act without a person. In between, send to review. Choose them from a validation set of past items that people have labelled. For each threshold, measure precision (of the items it flags, how many are real violations) and recall (of all real violations, how many it flags). In our assumptions, the auto-act threshold catches 40% of violations at 99% precision, so about 800 of 80,000 daily removals are mistakes. The review band catches 50% more, but only 1 item in 4 is a violation, so 400,000 items a day reach people.
Lowering the review threshold raises recall and the reviewer bill together. Raising the act threshold cuts mistakes and moves more work to people. So set thresholds per policy. A policy with severe harm gets a low review threshold and accepts more reviewer work. A mild one, such as spam, gets a high act threshold and few humans.
Watching for misses. Send a small random sample of allowed items to reviewers, 0.05% of posts, 10,000 a day. That gives a measured miss rate, which an accuracy figure on old data never does. Retune when the model or the content changes.
The review queue
Interviewer asks
How do you give reviewers work so that no item is shown twice, urgent items come first, and quality stays measured?
The review queue is a database table, not a plain message queue, because we need priority, leases and the ability to change an item's rank.
Read moreShow less: The review queue
- Lanes. Urgent policies, such as child safety, have their own lane with a 15-minute target. Everything else has a 24-hour target.
- Priority inside a lane. Reach first: a post that 20,000 people saw in the last hour ranks above one that 50 saw. But deadlines win near the end. Once an item is within 4 hours of its target, order by earliest deadline, so low-reach items never starve.
- Specialisation. Each item has a language and a policy. A reviewer group is defined by languages and policies it has been trained on, and the queue serves each group only its own items. Rotate people off the worst content, for their wellbeing.
Leases. Two reviewers must never take the same item. Take the next item with one atomic statement that picks the top open row, skipping any row another transaction has locked, and marks it leased to this reviewer until 5 minutes from now. If the reviewer closes the tab, the lease runs out and the item returns to the queue. Submitting a decision checks that the lease is still theirs.
Measuring quality. Send 2.5% of the queue, 10,000 items a day, to two reviewers who do not see each other's answer. The share of pairs that agree is the agreement rate. If it is low for a policy, the policy text is unclear, and training will not fix that. Disagreements go to a senior reviewer, and the answer is recorded. Mix in a few items with a known correct answer, so each reviewer's accuracy is tracked.
Capacity. At 420,000 items a day and 250 an hour, that is about 280 reviewers. Alert when the oldest item in a lane is at 80% of its target, and respond by pulling reviewers from a quieter group, or by raising the auto-act threshold for a mild policy for a day.
Decisions, propagation and appeals
Interviewer asks
How do you record decisions, make sure they take effect everywhere, and handle an appeal?
Treat every decision as a fact that happened, not as a field to overwrite.
Read moreShow less: Decisions, propagation and appeals
The decision log is append only. Each entry records the item and its version, the action, the policy, the actor, either a person or a named model version, the reason and the time. The current state of an item is its latest entry. An appeal adds a new entry that points at the old one. Nothing is edited or deleted, which is what an audit needs.
Version matters. A decision applies to the content as it was. If the author edits the post, the new version gets a new scan, and an old removal does not silently cover it.
Making it take effect. A decision must reach every place the content shows up: feeds, search, recommendations, notifications, shared links, and CDN-cached pages and images. The decision service sends one action keyed by the decision ID. The platform applies it, records that it did, and purges the caches by item ID. A repeated action is a no-op, so retries are safe. The target is 60 seconds from decision to gone.
Because a lost action leaves removed content visible, add a reconciler. Every few minutes it compares the latest decision for recently decided items with what the platform shows, and re-sends any that differ. Alert on any mismatch older than 60 seconds.
Appeals.
- The author appeals within a window such as 14 days. The decision service opens a task linked to the first decision.
- The task goes to a different reviewer, one who was not involved, and for a machine decision to any trained person. High-impact cases, such as account bans, go to a senior reviewer.
- If the removal is reversed, the post is restored and the strike taken back. The reversed item becomes a labelled example, so the false positive counts against the model version that made it.
- Every step is an entry in the log. Reading the log can show who saw what, which policy applied, and why.
Limit access to the log, because it holds sensitive content. Keep it for as long as law and policy say, and record who looks at it.
Step 5 of 6: Name the trade-offs
When to check
Hash-match before publishing, classify within seconds after over Hold every post until it is scanned
Known-bad images never go live, posting feels instant, and the 5-second classifier scan limits exposure. The cost is a short window for new harmful content, so urgent policies can use a hold for new accounts.
Image matching
Exact hash plus a perceptual hash over A classifier alone
Known content is caught in microseconds, with almost no mistakes. The cost is an index to maintain, and it never finds content nobody has seen.
Thresholds
Two thresholds, set per policy over One global cut-off
Each policy gets its own balance of reviewer work and mistakes. The cost is more tuning and more numbers to monitor.
Recording outcomes
An append-only decision log over A status field on the post
It gives an audit trail and makes appeals clean. The cost is more storage and a read that finds the latest entry.
Queue storage
A table with leases and priority over A plain message queue
Rank can change as reach grows, and a lease frees an abandoned item. The cost is a hot table that needs good indexes.
Step 6 of 6: Say what breaks next
Questions an interviewer asks once the design is on the board. Answer each one out loud first.
- How do you handle videos and live streams?
Sample frames and audio at intervals, run the same hash and classifier layers on them, and score the clip as a whole. For live streams, scan continuously and let a high score pause the stream and call a reviewer at once. The urgent lane exists for this.
- How do you find violations the classifier missed?
Let users report content, and treat a report as one more signal. Many reports on a post, or reports from trusted users, raise its priority in the queue. Sample the allowed items too, so you measure the miss rate yourself.
- How do you roll out a new classifier safely?
Run it in shadow mode first, scoring live traffic without acting, and compare it with the current model and with human decisions. Then let it act on a small share of traffic and watch precision and the appeal reversal rate. Keep the old model's version on every log entry so you can compare.
- What stops reviewers from being overwhelmed by harmful content?
Blur images by default and let the reviewer reveal them. Limit time on the worst categories and rotate people between queues. Track exposure per reviewer, and give access to support.