System design interview
Design a website uptime monitor
Staggered probes, multi-region confirmation, incident alerts and history.
Press Play to follow a request through the design, or step through it.
- Client and edge
- Service
- Data store
- Cache and queue
- The scheduler loads the checks in its shards, each with a fixed start offset inside its minute.
- When a check's second comes up, the scheduler queues a job with the check ID, the scheduled time and a region.
- A probe worker in that region takes the job and holds it under a 30-second lease.
- The worker requests the URL and checks the status code and any required text.
- The site answers 200 in 180 ms, and the worker records the timings.
- The worker sends the result, keyed by check ID and scheduled time, and releases the lease.
- The tracker keeps the state as up and appends the raw result to the store.
- Check config: URL, interval, rules
- Scheduler: start offset per check
- Check queue: leased jobs
- Probe workers: one pool per region
- State tracker: up, down, quorum
- Target site: the customer's server
- Notify: email, SMS, webhook
- Result store: raw, then daily
- Status page: public, cached
On this page
In 2 minutes
I would build the monitor around a queue of probe jobs. A scheduler, sharded by a hash of the check ID, gives every check a fixed start offset inside its minute. That turns 300,000 checks into a steady 5,000 probes a second instead of a spike at the top of the minute. Jobs go on a queue with a lease, so a crashed worker's job is retried, and results are keyed by check ID and scheduled time so a repeat never double counts.
Probe workers run in 5 regions and request the customer's site. A failure alerts nobody. The state tracker asks 2 other regions to re-check, and calls the site down only when a majority agree in 2 rounds in a row. That cuts 432,000 chance failures a day to almost no false incidents, at the cost of about 70 seconds.
Alerts fire on state changes, tied to an incident ID, so a long outage sends 2 messages, with flapping suppression on top. Raw results are kept 30 days, about 1.3 TB, and daily summaries for a year, about 5.5 GB. The status page is a static file on a CDN, so it survives our own outage.
Step 1 of 6: Clarify the scope
Interviewer asks
Design a service that checks customers' web addresses every minute from several places in the world, and tells them when a site is down. Customers want one clear alert per outage, a history of their uptime, and a public status page.
Questions to ask first
Ask these before you draw anything. Under each one is why its answer changes the design.
- How many checks are there, and how often does each one run?
The count and interval set the probe rate. How evenly runs are spread matters more than the average, because a burst at the top of each minute breaks schedulers.
- What counts as down: a timeout, any bad status code, a missing word in the page, an expired certificate?
The kind of check decides what a probe records and what the state machine compares. A status check and a content check need different results.
- How fast must an outage be reported, and how many false alarms can customers put up with?
A tight detection time leaves no room to double-check. Fewer false alarms need more confirmation, which costs delay.
- From where do we probe, and may a probe fetch any address a customer types?
Several regions are what let us tell a dead site from one bad network path. Probes that fetch any address also need guards against internal targets.
- How long is history kept, and must the status page survive an outage of our own?
Retention drives the roll-up design. A page people open during incidents must not fall over with our system.
Requirements
Functional
- Run each check, an HTTP request with an expected status and an optional text match, at its interval and from several regions.
- Decide that a site is down only after other regions confirm the failure.
- Send one message when an incident starts and one when it ends, by email, SMS or webhook.
- Keep results, and show uptime percentages and response times for the last 90 days.
- Publish a public status page for each customer.
Non-functional
- Run 300,000 checks a minute, each one starting within 5 seconds of its scheduled time.
- Confirm a real outage and alert within 2 minutes.
- No more than about 1 false incident a week across all checks.
- No check is lost when a scheduler or a probe worker crashes, and a check that runs twice is recorded once.
- The status page loads in under 1 second and stays up when the monitoring back end is down.
Step 2 of 6: Size it
The average load of 5,000 probes a second is modest, but a naive schedule would fire all 300,000 at once. Spread the start times, confirm before alerting, and keep raw results for weeks and daily summaries for a year.
Assumed for the exercise, not facts about a real system:
- 300,000 checks, each run every 60 seconds. A check runs from one region per round, and the region rotates between 5 regions.
- A healthy site fails 1 probe in 1,000 from a single region by chance, from lost packets or a congested path. Regions fail independently.
- A result row is about 100 bytes, and a daily summary row about 50 bytes.
- A probe takes 1 second on average, with a 10-second timeout, and one worker runs about 500 probes at once.
- Probes per second
- 300,000 checks ÷ 60 s
- ≈ 5,000/s. Fired all at second 0, that would be 300,000 in one second, 60 times as many
- Probe workers
- 5,000/s × 1 s = 5,000 in flight ÷ 500 per worker = 10
- 10 workers, about 2 per region. Run 4 per region for headroom
- Raw results per day
- 5,000 × 86,400 s = 432M rows × 100 bytes
- ≈ 43 GB a day
- Raw results, 30 days
- 43.2 GB × 30
- ≈ 1.3 TB. Keeping a year would be ≈ 15.8 TB
- Daily summaries, 1 year
- 300,000 rows × 50 bytes = 15 MB a day × 365
- ≈ 5.5 GB
- False first failures
- 432M probes × 0.001 = 432,000 a day. A 2-of-3 quorum keeps 0.001 × 0.002
- 432,000 a day, about 864 after the quorum
Step 3 of 6: Sketch the path of one request
Step through the diagram at the top of the page to watch a request move through these parts.
The Check config store holds every check: its URL, expected status and interval. The Scheduler finds the checks due this second and puts a job for each on the Check queue. Start times are spread across the minute, so the queue sees about 5,000 jobs a second.
Probe workers run in several regions and take jobs from the queue. A worker requests the URL on the Target site, the customer's own server, and measures what comes back. It sends the result to the State tracker, which writes it to the Result store.
The tracker keeps each check's state, up or down. A failed result alerts nobody yet. The tracker queues re-checks from two other regions, and when a majority agree it opens an incident and hands one message to Notify. The Status page reads state and daily uptime from the store.
Other paths through the system
Confirm an outage
- A probe in one region reports a timeout for a check that was up a minute ago.
- The tracker marks the check suspect and queues re-checks for two other regions.
- Probe workers in those two regions take the jobs within seconds.
- Both request the same URL from their own networks.
- Both time out as well, so a majority of the 3 regions now see a failure.
- Both report their failures to the tracker.
- After a second failed round a minute later, the tracker opens an incident and Notify sends one message.
Recover and show status
- A later round requests the site again.
- The site answers normally, 200 in 210 ms.
- The workers report success to the tracker.
- After 2 good rounds in a row the tracker closes the incident, and Notify sends one recovery message.
- The tracker writes the incident's end time and length to the store.
- When rebuilt, the status page reads the store and shows the site as up.
Step 4 of 6: Go deep on the hard part
Spreading and sharding the schedule
Interviewer asks
How do you run 300,000 checks a minute on time, once each, without everything firing at the top of the minute?
A simple scheduler wakes once a minute and queues every check.
Read moreShow less: Spreading and sharding the schedule
That puts 300,000 jobs on the queue in one second, 60 times the average of 5,000 a second. Spread the work instead.
- Offset by hash. Each check gets a start offset inside its interval: the hash of its ID, modulo 60 for a 60-second check. A check with offset 17 runs at 12:00:17, 12:01:17 and so on. Hashing spreads checks evenly with no coordination, and the timing stays steady for the customer.
- Shard the schedule. Split the checks into shards by a second hash of the ID, say 64 shards. Each scheduler instance owns some shards and loads only their checks.
- Tick every second. On each tick an instance queues the checks in its shards whose offset matches this second.
- Rotate regions. Each job names a region, picked from the scheduled minute, so a check is probed from all 5 regions over 5 minutes. Workers only read their own region's jobs.
Running once, even through crashes. A scheduler or a worker can die mid-task. Leases and keys handle both.
- Shard lease. An instance owns a shard only while it holds a lease, a row with an owner and an expiry that it renews every few seconds. If the instance dies, the lease runs out and another instance takes the shard. Due checks are claimed with one atomic update that moves the next run time forward by one interval and returns the rows it changed. A second instance running the same update gets nothing.
- Job key. Every job carries the check ID and the scheduled time. Results are stored under that pair, so a second run overwrites the first rather than adding to it.
- Worker lease. A worker that takes a job holds it for a 30-second visibility timeout, three times the 10-second probe timeout. If the worker dies, the job reappears and another worker runs it.
Falling behind. Watch the age of the oldest job in the queue, and add workers when it passes 5 seconds. If a job is already a full interval old when a worker reaches it, drop it, because a newer job is behind it.
Confirming an outage
Interviewer asks
A probe failed. How do you decide the site is really down, without paging people over one bad network path?
A single failed probe proves little.
Read moreShow less: Confirming an outage
The fault may be in the probe's own region: a congested link, a bad resolver, a worker short of sockets. We assumed 1 probe in 1,000 fails this way, which over 432 million probes a day is 432,000 failures.
The tracker confirms in three steps.
- Classify the failure. A timeout, a refused connection, a DNS error, a bad certificate, a wrong status and a missing text match are different kinds, so record which. A fault of our own, such as running out of sockets, must not count at all: mark it a probe error and queue the job again.
- Ask other regions. On the first failure the tracker queues re-checks from 2 other regions at high priority, so the answer comes in seconds, not a minute later. It also repeats the check once from the original region after a few seconds.
- Take a quorum. The round fails if a majority of the 3 regions report a failure. If only the first region failed, the round was a blip: log it and do not alert. If one region fails every round, the site may block that region, so show a regional problem, not an outage.
The arithmetic. A blip in the first region has probability 0.001. For a false round, at least one of the other two must also fail by chance: 1 − 0.999² ≈ 0.002. Together that is 0.001 × 0.002 = 2 in a million per probe. Over 432 million probes a day that is about 864 false rounds, down from 432,000, which is 500 times fewer.
864 a day is still too many, so add a second guard: two rounds in a row. A false alarm now needs two false rounds, 2 in a million squared, or 4 in a trillion per check per minute. Across 432 million check-minutes a day that is about 0.002 false incidents a day. Real failures are correlated, so treat this as a floor, not a promise. The price is delay: re-checks finish by about 10 seconds, the second round runs at 60 seconds, and confirmation lands by about 70 seconds, inside the 2-minute goal.
Recovery follows the same rule in reverse: 2 good rounds in a row close the incident. It stops a flaky site from toggling every minute. Let customers pick the number of rounds, with 2 as the default.
One alert per incident, history and the status page
Interviewer asks
How do you send one alert per outage, calm a flapping site, and keep a year of history cheaply?
Alert on state changes, not on failed checks.
Read moreShow less: One alert per incident, history and the status page
Each check has a state, up or down, kept in the tracker with the time of the last change. A message goes out only on up to down, which opens an incident, and on down to up, which closes it. A site that is down for an hour at one check a minute causes 2 messages, not 60.
- Incident row. Opening an incident writes a row with an ID, the check, the start time and the failure kind. The start time is that of the first failed round, not of the confirmation. Closing sets the end time.
- Idempotent sends. The notification key is the incident ID plus the event, opened or closed. Notify remembers the keys it has sent, so a retry after a crash never sends the same message twice.
- Channels. Email, SMS and webhooks fail in different ways. Retry each with backoff, and fall back to email if a webhook keeps failing.
Flapping. A site that fails every third minute would open and close incidents all day. Count state changes in a sliding 30 minutes. Past 4 changes, mark the check as flapping, send one message saying so, and hold further alerts until it has stayed in one state for 15 minutes.
History. At 432 million results a day, raw rows are 43 GB a day. Keep them 30 days, about 1.3 TB, in a store partitioned by day, so removing a day means dropping a partition. A nightly job rolls each finished day into one row per check: probes, failed probes, p50 and p95 response times, and minutes of confirmed downtime. A year of those rows is about 5.5 GB.
Uptime percentage. Compute it from confirmed downtime, not from raw failed probes, which would count blips the quorum threw out. A day with 3 confirmed minutes of downtime out of 1,440 gives 1 − 3 ÷ 1,440 ≈ 99.79%.
The status page. People open it when something is wrong, so it must not share fate with the checker. Render each customer's page to a static file, keep it in object storage, and serve it through a CDN with a 30-second cache. If the tracker is down, the last file still serves, with its time shown.
Step 5 of 6: Name the trade-offs
Start times
A fixed offset from the hash of the check ID over Every check at the top of the minute
Load stays near 5,000 probes a second. The cost is that a new check's first run can wait up to a minute.
Declaring a site down
A majority of 3 regions, in 2 rounds in a row over Alert on the first failed probe
False alarms drop from hundreds of thousands a day to almost none. The cost is about 70 seconds before a real outage is confirmed.
What triggers a message
State changes, tied to an incident ID over A message for every failed check
One message opens and one closes, and the ID makes retries safe. The cost is a state machine to maintain.
History
30 days of raw rows plus daily summaries for a year over Raw rows for a year
About 1.3 TB plus 5.5 GB, instead of about 15.8 TB. The cost is no per-probe detail for old incidents.
Status page
Static files behind a CDN over Rendered from the database on each request
It survives our own outage and a flood of visitors during an incident. The cost is up to 30 seconds of staleness.
Step 6 of 6: Say what breaks next
Questions an interviewer asks once the design is on the board. Answer each one out loud first.
- How do you stop customers from pointing probes at internal addresses?
Resolve the hostname inside the probe, check that the address is public, and connect to that exact address, so a second lookup cannot swap in a private one. Repeat the check after every redirect. Run workers in a network with no route to our own services.
- How would you support checks every 10 seconds?
The offset becomes the hash modulo 10, and those checks cost six times as much. If all checks ran that way, 5,000 probes a second would become 30,000. Charge more for short intervals and cap the share per customer. Workers and queue partitions scale out, because each job is independent.
- What if a customer's site is only slow, not down?
Track response time as a second signal. Alert when it stays above a threshold for several rounds, with the same quorum and incident logic. Keep it apart from downtime, so a slow day does not count against uptime.
- How do you watch certificate expiry?
Record the certificate's expiry date on every successful HTTPS probe. A daily job finds certificates that expire within 14 days and sends one reminder for each certificate and threshold. This is not an incident, since the site still works.