System design interview
Design a metrics and monitoring system
Pull or push, compression, downsampling, alerts and label cardinality.
Press Play to follow a request through the design, or step through it.
- Client and edge
- Service
- Data store
- Cache and queue
- Every 15 seconds the collector asks each target on its list for its current values. This is a pull.
- Each target replies with metric names, labels and numbers. A target that does not reply is itself a signal.
- The collector writes the batch to the queue, partitioned by a hash of the series, and moves on.
- An ingest worker reads the batches for its share of the series.
- It groups samples by series, compresses them and appends them to the database.
- Targets: servers and apps
- Collector: scrape or receive
- Dashboards
- Metrics queue: buffers bursts
- Alert rules: query, threshold, for
- Query service: picks resolution
- Ingest workers: batch and compress
- Alert evaluator: runs every minute
- Time-series DB: raw, then rollups
- Pager: on-call alerts
On this page
In 2 minutes
I would build the system as a pipeline. A collector pulls samples from every target every 15 seconds from a discovery list, so a missing target is itself a signal, and a small push gateway covers short jobs. The collector writes batches to a queue partitioned by a hash of the series, so a database outage costs no data.
Ingest workers group samples by series, compress them with delta-of-delta timestamps and XOR-encoded values, and append them to a time-series store. At 5 million series, that is about 333,000 samples a second, and compression takes 461 GB a day down to roughly 58 GB. I would keep raw data for 15 days, then roll it up into 5-minute points with min, max, sum and count for 13 months.
A query service picks the resolution from the time range, so a dashboard never reads more points than it can draw. An evaluator runs alert rules every minute with a for-duration, then groups, dedupes and routes notifications to the pager. I would cap label cardinality, because a single unbounded label like a user ID can create millions of series.
Step 1 of 6: Clarify the scope
Interviewer asks
Design a system that collects numbers from thousands of servers, such as CPU use and request counts. Engineers should be able to draw dashboards of the history, and be paged when something goes wrong.
Questions to ask first
Ask these before you draw anything. Under each one is why its answer changes the design.
- How many machines and services, and how many different metrics does each expose?
The number of distinct series, not the number of machines, is what sets memory and storage needs.
- How often do we sample, and how fresh must a dashboard or an alert be?
The interval sets the write rate. A freshness target of a minute allows batching that a one-second target would not.
- How long do we keep data, and at what resolution?
Keeping raw samples forever is costly. Older data is usually fine at a coarser step, which is what downsampling is for.
- Do systems we monitor sit behind firewalls, or are some short-lived batch jobs?
Short-lived jobs and locked-down networks are the cases where pulling metrics does not work and pushing is needed.
- Who gets paged, and how do we avoid a storm of duplicate alerts?
Alert routing, grouping and a duration condition are part of the product, not an afterthought.
Requirements
Functional
- Collect numeric samples with names and labels from servers and applications.
- Store them and answer range queries for dashboards, such as the last 6 hours or 30 days.
- Evaluate alert rules continuously and notify the right on-call person.
- Keep older data at a lower resolution to save space.
Non-functional
- Ingest 333,000 samples a second, with no data loss if storage is down for an hour.
- A new sample is visible to queries within 30 seconds.
- Dashboard queries for the last 6 hours: p99 under 1 second.
- Alerts fire within 2 minutes of a fault, and the monitoring system itself is more reliable than what it watches.
Step 2 of 6: Size it
This is a write-heavy, append-only workload on data that is ordered by time. Compression is what makes it affordable, and a queue and tiered retention keep a storage outage or a long dashboard range from becoming a crisis.
Assumed for the exercise, not facts about a real system:
- 10,000 servers each expose 500 series, so 5 million active series, sampled every 15 seconds.
- A raw sample is a timestamp and a value, 16 bytes in all. With compression we assume about 2 bytes a sample.
- Raw data is kept 15 days, then rolled up to 5-minute points for 13 months.
- Samples per second
- 5M series ÷ 15 s
- ≈ 333,000/s
- Raw data per day
- 333,333 × 86,400 = 28.8B samples × 16 bytes
- ≈ 461 GB a day, uncompressed
- Compressed data per day
- 28.8B samples × 2 bytes
- ≈ 58 GB a day
- Hot tier, 15 days
- 58 GB × 15
- ≈ 0.86 TB
- Rollups, 13 months
- 5-minute points are 20x fewer than 15-second ones: 58 GB ÷ 20 = 2.9 GB a day × 395 days
- ≈ 1.1 TB
- One hour of queue buffer
- 333,333 × 16 bytes = 5.3 MB/s × 3,600 s
- ≈ 19 GB
Step 3 of 6: Sketch the path of one request
Step through the diagram at the top of the page to watch a request move through these parts.
A Collector gathers samples from the Targets, the servers and applications. It scrapes each target on a schedule, so it knows what should exist, and it can also receive pushes from jobs that cannot be scraped. The collector puts batches on a Metrics queue, partitioned by series.
Ingest workers drain the queue. They group samples by series, compress them, and write them to the Time-series DB. The queue lets the database be slow or down for a while without losing data.
Reading has two clients. Dashboards call a Query service, which reads the database and picks a resolution that fits the time range. An Alert evaluator loads rules from the Alert rules store, runs their queries on a timer, and sends grouped notifications to the Pager.
Other paths through the system
Draw a dashboard
- A dashboard asks for one service's CPU over the last 6 hours, with a point every minute.
- The query service reads the matching series from the database at the best resolution for that range.
- The database returns compressed blocks, and the service decodes only the range it needs.
- The service returns about 360 points per line, and the chart draws them.
Fire an alert
- Every minute the evaluator loads the alert rules, each one a query, a threshold and a duration.
- It runs each rule's query against the database, for example the error rate over the last 5 minutes.
- The error rate is 7%, above the 5% threshold. The alert becomes pending.
- After the condition has held for the full duration it fires, and one grouped notification goes to the pager.
Step 4 of 6: Go deep on the hard part
Pull or push, and why a queue
Interviewer asks
How should metrics get from servers to storage, and what sits in between?
There are two ways to collect.
Read moreShow less: Pull or push, and why a queue
- Pull. The collector keeps a list of targets, from service discovery, and scrapes each one on a timer. It always knows what should be there, so a failed scrape is a clear 'this target is down' signal for free. It is easy to debug, since you can open a target's metrics page by hand. It needs network access to every target, and it struggles with short jobs that finish before the next scrape.
- Push. Each agent sends its metrics to the collector. This works through firewalls and for batch jobs that run for seconds. But silence is ambiguous: a quiet server might be down or just idle. Clients also decide the volume, so the collector must protect itself with limits and authentication.
Many systems use pull as the default and a small push gateway for short-lived jobs.
Between the collector and the database, put a queue. It gives you three things. It absorbs bursts, for example when a thousand servers restart together. It lets storage fail or be upgraded without losing samples, since the queue keeps them. And it lets other consumers, such as a stream processor, read the same data. In our estimate, an hour of buffer is about 19 GB.
Partition the queue by a hash of the series. All samples for one series then reach the same ingest worker in order, which makes batching and appending simple. When a worker falls behind, the queue length shows it, and you add workers.
Compression, downsampling and retention
Interviewer asks
How do you store billions of samples a day affordably, and still answer long-range queries fast?
Time-series data is special: samples arrive in time order, at regular steps, with values that change slowly.
Read moreShow less: Compression, downsampling and retention
Two tricks exploit that.
- Timestamps: delta-of-delta. Samples at 15-second steps have times t, t+15, t+30. The gaps are 15, 15, 15, and the change in the gap is 0. Store that zero in one bit.
- Values: XOR with the previous value. Two nearby floating-point numbers share most of their bits. XOR them, and an unchanged value gives all zeros, which costs one bit. A slightly changed value leaves a short run of different bits in the middle, and only that is stored.
Together these bring a 16-byte sample to a couple of bytes. We assumed 2, which turns 461 GB a day into 58 GB.
New samples go to an in-memory head block, backed by a write-ahead log in case of a crash. Every couple of hours the head is written out as an immutable compressed block. An inverted index maps each label pair, such as service=checkout, to the series that carry it, so a query finds its series without scanning.
Downsampling. Nobody needs 15-second points from last year. A background job rolls old data into 5-minute points. Keep the minimum, maximum, sum and count for each step, not just the average. An average hides a one-minute spike, and a maximum shows it. The average is still available as sum divided by count.
The query service picks the resolution from the range: 6 hours at 1-minute steps gives 360 points, and 30 days at 5-minute steps gives 8,640. Retention is then cheap, because whole blocks past their age are deleted, with no row-by-row cleanup.
Alerts, grouping and cardinality
Interviewer asks
How do you alert without flooding people, and what stops the system from running out of memory?
An alert rule is a query, a threshold and a for duration. 'Error rate above 5% for 5 minutes' ignores a single blip.
Read moreShow less: Alerts, grouping and cardinality
A rule moves through states: inactive, pending when the condition first holds, firing after the duration, and resolved when it clears. The evaluator runs on a timer, every minute here, and writes each state change out.
A real outage can trigger hundreds of alerts at once. The notifier tames them:
- Group alerts by shared labels such as cluster and service, so one page covers them all.
- Deduplicate by alert identity. Two evaluators can run for reliability, and the notifier sends one page for the same alert from both.
- Throttle repeats. Remind every few hours, not every minute.
- Inhibit details. If a whole data centre is down, suppress the per-server alerts below it.
- Route by severity: a page for customer impact, chat for the rest. Allow silences for planned work.
Also alert on missing data. A server that stops reporting will never cross a threshold, so add a rule that fires when no samples arrive. And watch the watcher: keep a tiny independent check that fires if the monitoring system stops evaluating at all.
Cardinality is the quiet killer. Every distinct combination of label values is a separate series, and each one costs memory in the head block and the index. 1,000 endpoints, 1,000 hosts and 10 status codes make 1,000 × 1,000 × 10 = 10 million series. Adding a label with unbounded values, such as a user ID or a request ID, multiplies that without limit. Set a cap on series per metric and reject or drop above it, check new labels in review, and keep high-detail data in logs and traces, not in metric labels.
Step 5 of 6: Name the trade-offs
Collection
Pull by default, with a small push path for short jobs over Push from every agent
The collector knows what should exist, so a missing target is an alert. The cost is network reach to every target and a push gateway for batch jobs.
Between collector and storage
A partitioned queue over Write straight to the database
Bursts and storage outages cost no data, and the data can be replayed. The cost is one more system to run and some added delay.
Storage engine
A purpose-built time-series store with compression over A general SQL table with one row per sample
It stores a sample in about 2 bytes and serves ranges quickly. The cost is a narrower query model than SQL.
Rollups
Keep min, max, sum and count per 5 minutes over Keep only the average
Spikes stay visible in old data. The cost is about four times the rollup storage of a single value.
Step 6 of 6: Say what breaks next
Questions an interviewer asks once the design is on the board. Answer each one out loud first.
- How do you monitor the monitoring system?
Run a small, separate watcher that checks whether the evaluator and the ingest path are alive. Add an alert that always fires, called a dead man's switch, and page if it stops arriving. Keep this path independent of the main stack.
- How would you scale to ten times as many series?
Shard the time-series database by a hash of the series, with replicas for each shard. The query service then fans a query out to the shards and merges the results. Ingest scales by adding queue partitions and workers.
- What about late or out-of-order samples?
Accept samples that are slightly late into the head block, within a window such as an hour, and reject older ones. When collectors stamp the time themselves, clock skew on the targets stops mattering. Clearly count and report anything rejected.
- Why not store logs and traces in the same system?
Metrics are small aggregates with a fixed shape, which is why they compress so well. Logs and traces are large, free-form records with their own indexing and cost. Link them by shared labels so an engineer can jump from a spike to the logs.