Design a metrics and monitoring system

In 2 minutes

I would build the system as a pipeline. A collector pulls samples from every target every 15 seconds from a discovery list, so a missing target is itself a signal, and a small push gateway covers short jobs. The collector writes batches to a queue partitioned by a hash of the series, so a database outage costs no data. Ingest workers group samples by series, compress them with delta-of-delta timestamps and XOR-encoded values, and append them to a time-series store. At 5 million series, that is about 333,000 samples a second, and compression takes 461 GB a day down to roughly 58 GB. I would keep raw data for 15 days, then roll it up into 5-minute points with min, max, sum and count for 13 months. A query service picks the resolution from the time range, so a dashboard never reads more points than it can draw. An evaluator runs alert rules every minute with a for-duration, then groups, dedupes and routes notifications to the pager. I would cap label cardinality, because a single unbounded label like a user ID can create millions of series.

The question

Clarifying questions

• How many machines and services, and how many different metrics does each expose? The number of distinct series, not the number of machines, is what sets memory and storage needs.
• How often do we sample, and how fresh must a dashboard or an alert be? The interval sets the write rate. A freshness target of a minute allows batching that a one-second target would not.
• How long do we keep data, and at what resolution? Keeping raw samples forever is costly. Older data is usually fine at a coarser step, which is what downsampling is for.
• Do systems we monitor sit behind firewalls, or are some short-lived batch jobs? Short-lived jobs and locked-down networks are the cases where pulling metrics does not work and pushing is needed.
• Who gets paged, and how do we avoid a storm of duplicate alerts? Alert routing, grouping and a duration condition are part of the product, not an afterthought.

Requirements

Functional

• Collect numeric samples with names and labels from servers and applications.
• Store them and answer range queries for dashboards, such as the last 6 hours or 30 days.
• Evaluate alert rules continuously and notify the right on-call person.
• Keep older data at a lower resolution to save space.

Non-functional

• Ingest 333,000 samples a second, with no data loss if storage is down for an hour.
• A new sample is visible to queries within 30 seconds.
• Dashboard queries for the last 6 hours: p99 under 1 second.
• Alerts fire within 2 minutes of a fault, and the monitoring system itself is more reliable than what it watches.

Back-of-the-envelope estimates

Assumptions for this exercise:

• These numbers are assumptions for the exercise, not facts about any real fleet.
• 10,000 servers each expose 500 series, so 5 million active series, sampled every 15 seconds.
• A raw sample is a timestamp and a value, 16 bytes in all. With compression we assume about 2 bytes a sample.
• Raw data is kept 15 days, then rolled up to 5-minute points for 13 months.

The arithmetic:

• Samples per second: 5M series ÷ 15 s → ≈ 333,000/s
• Raw data per day: 333,333 × 86,400 = 28.8B samples × 16 bytes → ≈ 461 GB a day, uncompressed
• Compressed data per day: 28.8B samples × 2 bytes → ≈ 58 GB a day
• Hot tier, 15 days: 58 GB × 15 → ≈ 0.86 TB
• Rollups, 13 months: 5-minute points are 20x fewer than 15-second ones: 58 GB ÷ 20 = 2.9 GB a day × 395 days → ≈ 1.1 TB
• One hour of queue buffer: 333,333 × 16 bytes = 5.3 MB/s × 3,600 s → ≈ 19 GB

High-level design

A Collector gathers samples from the Targets, the servers and applications. It scrapes each target on a schedule, so it knows what should exist, and it can also receive pushes from jobs that cannot be scraped. The collector puts batches on a Metrics queue, partitioned by series.

Ingest workers drain the queue. They group samples by series, compress them, and write them to the Time-series DB. The queue lets the database be slow or down for a while without losing data.

Reading has two clients. Dashboards call a Query service, which reads the database and picks a resolution that fits the time range. An Alert evaluator loads rules from the Alert rules store, runs their queries on a timer, and sends grouped notifications to the Pager.

How a request flows

Collect and store samples

1. Every 15 seconds the collector asks each target on its list for its current values. This is a pull.
2. Each target replies with metric names, labels and numbers. A target that does not reply is itself a signal.
3. The collector writes the batch to the queue, partitioned by a hash of the series, and moves on.
4. An ingest worker reads the batches for its share of the series.
5. It groups samples by series, compresses them and appends them to the database.

Draw a dashboard

1. A dashboard asks for one service's CPU over the last 6 hours, with a point every minute.
2. The query service reads the matching series from the database at the best resolution for that range.
3. The database returns compressed blocks, and the service decodes only the range it needs.
4. The service returns about 360 points per line, and the chart draws them.

Fire an alert

1. Every minute the evaluator loads the alert rules, each one a query, a threshold and a duration.
2. It runs each rule's query against the database, for example the error rate over the last 5 minutes.
3. The error rate is 7%, above the 5% threshold. The alert becomes pending.
4. After the condition has held for the full duration it fires, and one grouped notification goes to the pager.

Deep dive: pull or push, and why a queue

There are two ways to collect.

• Pull. The collector keeps a list of targets, from service discovery, and scrapes each one on a timer. It always knows what should be there, so a failed scrape is a clear 'this target is down' signal for free. It is easy to debug, since you can open a target's metrics page by hand. It needs network access to every target, and it struggles with short jobs that finish before the next scrape.
• Push. Each agent sends its metrics to the collector. This works through firewalls and for batch jobs that run for seconds. But silence is ambiguous: a quiet server might be down or just idle. Clients also decide the volume, so the collector must protect itself with limits and authentication.

Many systems use pull as the default and a small push gateway for short-lived jobs.

Between the collector and the database, put a queue. It gives you three things. It absorbs bursts, for example when a thousand servers restart together. It lets storage fail or be upgraded without losing samples, since the queue keeps them. And it lets other consumers, such as a stream processor, read the same data. In our estimate, an hour of buffer is about 19 GB.

Partition the queue by a hash of the series. All samples for one series then reach the same ingest worker in order, which makes batching and appending simple. When a worker falls behind, the queue length shows it, and you add workers.

Deep dive: compression, downsampling and retention

Time-series data is special: samples arrive in time order, at regular steps, with values that change slowly. Two tricks exploit that.

• Timestamps: delta-of-delta. Samples at 15-second steps have times t, t+15, t+30. The gaps are 15, 15, 15, and the change in the gap is 0. Store that zero in one bit.
• Values: XOR with the previous value. Two nearby floating-point numbers share most of their bits. XOR them, and an unchanged value gives all zeros, which costs one bit. A slightly changed value leaves a short run of different bits in the middle, and only that is stored.

Together these bring a 16-byte sample to a couple of bytes. We assumed 2, which turns 461 GB a day into 58 GB.

New samples go to an in-memory head block, backed by a write-ahead log in case of a crash. Every couple of hours the head is written out as an immutable compressed block. An inverted index maps each label pair, such as service=checkout, to the series that carry it, so a query finds its series without scanning.

Downsampling. Nobody needs 15-second points from last year. A background job rolls old data into 5-minute points. Keep the minimum, maximum, sum and count for each step, not just the average. An average hides a one-minute spike, and a maximum shows it. The average is still available as sum divided by count.

The query service picks the resolution from the range: 6 hours at 1-minute steps gives 360 points, and 30 days at 5-minute steps gives 8,640. Retention is then cheap, because whole blocks past their age are deleted, with no row-by-row cleanup.

Deep dive: alerts, grouping and cardinality

An alert rule is a query, a threshold and a for duration. 'Error rate above 5% for 5 minutes' ignores a single blip. A rule moves through states: inactive, pending when the condition first holds, firing after the duration, and resolved when it clears. The evaluator runs on a timer, every minute here, and writes each state change out.

A real outage can trigger hundreds of alerts at once. The notifier tames them:

• Group alerts by shared labels such as cluster and service, so one page covers them all.
• Deduplicate by alert identity. Two evaluators can run for reliability, and the notifier sends one page for the same alert from both.
• Throttle repeats. Remind every few hours, not every minute.
• Inhibit details. If a whole data centre is down, suppress the per-server alerts below it.
• Route by severity: a page for customer impact, chat for the rest. Allow silences for planned work.

Also alert on missing data. A server that stops reporting will never cross a threshold, so add a rule that fires when no samples arrive. And watch the watcher: keep a tiny independent check that fires if the monitoring system stops evaluating at all.

Cardinality is the quiet killer. Every distinct combination of label values is a separate series, and each one costs memory in the head block and the index. 1,000 endpoints, 1,000 hosts and 10 status codes make 1,000 × 1,000 × 10 = 10 million series. Adding a label with unbounded values, such as a user ID or a request ID, multiplies that without limit. Set a cap on series per metric and reject or drop above it, check new labels in review, and keep high-detail data in logs and traces, not in metric labels.

Trade-offs

• Collection: Pull by default, with a small push path for short jobs over Push from every agent. The collector knows what should exist, so a missing target is an alert. The cost is network reach to every target and a push gateway for batch jobs.
• Between collector and storage: A partitioned queue over Write straight to the database. Bursts and storage outages cost no data, and the data can be replayed. The cost is one more system to run and some added delay.
• Storage engine: A purpose-built time-series store with compression over A general SQL table with one row per sample. It stores a sample in about 2 bytes and serves ranges quickly. The cost is a narrower query model than SQL.
• Rollups: Keep min, max, sum and count per 5 minutes over Keep only the average. Spikes stay visible in old data. The cost is about four times the rollup storage of a single value.

Common follow-up questions

How do you monitor the monitoring system?
Run a small, separate watcher that checks whether the evaluator and the ingest path are alive. Add an alert that always fires, called a dead man's switch, and page if it stops arriving. Keep this path independent of the main stack.

How would you scale to ten times as many series?
Shard the time-series database by a hash of the series, with replicas for each shard. The query service then fans a query out to the shards and merges the results. Ingest scales by adding queue partitions and workers.

What about late or out-of-order samples?
Accept samples that are slightly late into the head block, within a window such as an hour, and reject older ones. When collectors stamp the time themselves, clock skew on the targets stops mattering. Clearly count and report anything rejected.

Why not store logs and traces in the same system?
Metrics are small aggregates with a fixed shape, which is why they compress so well. Logs and traces are large, free-form records with their own indexing and cost. Link them by shared labels so an engineer can jump from a spike to the logs.