Design an ad click aggregator

I would build two paths over one durable log. The click tracker stamps each click with an event time and a click ID created when the ad was shown, and appends it to a log partitioned by ad ID. Every event is also copied to an immutable raw archive. A stream job counts clicks per ad in one-minute windows by event time, using a watermark with two minutes of lateness, and keeps window state for ten minutes so slightly late events update the row.

It drops repeated click IDs using a one-hour set, about 3.3 GB over 24 partitions, and asks a fast fraud filter whether the click is valid. It checkpoints its state and log offsets every 30 seconds. It rewrites each open window's full count every 10 seconds and once more at the close, always as a versioned overwrite, so a restart or replay never counts anything twice. That gives dashboards in seconds.

For billing, a batch job reads the archive for each closed hour, removes duplicates globally, applies the final fraud rules, writes final counts and compares them with the stream. A difference above 0.1% raises an alert, because that error is half a million dollars a day. Billing reads only final rows, with corrections issued as adjustments.

These numbers are assumptions for the exercise, not facts about a real system:

• There are 1 billion clicks a day and the peak is 5 times the average. An event is 200 bytes.
• A window that has any click has 5 clicks on average. An output row takes 40 bytes.
• Duplicates are caught within a 1-hour dedupe window, using 16-byte click IDs.
• 99% of events arrive within 1 minute of the click, and 99.9% within 10 minutes.
• One stream worker handles 10,000 events a second. The average price is $0.50 per click.

The arithmetic:

Clicks per second

1B ÷ 86,400 s ≈ 11,574; × 5 for the peak

≈ 11,600/s on average, ≈ 58,000/s at the peak

Raw events per day

1B × 200 bytes

200 GB a day, 11.6 MB/s at the peak

Aggregate rows per day

1B clicks ÷ 5 per row = 200M rows × 40 bytes

8 GB a day, 25 times smaller than the raw events

Dedupe state

58,000/s × 3,600 s = 208.8M IDs × 16 bytes

≈ 3.3 GB, or about 140 MB on each of 24 partitions

Stream workers

58,000 ÷ 10,000 per worker = 5.8, rounded up to 6; × 2 for headroom

12 workers, over 24 partitions

Cost of a small error

0.1% of 1B = 1M clicks × $0.50

$500,000 a day

Count by event time, the moment the click happened, stamped by our own tracker. Do not count by processing time, the moment the job sees it. If the job stalls for ten minutes and then catches up, processing time would pile ten minutes of clicks into one minute, and a billing day could even be wrong. Never trust the user's device clock.

Use tumbling windows of one minute, such as 12:05:00 up to 12:06:00. Partition the log by ad ID, so every event for an ad meets the same task and the same state.

The job needs to know when a window is complete. That is the job of the watermark: the job's claim that no events older than this time will arrive. It is the largest event time seen, minus an allowed lateness, say 2 minutes. If the newest event is 12:08:10, the watermark is 12:06:10. The window ending at 12:06:00 is now closed, and its final result is written. The dashboard does not wait for this. While a window is open, the job fires early: every 10 seconds it overwrites the row with the count so far, marked provisional, so a click shows up within about 10 seconds and the newest minute only grows. The close writes the row once more as complete, and late events can write it again. With many partitions, take the lowest watermark, and treat a partition with no traffic as idle after a timeout, so it does not hold everything back.

Late events are those that arrive after their window closed. Handle them in three tiers:

• Slightly late. Keep window state for 10 minutes after the close. A late event updates the count and the job writes the row again. This is safe because the write replaces the row with the new full count.
• Very late. Past 10 minutes, send the event to a late output and do not touch the stream's count. With 99% of events arriving within 1 minute and 99.9% within 10, this tier catches almost everything.
• Always safe. The raw archive has every event, so the batch job includes late events in the final count for the billing period.

Billing needs a cut-off. Finalise an hour a fixed time after it ends, such as one hour. A click that arrives after the cut-off goes into an adjustment on the next invoice rather than changing a closed one.

Duplicates come from everywhere: a double tap, the tracker retrying after a timeout, the log delivering a message twice, and the job replaying after a restart. The defence is a click ID created at the source, when the ad is shown, and reused by every retry. Without it a duplicate cannot be told from a second real click.

Dedupe in the job. Keep a set of click IDs seen in the last hour, in state that is partitioned the same way as the log. If an ID is already there, drop the event and count it as a duplicate. Our estimate is 208.8M IDs × 16 bytes, about 3.3 GB, or about 140 MB on each of 24 partitions. Expire old entries by event time. A duplicate that arrives after an hour slips past the job, and the batch job catches it later.

Exactly once. Real exactly-once delivery does not exist. What we build is at-least-once delivery with effects that are safe to repeat. It takes two parts.

• Checkpoints. Every 30 seconds the job saves its state, the window counters and the dedupe set, to durable storage together with the log offsets it has read. After a crash it restores that snapshot and reads again from the saved offsets. State and offsets match, so the replay leads to the same result and nothing is counted twice.
• An idempotent sink. The events between the last checkpoint and the crash were already counted, and their output may already be written. If we added 1 to a counter for each click, the replay would add them again. Instead write the whole window's count to a row keyed by ad and window start. Writing the same value twice leaves one row. Add a version, so an older write never overwrites a newer one.

Take ad 77 at 12:05. The job writes 41 and then crashes. After the restore it counts the same events again, reaches 41 and writes the same row. If it reads two more late events it writes 43. At no point is there a 'plus one' that could be applied twice.

A transactional sink that commits offsets and output together also works, but it is heavier. The idempotent write is simpler.

The stream is fast, but its edges are loose: a lateness cut-off, a one-hour dedupe window, and a fraud rule set that was current at the time. A bug could skew it silently. Since 0.1% is half a million dollars a day, billing cannot rest on it alone. So there is a batch path over the immutable raw events, which is the source of truth.

For each closed hour the batch job reads the archive, selecting by event time. It removes duplicates by click ID over the whole period, applies the latest fraud rules, groups by ad and minute, and writes the result to the count store as final, together with the run ID and the rule version. The same input and the same rules always give the same output, so it can be rerun at any time. Share the counting code with the stream job, so the two cannot drift apart.

The job also compares its numbers with the stream's. Alert if any hour differs by more than, say, 0.1%. A difference is not an annoyance. It is a signal that one path has a bug. Billing reads only final rows. If a correction comes after an invoice was sent, issue a credit or an extra charge, never a silent edit of the invoice.

Raw events are kept for 90 days, in compressed columnar files partitioned by hour. At about 5 to 1 compression, 200 GB a day is 40 GB, or about 3.6 TB for 90 days. That is enough for recounts and disputes.

Fraud. Sources include bots, click farms, and a publisher clicking its own ads. In the stream, apply cheap in-memory rules: too many clicks from one device or network a minute, a click within a third of a second of the impression, known data-centre address ranges, and an odd user agent. Mark the click invalid with a reason. Never delete it, so counts show valid and invalid separately and a dispute can be explained. The batch job adds heavier analysis over the whole day, such as grouping devices that behave alike, and can flip a click to invalid after the fact. The stream must never wait on a slow check.