Design a centralized logging pipeline

The pipeline has four stages: agent, queue, processor, store. An agent on each host reads standard output, adds labels, batches, compresses and ships, and spools to a disk buffer if the pipeline is down. At 20 lines a second, a 1 GB buffer covers about 35 hours.

Batches go to a durable partitioned queue that absorbs peaks of 500,000 lines a second and holds six hours, about 0.86 TB. A processor parses fields, redacts secrets in the agent and again here, samples noisy sources by trace ID, and writes each line to two places: a time-partitioned search index, one index per day, holding the last 7 days, about 6 TB, and an archive of compressed files for 90 days, about 39 TB. The index holds exact-value fields and tokenized message text, and a flush every 5 seconds meets the 30-second freshness goal.

Retention is deleting whole days. A tier manager moves data and loads old days into a temporary index when someone searches them. The trade-off is cost against speed: only the recent week is fast, and the rest is cheap and slow.

These numbers are assumptions for the exercise, not facts about a real system:

• 5,000 hosts each write 20 lines a second on average, and the peak is 5 times the average.
• An average line is 400 bytes.
• The searchable form, with compression and an index, takes about 25% of the raw size. A compressed archive takes about 12.5% (8 to 1).
• Engineers search the last 7 days often. Logs are kept 90 days.

The arithmetic:

Lines per second

5,000 hosts × 20 lines; × 5 for the peak

100,000/s on average, 500,000/s at the peak

Bytes per second

100,000 × 400 B = 40 MB/s; 500,000 × 400 B = 200 MB/s

40 MB/s average, 200 MB/s at the peak

Raw volume per day

40 MB/s × 86,400 s

≈ 3.5 TB a day

Queue buffer, 6 hours

40 MB/s × 21,600 s

≈ 0.86 TB

Hot index, 7 days

3.456 TB × 25% = 864 GB a day; × 7

≈ 6 TB, before replicas

Archive, 90 days

3.456 TB ÷ 8 = 432 GB a day; × 90

≈ 39 TB in cheap object storage

Services should do nothing special: they write a line to standard output. An agent on each host reads it and does the shipping, so no application code changes.

The agent does four jobs. It adds labels (host, service, version, environment). It groups lines into batches, say 1 MB or 1 second, whichever comes first. It compresses each batch to save bandwidth. And it sends with retries and growing delays. It remembers how far it has read in each file, so after a restart it carries on without gaps or repeats, and it follows a log file when it is rotated.

When the pipeline is unreachable, the agent spools batches to a disk buffer. At 20 lines a second of 400 bytes, a host writes 8 KB a second, or 28.8 MB an hour. A 1 GB buffer therefore covers about 35 hours at the average rate, far beyond our 1-hour goal. If the buffer does fill, the agent drops the lowest-value lines first, debug before info, and never audit lines. It counts every drop and reports the count, so a loss is visible.

Delivery is at least once, so a retry after a timeout can send a batch twice. Give every line an ID built from the host, the file and the offset. The index uses it as the document ID, and a duplicate just overwrites the first copy.

In the middle sits the queue: a durable, partitioned log. It is there because indexing is the slow and bursty step, and agents must not wait for it. At the 5 times peak of 200 MB/s the queue simply grows. Six hours of average traffic is 40 MB/s × 21,600 s, about 0.86 TB, which is modest. A second reason: other consumers, such as a security tool, can read the same stream. Alert on consumer lag, since lag is the early warning that searches will fall behind.

Do not rely on arrival order. Search sorts by the event time written in the line, and each line also records its ingest time for debugging.

Searching by reading every line is far too slow, so build an inverted index: for each term, a list of the lines that contain it. The question is what to put in it.

• Structured fields such as service, level, host and trace ID are indexed as exact values. Filters on them are cheap and precise.
• The message text is split into words and indexed for full-text search.
• The raw line is stored compressed and returned for display. It is not searched.
• Do not index everything. Free-text fields full of unique IDs make the dictionary of terms huge. Give each field a rule, and keep unknown fields stored but not indexed.

Time-partitioned indices. Write to one index per day, or per hour for the biggest services. A query for the last 15 minutes touches only today's index, and a 7-day query touches seven. Old days are made read-only and compacted. Retention is cheap, because deleting a whole index is instant and needs no row-by-row cleanup. Size shards for about 40 GB each: 864 GB a day divided by 40 GB is about 22 shards per day.

Freshness. New lines become searchable after the index flushes a segment, which we set to every 5 seconds. Add the queue and batching delay and you land at roughly 10 to 15 seconds, inside the 30-second goal. Bulk-write thousands of lines at a time, not one by one.

Parsing. Apps that log JSON are easy. For older plain-text logs, write a pattern per service that pulls out the time, level and message. A line that fails to parse is kept whole and marked with a parse error, never dropped. Also guard against schema drift. If one service logs status as the number 200 and another as the text '200', the index will reject one of them, so agree on field names and types, and cap the number of fields per index at, say, 1,000.

For paging through results, use a cursor that remembers the last sort value, not an offset. An offset gets slower with every page.

Tiers. Keep the last 7 days hot: on fast disks, fully indexed, with a replica. Keep days 8 to 90 cold, as compressed files in object storage, with no index. To keep this simple, the processor writes every line to the archive on the way in, so the hot index is a fast copy and moving data down a tier just means deleting from it. Search over cold data loads files into a temporary index, which is slow but rare.

Compare the cost. Indexing 90 days would hold about 78 TB of fast storage (864 GB × 90). The tiered plan holds 6 TB hot plus 39 TB of cheap archive. Retention differs per service: audit logs may need a year or more in the archive, while debug logs need three days. Set that per service and day prefix with an object storage lifecycle rule.

Sampling. A few sources create most of the volume, often health checks and debug lines. Drop 90% of successful health-check lines. If they are 40% of the volume, the total falls to 60% + 4% = 64%, a 36% saving. Keep every warning and error. Sample by a hash of the trace ID, so a kept trace is complete and has no holes. Store the sample rate on each kept line, so a count of 1 can be read as 10. Never sample audit lines. A per-service rate cap stops one runaway loop from drowning the pipeline: when a service passes its quota, drop its lowest levels and emit one line saying so.

Redaction. Logs pick up passwords, tokens, card numbers and email addresses. Strip them before anything is stored. Apply rules by field name, so values of password and authorization are replaced with a marker, and by pattern, such as a 13 to 19 digit number that passes the Luhn check, or a known token format. Do it in the agent, so secrets never cross the network, and again in the processor as a safety net. No rule set catches everything, so treat the archive as sensitive too: encrypt it, limit who can search, and record every search.