Design a pastebin

I would split a paste into two parts: a small metadata row and a large, immutable body. The row lives in a database with the key, owner, expiry, visibility and status. The body lives in object storage, compressed. The key is 8 random characters, with a unique constraint and a retry, so links cannot be guessed.

A CDN sits in front of reads. Because bodies never change, the CDN can cache them for as long as the paste has left to live, and a viral link barely reaches the origin. At 1 million pastes a day and 10 reads each, the origin sees about 116 reads a second at peak. Expiry is a lazy check on every read plus a batch worker that deletes bodies first and rows last.

Burn after reading is one conditional update, so exactly one reader wins, and a reveal screen stops link previews from burning it. A queue feeds a scanner that looks for leaked secrets and abuse, and a flagged paste is blocked and purged from the CDN. The main trade-off is a short window where a bad paste is live before the scan finishes, in exchange for fast, always-available creates.

These numbers are assumptions for the exercise, not facts about a real system:

• 1 million new pastes a day, and each paste is read 10 times on average, so 10 million reads a day.
• The average paste is 10 KB and the cap is 512 KB. Text compresses about 3 to 1.
• A metadata row (key, owner, times, size, flags, body location) takes about 200 bytes.
• Peak traffic is 10 times the average, and a CDN answers 90% of reads from its cache.

The arithmetic:

Writes per second

1M ÷ 86,400 s ≈ 11.6

≈ 12/s, 120/s at a 10x peak

Reads per second

10M ÷ 86,400 s ≈ 116

≈ 116/s, 1,160/s at peak

Origin reads at peak

1,160/s × (1 − 0.9 cache hits)

≈ 116/s reach the service, a trickle

Body storage a year

1M × 10 KB = 10 GB a day; × 365 = 3.65 TB; ÷ 3 for compression

≈ 3.65 TB raw, ≈ 1.2 TB compressed

Metadata a year

1M × 200 B = 200 MB a day; × 365

≈ 73 GB, which fits on one database node

Key space

62^8 ≈ 2.2 × 10^14 keys; 10 years of pastes is 3.65 billion

≈ 0.002% of keys used after 10 years

Split the paste into two parts that behave very differently. The body is large, written once, never changed and read by key. The metadata is small, changes (status, expiry) and needs a unique key and quick lookups.

Object storage suits bodies: it is cheap per gigabyte, durable, and made for reading a whole object by name. A database suits metadata: a row holds the key, owner, created and expiry times, size, visibility, status and the body's location. At 200 bytes a row, a year of metadata is about 73 GB, so one primary with replicas is enough for a long time. Bodies go to a store that grows to terabytes without any work from us.

Enforce the size cap while the upload streams in, by checking Content-Length and cutting the stream at 512 KB. Do not read an unbounded body into memory first. Compress text with a fast algorithm such as gzip or zstd before storing it. Text shrinks about 3 to 1, and the CDN can pass the compressed bytes straight to browsers that accept them. Very small pastes, under a few kilobytes, can live in the metadata row to save a second round trip. That is an optimisation to add later, not a starting point.

Keys. Generate 8 random characters from a secure random source, using the 62 letters and digits. That gives about 2.2 × 10^14 keys, so a clash is rare, and a unique constraint on the key catches the rare one: the insert fails and the service tries another key. Do not use a counter. A counter makes links guessable, which would expose unlisted pastes.

Write order. Store the body first, then insert the row. A crash between the two leaves an orphan body that no row points to, which is harmless and found later by a sweep. The reverse order would leave a row that points at nothing, and readers would see errors.

Store an expiry time on each row. Expiry is then enforced twice, for two different jobs.

• Lazy check. On every read, compare the expiry time with now. An expired paste returns "gone" even if the cleanup has not reached it yet. This gives correctness.
• Batch cleanup. A worker runs every few minutes, finds rows past their expiry through an index on the expiry column, and deletes them in batches of about 1,000. It deletes the body first and the row last, and every step is safe to repeat. This gives the storage back.

A scan of the whole table would not scale, so the index matters. For a fixed menu of expiries, such as 1 hour, 1 day and 1 week, the object store's own lifecycle rules can remove bodies by age, with the database sweep removing rows.

Burn after reading. Two readers who open the link at the same moment must not both see the paste. Make the read a claim: one conditional update that sets the status to burned only where the status is still active, and returns the body location. Exactly one request gets a row back. The others see no row and get "gone". The winner fetches the body and returns it, and the worker deletes the data later.

Three details make this work in practice. First, a burn paste must be sent with a no-store cache header, or the CDN will serve a second reader from its copy. Second, many chat apps fetch a link to build a preview, and that fetch would burn the paste before the person arrives. So the first page is a small "reveal" screen, and only the button on it makes the claiming request. Third, if the winner's connection drops after the claim, the paste is lost. That is the defined behaviour of burn after reading, and the page should say so before the reader reveals it.

CDN reads. A paste never changes, so the URL for a key always returns the same bytes. That makes it ideal for a CDN: send a public cache header and the CDN keeps the copy close to readers. Set the cache lifetime to the smaller of the paste's remaining life and a fixed ceiling such as one hour, so a cached copy never outlives the paste. At a 90% hit rate, about 116 of 1,160 peak reads a second reach the origin. A viral link is cheaper still, because the CDN collapses simultaneous misses for one key into a single origin request.

Deleting early. A creator's delete or an abuse takedown removes the row, then calls the CDN's purge by key. The purge is what meets the target of unreachable within a minute. Without it, a short cache lifetime would be the only protection.

Visibility. There are three levels. A public paste can be listed. An unlisted paste is reachable by anyone with the key, and its 8 random characters are the secret. A private paste needs sign-in and a check that the viewer is the owner. Private responses carry a private cache header so shared caches never store them. Slow down guessing by rate limiting repeated not-found answers per client.

Abuse. Pastebins attract spam, malware links and accidentally leaked credentials. Scan after the create call returns, using the queue. Look for known secret formats, such as private key headers and cloud access keys, plus bad links and repeated spam text. A flagged paste is marked blocked, which stops serving it at once, and the CDN copy is purged. Per-client create limits and a report button cover the rest.

Serving safely. Return text as plain text, with a no-sniff header, from a separate domain from the main site. Then a paste containing script can never run with the site's cookies. Add a no-index header to keep spam pastes out of search results.