Design an image hosting service

The workload is heavy bytes, not heavy requests, so I keep the bytes off our servers. A client asks an image API for an upload slot. The API checks quota, records a pending image in a metadata database, and returns a presigned URL for one upload of up to 20 MB to a random key. The client sends the file straight to object storage, which emits an event to a queue.

An image worker checks magic bytes, decodes the file, limits pixel count, strips EXIF while applying rotation first, hashes the content to dedupe, makes three standard sizes and marks the image ready. Viewers get images from a CDN, with links that never change, so they cache for a year. On a miss, a thin image origin checks the metadata DB, returns 404 for deleted or pending images, finds the blob and reads or makes the variant. At 500 million views a day and a 95% hit ratio, the origin sees about 290 requests a second.

Rare sizes are made on demand, but only from an allowlist, with a strict cache key, so nobody can fill the cache. A delete marks the record, so the origin answers 404, purges the exact variant keys, deletes the image's own variants and removes the original when no record uses it. The trade-off is a two-step upload and a purge for every delete.

These numbers are assumptions for the exercise, not facts about a real system:

• 5 million images are uploaded a day, averaging 3 MB each.
• There are 500 million image views a day, and the average view sends 100 KB.
• Three standard variants are made for each image: a thumbnail of 15 KB, a medium size of 100 KB and a large size of 400 KB.
• The CDN answers 95% of views from its cache.

The arithmetic:

Uploads per second

5M a day ÷ 86,400 s ≈ 58

≈ 58/s on average, about 290/s at a 5x peak

Originals per day

5M × 3 MB = 15,000,000 MB

15 TB a day, about 5.5 PB a year (15 TB × 365)

Variants per day

(15 + 100 + 400) KB = 515 KB per image × 5M

≈ 2.6 TB a day, about 17% of the originals

Views per second

500M a day ÷ 86,400 s ≈ 5,787

≈ 5,800/s on average

Requests reaching the origin

5% misses × 5,800/s

≈ 290/s, easy for a small fleet

Egress bandwidth

500M × 100 KB = 50 TB a day; × 8 bits ÷ 86,400 s

≈ 4.6 Gbit/s on average, mostly carried by the CDN

Sending 15 TB a day through API servers would tie up connections and bandwidth for no benefit. Instead the client uploads directly to object storage with a presigned URL: a link the API signs, which allows exactly one action and expires soon.

The API fills the signature with rules. It names the exact storage key, which is a random ID and never the user's file name. It sets a time limit of about ten minutes. It caps the size, since object stores can enforce a content-length range in the signed policy. It also fixes the content type. Large files use multipart upload, so a dropped connection resumes from the last part. A rule on the bucket deletes pending objects that were never confirmed after a day.

Do not trust any of that for safety. The declared type and the file extension are claims from the client. The worker checks the real thing:

• Magic bytes. A JPEG begins with FF D8 FF, and a PNG with 89 50 4E 47. If the bytes do not match an allowed format, reject the file.
• A full decode. Open it with a well-maintained image library. A file that does not decode is not an image.
• A pixel limit. A small file can describe a huge picture. A PNG of 30,000 by 30,000 pixels is 900 million pixels, which needs about 3.6 GB of memory at 4 bytes a pixel. Read the dimensions from the header first and refuse anything above, say, 50 megapixels, which is about 200 MB decoded.
• A re-encode. Never serve the uploaded bytes as they are. Writing new pixels into a fresh file removes anything hidden inside, such as a script placed after the image data.

Serve images from a separate, cookie-free domain with a correct Content-Type and the header X-Content-Type-Options: nosniff, so a browser never treats an image as a page. Rate limit uploads per user, and keep the state pending until every check passes.

There are two ways to get a 640-pixel-wide copy.

• Ahead of time. The worker makes a fixed set, say 150, 640 and 1,600 pixels wide, when the image arrives. CPU use is steady and predictable, and the first viewer is never slow. The cost is wasted storage on sizes nobody opens, and a back-fill job whenever the set changes.
• On demand. The link carries the size, such as /abc123/w640.webp. On a CDN miss, the image origin makes it, returns it and the CDN keeps it. Nothing is wasted and any size is possible. The first viewer waits maybe 100 to 300 ms. The danger is abuse: an attacker can request widths 1, 2, 3 and up to 100,000, each a CDN miss and a CPU-heavy resize, and fill the cache with junk.

The defence for on-demand is a variant allowlist. Accept only sizes from a short list, such as 150, 320, 640, 1,024 and 1,600 pixels, and only the formats we choose, WebP and JPEG. Anything else gets a 400 before any work is done. That caps a single image at 5 sizes × 2 formats = 10 cached objects, instead of an unlimited number.

Next, control the cache key. It should be the image ID plus the variant name plus the format, and nothing else. Ignore unknown query parameters, or an attacker can add ?x=1, ?x=2 and so on to get a fresh miss each time. Collapse concurrent requests, so a thousand viewers who miss at once cause one resize, not a thousand. Cap the number of resizes running at once.

My pick is a hybrid. Make the three common sizes ahead of time, so almost every view is a plain storage read. Serve the rest of the allowlist on demand, by the image origin. Put a version or content hash in the link, so a link is immutable and can be cached for a year with the immutable directive.

Dedupe. The worker computes a SHA-256 hash of the cleaned original. If a blob with that hash already exists, it does not store a second copy. Each image record points to a shared blob of the original, and a reference count tracks how many records use it. The variants are not shared. Each record owns its own, made from the shared original, so deleting one image never touches another's sizes. The stable link names the image ID, so the origin must look the ID up in the metadata DB to find the blob. The bucket alone cannot answer that. Popular memes upload thousands of times, so this saves real space. It catches only exact copies. A resized or re-compressed version has a different hash. Do the check on the server only. If an upload could finish instantly when the file already exists, an observer could learn that someone else holds that file.

Metadata. Photos from a phone often carry an EXIF block with GPS coordinates, the camera's serial number and timestamps. Sharing a photo with an exact home location is a real harm. Strip all of it from everything we serve. One catch: the orientation tag. If you drop it without care, portrait photos turn sideways. So apply the rotation to the pixels first, then remove the tag. Convert colour profiles to sRGB so images look the same in every browser.

Deletes. A delete has more layers than it seems, so do them in order:

• Mark the record deleted in the metadata DB. This is the source of truth. Every CDN miss goes through the image origin, which reads the record and answers 404 for a deleted image, so a purged copy cannot come back.
• Purge the CDN. The record lists every variant key, at most about ten, so the purge is exact. Purges are not instant: allow seconds to a few minutes, which fits the 5-minute target.
• Delete the image's own variant objects at once, since only the original is shared. Lower the original blob's reference count. At zero, a cleanup job deletes it from storage.
• Backups age out at the end of their retention period.

Links are immutable and cached for a year, so a purge is the only way to remove a cached copy from the CDN. Be honest about the limit: a browser that already holds the image, or a person who saved it, keeps it. Say that plainly in the privacy policy.