System design interview
Design a content delivery network
Edge steering, a shield tier, request collapsing, hit ratios and purges.
Press Play to follow a request through the design, or step through it.
- Client and edge
- Service
- Cache and queue
- A browser looks up the file host. The steering DNS sees the resolver's address, and the viewer's subnet if the resolver passes it.
- It answers with the addresses of the two best healthy sites for that network, with a TTL of 30 seconds.
- The browser connects to the first address and asks for a file. The front end ends TLS.
- Meanwhile every site reports its probe results and load every 5 seconds. A site over 80% load gets less traffic, and a dead site gets none.
- Viewer
- Steering DNS: geo, health
- Edge front end: TLS, hash route
- Purge service: fan-out, confirm
- Cache servers: one owner a URL
- Regional shield: one miss/region
- Customer origin
On this page
In 2 minutes
I would build four layers. First, steering: DNS answers that depend on the resolver's location, the viewer's subnet when available, and each site's health and load, with a 30 second TTL and two addresses from two sites. Second, each site hashes cache keys to a single owner among its 20 servers, so one site holds 1 PB of distinct files. Third, a regional shield serves the misses from many sites, so a new file is fetched from the origin 5 times, not 300.
Collapsing makes all waiting requests share one fetch. With 5 million requests a second, a 92% edge hit ratio and an 80% shield hit ratio, origins see 80,000 requests a second, 1.6% of the total. Fourth, freshness: origin cache headers, stale-while-revalidate, and versioned file names with a one-year lifetime as the main update tool. For purges, a service sends a numbered purge down a tree with fan-out 20, which reaches 6,000 servers in about 450 ms, counts acknowledgements and retries.
Each tier records a purge time, so a shield cannot refill an edge with the old copy. The trade-offs are an extra hop on a miss, and DNS steering that trusts resolver location.
Step 1 of 6: Clarify the scope
Interviewer asks
Design a content delivery network. Customers point their websites, images and video segments at it. It keeps copies in sites around the world, sends each viewer to a nearby one, and fetches from the customer's own servers, the origin, only when it must. Updates and removals have to reach every site fast.
Questions to ask first
Ask these before you draw anything. Under each one is why its answer changes the design.
- What do we serve: small web files, large downloads, video segments, or all three?
Small files make the request rate the limit. Large files and video make bandwidth and disk the limit, and need range requests.
- How many sites will we run, and how many customers?
The number of sites sets the work of one purge and the number of origin fetches for one new file.
- How quickly must a purge take effect, and must we confirm it?
Seconds with a confirmation needs a fan-out tree and a record of who has applied it. Minutes would let us rely on short lifetimes.
- Is some content private, such as paid video or signed download links?
Private content needs a signature check at the edge and cache keys that ignore the signature but not the file.
- What is the goal for protecting the origin: a hit ratio, or a hard limit on its load?
A hit-ratio goal is a cache design. A hard limit adds collapsing, a shield and per-customer limits at the origin.
Requirements
Functional
- Serve a customer's files over HTTPS from a site near the viewer, and fetch from the origin on a miss.
- Honour the origin's cache rules, with per-customer overrides.
- Purge by URL, by path prefix or by tag, and report when the purge is complete.
- Check signed URLs at the edge for private content.
Non-functional
- A cache hit has a time to first byte under 50 ms at p95 for viewers near a site.
- At least 98% of requested bytes are served without touching an origin.
- A purge is applied on 99% of servers within 5 seconds, and confirmed when all have it.
- A failed site is out of rotation within 60 seconds, and viewers are not left without an address.
- A file that goes viral causes one origin fetch per region, not one per request.
Step 2 of 6: Size it
Hit ratio is the whole economy: moving from 92% to 93% at the edge cuts origin traffic by an eighth. Edge caches, a shield and request collapsing each remove a layer of repeated work, and purges need a tree, not a loop.
Assumed for the exercise, not facts about a real system:
- 5 million requests a second at the peak, averaging 250 KB each. They are spread across 300 sites, and each site has 20 cache servers with 50 TB of disk.
- The edge hit ratio is 92% of requests. Five regional shields answer 80% of what the sites miss. These ratios are assumptions for the exercise.
- A message between two machines in a purge tree takes about 150 ms.
- Egress to viewers
- 5M requests/s × 250 KB = 1.25 TB/s; × 8 bits
- 10 Tbit/s at the peak
- Origin load with edge caches only
- 5M × (1 − 0.92) = 400,000 requests/s; × 250 KB = 100 GB/s
- 400,000/s and 800 Gbit/s, far too much
- Origin load with a shield tier
- 400,000 × (1 − 0.80) = 80,000 requests/s; × 250 KB = 20 GB/s
- 80,000/s and 160 Gbit/s, which is 1.6% of requests, so 98.4% offload
- Origin fetches for one new, popular file
- 300 sites each fetch it once, against 5 shields
- 300 fetches, or 5 with a shield: 60 times fewer
- Unique content in one site
- 20 servers × 50 TB when URLs are spread by hash, against 50 TB when every server caches what it sees
- 1 PB against 50 TB of distinct objects
- Purge to every server
- 300 × 20 = 6,000 servers; a tree with fan-out 20 has 20 × 20 × 20 = 8,000 leaves at depth 3; 3 × 150 ms
- ≈ 450 ms to reach all, well inside 5 seconds
Step 3 of 6: Sketch the path of one request
Step through the diagram at the top of the page to watch a request move through these parts.
A Viewer asks Steering DNS for the address of the customer's file host. Steering knows each site's health and load and the viewer's rough location, and answers with one or two nearby healthy sites. The viewer connects to an Edge front end at that site, which ends TLS and checks signed URLs.
Inside a site, many servers share the cache. The front end hashes the cache key to find the Cache server that owns that file, so each file is stored once per site. On a miss, the cache server asks the Regional shield, a bigger cache that serves many sites. Only if the shield also misses does a single request reach the Customer origin. While a fetch is in progress, every other request for the same file waits for it.
A Purge service takes purge requests from customers, and sends them down a tree to every cache server and shield. Each server confirms, and the service reports completion. Edge servers also report their health and load back to steering, so that a failing or overloaded site is soon left out of its answers.
Other paths through the system
Serve a file on a miss
- A viewer asks for a video segment that nobody in this region has requested yet.
- The front end builds the cache key from the host, path, clean query and the encoding the viewer accepts, then hashes it to the owning cache server.
- The owner has no copy, so it asks the regional shield. Every other request for this key now waits behind this one.
- The shield misses too, and sends the first and only request for the file to the customer's origin.
- The origin returns the file with a cache lifetime. The shield stores it.
- The shield returns it to the owner, which stores it on disk and answers all waiting requests together.
- The front end gets the bytes and starts sending them to the viewer as they arrive.
- The viewer plays the segment. The next viewer who asks is a hit on the owning cache server, and the origin is not involved.
Purge a file everywhere
- A customer purges one image. The purge service numbers the purge and sends it down a tree to all 6,000 cache servers.
- Each cache server records the purge time and tells the regional shield, so the shield cannot hand the old copy back.
- The shield records it too and confirms.
- Each cache server confirms to the purge service. Late or missing confirmations are retried, and the purge is reported complete when every server has applied it.
Step 4 of 6: Go deep on the hard part
Steering viewers to a good site
Interviewer asks
How do you send each viewer to a site that is near, healthy and not overloaded?
There are two ways to do it, and many networks combine them.
Read moreShow less: Steering viewers to a good site
DNS steering. The CDN runs the authoritative DNS for the file host. Its answer depends on who asks. The resolver's address gives a rough location, and some resolvers also pass part of the viewer's own subnet, called client subnet, which is more precise at a small privacy cost. The weakness is that the CDN sees the resolver, not the viewer. A viewer in one country who uses a public resolver in another may be sent to the wrong continent. Answers carry a short TTL, about 30 seconds, so the CDN can change its mind quickly, at the price of more DNS queries.
Anycast. Every site announces the same IP address over BGP, and the network delivers each packet to the site that is closest by routing distance. Failover is quick: a failed site withdraws its announcement, and traffic moves in seconds. An attack is spread across many sites. The weaknesses: it is hard to steer by load, since the only levers are withdrawing or weakening an announcement. Also, a route change in the middle of a long connection can move it to another site, which resets it. Closest by routing is not always lowest latency either.
A sensible mix. Use DNS for decisions that need data, such as load and health, and let it return addresses that are anycast within a region. The DNS servers themselves are anycast, so they are always reachable.
Health and load. Each site reports probe results, CPU and bandwidth use every 5 seconds. Steering computes a weight for each site. A site above 80% load sheds part of its share to the next best site that is no more than about 30 ms slower. A site that fails its probes is dropped from answers, which takes effect after the 30 second TTL plus detection time, so about 60 seconds. To cover that gap, return addresses from two different sites, so a browser can try the second.
Where location comes from. Build the latency map from real measurements, taken by small scripts in pages that time a download from several sites, grouped by network prefix. That beats guessing from a map of countries.
Caches inside a site, above it, and the hit ratio
Interviewer asks
How do you lay out the caches so that the origin sees almost nothing, and what happens when a file goes viral?
The cache key. It is the host, path, a cleaned query string and the request headers that change the response.
Read moreShow less: Caches inside a site, above it, and the hit ratio
Remove tracking parameters and sort the rest. A Vary header on encoding becomes a few classes, such as gzip, brotli and plain. A Vary on User-Agent would create a copy for every browser version, so map it to two or three device classes, or do not cache. Never put cookies in the key unless the customer asked for it, and do not store responses that set one.
Inside a site. A site has 20 servers. If each cached whatever it happened to see, the same popular files would sit on all 20 and the site would hold about 50 TB of distinct objects. Instead, the front end hashes the key with consistent hashing, so each file has one owner. Distinct capacity becomes 20 × 50 TB, or 1 PB. When a server is added or removed, only about 1 in 20 files changes owner. The extra hop inside the site is cheap. For a handful of extremely hot files, keep a tiny copy in each front end's memory for a few seconds.
The shield. Sites send their misses to one of five regional shields, which are larger caches. A file new to a region is then fetched from the origin once per region, not once per site: 5 fetches instead of 300. In the estimate, 92% edge hits and 80% shield hits send 80,000 requests a second to origins, 1.6% of the total, so the origin load is 98.4% lower than with no CDN at all.
Collapsing. When a request misses, take a lock on that key. Every later request for it waits, up to several seconds, for the one fetch to finish, and then all are answered together. Without this, a viral file or a freshly purged one sends thousands of identical requests to the origin at once. Treat each video segment as its own object, and let a range request be served from a whole cached file.
Admission and the last point of hit ratio. Many files are requested only once and never again. Admit an object to the disk cache on its second request, tracked with a small filter, so one-hit wonders do not push useful files out. Each point matters. Going from 92% to 93% cuts edge misses from 8% to 7%, which is an eighth less traffic to the shields and origins.
Freshness and purging
Interviewer asks
How do updates reach viewers, and how do you remove a file from 6,000 servers in seconds and know it worked?
Lifetimes. Take the lifetime from the origin's cache headers, with a default per customer for files that have none.
Read moreShow less: Freshness and purging
After expiry, send a conditional request with the file's validator. A not-modified answer extends the life without a download. Two extensions help: stale-while-revalidate serves the old copy for a short time while one background request refreshes it, so no viewer waits, and stale-if-error serves the old copy when the origin is down.
Versioned file names are the main tool. Put a hash of the content in the name, such as app.4f9a1c.js, and cache it for a year as immutable. A new release means a new name, so no purge is needed. Only the page that mentions the names is cached briefly, say 60 seconds. Purge is for mistakes, legal removal and urgent changes, which is why it can be a heavier mechanism.
Purge types. By URL, by path prefix, and by tag: the origin labels responses, for example with a product id, and one purge call removes all files with that label. A tag purge needs an index of tags to keys on each server. A cheaper variant marks the files stale so the next request revalidates, rather than deleting them.
Fan-out. A loop over 6,000 servers is slow and fragile. Send the purge down a tree where each node forwards to 20 others. Three levels reach 8,000 servers, and at about 150 ms a hop that is roughly 450 ms. Each server applies the purge, acknowledges up the tree, and the service counts acknowledgements. It retries missing ones, and removes a site that cannot be reached from rotation until it catches up. Number the purges in a log, so a server that was off and sees a gap can ask for the ones it missed.
The refill race. If the edge purges first, it can refill straight from a shield that still holds the old file. So record a purge time for the key, and treat any copy that was fetched before that time as stale, on every tier. The shield then also revalidates, and the old file cannot come back. Report the purge as complete only when every tier has confirmed it.
Step 5 of 6: Name the trade-offs
Steering viewers
DNS answers shaped by health and load over Anycast only
We can move load deliberately and use measured latency. The cost is trust in resolver location, and a delay of about a minute to leave a failed site.
A shield tier above the sites
Five regional shields over Every site asks the origin directly
Origin fetches for a new file drop from 300 to 5. The cost is an extra hop on a miss, and more caches to run.
Placing files inside a site
Hash each URL to one owner server over Every server caches what it sees
A site holds 20 times more distinct files and has a higher hit ratio. The cost is an internal hop, and a server failure loses its share of the cache.
Updating content
Versioned names with long lifetimes, purge for emergencies over Short lifetimes and frequent purges
Most updates cost nothing and hit ratios stay high. The cost is that the customer's build must produce hashed names.
Step 6 of 6: Say what breaks next
Questions an interviewer asks once the design is on the board. Answer each one out loud first.
- How do you serve private files, such as paid video?
Use signed URLs. The link carries an expiry and a signature, and the front end checks both before looking in the cache. The signature is left out of the cache key, so one cached copy serves all valid links. Short expiries limit sharing.
- How do you protect the origin from a customer's own cold start?
Collapse requests, send misses through the shield, and limit the number of connections and the request rate to each origin. Offer a way to pre-load files before a launch. If the origin returns errors, serve stale copies where the customer allows it.
- How would you handle a denial-of-service attack?
Cache hits cost very little, and the load is spread over 300 sites, so most floods are absorbed. Add per-client rate limits at the front end and drop bad traffic before it reaches the cache servers. Requests that miss and bypass the cache, such as random query strings, are the real danger, so cap the cache key's variety.
- How would you run HTTPS for thousands of customer domains?
Keep certificates in a distributed store and load them on demand, selecting by the name the browser sends at the start of the handshake. Renew them automatically, and cache the loaded certificate in memory. Private keys stay inside a protected service and are never copied to disk in the clear.