Design a link preview service

In 2 minutes

I would build the service around two ideas: safe fetching and one fetch per link. A client asks a preview API, which cleans the URL into a standard form and looks it up in a cache. At 20 million requests a day, about 80% hit, so we need only about 46 fetches a second, and about 460 at the peak. On a miss, only one worker fetches a given URL. Everyone else waits for that result, so a link shared by 100,000 people causes one fetch.

The worker never connects on its own. It goes through an egress proxy in an isolated network that resolves the name once, refuses private, loopback and link-local addresses, checks every redirect, and caps the size at 2 MB and the time at 5 seconds. The worker reads the head of the page and prefers Open Graph tags, then Twitter tags, then the title tag. It treats every value as plain text. Images are never hotlinked.

An image proxy fetches the original, caps its pixels, strips metadata, re-encodes it at up to 1,200 pixels wide and stores it for the CDN. Cards live 24 hours with conditional refresh, and failures are cached for a few minutes. The trade-off is a card that can be a day old, in return for safety and almost no load on other sites.

Step 1 of 6: Clarify the scope

Questions to ask first

Requirements

Functional

• Given a URL, return a title, a description, a site name and an image, or a plain link when nothing can be read.
• Read Open Graph and Twitter card tags first, then fall back to the page title and the first large image.
• Serve every preview image from our own domain, resized, and never from the page's host.
• Let a user or support agent refresh a card or remove it.

Non-functional

• A cached card: p99 under 100 ms. A new link: p95 under 3 seconds.
• Never connect to a private, loopback or link-local address, including through a redirect.
• A fetch reads at most 2 MB, follows at most 5 redirects and gives up after 5 seconds.
• A link shared by 100,000 people in one minute causes one fetch.
• 99.9% availability, and one slow site never slows other previews.

Step 2 of 6: Size it

Most requests are cache reads, and the fetch load is modest: a few hundred at the peak. The work is in making each fetch safe, and in never fetching the same link twice at once.

Requests per second

20M ÷ 86,400 s ≈ 231

≈ 230/s, 2,300/s at the peak

Fetches per second

20M × 20% = 4M a day; 4M ÷ 86,400 ≈ 46; × 10 for the peak

≈ 46/s, 460/s at the peak

Fetches in flight at the peak

460/s × 0.8 s (arrival rate × time each takes)

≈ 370 at once, so about 8 workers at 50 connections each

Download bandwidth

4M fetches × 100 KB = 400 GB a day; ÷ 86,400 s ≈ 4.6 MB/s

≈ 37 Mbit/s on average, small

Cards held for 24 hours

4M × 1 KB; × 7 if every card took the 7-day maximum

≈ 4 GB, at most ≈ 28 GB, which fits in a cache cluster

Resized images held for 7 days

4M × 60% = 2.4M a day × 80 KB ≈ 192 GB a day; × 7

≈ 1.3 TB in object storage

Step 3 of 6: Sketch the path of one request

Step through the diagram at the top of the page to watch a request move through these parts.

A Client asks the Preview API for a card. The API cleans the URL into one standard form and looks it up in the Preview cache. A hit returns at once, and for most requests that is the whole story.

On a miss, the API hands the URL to a Fetch worker. Only one worker fetches a given URL at a time, and everyone else waits for that result. The worker never opens the connection itself. It sends the request through the Egress proxy, which sits in a network with no route to our internal systems and refuses private addresses. The proxy reaches the Remote sites. The worker reads the head of the page, pulls out the tags, writes the card to the cache and answers the API.

The card holds an image address on our own domain. When the browser asks for it, the Image proxy looks for a resized copy in the Image store. On the first request there is none, so it fetches the original through the same egress proxy, resizes and re-encodes it, and stores it. Later requests are served from the store with long cache headers.

Other paths through the system

Fetch a new page

1. On a miss, the API hands the URL to a fetch worker. If one is already fetching this URL, the API waits for that one.
2. The worker sends the request through the egress proxy, which resolves the name once and checks the address.
3. The proxy connects only to a public address. It reads at most 2 MB within 5 seconds, and checks every redirect the same way.
4. The head of the page comes back. The worker stops reading there and pulls out the title, description and image tags.
5. It writes the card to the cache under the standard URL, with a lifetime. A failure is written too, for a few minutes.
6. The worker returns the card to the API, which also answers everyone who was waiting.
7. The client shows the card, with an image address on our own domain.

Load the card image

1. The browser loads the card's image from our domain, using a hash of the original image address.
2. The request is routed to the image proxy.
3. The proxy looks for a resized copy in the image store. This is the first request, so there is none.
4. It asks the egress proxy for the original image, which applies the same address checks as for pages.
5. The original is downloaded, up to a size cap and a pixel cap.
6. The bytes arrive at the image proxy, which decodes them, strips metadata and shrinks them to at most 1,200 px wide.
7. It stores the re-encoded copy, so every later request is a plain read.
8. The image goes back through the API to the browser with a long cache lifetime.

Step 4 of 6: Go deep on the hard part

Fetching hostile URLs safely

A preview service makes web requests to addresses that strangers choose.

Parsing tags and proxying the image

Order of trust. Parse only the head section, with a forgiving HTML parser, because real pages are messy.

Caching, coalescing and failures

The key. Cache by the normalized URL: lowercase the scheme and host, drop the default port and the fragment, remove tracking parameters such as utm ones, and sort what is left of the query.

Step 5 of 6: Name the trade-offs

Step 6 of 6: Say what breaks next