Design a recommendation system

The design is a funnel with three stages, each slower per item than the one before and working on fewer items.

• Candidate generation. Several cheap sources each return their best items, and the results are merged. The main source is nearest neighbours over embeddings. A model learns a vector for each user and each item, so that items a user would like lie close to the user's vector. The index finds the closest 400 items by approximate search, in about 15 ms, giving up a few percent of recall for a large gain in speed. Other sources are items often viewed together, computed offline from watch histories into a table; trending items for the region; fresh items; and recent items from followed creators. Several sources mean that one weak spot does not empty the page.
• Ranking. A heavier model scores each of the roughly 1,000 candidates with many features: the user, the item, how they fit, the time of day and the device. It predicts several outcomes, such as the chance of a click and of a satisfied watch, and a weighted blend gives the score. 1,000 × 50 µs is 50 ms.
• Rules. Final code, not a model, picks the 20. It drops seen, blocked and unavailable items and caps items per creator. It also picks for variety. After choosing each item, it lowers the score of candidates that are very similar to those already chosen, so the page does not show ten near-copies.

Retrieval sets the ceiling. An item that retrieval does not return can never be shown, however good the ranker is. So measure the recall of retrieval: of the items a user later enjoyed, how many were among the candidates. Improve retrieval and ranking separately.

Budget. Features 10 ms, retrieval 40, ranking 60, rules 10, network 30 add up to 150 ms. That leaves 50 ms for slow requests. Give each stage a deadline. If the ranker is late, return candidates in retrieval order instead of an empty page.

• Online store. An in-memory key-value store with reads under 5 ms. It holds the user's embedding and recent views, and per-item counters such as views in the last hour.
• Offline store. A columnar history, partitioned by day, with every feature's value over time. Training reads from it.

Training-serving skew is the classic failure. If training computes views in the last hour with a SQL query, and serving reads a streaming counter, the two differ in subtle ways: time zones, duplicate events, late events. The model learns from one definition and is judged on another, and it does worse live than in testing. Two defences work together.

• One definition. Write each feature once, in code that both the batch job and the live service run.
• Log what was served. The API writes the exact feature values the ranker used for each impression to the event log. Training reads those logged values for the fast-moving features, and never recomputes them. What the model learns from is what it saw. The cost is the log, about 600 GB a day. For features that were not logged, join them as of the time of the impression, never as of the training run, or the model peeks at the future.

Where item features live. A page needs the features of 1,000 items. Read from a remote store, that is 4,630 requests × 1,000 = 4.6 million key reads a second at the peak. Instead, each ranker holds the item side in its own memory: 10 million items × 1 KB is 10 GB, refreshed from the feature store every minute. Only the user's features are fetched per request, in one read.

Freshness. Counters update within a minute through the stream. The user's recent views update in seconds. Embeddings are rebuilt nightly. Watch for drift: compare each feature's live distribution with its training distribution every day, and alert on a sudden rise in missing values.

A click is noisy, because a catchy thumbnail gets clicks that end in a quick exit. Treat a watch of more than half the item, or a purchase, as a stronger positive. Treat hide and not-interested as strong negatives. Predict several of these and blend them, so clickbait cannot win on clicks alone.

Position bias. Items in slot 1 get clicked more because of where they sit. If slot 1 is seen 90% of the time and slot 10 only 30%, a click on slot 10 is better evidence. Weight each training click by 1 ÷ the chance the slot was seen: 1.1 for slot 1, 3.3 for slot 10. Cap weights at 5 to keep training stable. Another way is to feed the position to the model as a feature during training and fix it to a neutral value when serving.

Selection bias and exploration. The model learns only from what we showed. An item never shown never gets data. So fill 5% of the slots, 1 in a page of 20, with exploration items, mostly new ones. With 5% of 2 billion impressions, that is 100 million a day, across 50,000 new items, which is 2,000 each. That is enough: at a true click rate of 5%, 2,000 impressions measure it to within about 1 point, since the standard error is √(0.05 × 0.95 ÷ 2,000) ≈ 0.5%.

Cold start. A new user has no history. Show popular items for their region and language, plus categories they chose at sign-up. After a few taps, average the embeddings of the items they touched into a rough vector. A new item has no history either, so give it an embedding from its title, tags and image, and rely on exploration.

Rollout. Retrain the model daily. Then pass four gates: offline metrics on held-out days, a shadow run that scores live traffic without showing it, a 1% A/B test for 2 days, and a ramp to 5%, 50% and 100%. Assign users to groups by hashing the user ID with the experiment name, so assignment is stable. Watch guardrails, such as watch time, hides, latency and errors, and roll back automatically on a drop. Stamp every impression with the model version, and keep a 1% holdback on an old model to measure long-run effects.