Design an LLM chat service

It passes through proxies and load balancers, and a browser reconnects by itself, sending the last event ID it saw. The user's message is a separate request, and so is a stop. Use a WebSocket only if the app needs two-way traffic during a reply, such as voice, since it needs sticky connections and more machinery.

Stateless gateways. A gateway keeps nothing that another one could not rebuild. While it streams, it appends each batch of tokens to the token buffer, keyed by conversation and reply, with an increasing sequence number and a TTL of 10 minutes. The event ID sent to the app is that number.

• Client drops. The client reconnects, to any gateway, with its last event ID. The gateway reads everything after it from the buffer and then follows live. The generation never paused, so nothing is lost.
• Gateway dies. The stream to the replica breaks. The next gateway takes the tokens saved in the buffer, and starts a new generation whose prompt is the original prompt plus those tokens. The replica often still has the original prompt in its cache, so the restart is cheap, and the saved tokens stay as they were.

Stop. A stop request reaches the gateway, which sends a cancel to the scheduler and on to the replica. The replica drops the sequence from its batch before the next step and frees its cache blocks, within one step of about 36 ms. The tokens already made are saved as the reply, marked stopped, and the quota is settled for what was produced. If a client has been gone for 30 seconds with no reconnect, cancel in the same way. Otherwise an abandoned reply holds GPU memory for up to a minute.

Saving turns. Save the user message before generation begins, so it is never lost. Save the assistant message when it completes, is stopped or fails, with a status. The key is conversation and turn number, so a retry overwrites the same row. When building the prompt, count tokens with the model's own tokenizer and keep as many recent turns as fit a budget, such as 6,000 tokens. Summarise older turns, or drop them.

Each step reads all the weights, 140 GB, from memory. That read, not arithmetic, takes the time. One request alone gets a token every 18 ms, about 57 a second, and the GPUs sit nearly idle. The fix is to serve many requests per step.

• Continuous batching. Static batching waits until every request in the batch finishes, so short replies wait for long ones. Instead, decide at every step: finished requests leave, and waiting ones join. The weights are read once for the whole batch. At 183 requests in the batch, a step takes about 36 ms, and every request gets 28 tokens a second. The replica makes about 5,000 tokens a second, 88 times as many as one at a time.
• KV cache. To avoid recomputing the past, the model keeps the keys and values of every earlier token. That is 320 KB a token in our assumptions. With 150 GB free after the weights and working memory, the cache holds about 458,000 tokens. That, not compute, is the real limit on concurrency.
• Paged memory. Reserving 4,000 tokens for every request fits 114 requests. Allocate the cache in small blocks as a request grows, and 458,000 ÷ 2,500 fits 183, which is 60% more.
• Prefix caching. Every request starts with the same 1,000-token system prompt. Compute its cache once, and let all requests share those blocks. The shared copy is about 330 MB. A private copy for each of 183 requests would be 60 GB, 40% of the cache. The estimate leaves this extra room unused.

Routing. A prefix cache helps only if the request lands on a replica that holds it. Send the follow-up turns of a conversation to the replica that served the last one, by consistent hashing on the conversation ID. Its earlier tokens are likely still cached. If that replica's queue is much longer than the others, send the request elsewhere. A hot replica is worse than a cold cache.

Prefill and decode. Processing the prompt, called prefill, is compute-bound and takes 0.14 s for 1,000 tokens. Run it in the same step as other requests' decode, and all of them stall for 140 ms and the stream stutters. Split prefill into chunks of 256 tokens and mix them in with decode steps. Or run prefill on separate replicas and move the cache over the network: 1,000 tokens is about 330 MB, 7 ms at 50 GB/s. Prefill is also why the replica count includes 4.9 replicas of compute besides decode.

So quotas count tokens, input and output, per user. Give each plan a daily budget and a per-minute limit, for example 30,000 tokens a day on a free plan and 500,000 on a paid one. Use a token bucket for the per-minute limit, which allows a burst. Also cap each user at 2 replies running at once.

Reserve, then settle. The reply's length is unknown at the start. Before generating, reserve the prompt size plus a cap on the reply, say 1,800 + 2,000 = 3,800 tokens, with one atomic update in a fast store. If the budget is too low, refuse at once with a clear message. After the reply, settle: charge the real use, 1,800 + 640 = 2,440, and give back the 1,360 unused. Give a reservation a 5-minute expiry. If a gateway dies, the budget is not locked for good, and a reaper settles from the replica's usage log.

Queue and priority. At full load the 12 replicas serve about 50 requests a second, against the 35 peak. When a spike passes that, requests wait. Keep a queue per plan, serve paid first, and within a lane go first in, first out. Estimate each request's wait as queue length divided by the service rate, and show it. A spike of 70 requests a second for one minute adds (70 − 50) × 60 = 1,200 waiting requests, and the last one waits about 24 seconds. Set a limit per plan, say 10 seconds for free, and past it choose: send to the small model, or refuse with a retry-after time. A refusal is better than a request that waits 3 minutes and times out after using the GPU.

Graceful degradation. A smaller model on its own pool answers about four times faster and costs a fraction as much. Use it for overflow, and for short or simple requests chosen by a cheap classifier. Tell the user which model answered. Add a lower tier of last resort: a cached or canned reply for the most common prompts.

Cost. At an assumed 2.50 dollars a GPU-hour, 48 GPUs cost 120 dollars an hour, or 2,880 a day. For 1 million messages, that is about 0.29 cents a message. The load is far lighter at night, so scale replicas with traffic, keeping warm capacity because loading 140 GB of weights into a new replica takes minutes.