Design an API gateway

Keep the whole table in each node's memory, compiled into a tree keyed by path segments, so a lookup costs a few comparisons. The longest matching prefix wins, and an exact match beats a prefix. At 2,000 routes the table is about 1 MB.

The control plane is the source of truth. An operator or a pipeline submits a change. Before saving, the control plane checks that it parses, that no two routes are equally specific for the same request, that each target service exists, and that every pattern is bounded, so a pathological pattern cannot burn CPU. It then replays a set of saved sample requests against the compiled table and compares the targets with what was expected. Only a table that passes becomes a new numbered version in the config store. Versions are immutable.

Delivery. The control plane pushes the version to each node over a long-lived stream, and nodes also poll every 30 seconds in case a push was lost. A node builds the new table next to the live one, validates it, and then swaps one pointer. Requests already running finish on the old table, and the next request sees the new one, so there are no locks and no half-applied state. Each node reports the version it runs, which lets the control plane say that 15 of 16 nodes are on version 42 and name the one that is not.

Safe rollout. Push to one canary node first, and compare its error rate and latency with the others for a couple of minutes. If it is worse, roll back by publishing the old contents as a new, higher version. Never reuse a version number, since a node could not tell which table is newer. A node that receives a table that fails its own checks keeps the old one and reports the problem.

When the control plane is down. The data path does not depend on it. Each node keeps the last good table in memory and on disk, so even a restarted node can start without it. The only cost is that no changes can be made until it returns.

It carries the user, the scopes, an expiry and the key id of the key that signed it. The gateway holds the matching public keys and checks the signature, the expiry (allowing about 30 seconds of clock difference), the issuer and the audience. That is about 50 microseconds, so 40,000 checks a second use about 2 cores. A call to an auth service for each request would add 2 ms and put 40,000 calls a second on one system.

Rotation. Keys are replaced on a schedule. The key service publishes the new public key before the identity service starts signing with it, and keeps the old public key until every token it signed has expired. A node caches keys by id for about 10 minutes. On an unknown key id it refetches, but at most once a minute, and it caches the failure. Without that limit, an attacker could send tokens with random key ids and turn every one into a fetch.

Revocation. Make tokens short-lived, around 10 minutes, so most revocation is just expiry. For an urgent block, such as a stolen account, push a small deny list of user ids and token ids to every node through the same channel as routes. It stays small because entries are dropped once the token would have expired anyway.

API keys for partners. A key is a random secret, shown to the partner once. We store only a hash of it. The gateway hashes the key it receives, and looks the hash up in a local cache with a 60 second lifetime, backed by the key service. Unknown keys are cached for 10 seconds as well, so a flood of guesses does not become a flood of lookups. Compare in constant time, and have the lookup return the partner's plan and limits.

Identity for backends. First, delete any identity header that arrived from the client. This is the classic mistake. Then set headers with the verified user, scopes, client id and a request id, and send them only over a mutually authenticated connection, so a backend can trust that they came from a gateway. Quotas use the plan from the token or key, and count per client in memory, synced between nodes about once a second.

Without limits, a slow backend holds connections until the gateway has none left for anything else. Give each backend its own connection pool, a bulkhead, so one cannot starve another.

Retries, carefully. Retry only when it is safe: GET and HEAD, other methods that carry an idempotency key, and failures where the request never reached the service, such as a refused connection. Never retry a POST that may have run. Retry on a different instance, after a short random wait. And set a retry budget: retries may not exceed 10% of the recent requests to that backend. Without a budget, three retries turn the 4,000 requests a second that one backend takes into 16,000 aimed at a service that is already struggling, which finishes it off. With a budget the worst case is 4,400.

Circuit breaker. Track the outcome of calls to each backend. When more than half of the last 10 seconds of calls fail, with at least 20 calls to judge from, the breaker opens and the gateway fails fast with a 503 and a retry time. After 30 seconds it lets a few probe requests through. If they succeed it closes, and if not it stays open. Failing fast gives the backend room to recover and gives the caller an answer in milliseconds, not seconds.

Load shedding by priority. Give each route a class, such as critical, normal and background. Each backend has a limit on requests in flight, found by measurement. When it is reached, reject the lowest class first. Checkout survives while an export job is turned away.

Hedging, optionally. For idempotent reads with a tight p99, send a second copy to another instance if the first has not answered by the p95 time, and cancel the slower one. It costs about 5% extra load and trims the slow tail. Use it only when the retry budget allows, and never for writes.