Design a feature flag service

The key choice is to move the rules to the readers. An app instance may check ten flags on every request, which at our assumed scale is 100 million checks a second, so a network call per check is out. Each app embeds an SDK that holds the whole ruleset, about 2 MB, in memory and evaluates locally in microseconds. Changes flow from an admin UI through a flag API into a store that keeps immutable, numbered versions and an audit log.

A publisher builds a snapshot for a CDN-backed cache and hands a small delta to a stream server, which pushes it to every SDK in about a second. Polling every 30 seconds is the safety net, and a version gap triggers a full fetch. If the service is down, the SDK keeps its last rules, so nothing fails. Percentage rollouts hash the flag key and the user key into one of 10,000 buckets, so a user stays in as the percentage grows.

A kill switch takes the same fast path with no batching. The trade-off is that rules live in every app, so SDKs must agree on the hash and validate every snapshot.

These numbers are assumptions for the exercise, not facts about a real system:

• 20,000 app instances run the SDK. At peak each handles 500 requests a second and checks about 10 flags per request.
• There are 2,000 flags in one environment, and compact rules average about 1 KB per flag.
• Teams make about 200 flag changes a day. Each touches one flag, so a change is a delta of about 1 KB.
• A local evaluation takes about 2 microseconds. A call to a remote service in the same region takes about 1 ms.

The arithmetic:

Flag checks per second

20,000 instances × 500 requests/s × 10 flags

100 million/s in total, far too many for a remote call each

Size of all rules

2,000 flags × 1 KB

≈ 2 MB, small enough to hold in every app's memory

Cost per request, local

10 flags × 2 µs

20 µs, about 0.02 ms

Cost per request, remote

10 flags × 1 ms, one call each

≈ 10 ms added to every request

One change, full ruleset pushed

20,000 instances × 2 MB

40 GB for a one-line change

One change, delta pushed

20,000 instances × 1 KB

20 MB, 2,000 times less

The core idea: ship the rules to the app, not the question to the service. The SDK keeps the whole ruleset in memory. Evaluation is a pure function. It takes the flag's rules and a context, such as the user ID and country, and returns a value. There is no network call, so the check cannot be slow and cannot fail because of the service.

Delivery has two paths. On start, the SDK downloads the latest snapshot from the cache, which is a static file that a CDN serves cheaply. After that it learns about changes in one of two ways:

• A stream. The SDK holds a long-lived connection, such as server-sent events. The server pushes a small delta when a flag changes. Changes arrive in about a second. The cost is 20,000 open connections, which at an assumed 10,000 per node is three nodes with one spare.
• Polling. Every 30 seconds the SDK asks the cache for the latest version, sending the version it holds. If nothing changed, the answer is a tiny 'not modified'. It is simple and works through any proxy, but a change can take up to 30 seconds. At 20,000 instances, that is about 670 requests a second.

Use both. The stream gives speed, and the poll is the safety net if a push is lost. Each delta carries a version number. If the SDK sees a gap, say it holds version 411 and receives 413, it fetches the full snapshot rather than guess.

When the service is down, the SDK keeps its last ruleset. It can also write the ruleset to local disk, so an instance that restarts during an outage still starts with real rules. If it has nothing at all, it returns the default that the caller passed in. After a long outage, reconnects should wait a random time, so 20,000 apps do not hit the stream server in the same second.

Browsers and phones are different. Shipping the full rules would leak targeting details, such as staff lists. So the server evaluates for that one user and sends back only the resulting flag values, which the client caches.

Do not pick randomly on each request. A user would see the new checkout on one page and the old one on the next. Instead, give every user a fixed position and compare it to the percentage.

The position is a bucket. Hash the flag key and the user key together, and take the result modulo 10,000. The same inputs always give the same bucket. A user is in the rollout when their bucket is below the percentage times 100. At 10% the cut-off is 1,000, and at 25% it is 2,500.

A worked example. User 42 hashes to bucket 1,834 for this flag. At 10% the cut-off is 1,000, so they are out. When the team raises the flag to 25%, the cut-off is 2,500 and they are in. Nobody who was in at 10% is removed, because the buckets below 1,000 are still below 2,500. Growing a rollout only adds people. Lowering it removes the people with the highest buckets first.

Details that matter:

• Include the flag key in the hash. If every flag hashed only the user, the same 10% of users would be first in line for every experiment.
• Use a stable key. A user ID works. A session ID or an IP address changes, so the person flips in and out. For signed-out visitors, store a random device ID in a cookie.
• Use the same hash in every SDK. A Java server and a JavaScript server must put user 42 in the same bucket. Pick one simple, non-cryptographic hash, write down the exact bytes that go into it, and publish test vectors that every SDK must pass.
• Allow a reseed. A salt field on the flag lets you re-draw the groups on purpose, for a new experiment.

Rules run in order: a user on the blocked list first, then specific users, then attribute rules such as country, then the percentage, then the default. For a flag with three variants, split the 10,000 buckets by weight, say 0 to 4,999, 5,000 to 7,999 and 8,000 to 9,999.

A kill switch is an ordinary flag that wraps risky code and defaults to the safe path. Its value is that turning it off is fast and needs no deploy. Speed is the requirement, so the path must have no slow step. Budget it: saving the change takes about 50 ms, the publisher about 100 ms, and the push to three stream nodes with about 6,700 connections each takes under a second. The SDK applies it in microseconds. That is a couple of seconds end to end, well inside the 5-second target.

Keep the kill-switch path free of batching. A delay that groups changes for a few seconds saves nothing here, so changes go out the moment they are saved. The polling fallback sets the worst case at 30 seconds if a push is lost.

Versions. Every save creates a new immutable version of the environment's ruleset, with a number that only goes up. The SDK remembers which version it holds. To roll back, republish the old contents as a new version, for example 415 with the contents of 412. Never reuse a number, or an SDK that holds 413 could not tell what changed.

Make rollouts observable. Each SDK reports the version it runs in a periodic heartbeat. The dashboard can then say '19,800 of 20,000 instances are on version 415' and name the ones that are not. Without it, 'it should have reached everyone' is a guess.

Audit. Write an append-only record for every change: who, when, which flag, the value before and after, and a reason or ticket. Never edit or delete these rows. Add permissions per environment, so more people can edit staging than production, and an optional approval step for production changes. Validate rules against a schema when they are saved. The SDK should also refuse a snapshot that fails to parse and keep the last good one.