System design interview
Design a code deployment system
Immutable builds, staged waves, health gates, host agents and fast rollback.
Press Play to follow a request through the design, or step through it.
- Client and edge
- Service
- Data store
- Cache and queue
- A developer merges a commit. The build system builds it and runs the tests.
- It uploads the result under the hash of its bytes, for example 9f3c, signs the hash, and never overwrites a name.
- The store copies the build to every regional mirror, and each mirror checks the hash after the copy.
- The store confirms that all 5 regions hold the exact bytes. Until then, the build cannot be rolled out.
- The build system registers build 9f3c with the control plane, as a candidate with its test results.
- Developer
- Build system: build once
- Control plane: waves, gates
- Artifact store: by content hash
- Metrics: error rate, p99
- Deploy state: desired versions
- Regional mirror: copy per region
- Host agents: on every host
On this page
In 2 minutes
I would separate building, distributing and installing. A build is made once and stored as an immutable artifact named by the hash of its contents, signed, and copied to a mirror in every region before any rollout starts, so the exact bytes that passed testing are the bytes that ship. A control plane runs the rollout in waves: 10 canary hosts, 1%, 10%, one region, then the rest, which takes about 85 minutes for 10,000 hosts. After each wave it compares the new hosts with the old-version hosts over the same minutes, on error rate, latency and a business signal.
A clear failure halts the rollout and writes the old version back. The control plane only records a desired version per group. An agent on each host asks for it every 10 seconds, so 20,000 hosts cause 2,000 small requests a second. The agent downloads from the nearest mirror, verifies hash and signature, drains traffic, switches, checks health and reports.
At most 20% of a region switches at once, and the last two builds stay on disk, so rolling back a region takes about 200 seconds. If the control plane is down, hosts keep serving and reboot into the last desired version. The trade-off is a slow release and two versions running together, which means changes must be compatible both ways.
Step 1 of 6: Clarify the scope
Interviewer asks
Design a code deployment system. Engineers merge code, and the system builds it and ships it to thousands of servers in several regions. A bad release must be caught on a few servers, not all of them, and rolling back must take minutes.
Questions to ask first
Ask these before you draw anything. Under each one is why its answer changes the design.
- How many hosts, regions and services, and how many deploys a day?
The fleet size sets download traffic and how many hosts poll for changes. The deploy rate sets how much automation we need.
- What is deployed: containers, packages, or static files? How big is a build?
Artifact size drives replication cost and how long a host needs to install, which shapes the wave timing.
- Must a rollout stop automatically on a bad release, or may a person decide?
Automatic halting needs a trusted health signal and a rule for judging it. A human-only gate is simpler but slower.
- Can old and new versions run side by side during a rollout?
They will, for an hour or more, so versions must be compatible. If not, we need a different strategy for the change.
- What happens to servers if the deploy system itself is down?
The fleet must keep serving. The deploy system should only be needed when we want to change something.
Requirements
Functional
- Build a commit once, and store it as an immutable artifact.
- Roll a build out in waves, to groups of hosts in order, across regions.
- Compare each wave's health with the old version, and halt or roll back on a bad result.
- Install a build on each host safely: download, verify, drain, switch, check health.
- Show which version runs on which host, and who started each rollout.
Non-functional
- A bad release is caught on at most 1% of hosts before it spreads further.
- Rolling back a region takes under 5 minutes.
- The same bytes that passed testing are the bytes that reach production, verified by hash.
- Hosts keep serving traffic, and reboot into the right version, while the control plane is down.
- No more than 20% of a region's hosts are being switched at any moment.
Step 2 of 6: Size it
Moving the bytes is cheap, and the hard limit is risk. Waves cost about 85 minutes, and in return a bad build touches 10 hosts, then 100, before anyone else. Rollback is quick because the old build is already on the disk.
Assumed for the exercise, not facts about a real system:
- The largest service runs on 10,000 hosts in 5 regions, so 2,000 per region. A build is 400 MB. The company runs 20,000 hosts in all.
- Each region's artifact mirror can send about 3 GB/s. Agents ask for their desired version every 10 seconds.
- Waves: 10 canary hosts (15 minutes), 1% (10 minutes), 10% (10 minutes), one region (20 minutes), then the other four regions two at a time (15 minutes each).
- Switching a host takes 30 seconds to drain and restart, plus 10 seconds of health checks. The last two builds stay on its disk.
- Agent polling
- 20,000 hosts ÷ 10 s
- 2,000 requests/s for a tiny document, easy
- Copying one build to the mirrors
- 400 MB × 5 regions
- 2 GB, done before the rollout starts
- Hosts pulling a full rollout
- 10,000 × 400 MB = 4 TB; per region 2,000 × 400 MB
- 4 TB in all, 800 GB per region
- If a whole region pulled at once
- 800 GB ÷ 3 GB/s
- ≈ 270 s, about 4.5 minutes of full mirror load, which waves avoid
- Time to finish the rollout
- 15 + 10 + 10 + 20 + 15 + 15 minutes
- 85 minutes, plus the downloads
- Time to roll back one region
- 20% of 2,000 = 400 hosts per batch; 2,000 ÷ 400 = 5 batches × 40 s
- 200 s, about 3.3 minutes
Step 3 of 6: Sketch the path of one request
Step through the diagram at the top of the page to watch a request move through these parts.
A Developer merges a commit, and the Build system builds it once and puts the result in the Artifact store. The artifact is named by the hash of its contents, so it can never change. The store copies it to each Regional mirror before the rollout begins. The build system also registers the build with the Control plane.
The control plane runs the rollout. For each wave it writes a desired version for a group of hosts into the Deploy state store. It then waits, reads health from Metrics, and compares the new version with the old one running at the same time. If the wave is healthy, it moves on to the next. If not, it stops and writes the old version back.
On every host, an Agent asks the state store for its desired version every 10 seconds. If it differs from what runs, the agent downloads the build from the nearest mirror, checks its hash, drains traffic, switches, runs health checks, and reports the result to metrics. The control plane never has to reach into a host.
Other paths through the system
Roll out in waves
- A developer asks to deploy build 9f3c to production, with the standard plan of canary, 1%, 10%, one region, then all.
- The control plane writes version 9f3c as the desired version for the 10 canary hosts. Every other group keeps the old one.
- Within 10 seconds, a canary host's agent reads the new desired version.
- The agent downloads the build from its regional mirror, checks the hash and signature, and switches over after draining traffic.
- The agent reports that it runs 9f3c and is healthy, and the host's error rate and latency flow to metrics.
- After 15 minutes, the control plane compares the canary's numbers with the hosts still on the old version.
- The canary is as good as the baseline, so the control plane raises the desired version for the 1% group, and repeats for each wave.
A bad canary rolls back
- The canary hosts report a 4% error rate. The hosts on the old version report 0.3%.
- The control plane sees that the gate has failed on three checks in a row, and stops the rollout.
- It writes the previous build, 8a21, as the desired version for the canary group, and pages the owner.
- The agents read the new desired version at their next poll.
- Each agent looks for build 8a21 on its own disk first, and downloads it from the mirror only if it is missing. Most hosts switch back in about 40 seconds.
Step 4 of 6: Go deep on the hard part
Build once, promote the same bytes
Interviewer asks
How do you make sure that what was tested is what runs in production?
Build once. Build the commit a single time, and store the result as an immutable artifact named by the hash of its contents.
Read moreShow less: Build once, promote the same bytes
The name is the proof: if a file has hash 9f3c, it has the same bytes everywhere and always. The store never overwrites a name, and only the build system's identity may write to it.
Promote, never rebuild. Moving a build from staging to production is a change of a pointer in the control plane, not a new build. A rebuild produces different bytes, because of a changed dependency, a clock stamp or a toolchain update, so the tests would have judged something else. Environment settings stay out of the artifact. They are supplied at start-up, so one build can run in every environment.
Signing. The build system signs the hash, along with facts such as the commit and that the tests passed. Agents check the signature and the hash before switching. A mirror that was tampered with, or a download that was cut short, is then rejected instead of run.
Replication. The store copies each build to every regional mirror straight after it is built, long before any rollout, since 2 GB is cheap. It is marked deployable only when every mirror has confirmed the hash. Without that rule, a region could be told to install a build it does not yet have. A host first looks on its own disk for the hash, so a rollback to a build it ran an hour ago costs no download at all.
Cleanup. Keep the latest builds, and every build that any group still wants as its desired version. Garbage collection asks the control plane for the list, and never deletes a hash that is in use. Rollbacks then always have something to roll back to.
Waves, health gates and halting
Interviewer asks
How do you roll out so that a bad release hurts as few users as possible, and stops by itself?
Waves. Start with 10 canary hosts, then 1%, then 10%, then one region, then the rest.
Read moreShow less: Waves, health gates and halting
Each wave raises the number of users at risk by about ten times, only after the last wave proved healthy. The canary hosts should serve real traffic and sit in different zones, since a canary that gets no traffic proves nothing. Start in a quiet region, and never change all regions at the same hour.
The gate. For each wave the control plane waits until 95% of hosts report the new version as ready, then lets it bake. It then compares the new hosts with the old-version hosts over the same minutes: error rate, p99 latency, crashes and restarts, and one business signal such as successful checkouts. Comparing with the old hosts is better than a fixed threshold, because a traffic spike or a slow dependency hurts both groups and cancels out. Require a minimum number of requests, say 10,000, before judging. Fail the gate if the new hosts are clearly worse, for example by half a percentage point of errors or double the latency, on three checks in a row. The exact numbers are tuned per service.
Halting. A failed gate stops the rollout at once, writes the previous version back to the hosts that changed, and pages the owner. A rollout that merely stalls, say because only 80% of hosts became ready, pauses and asks a person. Automatic action is for clear failures, and a human decides the grey cases.
Locks and freezes. One rollout per service at a time, so two releases cannot overlap. Freeze windows block normal rollouts during big events, and an emergency fix can override one with an approval that is logged.
Compatibility. For about 85 minutes both versions serve traffic. So a change must be safe in both orders. For database changes, add the new column first, deploy code that uses it, and remove the old column later. Use feature flags so a deploy and a launch are separate decisions.
Installing on a host
Interviewer asks
How does a new build actually get onto thousands of hosts, and what if the control plane is down?
Desired state, pulled by agents. The control plane writes, for each group, the version it should run.
Read moreShow less: Installing on a host
Every host runs an agent that asks for it every 10 seconds, with a version tag so that a no-change answer is tiny. 20,000 hosts make 2,000 requests a second. If the desired version differs from the running one, the agent reconciles. It does not need to be told how, and a host that was off while a rollout passed simply catches up when it returns.
The install steps. Download from the nearest mirror, resuming if cut off. Verify the hash and signature. Unpack beside the running version. Drain: remove the host from the load balancer, stop new connections, let in-flight requests finish for up to 20 seconds, and ask long-lived connections to reconnect elsewhere. Start the new version, and require the health check to pass several times in 10 seconds. Add the host back. If the health check fails, switch back to the previous build on disk and report the failure with a reason.
Limits. At most 20% of a region's hosts may be switching at once, or capacity would drop. The agent takes one of a fixed number of leases from the state store before it drains, and returns it afterwards. Agents also wait a random few seconds before polling, so 2,000 hosts do not ask in the same instant.
Reporting. Each agent reports its host, the version it runs and its health. That gives the control plane the facts for gates and gives people a map of the fleet. A host that stays on the wrong version for too long raises a drift alert.
When the control plane is down. Nothing on the hosts depends on it. They keep serving the version they run, and a host that reboots starts the last desired version it saved on disk. Rollouts pause, which is safe. Run the control plane in three zones so this is rare.
Step 5 of 6: Name the trade-offs
How builds reach hosts
Agents pull the desired version over The control plane pushes to each host
Hosts that were down catch up on their own, and the control plane tracks no connections. The cost is up to 10 seconds of delay and a polling load of 2,000 requests a second.
Moving between environments
Promote the same artifact over Rebuild for each environment
What passed testing is exactly what ships, verified by hash. The cost is keeping settings outside the build and supplying them at start-up.
Judging a canary
Compare with old-version hosts at the same time over Check against fixed thresholds
Traffic spikes and slow dependencies affect both groups and cancel out. The cost is needing enough traffic on both sides, and a minimum sample before judging.
Rollout speed
Staged waves, about 85 minutes over Update every host at once
A bad build reaches 10 hosts, then 100, and is stopped. The cost is a slower release, and two versions running together for over an hour.
Step 6 of 6: Say what breaks next
Questions an interviewer asks once the design is on the board. Answer each one out loud first.
- How would you ship a hotfix faster?
Use the same path with shorter bake times and an approval, and keep the health gate on. The artifact, hash check and drain steps stay. Skipping the gate is how a hotfix becomes a second outage.
- How do you deploy a service that holds state, such as a database?
Go one host at a time, wait until replication has caught up before moving on, and update the leader last, after a controlled hand-over. Gates look at replication lag as well as errors. The 20% limit becomes one replica at a time.
- How do you handle a build that passes the canary but fails an hour later?
Keep watching after a rollout finishes, and keep the previous build on disk for a day. A later alert can trigger the same rollback path, which is only a change of desired version. Slow leaks, such as memory growth, are why the canary bake time is more than a few minutes.
- How do you know which version is running where?
Agents report the version and health of each host, and the control plane keeps the latest report for each. A dashboard groups hosts by version, and drift alerts fire when a host differs from its desired version for too long. Each rollout is recorded with who started it and which gates it passed.