Design file storage and sync

In 2 minutes

I would separate the bytes from the metadata. The client splits files into chunks of about 4 MB, hashes each with SHA-256, and uploads only chunks the server lacks. Chunks live in object storage under their hash, so uploads are resumable and idempotent, small edits cost one chunk, and identical data is stored once. A file is a manifest, an ordered list of hashes, held in a metadata database sharded by namespace. A commit sends the base version it started from. The server checks it is still current, writes the new version and appends to a per-namespace journal with an increasing cursor, all in one transaction. Devices remember their last cursor and ask for changes after it, so offline catch-up is the same call as normal sync. To get changes in seconds, devices hold a long-poll request on a notify service, and a commit wakes them. The signal carries no data, so losing it is harmless. If two devices edit one file, the second commit is refused because its base is stale, and the client saves its edit as a conflicted copy, so no work is lost. Old chunks are removed by a slow, delayed sweep.

The question

Clarifying questions

• How large can a single file be, and how many files does a typical user have? Large files need chunking and resumable uploads, and the file count decides how big the metadata store becomes.
• Can folders be shared between people? Sharing means a folder can belong to several users, so changes must be tracked per shared folder and not only per account.
• How quickly must a change on one device show up on another? Seconds rather than minutes means the server must push a signal, not wait for the devices to poll.
• Do we keep old versions, and for how long? Version history means chunks are shared between versions, so deleting them needs care.
• Is end-to-end encryption required? If the server cannot read file contents, it cannot deduplicate chunks across users.

Requirements

Functional

• Upload, download, rename, move and delete files and folders.
• Sync a folder across a user's devices, including edits made offline.
• Resume an interrupted upload or download without starting over.
• Keep recent versions of each file so a change can be undone.

Non-functional

• A change appears on other online devices within 5 seconds at p99.
• Durability: no stored file is ever lost.
• A small edit to a large file transfers only the changed part.
• Resumable: an interrupted upload restarts from the last 4 MB chunk, never from zero.
• Syncing never silently overwrites another device's changes.

Back-of-the-envelope estimates

Assumptions for this exercise:

• These numbers are assumptions for the exercise, not facts about any real service.
• 10 million users store 2 GB each on average. Deduplication saves about 25%.
• Files are split into chunks of 4 MB on average. An index entry takes about 40 bytes: a 32-byte hash and a pointer.
• 2 million users are active each day with 20 file changes each. Peak traffic is 10 times the average. 3 million devices keep a connection open.

The arithmetic:

• Stored data after deduplication: 10M × 2 GB = 20 PB; × 0.75 → 15 PB
• Number of chunks: 15 PB ÷ 4 MB → 3.75 billion chunks
• Chunk index size: 3.75B × 40 bytes → 150 GB, so the index is sharded
• Commits per second: 2M × 20 = 40M a day ÷ 86,400 s ≈ 463 → ≈ 460/s on average, about 4,600/s at a 10x peak
• Saving from chunking: Editing one 4 MB chunk of a 100 MB file: 100 ÷ 4 → 25 times less data uploaded
• Polling load avoided: 3M devices asking every 10 seconds → 300,000 requests a second saved by long polling

High-level design

A Desktop client watches a folder. When a file changes, it splits the file into chunks and hashes each one. Everything goes through the API gateway. The Block service accepts chunks and writes them to Object storage, where each chunk is stored under its own hash. A chunk already stored is never stored again.

The Metadata service records the files. A file is just a name, a version, and an ordered list of chunk hashes. Committing a new version writes it to the Metadata DB and appends an entry to a journal of changes with an increasing cursor number.

The Notify service tells other devices. Each device holds a long-poll request open. When the journal moves, the Notify service wakes it, the device asks for changes after its cursor, and then it downloads only the chunks it lacks.

How a request flows

Upload a changed file

1. The client splits the changed file into 4 MB chunks, hashes each, and sends only the chunks the server does not have.
2. The gateway passes the chunks to the block service.
3. The service checks each hash, then stores each new chunk under its hash. Chunks already stored are skipped.
4. Object storage confirms the chunks are durable.
5. The block service tells the gateway that all chunks are stored.
6. The client now knows every chunk of the new version is safe on the server.

Commit the new version

1. The client commits: the file path, the list of chunk hashes, and the version it started from.
2. The gateway routes the commit to the metadata service.
3. The service checks with the block service that every listed chunk exists.
4. All chunks are present.
5. In one transaction it checks the base version is still current, writes the new version and appends a journal entry.
6. It publishes a short event, 'this folder changed', to the notify service.
7. The service answers the gateway with the new version number.
8. The client marks the file as synced.

Another device syncs

1. A commit has just published a folder-changed event to the notify service.
2. The notify service answers the long-poll request that device B had been holding open.
3. Device B learns that something changed after its last cursor.
4. It asks for the list of changes after its cursor, say 9,041.
5. The gateway routes the request to the metadata service.
6. The service reads the journal entries after that cursor.
7. The database returns the changed files and their chunk lists.
8. The device receives the list, then downloads only the chunks it does not already hold.

Deep dive: chunking and content hashes

Split every file into chunks, hash each chunk, and store the chunk under its hash. This is content addressing: the name of the data is a fingerprint of the data. A file is then just an ordered list of hashes, called a manifest.

This one idea gives several things:

• Resumable upload. Chunks are uploaded independently, and storing a chunk is idempotent, because the same hash always means the same bytes. After a dropped connection, the client asks which hashes are still missing and sends only those. The server re-checks each hash on arrival.
• Delta sync. If one chunk of a 100 MB file changes, only that chunk, 4 MB, is sent. That is 25 times less.
• Deduplication. A chunk that already exists is never stored twice, whether it came from another file or another version.

Fixed or content-defined chunks? With fixed 4 MB boundaries, inserting a single byte at the start shifts every later boundary, so every chunk changes and the whole file is re-uploaded. Content-defined chunking places boundaries where a rolling hash of the nearby bytes meets a rule, so boundaries move with the content. An insertion changes one or two chunks and the rest keep their hashes. It costs more CPU and code, and is worth it for files that are often edited in the middle.

A privacy warning: if the server tells a client 'I already have this chunk', an attacker can learn that someone stored a particular file. Limit deduplication to one account or organisation, or ask the client to prove it holds the chunk, for example by hashing parts chosen by the server. If you offer end-to-end encryption, cross-user deduplication is not possible.

Each chunk is compressed and encrypted at rest. Object storage keeps its own redundancy, so durability does not depend on the application.

Deep dive: the metadata service and the change journal

The metadata database holds the namespace: a row per file or folder with an ID, a parent ID, a name, a version, the chunk list and a deleted flag. Use IDs, not paths, for identity, so a rename or move is a small update and not a rewrite of every child. Shard by namespace, where a shared folder is its own namespace, so one transaction stays on one shard.

Every change also appends an entry to a journal for that namespace, with a cursor that always grows: 9,041, 9,042 and so on. A commit does two things in one transaction: check that the base version the client started from is still the current one, then write the new version and the journal entry. This is optimistic concurrency, and it is what stops silent overwrites.

A device remembers the last cursor it processed. To catch up, it asks for 'changes after cursor N'. This is the same idea as sequence numbers in a chat app, and it makes offline sync simple: a device that was off for a week asks the same question.

Telling devices. Polling every 10 seconds from 3 million devices is 300,000 requests a second, nearly all of which find nothing. Use long polling instead. A device sends a request 'tell me when the cursor moves past N' and the notify service holds it open for up to a minute. When a commit happens it answers the waiting requests, and the device then fetches the changes. Each notify machine can hold about 100,000 connections, so 3 million need around 30.

The signal carries no data, only 'something changed'. If one is lost, nothing breaks, because devices also check on a slow timer and on start-up. The journal, not the signal, is the source of truth.

Deep dive: conflicts, versions and cleanup

Both devices start from version 7 and both commit. The first commit succeeds and creates version 8. The second carries a base version of 7, which is no longer current, so the server refuses it.

The losing client does not discard its work. It downloads the winning version 8 as the file, and saves its own changes beside it as a new file, for example 'report (conflicted copy from Laptop, 2026-10-15).docx'. It uploads that as a normal new file. Nothing is lost, and the user sees both and decides.

Why not merge automatically? For arbitrary files, such as images, spreadsheets or compressed archives, there is no safe way to combine two edits. A wrong merge is worse than a visible conflict. Text-heavy documents are better served by real-time collaboration, which is a different system.

Other conflicts follow the same principle: never destroy data. If one device deletes a file while another edits it, keep the edit and restore the file. Renames and moves act on file IDs, so they rarely clash. Two devices creating the same name in the same folder get a numbered suffix on one of them.

Versions and cleanup. Keep older versions for a set period, such as 30 days, so people can undo a change. Different versions share most of their chunks, so deleting a version cannot simply delete its chunks. Track how many manifests reference each chunk, or run a periodic mark-and-sweep that finds chunks no live version refers to. Delete a chunk only after a delay, as a safety net against a race with a new upload. Run the sweep slowly, because a bug here destroys user data.

Trade-offs

• Chunk boundaries: Content-defined chunks over Fixed 4 MB chunks. An edit near the start of a file changes only a couple of chunks. The cost is more CPU on the client and more complex code.
• Telling devices about changes: Long polling on a journal cursor over Devices poll on a timer. Changes arrive in seconds and idle devices cost almost nothing. The cost is a fleet of servers that hold open connections.
• Conflicts: Keep both, as a conflicted copy over Last write wins. No edit is ever lost. The cost is that the user must tidy up the extra file.
• Deduplication: Within one account or team over Across all users. It avoids the leak that tells someone a file exists. The cost is some extra storage.

Common follow-up questions

How do you handle a 50 GB file?
At 4 MB, that is 12,500 chunks, so upload them in parallel and resume per chunk. The manifest is long, so store it in pages or as a small tree of hashes. Clients can then start downloading and verifying chunks before the whole manifest has arrived.

How does a shared folder work?
Make the shared folder its own namespace with its own journal and a list of members and permissions. Each member's device follows that journal like any other. A commit there notifies every member's devices.

How do you really delete data?
A delete first marks the file deleted and starts a retention period. After it, the version stops referencing its chunks, and the sweep removes chunks nobody uses. For legal deletion requests, use a shorter path, but keep the same safety checks.

How would you scale the metadata database?
Shard by namespace, because commits and journals never cross namespaces. Add read replicas for listings. A very busy shared namespace is the hard case, and you handle it with more headroom on its shard and by batching journal reads.