Design a secrets manager

Each secret version gets its own random data key, and the value is encrypted with it using an authenticated cipher such as AES-GCM. The path and version are bound in as extra authenticated data, so a ciphertext cannot be copied into another record unnoticed. The data key is then wrapped, which means encrypted, by a master key held in the key service, ideally inside a hardware security module. We store the ciphertext, the wrapped key, and the id of the master key version that wrapped it.

Why a stolen database is useless. An attacker gets ciphertext and wrapped keys. Unwrapping needs the master key, and the master key never leaves the key service. The key service only does the operation for an authorised caller and logs it. So the attacker must break into the key service as well, which is a separate system with separate access.

Why not encrypt every secret directly with the master key. It would work, but envelope encryption wins in three ways. First, the master key touches only tiny data keys, so bulk data never passes through the module. Second, replacing the master key means re-wrapping 500,000 small keys, about 17 minutes at 1,000 operations a second, rate-limited so reads keep priority, and not re-encrypting every value. Third, a leaked data key exposes one version of one secret and nothing else.

Master key changes. Create a new master key version. New writes use it at once. The job re-wraps old records in the background. Old master versions stay enabled until the last record is re-wrapped, and are then disabled, not deleted, for a retention period so an old backup can still be read if needed. Deleting a master key on purpose makes every copy of data under it unreadable, which is a deliberate way to destroy data.

Starting the key service. It needs to reach its master key after a restart. With a hardware module, it authenticates to the module. Without one, the key is split into shares, and a quorum of operators, say 3 of 5, must provide them to unseal it. Either way, no single person or file holds the whole key.

So the workload holds no long-lived credential. The platform that runs it, such as the orchestrator or the cloud provider, vouches for it by signing a short-lived token that names the workload: its service account, environment and namespace. The token lives for about 10 minutes. The secrets API checks the signature with the platform's public keys, which it caches, and checks the expiry and that the token was meant for this service. A stolen token is useful for minutes, and it can also be tied to the workload's mutual TLS certificate.

Policies. Deny by default. A policy says that an identity pattern may do certain actions on a path prefix. For example, the payments service in production may read anything under the payments path, and nothing else. Actions are separate: read, write, list and rotate. An explicit deny beats any allow. The person who writes a secret should not be the identity that reads it. Evaluate on every read. Nodes may hold policies in memory, but they are updated by a push with a version number, and an old policy is never used for more than about a minute.

Humans. People get access through approval, with a short time limit, and the approval itself is logged. A break-glass path exists for emergencies. It works at once but pages the security team and is reviewed afterwards.

The audit log. Each read adds a record: time, caller identity, path, version, action, decision, request id and source address. Never the value. Write it before returning the secret, and if the write fails, deny the read, because a read nobody can see is the thing the log exists to prevent. Make it tamper-evident: each record includes a hash of the one before, and a signed checkpoint is copied every few minutes to a separate account that the secrets admins cannot write to. At 300 bytes a record, a year is about 3.1 TB, which cold storage handles easily. Alert on new readers of a sensitive path, and on a sudden jump in volume.

It creates version n+1 and moves the current pointer. The previous version stays readable for a while. A caller can ask for current or for a specific version.

The overlap. Clients cache a secret for about 5 minutes, and a rolling deploy can last longer. If the old password stopped working the moment the new one appeared, every instance with a cached copy would fail. So rotation is a sequence, and the target system accepts both credentials in the middle of it:

• Create first. The job creates the new credential in the target system. The old one remains valid.
• Test. It uses the new credential to log in. If this fails, stop. Nothing has changed.
• Publish. It stores the new value as the current version and keeps the old one as the previous version.
• Wait. Hold for a grace period longer than the client cache and a deploy, such as an hour.
• Revoke. It disables the old credential in the target and marks the old version retired.

The job records its progress in the store after each step, so a crash resumes at the right step and every step can be repeated safely. For databases, two users used in turns work well: rotate the idle one, switch the current pointer, and leave the other valid until the next cycle.

Clients. The client library caches the secret with a TTL plus random jitter, and refreshes before it expires. If the target rejects the credential, the library fetches again at once, one time. That repairs a missed rotation without a human.

Failing safe. If the secrets service cannot be reached, the client keeps using its cached value, even past its TTL, for up to an hour, and raises an alert. An instance starting with no cache and no service stops and retries, because starting with a blank password helps nobody. Run the API and the key service in three zones, so reads stay up when a zone is lost.