Root in the vault, intermediate on the wire: building a two-tier MongoDB PKI with Azure Key Vault

Three Ubuntu VMs, one in UAE North, one in South Africa North, one in West Europe. Private DNS resolves each by hostname across a VNet mesh. NSGs deny everything except SCRAM traffic between the three subnets and SSH from a single operator network. No public surface for the database, anywhere.

That's where the cluster lives. What it doesn't have yet is identity. Nothing on those VMs can prove to anything else which node it is or who is asking. Until that's fixed, the cluster cannot meaningfully be called secure.

MongoDB offers two ways to authenticate replica set members and clients: a shared keyfile, or x.509 certificates. The keyfile is the path most tutorials take because it fits in one paragraph: generate random bytes, copy the file to every node, set permissions, done. It works.

It is also the wrong answer for any deployment that has to defend itself to an auditor or a recruiter. A keyfile is a single shared secret. Every node has the same one. Every member is interchangeable from the cluster's perspective. There is no per-node identity, no per-client identity, no audit trail that says "node A connected" rather than "someone holding the keyfile connected." If the keyfile leaks from any one node, the entire cluster's authentication is compromised and the only remediation is regenerating and redistributing the keyfile to every node simultaneously.

x.509 is the alternative. Every node gets its own certificate, signed by a certificate authority you control. Every client — humans, applications, monitoring agents — gets a separate certificate with a different cert class. Every TLS handshake records the full Distinguished Name of the connecting party in the audit log. Compromise of a single cert has a bounded blast radius: revoke that cert, issue a new one, the rest of the system is unaffected.

The cost of x.509 is that you now have to run a certificate authority. That's what this piece is about.

The simplest CA design is one tier: a single self-signed root that signs every leaf cert directly. It works at any scale, and you can build one in twenty minutes with openssl req -x509. The problem is that the root's private key has to be online — readable by something — every time it signs a new leaf. For a 4-cert deployment that rotates once a year, that's five signing events in the root's lifetime. For anything larger, it's hundreds.

A root that's exposed to that many signing operations is a root whose blast radius equals the entire trust chain. If the key is compromised, every certificate it ever signed is worthless. You rebuild from zero.

Two-tier separates the problem. A root CA signs one intermediate CA, then retires to cold storage. The intermediate handles every day-to-day signing operation. If the intermediate is ever compromised, you revoke it, the offline root signs a new one, you re-issue the leaves, and the root of trust survives the incident. The root might be touched three times in its entire 20-year life: at creation, at the first intermediate signing, and at one or two replacement intermediate signings.

This is how every serious PKI is built — web PKI, government PKI, financial PKI. It's the only design that makes the audit story "we have a recoverable trust hierarchy" rather than "we have a single point of catastrophic failure."

For three MongoDB nodes and one operator, it's mild overkill. For an article series framed as financial-grade architectural thinking, it's the only honest answer.

The two-tier design has one load-bearing operational property: the root key must be hard to touch. The whole point of putting the intermediate in front of it is to reduce how often the root is exposed. If the root key sits on a laptop in plaintext, the entire design is theater.

The textbook answer is a Hardware Security Module. A dedicated network-attached HSM (Thales, Entrust) is tens of thousands of dollars in capex. Azure offers Managed HSM at roughly three to five dollars per hour, which works out to $2,000–$3,500 a month. Both are out of scope for anything that isn't already a regulated production workload.

The home-lab textbook answer is an encrypted USB stick with the root key on it, stored offline, used during an air-gapped ceremony when the intermediate needs signing. That's a real and defensible pattern. It's also a pattern that requires physical materials, a clean machine to boot from, and the operational discipline to run a ceremony once every few years without forgetting how. For a series being written and published in real time, it adds friction without obviously improving the security story for the audience.

The pragmatic answer for an Azure-anchored deployment is Azure Key Vault standard tier with the root key generated inside the vault, marked non-exportable. The key is software-protected (FIPS 140-2 Level 1), not hardware-protected, but it has a property that's operationally more important: it cannot be extracted, ever. Not by you, not by an admin, not by Microsoft, not by any API. The vault exposes one operation against the root key: sign this digest with it. The key bytes themselves never leave the vault's process boundary.

This is the right answer for this deployment. It's also worth being honest about what it's not.

A regulated financial deployment running real money would use Azure Key Vault Managed HSM or a dedicated on-prem HSM. The cryptographic difference (Level 1 software vs Level 3 hardware) matters when the threat model includes adversaries with sufficient resources to compromise the cloud control plane or exfiltrate key material through software-level attacks. The article you're reading is below that bar, intentionally. That gap is named explicitly later in this piece, alongside every other gap.

The vault itself is one CLI command. The flags matter:

Three of those flags are load-bearing. --enable-rbac-authorization true opts into the modern role-based authorization model rather than the legacy per-vault access-policy ACL — this is the audit-friendly model. --enable-purge-protection true makes the vault irrecoverably resistant to a "delete then recreate empty" attack, at the cost of being irrevocable itself: once on, the vault cannot be force-deleted within its retention window even by an Owner-role identity. --default-action Deny plus the explicit AzureServices bypass means the vault firewall is up by default and only Microsoft platform services can reach it. The operator's IP gets added explicitly as a /32 in a separate call.

That last point is worth a paragraph. The SSH NSG rule on the VMs uses a /24 to accommodate a dynamic public IP. The Key Vault firewall, for the same operator, uses /32. The asymmetry is deliberate. SSH is primarily defended by key-based authentication; the source IP range is a secondary control, and a /24 is acceptable because the primary control is strong. The Key Vault data plane is primarily defended by the source IP range plus Azure AD identity; the cost of false positives (your IP rotated outside the allowlist) is low, and the cost of a wider range is real. The aperture matches what each control is actually protecting, not cross-resource symmetry.

The RBAC role assignment is where most tutorials get sloppy. The instinct is to grant the operator account Key Vault Administrator — the all-access role. That works. It is also wildly broader than necessary. The operator needs to create and read certificate objects, and later to invoke the sign operation on the underlying key. Those are two narrower roles:

• Key Vault Certificates Officer — create, manage, read certificate objects

• Key Vault Crypto User — sign and verify with keys, no management

Both scoped to the single vault, not the resource group, not the subscription. Anything broader violates least privilege without buying anything operationally.

So far, none of this is news. What is news — and this is the part most cloud-PKI tutorials skip past with a hand-wave — is what happens when you try to create a CA certificate using the Azure CLI.

A normal Key Vault certificate policy looks structurally fine for a CA cert. You declare a self-signed issuer, set the right key usage flags (keyCertSign, cRLSign), declare the cert as a CA in basicConstraints, set the validity. You write the policy to a JSON file, run az keyvault certificate create --policy @policy.json, and the certificate gets generated successfully inside the vault.

It is not a CA certificate.

Inspect the cert with openssl x509 -text and you'll find Basic Constraints: CA:FALSE. You'll also find an Extended Key Usage extension stamped on for you — TLS Web Server Authentication, TLS Web Client Authentication — that you did not ask for and that no root CA should ever carry. The Azure CLI silently strips basicConstraints.ca: true from the policy on its way to the API, and Key Vault's default template stamps TLS-leaf EKUs on certificates that don't override them.

This is a documented gap in the Azure CLI's certificate API surface. It has been an open issue since 2021. The fix is to bypass the CLI entirely and call the Key Vault REST API directly, where the basic_constraints field is honored:

The REST payload uses snake_case field names (key_props, validity_months, basic_constraints) instead of the CLI's camelCase, and explicitly sets "ekus": [] to suppress the default TLS-leaf usages. That second field is non-obvious — Key Vault will populate EKUs for you if you don't override them with an empty array.

The two soft-deleted cert names in the vault are the receipt for the attempts that taught me this. They'll sit there for ninety days under purge protection before they can be re-used. That's the cost of learning the gap; it's also worth seeing in the audit trail.

Once the REST API call lands correctly, the cert that comes back is a real root CA: Basic Constraints: critical, CA:TRUE, Key Usage: critical, Certificate Sign, CRL Sign, no Extended Key Usage, 4096-bit RSA, twenty-year validity, signed by itself. The private key is inside the vault, marked non-exportable, and will never leave.

That's the root.

This is the architectural insight that the rest of the build hangs on, and it's the part most cloud-PKI tutorials get wrong.

Cloud Key Vaults — Azure's, AWS KMS, Google Cloud KMS, Managed HSMs of any flavour — are not certificate authorities in the sense that they issue and assemble X.509 certificates for arbitrary CSRs. They are signing oracles. The primitive they expose is: here is a key; given a digest, return a signature over that digest with this key. That is the whole API.

A certificate authority on top of a signing oracle is something you build. You construct the X.509 to-be-signed structure locally — issuer DN, subject DN, public key from the CSR, serial number, validity, all extensions including the right basicConstraints for whatever class of cert you're issuing. You hash it. You send the hash to the vault. The vault signs it. You receive the signature back. You assemble the final cert by concatenating the TBS structure with the signature in the appropriate ASN.1 envelope.

This pattern is universal across HSM-backed PKIs. AWS CloudHSM works this way. Google Cloud KMS works this way. Azure Managed HSM works this way. Every serious enterprise CA that uses an HSM at all uses this pattern. The reason it doesn't show up in most cloud-Key-Vault tutorials is that the tutorials are written about Key Vault as a cert store — a place to put existing certs you've gotten from a real CA — rather than as the root of trust for a CA you're building.

The implementation is roughly seventy lines of Python using the cryptography library, plus a urllib HTTP call to Key Vault's /sign endpoint. The script reads the intermediate CSR, reads the root cert (for its Subject DN, which becomes the new cert's Issuer DN), builds a certificate body with the right extensions and a five-year validity, hashes the body with SHA-256, sends the hash to Key Vault, verifies the returned signature locally against the root's public key (defense in depth — never trust a signature you haven't checked yourself), and writes the assembled certificate to disk.

There is one network-architecture detail that this operation forces. By default, Key Vault's data plane is reachable over its public endpoint, restricted by the IP allowlist. That works for an operator running az commands from a laptop. It does not work for a VM running the signing script: the VM's public IP isn't in the allowlist, and adding every VM's public IP to the allowlist would widen the aperture on the trust anchor's network ACL without good reason.

The right answer is a Private Endpoint. A Private Endpoint puts a network interface inside the West Europe VNet with a private IP, and Azure's resolver — once a Private DNS Zone for privatelink.vaultcore.azure.net is linked to the VNet — returns that private IP when anything in the VNet queries the vault's hostname. The vault's public endpoint is unchanged for operator access from outside; the VM's path to the vault stays entirely on Azure's private backbone. The same Private DNS Zone is linked to the UAE and South Africa VNets too, so that the other nodes can resolve the vault privately when intermediate rotation happens in five years.

This is also the moment to acknowledge an honest tradeoff. The Key Vault firewall still has --bypass AzureServices enabled, which means certain Microsoft platform services can still reach the public endpoint regardless of the allowlist. For a tighter posture, that bypass can be removed entirely once the Private Endpoint is in place. I left it on for this article because it doesn't materially weaken the design for a personal-scale deployment, and the article should be honest about what's tightened and what isn't.

With the root in the vault and the network plumbing in place, the intermediate is straightforward — it's just a smaller version of the root build, with the signing step delegated to the script we just wrote.

The intermediate's private key is generated locally on vm-mongodb-westeurope using openssl genpkey, RSA-4096, encrypted at rest with AES-256-CBC using a passphrase that exists only on paper, in two physically separate locations. The CSR is built with openssl req, requesting basicConstraints: critical, CA:true, pathlen:0 (this intermediate cannot sign sub-intermediates) and keyUsage: critical, keyCertSign, cRLSign. The pathlen:0 is the structural property that constrains the intermediate to only signing leaves — anything else gets rejected by compliant verifiers.

The signing script runs on the same VM, against the CSR we just built, using a short-lived Azure AD bearer token transferred from the operator laptop. The script's verification step is worth pausing on: after receiving the signature from Key Vault, the script verifies the signature against the root's public key locally before writing the cert to disk. Trust but verify, applied to the trust anchor itself.

The token gets unset immediately after the script completes. The intermediate's encrypted private key now lives on disk under /etc/pki/meetadev-mongo-ca/private/, mode 600, owned by the operator account. The signed intermediate cert lives under certs/, alongside the freshly-assembled trust chain (ca-chain.cert.pem — intermediate + root concatenated in that order).

A two-tier chain is verified the same way every chain is:

That single word — OK — is the proof that the chain links cryptographically. The intermediate's Authority Key Identifier matches the root's Subject Key Identifier byte-for-byte. The signature on the intermediate validates against the root's public key. The chain works.

With the intermediate signed and online, leaf cert issuance becomes routine openssl ca work — no more Key Vault REST API gymnastics, no more byte-level X.509 assembly. The CA workspace was set up on the West Europe VM with a proper openssl.cnf defining the policy, the extension templates for member and client certs, and the database/serial/archive structure that openssl ca requires.

The MongoDB leaf identity model has two cert classes with one critical distinguishing property and two supporting ones.

The Subject DN. All certs in the cluster — members and clients — share the same O, the same DC components, and differ only in OU:

The OU distinction is what tells mongod "this cert is a peer trying to join the replica set" versus "this cert is a client trying to use the cluster." MongoDB enforces this at the authentication layer. I'm using OU as the discriminator rather than O or DC because it reads most cleanly in audit logs and matches MongoDB's own documentation conventions — but O or DC would also work as long as at least one component differs between the two classes.

The Extended Key Usage. This is where the security boundary gets enforced at the TLS protocol level, below MongoDB's auth layer.

• Member certs: serverAuth, clientAuth. A mongod is both a TLS server (accepting inbound connections from peers and clients) and a TLS client (initiating outbound connections to other peers for replication). Both EKUs are required, or replica set formation breaks.

• Client certs: clientAuth only. A stolen client cert physically cannot complete a TLS handshake as a server — the TLS layer itself refuses it. The DN-based discriminator (OU=cluster-clients) is enforced at MongoDB's auth layer; the EKU-based discriminator is enforced at the TLS layer. Defense in depth: both layers must be compromised for a client cert to be misused as a member cert.

The Subject Alternative Name. Member certs carry the node's FQDN, plus localhost and 127.0.0.1 for on-host operations. No external IPs — the network design from the Azure foundation piece ensures nodes only ever address each other by FQDN within the VNet mesh. Client certs carry a minimal SAN matching the CN.

The signing flow for each leaf is the same. On the owning machine — UAE VM, SA VM, WEU VM, or operator laptop — generate the keypair locally (encrypted with a unique passphrase, written down on paper, never transmitted), build the CSR with the appropriate subject and extensions, transfer the CSR to the WEU CA host via SCP through the operator laptop, run openssl ca with the appropriate extension template (v3_member_cert or v3_client_cert), transfer the signed cert back to the owning machine. The private key never leaves the machine it was generated on. The CA only ever sees public material — CSRs and signed certs.

Four leaves issued, sequential serial numbers, all chaining to the same intermediate:

Each cert verifies locally on its owning machine against the trust chain:

That's the property mongod will rely on at TLS handshake time, when Part 4 of this build actually turns on the database. If a leaf cert verifies locally with openssl, it will verify under mongod too — the two use the same library and the same chain validation logic.

A PKI without an operations plan is theatre. The validity periods on this build look like this:

• Root CA: 20 years. Touched maybe 3 times in its lifetime. Lives in Key Vault, non-exportable.

• Intermediate CA: 5 years. The next rotation happens in 2031.

• Member and client certs: 1 year. Annual rotation, all leaves rotated in a single ceremony so they expire on the same day.

The rolling rotation procedure for members maintains quorum throughout — rotate one secondary, verify it rejoins the replica set, rotate the next secondary, verify, step down the primary, rotate the former primary. The cluster never goes below two of three nodes online. Calendar entries at month 9 (prepare), month 11 (execute), month 11.5 (verify) — never reactive, always with a three-month buffer.

Revocation in this build is deliberately scoped to what's defensible. There is no CRL infrastructure. There is no OCSP responder. The reason is honest: running revocation infrastructure correctly — monitored, with uptime SLAs, with regular enforcement testing — exceeds what a single engineer can sustain for a 4-cert deployment, and revocation infrastructure run badly is worse than none. It gives the false confidence of compliance without the operational discipline that makes the compliance real.

In place of live revocation, there's a four-layer approach. Routine annual rotation limits the maximum exposure window to roughly eleven months. Suspected compromise triggers an out-of-cycle rotation under the same procedure as scheduled rotation, just compressed. Network-level access denial (NSG rules, Azure Bastion access revocation) provides the immediate "stop the bleeding" capability while certs are reissued. The MongoDB audit log, capturing full DNs of every connecting party, makes detection of misuse possible.

The threshold at which this trade-off flips is roughly twenty to fifty certs, or any deployment where cert holders exist outside the operator's direct security perimeter. At that point, CRL/OCSP infrastructure becomes worth running. This deployment is well below that threshold.

The full operational procedure — including the exact commands for routine rotation, intermediate rotation, leaf compromise response, intermediate compromise response, and root compromise response — lives in a separate runbook document alongside the CA workspace. That runbook is the executable reference; this article is the design rationale.

This deployment is intentionally below true financial-grade in several known ways. Each gap is acceptable for the current scale and operator model. Each one would have to close in a regulated production environment.

• Root CA in Azure Key Vault standard tier (FIPS 140-2 Level 1, software-protected), not Managed HSM or dedicated on-prem HSM. The cryptographic protection is software-level; the operational protection (non-exportable, audit-logged) is real.

• Intermediate CA private key on VM disk, encrypted at rest with AES-256-CBC, but not hardware-protected. A compromise of the West Europe VM during the rare moments the intermediate is decrypted for signing is a compromise of the intermediate.

• No CRL infrastructure. No OCSP responder. Revocation is via trust-list pruning, scheduled rotation, NSG-level access denial, and audit-based detection.

• No Shamir's Secret Sharing or multi-party control for root key recovery. Single-party control via Azure AD identity and RBAC.

• No formal cert ceremony with witnesses, video, or signed ceremony logs. Root signing operations are logged in Azure Activity Log only; that log is the audit trail.

• No documented revocation SLAs. Time-to-revoke is best-effort, not contractually defined.

• No periodic revocation enforcement testing on audit-grade cadence. The failure-drills part of this build series will exercise some of this, but not formally.

• Client cert lifetime is 1 year, not 90 days. The 90-day cadence is the right answer once rotation is automated; this deployment isn't automated yet, and one-year cycles are the honest compromise for manual operation.

A regulated financial deployment running real money would close all of these. A staged upgrade path is plausible — most starts with Managed HSM for the root, then adds OCSP stapling for live revocation, then formalises ceremony procedures with second-operator witnesses. None of those upgrades requires redesigning what's already built; they extend it.

Naming these gaps explicitly is, I think, the strongest version of this article. The PKI design that knows where it sits relative to the gold standard is more trustworthy than one that doesn't.

The next piece in this build installs mongod against the trust chain we just assembled. The TLS config has been waiting for this: net.tls.CAFile points at ca-chain.cert.pem, net.tls.certificateKeyFile points at each node's node-<region>.cert.pem concatenated with its encrypted private key, net.tls.allowConnectionsWithoutCertificates is false, security.clusterAuthMode is x509. The cluster will not accept a single connection that doesn't present a cert signed by the intermediate.

The cert design from this article is the load-bearing foundation for everything from this point onward. The replica set initiation, the RBAC model, the audit log structure, the financial-transactions schema, the backup procedures — all of it depends on the per-node and per-client identity that x.509 provides and a shared keyfile cannot.

Without this layer, there is no audit. Without an audit, there is no defensibility. Without defensibility, "financial-grade" is a marketing word, not an architecture.

The work this piece documents is the difference.

Three tiers down, one to go. 🎯

The root sits in the vault and refuses to come out — exactly what we asked of it. The intermediate handles the day job. Four leaves carry the identities. Every chain verifies. Not bad for a Saturday.

The decisions that got us here were deliberate. Two tiers because one is theatre. 🎭 Key Vault as a signing oracle because cloud-PKI tutorials hand-wave past the part that actually matters. EKU separation because defense in depth at the protocol layer is cheap and regret isn't. The Azure CLI tried to give me a TLS leaf when I asked for a root CA — twice. We have the soft-deleted certs to prove it. 🪦🪦

If you missed the network foundation under this PKI: Three regions, one private network. The architecture and threat model that started it all: What a wiped MongoDB cluster in Bahrain taught me about cloud architecture. 📖

The certificates exist. The chain validates. The cluster still doesn't run. 😅 Next: mongod finally meets the certs we issued, the replica set goes live with clusterAuthMode: x509, no SCRAM, no shared secrets, no second chances. 🔒

Drop a comment with your own PKI choices — especially if you've taken a different stance on revocation, root storage, or the member/client discriminator. ⚡ Follow on Meetadev. Connect on LinkedIn. 👋