Secrets in Azure Key Vault

IdentityMesh ships with two secret-store backends:

This document covers the Key Vault provider.

Why Key Vault

The DPAPI provider encrypts each secret blob with the host’s LocalMachine key. That key never leaves the host, so the blobs are not portable: a re-image, a DR rebuild, or a cluster failover to a second host all require re-provisioning every connector secret by hand.

Key Vault keeps the ciphertext in the vault, not on the host. Any authorised IdentityMesh process — original host, replacement host, DR-region host — can read the same secret using the same vault URI and an authenticated managed identity.

Enabling Key Vault

Two things are needed, in this order:

1. Sanction the vault on every host. In the appsettings.json of the Admin API (C:\Program Files\IdentityMesh\api\) and of the Sync Engine (C:\Program Files\IdentityMesh\engine\), name the vault the host may talk to, then restart that service:

   {
     "Secrets": {
       "KeyVault": {
         "VaultUri": "https://contoso-im.vault.azure.net/"
       }
     }
   }

   A host refuses any vault URI its own file does not sanction, so this step cannot be skipped.

2. Switch on the Secrets page. The active provider is chosen in the console on Settings → Secrets: pick Azure Key Vault as the target, press Test, migrate, then switch. Until you switch, every host keeps using the DPAPI store and the Key Vault settings are ignored. The steps are under Migration from DPAPI to Key Vault.

Secrets:Provider in appsettings.json only seeds the choice on a host’s first start after the upgrade; the page is the switch.

Authentication

A host reaches the vault with the app registration from the Secrets page when a Tenant ID and client id are set there. Otherwise it uses the Azure SDK’s default chain, which tries the AZURE_* environment variables first, then the host’s managed identity, then an Azure CLI session.

1. An app registration entered on the Secrets page. Settings → Secrets → Target provider → Azure Key Vault takes a Tenant ID, an Application (client) ID and a client secret. The ids are ordinary settings. The secret is encrypted to each host’s own key and delivered through the database: every host, including engines on other machines and hosts installed later, receives its own copy automatically, and the page shows which hosts have it. The secret is never shown again; save a new one to rotate it, or tick Clear stored secret to drop it. This is the path for on-prem hosts, which have no managed identity.
2. Environment variables AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET set on both Windows services, with the page’s ids left blank. Same credential as option 1, maintained by hand on every host.
3. Managed identity assigned to the host (system-assigned VM identity, or a user-assigned identity referenced by AZURE_CLIENT_ID). Leave the Tenant ID and client id blank. This is the recommended setup when the hosts run in Azure.

A host can also carry Secrets:KeyVault:ClientSecret in its own appsettings.json; a copy delivered from the page takes precedence over it, so a rotated secret reaches a host with a stale file.

With ids set but no secret on a host yet (the first minute after an upgrade, or a host whose key was re-minted after a rebuild), that host keeps its previous store and reports Secrets:KeyVault:ClientSecret has not been delivered to this host yet on the Secrets page until the next poll delivers it. If the API host itself was rebuilt, the page asks you to re-enter the secret.

Whichever identity the host ends up using needs the Key Vault Secrets Officer role on the vault: the Secrets page writes during migration and the probe writes a test secret. The vault URI must be sanctioned in each host’s appsettings.json (Secrets:KeyVault:VaultUri or Secrets:AllowedEndpoints) whatever the identity.

The vault URI chosen on the Secrets page is accepted on a host only when that host’s appsettings sanction it: set the same URI under Secrets:KeyVault:VaultUri, or add host or host:port to Secrets:AllowedEndpoints (semicolon-separated). The URI must be https; http is allowed only for localhost and 127.0.0.1.

Secret names are hashes of the IdentityMesh reference, so a reference that was deleted and is written again (a purge followed by a move back to Key Vault, an integration key cleared and set again, a partition renamed back) lands on its own soft-deleted name. The store recovers the soft-deleted secret and then writes the new value. That needs the recover permission, which Key Vault Secrets Officer includes; with access policies, grant Recover alongside Set. A name already purged from the vault is simply created again.

Secret naming

Callers pass arbitrary refs (e.g. secret://ad/svc/password) but Key Vault names allow only [A-Za-z0-9-] and must start with a letter. IdentityMesh maps each ref to a stable Key Vault name:

secretRef  →  "im-" + lowercase-hex(SHA-256(secretRef))[:32]

For example, secret://ad/svc/password becomes im-3f2a1b... — predictable, collision-resistant, and never truncates a long ref. The engine logs the mapping at INFO when it writes a secret, so you can correlate refs to vault entries when needed.

The mapping is one-way (you can’t recover the original ref from the vault name). Keep the ref strings in your connector configs as the canonical identifier; the vault name is an implementation detail.

Provisioning a secret

Secrets are still provisioned via the Admin UI / Admin API exactly as they are with the DPAPI store. The engine handles the encoding-and-storage step transparently:

1. Operator enters the cleartext in the Admin UI.
2. The Admin API writes the value through the secret store.
3. With Secrets:Provider = AzureKeyVault, that resolves to the Key Vault store, which Base64-encodes the bytes and writes the secret to the configured vault.
4. The connector config keeps only the ref. Rotation is ref-stable: re-setting the same ref creates a new vault version, the engine always reads the current version.

What if the vault is unreachable

The engine attempts to read the secret on each connector run. If Key Vault is unreachable (network outage, identity revoked, vault deleted) the connector run fails with the underlying SDK error surfaced. The watermark is not advanced, so the next run re-tries from the last good checkpoint.

For longer outages, vault availability — not IdentityMesh — is the SLA you should monitor. Use the Azure status page and per-vault metrics to decide whether the engine is the right place to alert.

Migration from DPAPI to Key Vault

What the upgrade adds

The installer’s database step creates the IM_SecretsState table (per-host status, candidate provider, last run, probe requests) and the secrets.manage permission (Admin only). There is nothing extra to do during a normal upgrade. A database updated outside the installer needs the database step run before the services start; until then the Secrets tab cannot load.

This release also routes the four kinds of value that used to be DPAPI-encrypted straight into IM_Secrets through the secret store: connector credentials, AD partition credential renames, and the CyInsights, Datadog and SIEM webhook keys. Once a vault is active, it receives everything saved from the UI. Secrets saved before this release still sit in IM_Secrets, and those are what the migration moves.

Run it from Settings → Secrets (permission secrets.manage, Admin by default).

1. Hosts shows what each host is running. Both the API and the engine must be live (status younger than 24 hours) before you can move. Switch refuses when no host has reported in 24 hours.
2. Add the endpoint to every host first. An Azure Key Vault endpoint taken from the database is accepted on a host only when that host’s own appsettings sanction it. Either set the same URI under Secrets:KeyVault:VaultUri (host and port), or add host or host:port to Secrets:AllowedEndpoints (semicolon-separated). The endpoint must be https; http is allowed only for localhost and 127.0.0.1. A refused endpoint shows as a build error for that host, and Test fails naming the key to set.
3. Target: choose Azure Key Vault, enter the vault URI, Save, then Test. Each host probes the candidate from its own process: the API at once, every live engine on its next poll (within 30 s). A probe writes one random secret, reads it back and deletes it. Results are kept per role and machine (Api:SRV1, Engine:SRV1), so an API and an engine on one server are tested separately.
4. Migrate copies every secret the catalog lists (connector credentials, AD partition credentials, vaulted account passwords, the email credentials, and the integration keys CyInsights:CyApiKey, Datadog:ApiKey and Siem:Webhook:AuthToken) and verifies each by reading it back. Nothing is deleted and nothing switches. An entry marked Unreadable names the host whose DPAPI minted it. Run the pass from that host with IdentityMesh.Service.exe --migrate-secrets, or re-enter the value, then run again. On a split install (API and engine on different servers) each run keeps the other host’s proof: a secret this host cannot decrypt, which the other host’s run verified between the same two providers and which the target still holds, stays Verified and reads “verified on SRV2”. Run Migrate on the page and --migrate-secrets on the engine, in either order; the second run is the one Switch reads.
5. Switch is enabled only when every entry verified and every live host has a fresh successful Test of the candidate after the last Save. Before cutting over, Switch re-reads every migrated secret and compares it byte for byte with the source. If any changed since the run, it refuses with “changed since the migration ran; run Migrate again”. A secret verified on the other host of a split install, which this host cannot decrypt, is only checked for presence in the target; an edit made on that host after its run is not detected from here, so run its pass again after any such edit. Both hosts follow within one poll; no restart. The one exception is the engine’s Datadog and SIEM webhook log sinks: they read their key once, at engine start, so they pick up the new provider when the engine restarts. Restart the engine after a switch if either is enabled.
6. Purge deletes the DPAPI copies of the verified entries, plus any orphan rows you tick, after you type DPAPI to confirm. A secret is deleted from the old provider only when the old and active copies are byte-equal. The rest are reported as diverged and left in place. Purge is refused while any live host still reports the provider being purged (it has not picked up the switch yet) or runs on Secrets:ProviderOverride; the message names the host.

Every action that changes something (Save, Test, Migrate, Switch, Purge) is written to the audit log. Reading the status is not.

The engine command

IdentityMesh.Service.exe --migrate-secrets [--to DPAPI|AzureKeyVault|HashiCorpVault]

Run it on the engine host when that host minted rows the API cannot decrypt. Without --to it uses the candidate saved in Settings. The exit code tells you the result:

The reverse move (Azure Key Vault back to DPAPI) is the same steps with DPAPI as the target.

Managed identity needs no per-host secret.

If the vault is unreachable after a switch, set "Secrets": { "ProviderOverride": "DPAPI" } in a host’s appsettings.json. That host is back on its DPAPI rows at the next poll (within 30 s) and the page shows the override in red. See secrets-and-dpapi.md.

Related

• secrets-and-dpapi.md — the default on-prem provider.
• secret-rotation.md — rotation runbook (ref-stable rotation works the same way for both providers).
• deployment-architecture.md — where secrets fit in the broader topology.