Secrets in HashiCorp Vault

IdentityMesh ships with three secret-store backends:

This document covers the HashiCorp Vault provider.

Why HashiCorp Vault

The DPAPI provider encrypts each secret blob with the host’s LocalMachine key. That key never leaves the host, so the blobs are not portable: a re-image, a DR rebuild, or a cluster failover to a second host all require re-provisioning every connector secret by hand.

Azure Key Vault solves the portability problem in Azure environments. HashiCorp Vault is the equivalent option for shops that don’t run on Azure or that already operate Vault as their organisation-wide secret broker — including AWS, GCP, and pure on-prem deployments where an Azure dependency would be out of place.

Enabling HashiCorp Vault

Two things are needed, in this order:

1. Sanction the vault and place the credential on every host. In the appsettings.json of the Admin API (C:\Program Files\IdentityMesh\api\) and of the Sync Engine (C:\Program Files\IdentityMesh\engine\), name the Vault address the host may talk to and give it its AppRole credential, then restart that service:

   {
     "Secrets": {
       "HashiCorpVault": {
         "Address": "https://vault.example.com:8200",
         "RoleId": "<approle-role-id>",
         "SecretId": "<approle-secret-id>",
         "MountPath": "secret",
         "PathPrefix": "identitymesh/"
       }
     }
   }

   A host refuses any Vault address its own file does not sanction, and the credential is never stored in the database, so this step cannot be skipped.

2. Switch on the Secrets page. The active provider is chosen in the console on Settings → Secrets: pick HashiCorp Vault as the target, press Test, migrate, then switch. Until you switch, every host keeps using the DPAPI store and the Vault settings are ignored. The steps are under Migration from DPAPI to HashiCorp Vault.

Secrets:Provider in appsettings.json only seeds the choice on a host’s first start after the upgrade; the page is the switch.

Configuration keys

The engine fails at startup with a clear error (logged to the IdentityMesh Windows event log) if Address is missing or if neither Token nor (RoleId + SecretId) are configured.

Authentication

Two auth methods are supported:

1. Token auth — set Secrets:HashiCorpVault:Token to a Vault token. Convenient for dev workstations and first-touch provisioning, but the token has whatever lifetime Vault gives it, so this is not the recommended production setup.

2. AppRole auth — set Secrets:HashiCorpVault:RoleId and Secrets:HashiCorpVault:SecretId. The engine exchanges them for a short-lived token at startup and on renewal. This is the recommended production configuration: the role ID is non-secret and lives in config, the secret ID is delivered out-of-band (CD pipeline, response wrapping, or Vault Agent sidecar) and rotates independently of the engine.

The token or AppRole SecretId is read from each host’s own appsettings or environment (Secrets:HashiCorpVault:Token, Secrets:HashiCorpVault:SecretId) and is never stored in the database. The Secrets page reports per host whether it is present. Set it on the API host and on every engine host.

The address must also be sanctioned by each host’s appsettings before a database-supplied value is accepted: set the same address under Secrets:HashiCorpVault:Address, or add host or host:port to Secrets:AllowedEndpoints (semicolon-separated). The address must be https; http is allowed only for localhost and 127.0.0.1.

Whichever identity the engine ends up using needs a Vault policy that allows create, update, and read on <MountPath>/data/<PathPrefix>im-* (and delete if you intend to prune secrets out-of-band).

Only the address is sanctioned by the host file. MountPath and PathPrefix chosen on the Secrets page come from the database and are not checked against appsettings, so anyone who can write those settings can point the hosts at another path in the same vault. The token or AppRole policy is the boundary: grant it exactly the paths above and nothing wider.

Storage layout: KV v2

IdentityMesh writes to the KV v2 secrets engine. Each secret is stored as a single dictionary key:

{
  "value": "<base64-encoded byte[]>"
}

KV v2 versions every write, so re-setting the same ref creates a new version rather than overwriting in place — useful for audit trails and emergency rollback. The engine always reads the current version.

Path mapping

Callers pass arbitrary refs (e.g. secret://ad/svc/password). Vault paths are hierarchical (slashes have semantic meaning), so mirroring those slashes into Vault would create awkward, unreviewable hierarchies and would couple the on-vault layout to how callers happen to format their refs today. Instead, IdentityMesh maps each ref to a stable flat path:

secretRef  →  PathPrefix + "im-" + lowercase-hex(SHA-256(secretRef))[:32]

For example, with PathPrefix = "identitymesh/", secret://ad/svc/password becomes identitymesh/im-3f2a1b... — predictable, collision-resistant (128 bits), and never truncates a long ref. The engine logs the mapping at INFO when it writes a secret, so you can correlate refs to vault paths when needed.

The mapping is one-way (you can’t recover the original ref from the vault path). Keep the ref strings in your connector configs as the canonical identifier; the vault path is an implementation detail.

Provisioning a secret

Secrets are still provisioned via the Admin UI / Admin API exactly as they are with the DPAPI and Key Vault stores. The engine handles the encoding-and-storage step transparently:

1. Operator enters the cleartext in the Admin UI.
2. The Admin API writes the value through the secret store.
3. With Secrets:Provider = HashiCorpVault, that resolves to the HashiCorp Vault store, which Base64-encodes the bytes and writes the secret to the configured KV v2 mount.
4. The connector config keeps only the ref. Rotation is ref-stable: re-setting the same ref creates a new KV v2 version, the engine always reads the current version.

What if the vault is unreachable

The engine attempts to read the secret on each connector run. If Vault is unreachable (network outage, sealed vault, AppRole revoked, mount deleted) the connector run fails with the underlying Vault error surfaced. The watermark is not advanced, so the next run re-tries from the last good checkpoint.

Common failure modes to monitor:

• Vault sealed — every read fails with HTTP 503. Unseal the vault; the engine recovers on the next run.
• Token / AppRole secret expired — every read fails with HTTP 403. Re-issue credentials and restart the engine.
• Mount path moved — every read fails with HTTP 404. IdentityMesh treats 404 as “secret not present” (consistent with the Azure store), so connector runs will see refs as unprovisioned. Restore the mount or update MountPath.

For longer outages, vault availability — not IdentityMesh — is the SLA you should monitor. Use Vault’s own health endpoints and metrics to decide whether the engine is the right place to alert.

Migration from DPAPI to HashiCorp Vault

What the upgrade adds

The installer’s database step creates the IM_SecretsState table (per-host status, candidate provider, last run, probe requests) and the secrets.manage permission (Admin only). There is nothing extra to do during a normal upgrade. A database updated outside the installer needs the database step run before the services start; until then the Secrets tab cannot load.

This release also routes the four kinds of value that used to be DPAPI-encrypted straight into IM_Secrets through the secret store: connector credentials, AD partition credential renames, and the CyInsights, Datadog and SIEM webhook keys. Once a vault is active, it receives everything saved from the UI. Secrets saved before this release still sit in IM_Secrets, and those are what the migration moves.

Run it from Settings → Secrets (permission secrets.manage, Admin by default).

1. Hosts shows what each host is running. Both the API and the engine must be live (status younger than 24 hours) before you can move. Switch refuses when no host has reported in 24 hours.
2. Add the endpoint to every host first. A HashiCorp Vault endpoint taken from the database is accepted on a host only when that host’s own appsettings sanction it. Either set the same URI under Secrets:HashiCorpVault:Address (host and port), or add host or host:port to Secrets:AllowedEndpoints (semicolon-separated). The endpoint must be https; http is allowed only for localhost and 127.0.0.1. A refused endpoint shows as a build error for that host, and Test fails naming the key to set.
3. Target: choose HashiCorp Vault, enter the address, mount path, path prefix and auth mode, Save, then Test. Each host probes the candidate from its own process: the API at once, every live engine on its next poll (within 30 s). A probe writes one random secret, reads it back and deletes it. Results are kept per role and machine (Api:SRV1, Engine:SRV1), so an API and an engine on one server are tested separately.
4. Migrate copies every secret the catalog lists (connector credentials, AD partition credentials, vaulted account passwords, the email credentials, and the integration keys CyInsights:CyApiKey, Datadog:ApiKey and Siem:Webhook:AuthToken) and verifies each by reading it back. Nothing is deleted and nothing switches. An entry marked Unreadable names the host whose DPAPI minted it. Run the pass from that host with IdentityMesh.Service.exe --migrate-secrets, or re-enter the value, then run again. On a split install (API and engine on different servers) each run keeps the other host’s proof: a secret this host cannot decrypt, which the other host’s run verified between the same two providers and which the target still holds, stays Verified and reads “verified on SRV2”. Run Migrate on the page and --migrate-secrets on the engine, in either order; the second run is the one Switch reads.
5. Switch is enabled only when every entry verified and every live host has a fresh successful Test of the candidate after the last Save. Before cutting over, Switch re-reads every migrated secret and compares it byte for byte with the source. If any changed since the run, it refuses with “changed since the migration ran; run Migrate again”. A secret verified on the other host of a split install, which this host cannot decrypt, is only checked for presence in the target; an edit made on that host after its run is not detected from here, so run its pass again after any such edit. Both hosts follow within one poll; no restart. The one exception is the engine’s Datadog and SIEM webhook log sinks: they read their key once, at engine start, so they pick up the new provider when the engine restarts. Restart the engine after a switch if either is enabled.
6. Purge deletes the DPAPI copies of the verified entries, plus any orphan rows you tick, after you type DPAPI to confirm. A secret is deleted from the old provider only when the old and active copies are byte-equal. The rest are reported as diverged and left in place. Purge is refused while any live host still reports the provider being purged (it has not picked up the switch yet) or runs on Secrets:ProviderOverride; the message names the host.

Every action that changes something (Save, Test, Migrate, Switch, Purge) is written to the audit log. Reading the status is not.

The engine command

IdentityMesh.Service.exe --migrate-secrets [--to DPAPI|AzureKeyVault|HashiCorpVault]

Run it on the engine host when that host minted rows the API cannot decrypt. Without --to it uses the candidate saved in Settings. The exit code tells you the result:

The reverse move (HashiCorp Vault back to DPAPI) is the same steps with DPAPI as the target.

Put the token or AppRole SecretId on each host before you press Test. It is never stored in the database, and the Secrets page reports per host whether it is present. See Authentication.

If the vault is unreachable after a switch, set "Secrets": { "ProviderOverride": "DPAPI" } in a host’s appsettings.json. That host is back on its DPAPI rows at the next poll (within 30 s) and the page shows the override in red. See secrets-and-dpapi.md.

Related

• secrets-and-dpapi.md — the default on-prem provider.
• secrets-keyvault.md — the Azure Key Vault provider.
• secret-rotation.md — rotation runbook (ref-stable rotation works the same way for all three providers).
• deployment-architecture.md — where secrets fit in the broader topology.