Maintenance Operations

Operator runbook for the manual and periodic maintenance surfaces IdentityMesh exposes: uniqueness re-resolve, duplicate-orphan cleanup, audit-backlog purge, export staging / gating, and the interactive discovery calls behind the connector wizard.

These are the levers you reach for after a bulk import, a messy migration, or an out-of-band change — the things you run deliberately, not the sync that runs itself. Each one is either dry-run-first or read-only-first so you can look before you leap.

Uniqueness re-resolve

IsUnique attributes (samAccountName, mail, userPrincipalName, employeeId, …) are allocated once when a mesh object is first resolved. If duplicates were imported before the uniqueness rule existed — or two sources contributed the same value in the same import window — those duplicates survive: the assign-once path doesn’t revisit an attribute it already set. The symptom is IsUnique attributes that still show collisions after a clean full import.

Re-resolve re-runs the uniqueness allocator across the existing mesh and clears those surviving duplicates.

Periodic

UniquenessMaintenanceService runs the re-resolve on a background cadence inside the Sync Engine, so a steady deployment self-heals without operator action. The manual endpoint is for when you need it now (post-migration) or want to preview the impact first.

On demand

# Dry-run (default): forecast only, writes nothing.
curl -X POST https://identitymesh.example.com/api/admin/reresolve-uniqueness \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{ "dryRun": true }'

# Apply: re-allocate and clear duplicates.
curl -X POST https://identitymesh.example.com/api/admin/reresolve-uniqueness \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{ "dryRun": false }'

POST with no body defaults to dryRun: true — the endpoint never mutates unless you explicitly pass { "dryRun": false }. Run the dry-run, read the forecast (how many values it would re-allocate), then re-run with dryRun false to apply.

The endpoint is gated on rules.write (re-allocating a unique value is a rules-side change). The read-only reconciliation report that surfaces duplicate/orphan symptoms is gated on audit.read (see reconciliation.md) — an auditor can see the drift; a rules operator applies the fix.

Duplicate-orphan cleanup

A duplicate orphan is a management-space row that is an orphaned duplicate of an object already linked to a mesh object — the same source identity represented twice, where one copy carries the live join and the other is dead weight. These accumulate from re-imports after a source-side rename, or from disabled accounts that were re-created upstream.

Report first

# CSV of every eligible duplicate for a connector (no row cap).
curl -X GET "https://identitymesh.example.com/api/admin/cleanup-duplicate-orphans/report?connectorId=$CONNECTOR_ID" \
     -H "Authorization: Bearer $TOKEN" \
     -o duplicate-orphans.csv

The report is a dry-run that returns the full eligible set as CSV (SourceLocator, SourceName, ExternalId, IsDisabledAtSource, DuplicateOfMeshObjectId). Review it before applying anything.

Apply

# Dry-run apply (default): count what would be removed.
curl -X POST https://identitymesh.example.com/api/admin/cleanup-duplicate-orphans \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{ "connectorId": "'"$CONNECTOR_ID"'", "dryRun": true, "requireDisabled": true }'

# Real apply, restricted to duplicates that are disabled at the source.
curl -X POST https://identitymesh.example.com/api/admin/cleanup-duplicate-orphans \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{ "connectorId": "'"$CONNECTOR_ID"'", "dryRun": false, "requireDisabled": true }'

requireDisabled: true is the safe default posture — it restricts the delete to duplicates whose source object is disabled at the source, so you never remove a duplicate that might still be live upstream. Set it to false only when you have reviewed the report and are certain the duplicates are dead regardless of source state.

Both the apply and the report are gated on rules.write.

Audit-backlog purge

The hourly AuditRetentionService keeps IM_ObjectAudit and IM_AdminAudit trimmed to the license retention window — but it deletes only 1,000 rows per sweep, which is far too slow to clear a multi-ten-million-row backlog (e.g. after a period with retention effectively disabled, or a first upgrade onto retention).

For that one-time bulk catch-up there is a standalone, run-by-hand script:

IdentityMesh.Infrastructure.Sql/sql/Manual_PurgeAuditBacklog.sql

It batch-deletes IM_ObjectAudit down to the retention window (default 30 days) in < 5000-row auto-committed batches, with a tunable inter-batch delay so the engine and Admin API can still query the table. It is not wired into Deploy_All.sql — run it deliberately, off-hours, after reading the header notes:

• Pause services first (Settings > System > Pause All Services, or export staging below).
• Mind the transaction log under FULL recovery — run frequent log backups during the purge, or switch to SIMPLE for the run and take a FULL backup afterward (this breaks the log-backup chain).
• IM_ObjectAudit is the hash-chained tamper-evident table (PreviousRowHash/RowHash). Purging the oldest rows is exactly what retention already does — after a purge, chain verification is meaningful from the earliest retained row forward. This is expected for any retention purge.

After the manual catch-up, the AuditRetentionService maintains steady state on its own. The full retention model — tables, license-driven vs operator-config windows, and verification — is documented in audit-retention.md.

Export staging / gating

By default the engine imports, projects, and exports in one continuous flow. During onboarding, a migration, or a risky change you often want to stage exports: let imports and projections populate the mesh and fill the export queue, review what would be written, then release it to the target systems on your signal. The /api/system family provides the gates.

Global service pause

curl -X POST https://identitymesh.example.com/api/system/pause  -H "Authorization: Bearer $TOKEN"
curl -X POST https://identitymesh.example.com/api/system/resume -H "Authorization: Bearer $TOKEN"

Sets ServicePaused — the whole engine (imports, projections, exports) stops. Use this for maintenance windows and before the audit-backlog purge. For a named engine instance, pause just that instance with POST /api/instances/{id}/pause (and /resume); see multi-instance for named-instance workload distribution.

Export-only staging pause

# Hold all exports; imports and projections keep running and the queue fills.
curl -X POST https://identitymesh.example.com/api/system/exports/pause  -H "Authorization: Bearer $TOKEN"
# Release: exports flush to targets.
curl -X POST https://identitymesh.example.com/api/system/exports/resume -H "Authorization: Bearer $TOKEN"

Sets ExportsPaused. Unlike the global pause, the mesh keeps building — nothing is written to any target connector until you resume. This is the core staging switch: stage the whole mesh, review, then flush when ready.

Per-connector allow-list

While staging is on, you can let a single connector export live (a canary target) while the rest stay held:

curl -X POST https://identitymesh.example.com/api/system/connectors/$CONNECTOR_ID/exports/allow    -H "Authorization: Bearer $TOKEN"
curl -X POST https://identitymesh.example.com/api/system/connectors/$CONNECTOR_ID/exports/disallow -H "Authorization: Bearer $TOKEN"

An allow-listed connector (ExportsAllowed:{connectorId} = true) flushes its queue even while ExportsPaused is on; every other connector stays paused. Verify the canary, then allow the next, or lift the global staging pause to release everything.

Staging state

curl -X GET https://identitymesh.example.com/api/system/staging-state -H "Authorization: Bearer $TOKEN"

{
  "exportsPaused": true,
  "pendingExports": 5120,
  "allowedConnectorIds": ["9c1c..."],
  "pendingByConnector": [
    { "connectorId": "9c1c...", "pending": 5000 },
    { "connectorId": "00000000-0000-0000-0000-000000000000", "pending": 120 }
  ]
}

pendingByConnector tells you how many items each connector would flush on release. Queue items with no connector (rare) are split out under the empty GUID (00000000-0000-0000-0000-000000000000) so they’re visible rather than hidden inside a total — investigate any non-zero null-connector bucket before releasing.

Drain-worker tuning

The dedicated ExportDrainService empties the queue on its own cadence. Tune its throughput when a large staged batch needs to flush faster (or slower, to spare a fragile target):

curl -X GET https://identitymesh.example.com/api/system/export-drain-settings -H "Authorization: Bearer $TOKEN"

curl -X PUT https://identitymesh.example.com/api/system/export-drain-settings \
     -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
     -d '{ "parallelism": 8, "intervalSeconds": 30 }'

parallelism is clamped to 1–32 (default is min(CPU, 8)), intervalSeconds to 5–3600 (default 60). The engine reads these live each drain cycle — no restart needed.

Import-run timeout

curl -X GET https://identitymesh.example.com/api/system/import-run-timeout -H "Authorization: Bearer $TOKEN"

curl -X PUT https://identitymesh.example.com/api/system/import-run-timeout \
     -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
     -d '{ "minutes": 120 }'

Caps how long a single import run may execute before the engine abandons it. Clamped to 1–1440 minutes (default 60). Raise it for a slow, very large first import; lower it to fail fast on a wedged source.

Staging pattern

The supported end-to-end pattern:

1. POST /api/system/exports/pause — stage. Imports and projections run; the export queue fills; nothing reaches a target.
2. Review — GET /api/system/staging-state for queue depth per connector, plus the mesh-object export-preview in the Admin UI, plus a reconciliation report.
3. Release progressively — allow a canary connector, verify the real writes landed, then either allow the rest or POST /api/system/exports/resume to flush everything.

Permissions

The write endpoints are additionally rate-limited (the sensitive limiter) — they are operator-scale actions, not high-frequency calls.

Interactive discovery

The connector wizard in the Admin UI (and relay agents, for relay-routed connectors) uses a set of live discovery calls so an operator can point at a source and browse it before committing a connector config. They connect to the live source with the supplied (or secret-referenced) credentials and return what they find; they never write. All are gated on connectors.write.

For a relay-routed connector these run on the relay agent against the remote source rather than on the API host — see relay-agent.md. The credentials the discovery calls use are resolved the same way the connector will resolve them at runtime, so a successful browse is also a credential sanity check.

Related

• reconciliation.md — the read-only drift/orphan/dangling report (and the operations posture dashboard) that tells you whether you need these operations.
• audit-retention.md — the automatic retention sweep the manual audit-backlog purge complements.
• relay-agent.md — where discovery and export run for relay-routed connectors.