Reconciliation report (sync-completeness signal)

IdentityMesh exposes an on-demand reconciliation endpoint that surfaces drift between what the mesh believes and what the most recent successful sync run reported. This is the auditable sync-completeness signal that satisfies SOC 2 Processing Integrity (PI1.x) and ISO 27001 Annex A 8.16: “yesterday’s run touched N objects; current count is M; here’s the difference.”

What it is

A read-only aggregation over existing tables (IM_MeshObjects, IM_ManagementSpaceObjects, IM_RunHistory, IM_Connectors). No new state is written. The endpoint is cheap enough to run on demand against a healthy production database; an operator running it after a DR restore, after a known out-of-band SQL edit, or as a weekly scheduled health check is the supported workflow.

What it detects (phase 1)

1. Run-vs-mesh count drift — most recent successful run for a connector reported N objects touched (Added + Updated + Deleted); current IM_ManagementSpaceObjects count for that connector is M. The difference (M - N, signed) is the drift since the last sync. Healthy steady state is zero or close to it; non-zero means something changed without going through the engine (a partial subsequent run, an out-of-band SQL edit, manual cleanup) or the mesh was tampered with.
2. Orphan mesh objects — rows in IM_MeshObjects with ErasedAtUtc IS NULL that have no IM_ManagementSpaceObjects row referencing them. In a healthy deployment every live mesh object is owned by at least one management-space object; an orphan indicates a partial delete or a broken import.
3. Dangling management-space rows — rows in IM_ManagementSpaceObjects whose MeshObjectId is non-null but the referenced mesh object doesn’t exist (or is erased). A dangling row is a broken join — the projection engine will misbehave on it until the broken state is repaired.
4. Per-object-type breakdown — current counts grouped by ObjectType so an operator can see “12,438 Users / 1,250 Groups” rather than a single number.

Source drift (phase 2)

Phase 1 compares two numbers that both come from IdentityMesh’s own database. Phase 2 asks the source system itself how many objects it holds and compares that with the rows the mesh keeps for the connector. This is the comparison that shows whether IdentityMesh holds everything the source has.

Operations → Reconciliation shows two drift columns per connector:

• Drift — the connector’s live management-space rows (rows not flagged deleted) against what the last successful full import returned. Rows a confirming import has marked deleted stay in the table but are exactly the rows that import did not return, so they are not drift. Delta imports and exports never move it. It reads “No baseline” until a full import has recorded an object count.
• Source drift — the same live rows against the source system’s live count.

The Live rows column beside them shows the live management-space rows that both drifts are measured against. Rows an administrator has isolated still exist at the source, so they stay inside Live rows; when any connector has isolated rows or open export holds, an Isolated / Held column appears with both counts (see isolate-disconnect.md).

The dashboard’s reconciliation card has a Source drift column too. Its drift-signals total counts a connector’s source drift when one is known and its run-vs-mesh drift otherwise, never both.

Counting the source

The Count source button on a connector row, and Count all sources above the table, ask the source for its current count. Both require the sync.execute permission.

• The count runs on the engine instance, or the relay agent, that owns the connector, using the connector’s saved configuration and credentials. Nothing is typed in at count time.
• The time budget is 25 seconds inside the connector, 30 seconds at the engine or relay agent, and 45 seconds at the API.
• A count that cannot finish inside the budget is shown as ”≥ N (partial)”. It is a lower bound, so it carries no source drift.
• A failed attempt keeps the last good count and its drift. The row adds “Last attempt failed” with the time and the reason.
• “No count yet” means nobody has counted this connector.
• “This connector version does not report source counts.” means the connector plugin predates the feature. Update it to a build from the same MSI as the engine.
• The Counted column shows when the count ran and where.

What “live” means per connector

The source count is chosen to mean the same thing as the live rows the import keeps.

The comparison is by total, not by object type. Management-space rows for an object type the connector no longer imports therefore still count as drift. That is intended: the source no longer feeds them. Use the orphan cleanup for those rows.

Upgrade note

Source counts are stored in a new database table, IM_ConnectorSourceCounts. The installer’s database step creates it during the upgrade; there is nothing extra to do. A database updated outside the installer needs the database step run before the upgraded Admin API starts.

How to tell the table is missing: the Reconciliation page still loads but shows no source counts, one warning is logged, and Count source cannot store a result. Run the database step and restart the Admin API.

The engine and the connectors ship together in the same release, so install them from the same MSI.

Endpoints

GET /api/admin/reconciliation — overall report

Returns the full reconciliation report: total counts, orphan + dangling counts, plus a per-connector summary.

curl -X GET https://identitymesh.example.com/api/admin/reconciliation \
     -H "Authorization: Bearer $TOKEN"

Response shape:

{
  "reportedAtUtc": "2026-04-26T14:33:21Z",
  "totalMeshObjects": 13688,
  "totalErasedMeshObjects": 4,
  "orphanMeshObjectCount": 0,
  "danglingConnectorSpaceCount": 0,
  "connectors": [
    {
      "connectorId": "9c1c...",
      "connectorName": "Entra-Production",
      "connectorType": "EntraId",
      "currentMeshObjectCount": 12438,
      "lastRunObjectsTouched": 12438,
      "lastSuccessfulRunUtc": "2026-04-26T03:15:00Z",
      "runVsMeshDrift": 0
    },
    {
      "connectorId": "a234...",
      "connectorName": "AD-OnPrem",
      "connectorType": "ActiveDirectory",
      "currentMeshObjectCount": 1250,
      "lastRunObjectsTouched": 1250,
      "lastSuccessfulRunUtc": "2026-04-26T02:45:00Z",
      "runVsMeshDrift": 0
    }
  ]
}

A drift report (something out of sync) looks like:

{
  "reportedAtUtc": "2026-04-26T14:33:21Z",
  "totalMeshObjects": 13690,
  "totalErasedMeshObjects": 4,
  "orphanMeshObjectCount": 2,
  "danglingConnectorSpaceCount": 1,
  "connectors": [
    {
      "connectorName": "Entra-Production",
      "currentMeshObjectCount": 12440,
      "lastRunObjectsTouched": 12438,
      "runVsMeshDrift": 2
    },
    {
      "connectorName": "AD-OnPrem",
      "currentMeshObjectCount": 1250,
      "lastRunObjectsTouched": 1250,
      "runVsMeshDrift": 0
    }
  ]
}

The runVsMeshDrift: 2 on Entra-Production paired with the orphanMeshObjectCount: 2 is the operator’s lead — there are two mesh rows that exist on this side without a management-space pairing on the other.

GET /api/admin/reconciliation/{connectorId} — per-connector detail

Returns the same summary plus a per-ObjectType breakdown and the five most recent runs (any status — useful for spotting a string of failed runs preceding the drift).

curl -X GET https://identitymesh.example.com/api/admin/reconciliation/9c1c... \
     -H "Authorization: Bearer $TOKEN"

Response shape:

{
  "summary": {
    "connectorId": "9c1c...",
    "connectorName": "Entra-Production",
    "connectorType": "EntraId",
    "currentMeshObjectCount": 12440,
    "lastRunObjectsTouched": 12438,
    "lastSuccessfulRunUtc": "2026-04-26T03:15:00Z",
    "runVsMeshDrift": 2
  },
  "byObjectType": [
    { "objectType": "Group", "currentCount": 1250 },
    { "objectType": "User",  "currentCount": 11190 }
  ],
  "recentRuns": [
    { "runId": "...", "status": "Success", "added": 5, "updated": 12433, "deleted": 0 },
    { "runId": "...", "status": "Success", "added": 0, "updated": 12438, "deleted": 0 },
    { "runId": "...", "status": "Failed",  "added": 0, "updated": 0, "deleted": 0 }
  ]
}

Operator workflow

Recommended cadence:

• Weekly — schedule a GET against /api/admin/reconciliation and feed the response into your monitoring stack. Alert on orphanMeshObjectCount > 0, danglingConnectorSpaceCount > 0, or any runVsMeshDrift outside an acceptable threshold (the threshold depends on natural change-rate for your tenant; start with abs(drift) > 5% of total).
• Post-DR / post-restore — run immediately after a database restore to confirm row counts are coherent before letting sync runs resume.
• Post-incident — after any known out-of-band SQL edit (DBA cleanup, bulk attribute fix, etc.) run reconciliation to confirm no joins were left dangling.

When drift is detected:

1. Pull the per-connector detail (/api/admin/reconciliation/{id}). Cross-reference recentRuns: a failed run preceding the drift often explains it (the engine started a delta, deleted some tracking rows, then crashed before repopulating them).
2. Pull the audit log (/api/admin/audit) for the same window — any ErasureRequest rows account for totalErasedMeshObjects moving up; any DBA-tagged actor entries account for out-of-band edits.
3. To clean up dangling management-space rows safely, use the duplicate-orphan cleanup in the console: run its report first, compare its counts with the orphan / dangling counts in the reconciliation report, then apply. Re-run the reconciliation report to confirm the counts dropped to zero. Do not delete rows by hand in SQL.
4. If drift persists across runs and can’t be attributed to out-of-band edits, treat it as a sync-engine bug — capture the reconciliation snapshot + run history and open a support ticket.

Import drift detection

The on-demand report above is the point-in-time snapshot. The engine also reconciles continuously during import. On every join-hit refresh — each time an incoming source object re-matches an existing mesh object — the engine asks the export projection logic a single question: “given the mesh object’s full attribute set, would export write anything to the target right now?” If the answer is yes, the object’s outbound state has drifted from what the target holds, and the engine auto-enqueues an ImportDrift export to close the gap.

The important property is that the drift detector and the actual export share one code path — the same export projection logic. They project from the same full mesh attribute set, so they cannot disagree. This rules out false-positive churn, where a detector projecting from only the attributes in one import would flag “drift” that the exporter, projecting from the full stored attributes, then declined to write — parking the queue at thousands of Pending items that never produced a real export. A detected drift always corresponds to a real pending write, and a clean object is never enqueued.

Operationally this means import-driven exports appear in the queue tagged as ImportDrift without an explicit schedule triggering them — that is expected self-healing, not churn. If you see ImportDrift items that never drain, treat it as an export-side problem (target unreachable, connector disallowed under export staging), not a detector bug.

Composer membership reconciliation

Dynamic groups built by MeshComposer match members by criteria. A group can be exported to its target before all of its members are resolvable — a member’s mesh object may not yet carry the anchor the target needs (it joins on a later import), so the group lands at the target with an empty or partial membership and is not automatically revisited when the members become resolvable.

A periodic self-heal pass reconciles this: it re-queues dynamic groups whose members only became resolvable after the group was first exported, so the membership is re-projected and the target group fills in. This is a durable reconciliation pass — it runs on its own cadence and closes the gap without an operator re-running the composer rule by hand.

If a group’s membership at the target looks short right after onboarding, this pass is what corrects it on the next cycle; a membership that stays short across several cycles points at an unresolved member (a join that never landed), which the orphan / dangling signals above will surface.

Operations posture dashboard

The Admin UI surfaces all of the above in one place at /identity/operations. Rather than curl the endpoints, an operator gets a single red / amber / green posture view:

• Overall posture — a rolled-up health signal (green = coherent, amber = drift within threshold, red = orphans / dangling rows / failed chain) so an on-call operator can triage at a glance.
• License / grace status — current license tier and whether the deployment is inside its grace window, alongside the posture (a lapsed license is itself an operational condition).
• One-click chain verification — runs the hash-chain verifier (see audit-chain.md) and shows pass/fail without leaving the page.
• Drift listings — the orphan mesh objects, dangling-management-space rows, and run-vs-mesh drift from the reconciliation report, rendered as actionable lists rather than raw counts.

The dashboard reads the same audit.read-gated reconciliation and audit surfaces described here — it is a presentation over them, not a separate data path.

Permissions

The reconciliation endpoint is gated on the same audit.read permission as the audit chain verifier (audit-chain.md) and the audit log itself — same audience, same evidence-grade trust level.

Compliance mapping

The reconciliation response is itself ephemeral — it’s recomputed on each call and not persisted. For a long-term record, fold the response into your monitoring system’s metrics history (the reportedAtUtc field is the timestamp authoritative for the snapshot).

Related

• audit-chain.md — the hash-chained audit log that records every state change. Reconciliation reads only; audit-chain is the write-side evidence.
• audit-retention.md — how long historical run / audit rows survive (relevant when investigating drift against an older window).
• dsar-erasure.md — the totalErasedMeshObjects field counts subjects erased through the data-subject erasure workflow; this is the link between the privacy-rights surface and the reconciliation signal.
• maintenance-operations.md — the operator actions (uniqueness re-resolve, duplicate-orphan cleanup, export staging) you run once reconciliation surfaces a problem.