Building reliable (and fast) directory sync

Directory sync is the process of copying an organization’s users, groups, and memberships from an identity provider into an application and keeping that copy current. The practical challenge is representing nested groups so membership checks are efficient - flattening group memberships gives each user a row for every group they belong to, direct or indirect - and then reliably ingesting changes. SCIM, the standard push-based approach, defines REST endpoints for provisioning but leaves provider behavior undefined. In practice providers diverge wildly (different PATCH semantics, type formats, unsupported features) and deprovisioning behavior varies (Okta, Entra, JumpCloud, OneLogin all behave differently), forcing per-provider shims behind the SCIM API and exposing the app to missed events if it’s briefly unavailable.

A pull-based sync engine avoids those pitfalls by having servers call each provider’s API on a schedule or on demand, walk users/groups/members, and reconcile into the database. Important implementation details include handling pagination, matching on stable IDs (not email), detecting cycles in nested groups, and recomputing flattened memberships so removals are correct. For large directories a checkpointed full sync uses an epoch timestamp: write pages stamped with the epoch, then delete records older than it to avoid partial-write races. That approach favors control, resilience, and a shared codepath across providers.