Sync Architecture

LucidPal sync is a Pro-only end-to-end encrypted, cursor-based delta sync between iOS devices via a Cloudflare Workers API backed by D1. All data is encrypted on-device before leaving the app — Cloudflare only ever stores AES-GCM ciphertext and a passphrase-wrapped key blob it cannot decrypt.

Overview

iOS Device A                    lucidpal-api (Cloudflare)              iOS Device B
────────────                    ─────────────────────────              ────────────
SyncManager                     D1 (sync_records, sync_keys)           SyncManager
    │                                        │                              │
    ├── push (PUT /sync/batch) ──────────────►                              │
    │                                        ├── WebSocket fanout ──────────► (foregrounded)
    │                                        ├── silent APNs push ──────────► (backgrounded)
    │                                        │                              │
    │◄── pull (GET /sync/manifest) ──────────┤                              │
    │                                        │◄── push ─────────────────────┤
    │                                        ├── WebSocket fanout ──────────►
    │◄── pull ───────────────────────────────┤                              │

Key Components

SyncCoordinator (actor)

Services/SyncCoordinator.swift — a Swift actor that serialises all outbound sync network operations. Multiple callers (SyncManager, ChatHistoryManager, AIMemoryStore) share one coordinator to prevent concurrent push/pull races.

SyncCoordinator.shared.acquire()  →  suspends until lock is free
SyncCoordinator.shared.release()  →  resumes next waiter or unlocks

Pattern: callers await acquire() at the start of the sync operation and call release() in both success and error paths (via the same defer-equivalent try/catch structure).

SyncManager (@MainActor)

Services/SyncManager.swift — the iOS sync orchestrator. Publishes sync state for UI consumption.

swift

@Published private(set) var state: SyncState
@Published private(set) var keyState: SyncKeyState   // drives passphrase UI
@Published private(set) var lastSyncDate: Date?
@Published private(set) var conflictCount: Int
@Published private(set) var conflictedItems: [(entityType: String, title: String)]

SyncState machine:

State	Meaning
`.idle`	No sync in progress; last sync succeeded
`.syncing`	Sync cycle running
`.offline(pending: N)`	Network unavailable; N deferred cycles
`.error(String)`	Last sync failed with this message

SyncKeyState machine:

State	Meaning
`.ready`	UEK in memory/Keychain — sync fully operational
`.needsSetup`	No wrapped UEK on server — first-time passphrase setup
`.needsPassphrase`	Wrapped UEK on server but not yet unlocked on this device

Sync is skipped (performSync returns early) unless keyState == .ready.

Device ID — a stable UUID string persisted in UserDefaults under syncDeviceId. Sent as X-Device-Id header on every PUT /sync/batch so the Durable Object can suppress echo (device A's own write notification is not broadcast back to device A).

SyncCrypto (@MainActor)

Services/SyncCrypto.swift — zero-knowledge encryption using a server-wrapped UEK model (1Password / Bitwarden style). The server stores a passphrase-encrypted blob it cannot decrypt.

Key derivation chain:

User passphrase
    │
    ▼ PBKDF2-SHA256(passphrase, salt, 600k iterations)
Wrapping Key (WK)  — 256-bit, never stored
    │
    ▼ AES-GCM(WK, UEK)
wrapped_uek  — stored in D1 sync_keys table (server cannot decrypt)
    │
    ▼ on unlock: AES-GCM.open(WK, wrapped_uek) → UEK
UEK  — 256-bit SymmetricKey, cached in memory + device-local Keychain
    │
    ▼ AES-GCM(UEK, plaintext)
ciphertext  — stored in Cloudflare D1 / R2

Setup flow:

setupWithPassphrase(_:) — generates a random UEK, wraps it, uploads wrapped_uek to PUT /sync/key-setup, saves UEK to local Keychain.
unlockWithPassphrase(_:) — fetches wrapped_uek from GET /sync/key-setup, derives WK from passphrase, unwraps UEK, saves to local Keychain.
resolveKeyState() — checks memory → Keychain → server to determine SyncKeyState.
resetSync() — deletes wrapped_uek from server (DELETE /sync/key-setup) and wipes local Keychain. All existing ciphertext becomes permanently unreadable.

Passphrase format — generatePassphrase() returns 6 groups of 4 hex chars (e.g. a3f2-91bc-...), providing 96 bits of entropy. Shown once during setup; user must save it.

PBKDF2 is implemented via CCKeyDerivationPBKDF (CommonCrypto) — no external dependencies, same algorithm available in Android WebCrypto for future cross-platform support.

Keychain attributes (device-local, no iCloud sync):

Attribute	Value
`kSecAttrService`	`app.lucidpal.sync`
`kSecAttrAccount`	`lp_uek`
`kSecAttrAccessible`	`kSecAttrAccessibleAfterFirstUnlock`

No kSecAttrSynchronizable — the UEK is intentionally device-local. The server-wrapped blob is the cross-device transport mechanism, not iCloud Keychain.

SyncRealtimeClient (@MainActor)

Services/SyncRealtimeClient.swift — maintains a persistent WebSocket to the SyncSession Durable Object for sub-second change notifications when both devices are foregrounded.

Connects on UIApplication.didBecomeActiveNotification; disconnects on didEnterBackgroundNotification.
On {type:"changed"} message → calls SyncManager.syncNowDebounced(delay: 0).
30-second ping keepalive ({type:"ping"} → {type:"pong"}).
Exponential reconnect backoff: 1s → 2s → 4s → … → 30s cap.
Task identity guard in startReceiving and startPing (while task === self?.socket) prevents stale loops from acting on a replaced socket.
connect() cancels any pending reconnect task before opening a new socket, preventing double-socket races.

SyncSession (Durable Object)

src/durable-objects/sync-session.ts — one DO instance per user (idFromName(userId)), fans out change signals to all foregrounded devices via WebSocket.

Uses the WebSocket Hibernation API (state.acceptWebSocket) — the DO is evicted from memory when all sockets are idle, incurring no cost between sessions.

Message protocol:

Direction	Message
server → client	`{type:"changed", since:"<ISO>", originDevice:"<id>"}`
server → client	`{type:"pong"}`
client → server	`{type:"ping"}`

Echo suppression — originDevice from the PUT /sync/batch request is stored as a WebSocket tag (state.acceptWebSocket(server, [deviceId])). On broadcast, the originating socket is skipped (tags.includes(payload.originDevice)). Only non-empty originDevice values trigger suppression.

Wrangler binding:

toml

[[durable_objects.bindings]]
name = "SYNC_SESSION"
class_name = "SyncSession"

[[migrations]]
tag = "v2"
new_sqlite_classes = ["SyncSession"]

Five Sync Triggers

Trigger	Initiator	Behaviour
WebSocket fanout	`SyncRealtimeClient.onRemoteChange`	Fires `syncNowDebounced(delay: 0)` — sub-second when foregrounded
Foreground	`NWPathMonitor.pathUpdateHandler`	Fires `syncNow()` whenever network becomes reachable
Write debounce	Stores call `syncNowDebounced(delay: 2)` after every local write	Batches rapid edits; debounce cancels prior pending trigger
Background App Refresh	iOS BGAppRefreshTask scheduled on each foreground	Calls `syncNow()` in background
APNs silent push	Server fires `content-available: 1` after any `PUT /sync/batch`	`AppDelegate.didReceiveRemoteNotification` → `syncNow()`

syncNow() is idempotent — a second call while a sync is in flight is a no-op (inFlightTask != nil guard).

syncNowDebounced() cancels any pending debounce task before scheduling a new one.

Setup requirements (Info.plist):

xml

<key>BGTaskSchedulerPermittedIdentifiers</key>
<array>
    <string>app.lucidpal.sync-refresh</string>
    <!-- also required for other BGTasks: daily-briefing, calendar-refresh, widget-refresh, model-download -->
</array>
<key>UIBackgroundModes</key>
<array>
    <string>fetch</string>
    <string>remote-notification</string>  <!-- required for APNs silent push trigger -->
</array>

The task identifier constant is AppDelegate.syncRefreshTaskID = "app.lucidpal.sync-refresh". Missing either the BGTaskSchedulerPermittedIdentifiers entry or remote-notification in UIBackgroundModes silently disables the respective trigger.

Sync Cycle — Step-by-Step

performSync()
 1. Guard: Pro entitlement (premiumManager.canUseDeviceSync)
 2. Guard: keyState == .ready (else return — passphrase not set up or unlocked)
 3. Guard: JWT token in Keychain
 4. Guard: network online (else .offline)
 5. Guard: Wi-Fi check (if syncOnWifiOnly && !isOnWifi → skip)
 6. await SyncCoordinator.shared.acquire()
 7. Reset conflictCount and conflictedItems
 8. Capture prePullCursor (before any network calls)
 9. push(token:cursor:)           — entity records, paginated at 200/page
10. Prune tombstones (30-day TTL) for notes, abilities, pinned prompts, habit entries
11. pushSessions(token:cursor:)   — chat session metadata + blobs
12. Prune session tombstones (30-day TTL)
13. pull(token:cursor:) → newCursor
14. pullSessions(token:cursor:)
15. Commit cursor (UserDefaults) only after both pull phases succeed
16. Update lastSyncDate; reset deferredSyncCount; state = .idle
17. SyncCoordinator.shared.release()

The cursor is committed after both pull phases to ensure the cursor never advances past session data that pullSessions failed to apply.

Key state guard (step 2) replaced the former isKeyPersisted / .keyUnavailable iCloud Keychain guard. keyState is resolved on app launch and on each didBecomeActive via SyncManager.refreshKeyState(), which calls SyncCrypto.resolveKeyState(). The Settings → Sync UI reacts to keyState to show the appropriate passphrase setup or unlock card.

Entity Types

`entity_type`	Domain Type	Storage	Sync Mode
`note`	`NoteItem`	JSON in `sync_records`	Incremental (cursor)
`habit`	`HabitDefinition`	JSON in `sync_records`	Incremental (cursor)
`habit_entry`	`HabitEntry`	JSON in `sync_records`	Incremental (cursor)
`pinned_prompt`	`PinnedPrompt`	JSON in `sync_records`	Full (small collection)
`ability`	`Ability`	JSON in `sync_records`	Full (small collection)
`ability_chain`	`AbilityChain`	JSON in `sync_records`	Full (small collection)
`agent_session`	`AgentSession`	JSON in `sync_records`	Singleton (id = "singleton")
`settings`	`SyncableSettings`	JSON in `sync_records`	Singleton (id = "singleton")
`chat_session`	`ChatSession`	Metadata in D1 + encrypted blob in R2	Incremental (cursor), blob on demand

ConversationTemplates are built-in static constants — no user-created instances, no sync required.

Push — Record Encoding

For each entity, the push phase:

Filters by cursor date (skip records not updated since last sync). Pinned prompts, abilities, and chains are always pushed in full (small collections).
JSON-encodes the domain model.
SyncCrypto.encrypt(_:) → base64 AES-GCM blob.
Builds SyncRecord { id, entity_type, payload, updated_at, deleted_at }.
Pages into batches of 200 records (PUT /sync/batch).

Session push is separate:

Pushes metadata (PUT /sync/sessions/meta/batch) for sessions changed since cursor.
For each changed active session, pushes the encrypted blob (PUT /sync/sessions/{id}/blob, max 10 MB).
Tombstoned sessions only appear in the metadata batch — no blob needed.

Pull — Conflict Resolution

Policy: remote wins. If the server has a newer version of a record, the local version is discarded.

For notes and prompts, mergeFromSync() returns true when a conflict occurred (local was overwritten). These are collected as conflictedItems and surfaced in the Settings → Sync UI.

Habits, habit entries, abilities, and chains use last-write-wins without conflict notification (non-destructive merges).

agent_session singleton: remote is applied only if decoded.updatedAt > local.updatedAt.

settings singleton uses per-field last-write-wins:

Each field has an entry in fieldTimestamps: [String: Double] (Unix epoch seconds), updated by touchField(_:) on every local write.
mergeFromSync(_:) compares remoteTs[field] vs localTs[field] for each of the 16 syncable fields and applies only those where the remote timestamp is newer.
Timestamp maps are merged by taking max per key — neither device loses a write history entry.
The global settingsUpdatedAt is advanced to max(local.settingsUpdatedAt, remote.updatedAt, latestFieldTimestamp).

This means two devices editing different settings while offline never clobber each other — only the fields that were actually changed on the other device are applied.

Tombstone Lifecycle

All syncable entities use soft delete — a deletedAt timestamp is set rather than hard-deleting the row. This allows other devices to learn about the deletion during their next pull.

TTL: Tombstones are pruned locally after 30 days. The pruning calls happen during each sync cycle's push phase, after the push is complete.

Pruned entity types:

Method	Entity
`notesStore.pruneTombstones(olderThan: 30)`	Notes
`abilityStore.pruneTombstones(olderThan: 30)`	Abilities + Chains
`pinnedPromptsStore.pruneTombstones(olderThan: 30)`	Pinned Prompts
`habitStore.pruneEntryTombstones(olderThan: 30)`	Habit Entries
`sessionManager.pruneTombstones(olderThan: 30)`	Chat Sessions

Wi-Fi Only Mode

AppSettings.syncOnWifiOnly: Bool — stored in UserDefaults (device-local, not synced).

When syncOnWifiOnly == true and the device is not on Wi-Fi (isOnWifi == false), performSync() returns without acquiring the coordinator or touching the network. isOnWifi is tracked via NWPathMonitor's path.usesInterfaceType(.wifi).

This setting is intentionally excluded from SyncableSettings — it is a per-device preference.

Passphrase & Key Setup

First-time setup (new account)

SyncManager.refreshKeyState() calls SyncCrypto.resolveKeyState() → returns .needsSetup.
Settings → Sync shows "Set Up Sync" card (SyncPassphraseView in .setup mode).
User is shown a generated passphrase (6×4 hex groups, 96-bit entropy) and must confirm they have saved it.
On confirm: SyncCrypto.setupWithPassphrase(_:) → generates UEK → wraps with PBKDF2 → uploads to PUT /sync/key-setup → saves UEK to device Keychain → keyState = .ready.

New device (existing account)

resolveKeyState() finds the wrapped UEK on server but no UEK in local Keychain → returns .needsPassphrase.
Settings → Sync shows "Unlock Sync" card (SyncPassphraseView in .unlock mode).
User enters passphrase.
SyncCrypto.unlockWithPassphrase(_:) → fetches wrapped_uek → derives WK → unwraps UEK → saves to local Keychain → keyState = .ready.

Reset

SyncCrypto.resetSync():

DELETE /sync/key-setup — server deletes sync_keys row AND all sync_records for the user atomically (D1 transaction).
Wipes local Keychain entry.
_cachedKey = nil.

Warning: After reset, all existing ciphertext in D1/R2 is permanently unreadable. A new passphrase must be set up and a fresh full push performed.

Account Deletion

SyncManager.deleteAccount() async -> Bool:

Calls DELETE /account with the user's JWT.
The API handler (src/index.ts) deletes the users row — all child tables cascade via FK constraints: sync_records, chat_session_meta, device_push_tokens, notes, habits, habit entries, etc.
On HTTP 200: cancels all in-flight tasks, cancels the network monitor, deletes the JWT from Keychain, removes the sync cursor and last-sync-date from UserDefaults, resets all published properties to their zero state.
Caller navigates to the unauthenticated flow.

API Endpoints

All sync endpoints require a valid JWT (authMiddleware) and a Pro plan (requirePlan('pro')).

Method	Path	Purpose
`GET`	`/sync/key-setup`	Return wrapped UEK blob `{configured, wrapped_uek, pbkdf2_salt, pbkdf2_iterations}` or `{configured:false}`
`PUT`	`/sync/key-setup`	Store wrapped UEK (idempotent — overwrites on passphrase change)
`DELETE`	`/sync/key-setup`	Reset sync: atomically wipe `sync_keys` + all `sync_records` for user
`GET`	`/sync/ws?deviceId=<id>`	WebSocket upgrade → `SyncSession` DO for real-time fanout
`GET`	`/sync/manifest?since=ISO`	Pull delta of all entity records since cursor
`PUT`	`/sync/batch`	Push entity records (triggers DO WebSocket fanout + silent APNs push)
`PUT`	`/sync/device-token`	Register APNs device token
`DELETE`	`/sync/device-token`	Unregister APNs device token
`GET`	`/sync/sessions/list?since=ISO`	Pull session metadata delta
`PUT`	`/sync/sessions/meta/batch`	Push session metadata
`PUT`	`/sync/sessions/:id/blob`	Push encrypted session blob (max 10 MB)
`GET`	`/sync/sessions/:id/blob`	Pull encrypted session blob
`DELETE`	`/account`	Delete account + cascade all data

PUT /sync/batch headers:

Header	Required	Purpose
`X-Device-Id`	Recommended	Stable device UUID; used by DO to suppress echo to originator

Key Files

File	Role
`Services/SyncManager.swift`	Orchestrates sync cycle, exposes published state for UI
`Services/SyncCoordinator.swift`	Actor-based mutex preventing concurrent sync operations
`Services/SyncCrypto.swift`	Wrapped UEK model — PBKDF2 key derivation, AES-GCM encrypt/decrypt, Keychain
`Services/SyncRealtimeClient.swift`	WebSocket client to `SyncSession` DO — real-time change notifications
`ViewModels/AppSettings.swift`	`syncOnWifiOnly` (device-local), `SyncableSettings` for cross-device settings
`App/AppDelegate.swift`	APNs silent push handler → `syncNow()`
`App/LucidPalApp.swift`	Foreground trigger, BGAppRefreshTask registration
`Views/SyncPassphraseView.swift`	Passphrase setup (`.setup`) and unlock (`.unlock`) flows
`Views/SettingsView.swift`	Danger zone section — delete account button + confirmation alert
`Views/SettingsView+Sync.swift`	Sync status card, passphrase cards, Wi-Fi toggle, conflict banner
`src/routes/sync.ts`	All sync API routes including `/key-setup` and `/ws`
`src/services/sync-key.service.ts`	`SyncKeyService` — get/set/delete wrapped UEK
`src/repositories/sync-key.repository.ts`	D1 CRUD for `sync_keys`; atomic `deleteAndWipeSyncRecords` transaction
`src/durable-objects/sync-session.ts`	Per-user DO — WebSocket Hibernation API, broadcast, echo suppression
`src/services/sync.service.ts`	Server-side sync business logic
`src/repositories/sync-record.repository.ts`	D1 CRUD for `sync_records`
`src/services/apns.service.ts`	ES256 JWT + APNs HTTP/2 silent push
`src/repositories/device-push-token.repository.ts`	`device_push_tokens` CRUD
`src/db/migrations/0009_sync_keys.sql`	Creates `sync_keys` table with FK cascade to `users`

D1 Schema

sql

-- sync_keys: one row per user, stores passphrase-wrapped UEK
CREATE TABLE sync_keys (
  user_id         TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
  wrapped_uek     TEXT NOT NULL,       -- base64 AES-GCM(WK, UEK)
  pbkdf2_salt     TEXT NOT NULL,       -- base64 random 16-byte salt
  pbkdf2_iterations INTEGER NOT NULL DEFAULT 600000,
  created_at      TEXT NOT NULL DEFAULT (datetime('now')),
  updated_at      TEXT NOT NULL DEFAULT (datetime('now'))
);

Migration: src/db/migrations/0009_sync_keys.sql. Applied automatically by CI/CD pipeline (wrangler d1 migrations apply) before each deploy.

Deployment Notes

D1 migration — runs automatically in Woodpecker before wrangler deploy for both dev (on push to main) and prod (on manual trigger).

DO migration — SyncSession v2 migration runs automatically inside wrangler deploy. No manual step.

No new Worker secrets — SYNC_SESSION is a DO binding declared in wrangler.toml, not a secret. Cloudflare provisions it automatically on deploy.

Rules

Never push unencrypted content. Every payload field is a base64 AES-GCM sealed box.
Never advance the cursor past a partially-complete sync. Commit the cursor only after pull and pullSessions both succeed.
Never hard-delete a syncable entity. Set deletedAt and let the 30-day pruner clean it up.
Never sync syncOnWifiOnly — it is a device-local preference.
The keyState == .ready guard must fire before the coordinator is acquired. Acquiring the coordinator and then finding no key would deadlock if another caller is waiting.
DELETE /account must succeed on the server before clearing local state. If the server returns non-200, deleteAccount() returns false and local state is preserved.
The server never sees the user's passphrase or the unwrapped UEK — only the wrapped_uek blob. This is the zero-knowledge guarantee.
DELETE /sync/key-setup wipes sync_keys AND sync_records atomically. After reset, existing ciphertext in D1/R2 is permanently unreadable.

Sync Architecture ​

Overview ​

Key Components ​

SyncCoordinator (actor) ​

SyncManager (@MainActor) ​

SyncCrypto (@MainActor) ​

SyncRealtimeClient (@MainActor) ​

SyncSession (Durable Object) ​

Five Sync Triggers ​

Sync Cycle — Step-by-Step ​

Entity Types ​

Push — Record Encoding ​

Pull — Conflict Resolution ​

Tombstone Lifecycle ​

Wi-Fi Only Mode ​

Passphrase & Key Setup ​

First-time setup (new account) ​

New device (existing account) ​

Reset ​

Account Deletion ​

API Endpoints ​

Key Files ​

D1 Schema ​

Deployment Notes ​

Rules ​