uid: T-195 title: Person-record data migration — move person-facts onto the Individual, group into maps, backfill the placeholder abbreviations status: done assignee: User Management area: user-management created: 2026-07-30 owner: girafeev1 related: T-188, T-183, T-048

T-195 — The actual moving of data¶

Why this is its own task¶

T-188 designs the target: users/{uid} becomes a pure access record, individuals/{id} becomes the only home of person-facts, fields group into maps, the two point at each other. That is the shape. This task is the execution — the transformation of every document that already exists in production so it matches the shape — kept separate because it is large, staged, irreversible once run, and gated on approvals the design work is not.

Owner, 2026-07-30: "We have created multiple steps of data migration … have we marked it as a task?" — it was only Phase P3 of T-188 and one bullet of the brief. Promoted here so the moving of data is tracked as its own unit, not buried in a design task.

This changes existing production documents, so every step below is under the AGENTS.md Firestore-structure gate: the whole-document current-vs-after goes to the owner and is approved before any code runs. This task file is the plan; it is not approval to run it.

Migration expectations (moved here from the T-188 brief, owner's instruction)¶

Originally recorded in the T-188 brief; extracted to this task on 2026-07-30 when the owner ruled the migration is not a subtask of T-188. The owner's expectations, in full:

Backfill onto blank Individual fields first; where both documents hold a value and they disagree, produce a conflict list for the owner rather than silently overwriting.
Re-point every reader before removing anything.
Only then remove the duplicated fields from users/{uid} — that removal is a Firestore structure change requiring whole-document current-vs-after approval before code.
Suggested phasing (owner): job title first → phone / whatsapp / organization → the write-once gates → the profile restructure. Smallest, highest-drift field first; the surfaces that consume the final shape last.

What moves (each is a migration step)¶

Person-facts off users/{uid} onto individuals/{id}. jobTitle, organization, phoneNumber, whatsappNumber, contactNotes, preferredContactMethod, isExternal — copied onto the bridged Individual where the Individual's field is blank, then removed from the user doc. Where both hold a value and they disagree, produce a conflict list for the owner — never a silent overwrite.
Audit timestamps into a history map on both documents — createdAt, approvedAt, approvedBy, acceptedAt and the like become history.* (owner, 2026-07-27: a map on the document, not a subcollection). accessExpiresAt / accessGrantedAt / accessGrantedBy stay put — they are live access control, read on every request, not history.
Group the remaining flat fields into maps per the no-flat-layout rule (access.* for role / badges / subsidiary access / status), owner-approved shapes only.
Backfill the placeholder abbreviations. Every individuals/{id} created by onboarding today carries abbreviation = id.slice(0,8).toUpperCase() (verified — crud.server.ts:53). T-188 P1 stops new ones; this step fixes the records that already exist, assigning a real key (owner- or bookkeeper-chosen, uniqueness-checked) and keeping the placeholder as a resolvable alias so nothing pointing at the old key breaks.
Re-point every reader from the user-doc person-facts to the Individual, so removal in step 1 is safe. Known consumers (grep-verified per the brief): UserProfileDrawer, ProfileApp, StaffDirectoryContent, ReceiptsTab, pages/api/users/[uid], pages/api/auth/staff-directory, pages/api/auth/profile, pages/api/receipts/index, lib/people/roster, and the resolver lib/individuals/personalFields.ts itself (which loses its legacy fallback at the end).
Chinese name flat → map (the name-field naming convention, T-188 2026-07-30). Today the Chinese name is two flat fields legal.chineseLastName + legal.chineseFirstName; the convention folds them into legal.chineseName.{ firstName, lastName } to match legalName's shape. Blast radius verified 2026-07-30: ~30 references across 6 files (chineseName.ts pure helper, payeeOverlay.ts, crud.server.ts, merge/compose.ts, types.ts, components/contacts/IndividualsContent.tsx) plus the stored-data move. legalName, preferredName need NO migration — they already conform.
Structure the preferred name into a preferredName map (owner, 2026-07-30, REVERSING the earlier same-day "consolidate to a single string": "let's have preferred first name and last name then"). The end state is basic.preferredName.{ firstName, lastName } — a map, matching legalName / chineseName per the T-188 naming convention. Migration per record:
the legacy flat basic.preferredFirstName / preferredLastName move into the map (preferredName.firstName / .lastName);
where only the single-string basic.preferredName exists (onboarding filler), its value seeds preferredName.firstName;
the field's type changes string → map on the same key — the migration writes the map shape and removes the two flat fields. Caveat (owner's own never-silently-overwrite rule): where a record carries BOTH a meaningful single-string preferredName AND a flat preferredFirstName that differ, surface it in the conflict list rather than guess which seeds .firstName. Blast radius to carry into the P2 composer follow-up: preferredName is read as a string in several places (PersonDetailsDrawer, ClientCompanyContent, IndividualsContent, the naming adapter/composer) — all move to the map shape.

The order is the safety (non-negotiable)¶

Expand → migrate → contract, and never sort on a field before every document carries it:

Expand — write the new location, keep the old; readers prefer new, fall back to old.
Backfill — populate the new location on every existing document.
Verify — confirm 100% coverage before anything reads the new location authoritatively or sorts on it. This is not optional: Firestore omits documents missing a sort field rather than erroring, so a premature switch produces empty lists and alerts reaching nobody, silently. The users indexes on history.createdAt were created ahead of time (T-188, 31a19bb2) for exactly this — the index is READY, waiting for the field.
Re-point — flip readers to the new location.
Contract — remove the old fields (the structure change; its own owner approval).

A conflict list (step 1) is produced and resolved before the contract of any field it covers.

Blocked on (two owner approvals, each its own question)¶

invitations/{id} — the invite / invitee / access / trail map layout (T-188, put to the owner 2026-07-27). Needed for the invite-time identity that step 4's future half depends on.
users/{uid} + individuals/{id} — the field layout the owner is composing. Needed for steps 1–3.

Also downstream of T-183 (the PaymentDetailsCard + /api/profile/payment-details surface T-188 absorbs), per the brief.

Verification per step¶

Unfiltered tsc (confirm npx tsc --version matches the pinned 5.x — a fresh container can pull TS 6 which aborts before checking, see T-188), vitest green, eslint at baseline, one PR per step, a dry-run that reports the planned writes and the conflict list before any document is written, and deploy/run only on explicit owner command. A migration script is reviewable code, not a console action.

BACKFILL RAN (2026-07-30) — applied, verified, one bug caught & fixed¶

Owner: "Run the backfill first, then create the PR." Ran scripts/t195-backfill.ts with T195_APPLY=1 against live aote-system. Applied the additive new-map writes to 29/29 individuals + 5/5 users (100% coverage, verified via individualIsMigrated/userIsMigrated), including the owner's JC phone/WhatsApp resolution (phone 5501 7113, WhatsApp 6469 0691). All new maps are new keys (access/telegram/history on users; company/chineseName/history on individuals); the old flat fields were left intact.

⚠️ Bug I introduced and fixed same-session (full transparency): the engine originally wrote basic.preferredName (the nickname) as a {firstName,lastName} map inside the additive writes — but that REUSES an existing key whose type changes string→map, and the deployed app reads basic.preferredName as a string (PersonDetailsDrawer calls .charAt()/.trim() on it and renders it as a React child). So the backfill briefly turned a string into a map on a key the live Contacts UI would crash on. Caught immediately on post-run verification; restored all 29 preferredName fields to their original strings (recovered losslessly from the map's firstName, e.g. "Carrie Tsui", "Jeffero Chan"). Then fixed the engine: preferredName's string→map conversion is now a contractWrites op (a new plan category for same-key type changes), applied atomically with the app deploy that reads the map — never by the additive backfill. A test locks this in.

Current prod state: a clean additive expand — new maps present as new keys, every old field intact, preferredName a string. The live (old) deploy is unaffected. Nothing was lost.

DRY-RUN RESULT + owner conflict resolutions (2026-07-30)¶

Phase-2 dry-run (read-only, scripts/t195-migration-dryrun.ts) over live aote-system: 5 users, 29 individuals (5 linked, 24 login-less contacts), 0 orphans. The entire conflict/review list was two items, both owner-resolved:

organization review — one account held free-text organization: "Establish Records Limited" (= the ERL subsidiary, i.e. the employer, already stored as primarySubsidiary). Owner (2026-07-30): discard. Not carried forward.
phone conflict on individual oS1b0stJGPc0aCoFEb5N (Jeffero Chan / me@jefferochan.com) — the directory record's basic.phone held 6469 0691 while the account held phone 5501 7113 and WhatsApp 6469 0691. Owner (2026-07-30): "5501 7113 is phone while 6469 0691 is WhatsApp" — the directory record has them swapped. Resolution (overrides the default keep-individual): the backfill sets basic.phone = { countryCode:'852', number:'55017113' } and basic.whatsapp = '+85264690691' for this record — an explicit, auditable per-record override in the backfill script (RESOLUTIONS map keyed by docId), not a silent clobber.

Both are encoded for the phase-3 backfill; no other record needs a human decision.

APPROVED SCHEMA (2026-07-30) — gates 1 & 2, owner-reviewed field-by-field¶

Owner reviewed the whole-document before/after for both person records and approved ("proceed for the whole thing", 2026-07-30). This is the AGENTS.md gate satisfied for these two documents. The invitation (invitations/{id}) is a separate gate, not yet drawn.

`users/{uid}` (aote-system) — AFTER¶

uid, email, displayName, photoURL, onboardingCompleted?, notificationPrefs?     (top-level, unchanged)
access   { role, roles?, badges?, status, subsidiaryAccess, primarySubsidiary?, vendorAccess?,
           lineManagerUid? (until T-190), expiresAt? (was accessExpiresAt), grantedAt?, grantedBy?, notes? }
telegram { linkAllowed?, id?, username?, linkStatus?, linkApprovedAt?, linkApprovedBy? }   (de-prefixed)
history  { createdAt, approvedAt?, approvedBy?, lastLoginAt?, updatedAt?, updatedBy? }
− firstName, lastName            → dropped (individual's name is authoritative)
− jobTitle                       → merges to individuals.system.position
− organization, isExternal, phoneNumber, whatsappNumber, contactNotes,
  preferredContactMethod, groups → move to individuals (below); `organization` NOT re-created (review-list)

`individuals/{id}` (aote-system) — AFTER¶

id, abbreviation, subsidiaries                                                   (top-level, unchanged)
basic   ├ preferredName { firstName, lastName }         (was string → map)
        ├ namePreferences?, sex, email, phone, whatsapp, birthDate, address, aliases   (unchanged)
        ├ contactNotes?, preferredContactMethod?, groups?     (arrive from users)
        − preferredFirstName, preferredLastName          (fold into preferredName)
legal   ├ legalName { title, firstName, lastName } | null                        (unchanged)
        ├ chineseName { firstName, lastName } | null      (was flat chineseLastName/chineseFirstName)
        ├ hkid, maritalStatus, spouseName, spouseIdNumber, bankAccounts           (unchanged)
company ├ repOfCompanyIds  string[]        ~ MOVED from system  (companies this person represents)
        └ affiliatedCompanyId string|null  ~ MOVED from system  (company billed instead of them)
system  ├ relatedParty, director, userUid                                        (unchanged)
        ├ position?  ← users.jobTitle merges here (conflict-list if both set & differ)
        └ isExternal?  (arrives from users)
history { created {at,by}, updated? {at,by} }             (regrouped from top-level created/updated)

Migrating person-fact map (backfill spec): jobTitle→system.position (merge/conflict-list), phoneNumber→basic.phone (parse), whatsappNumber→basic.whatsapp, contactNotes/preferredContactMethod/ groups→basic.*, isExternal→system.isExternal, organization→ review-list (link an Organization or discard), firstName/lastName→ dropped. New map name company (owner's choice; holds the two company-link fields).

Execution plan (staged — expand → verify → contract)¶

Expand (code, safe): add the new nested shapes additively; dual-read adapters prefer new, fall back to old; writers write both. No field removed, behaviour-preserving. PR + tsc/tests.
Migration script with DRY-RUN: reports every planned write + the full conflict list, writes nothing. Owner reviews the dry-run output.
Backfill run (owner-gated): on the owner's explicit go after the dry-run, populate the new locations on every document. Verify 100% coverage.
Re-point readers to the new locations (the reader list in step 5 above).
Contract (owner-gated schema removal + deploy): remove the old flat fields once coverage is verified and readers are moved.

The two owner-gated points (the production backfill run, and the contract/deploy) are the safety design, not optional ceremony — a bulk rewrite of production person records happens only after a reviewed dry-run and on command.

EXPAND CUTOVER WIRED (2026-07-30) — dual-read + dual-write, flat-authoritative¶

Owner: "Do the rest as one big PR instead" / "Proceed up until a new deployment is required" / "proceed with caution." The application-side expand cutover is now wired in one PR. It is a behavioural no-op when deployed and fully reversible (no old field is removed). Merged un-deployed per the deploy-gate; the owner triggers the deploy.

Precedence REVISED — flat-authoritative, not "prefer new". Steps 1/213 above said readers "prefer new, fall back to old." That is unsafe here: the backfill already ran, but the live (old) deploy keeps mutating the flat fields without the reconcile writer until this PR deploys, so a new map can lag its flat field. On the auth path a stale access.role/access.status read new-first is an authorization bug. So the dual-read reads the FLAT (legacy) field first and the new map only once the flat field is deleted at contract. When the two agree (the steady state) both orders are identical — so this is still a no-op during expand — but a stale map can never win. It also needs no precedence flip at contract: the contract delete removes the flat field and reads fall through to the map. (lib/individuals/personRecordNormalize.ts, tests assert the both-present case.)

Wired: - Dual-read at the two boundary mappers so in-memory UserProfile / Individual shapes stay byte-identical: normalizeUserProfileRecord (lib/rbac/claims.ts) and toIndividual (lib/individuals/store.server.ts). basic.preferredName is left a string (its string→map is a contract-phase change, readPreferredName type-dispatches). - Dual-write — best-effort reconcile{User,Individual}Maps (personRecordReconcile.server.ts) re-derive the new maps from the just-written flat fields after every write site: user → create/update/updateAuth/getOrCreate×2 (claims.ts), login lastLoginAt ([...nextauth].ts), telegram link (telegram/link.ts); individual → create/updateIndividualServer (crud.server.ts). A failure logs and returns false — it never breaks the caller's write. The individual reconcile writes ONLY the self-derived maps (company/chineseName/history); the user→individual person-fact backfill is a one-time migration step, not a per-write op. Under flat-authoritative this dual-write is warmth only (reads never depend on it), so a few un-hooked paths (org rep-list edits changing system.repOfCompanyIds, a direct swapAbbreviation) are acceptable — the contract script's full re-derive (below) is the correctness net. - resolvePersonalFields re-point (personalFields.ts): the person-facts now fall back to the Individual (jobTitle→system.position, contactNotes, preferredContactMethod, groups, isExternal). Legacy (user-doc) value still wins while present, so display is unchanged during expand; the Individual becomes the source only once contract deletes the user-doc copy. The Individual type gained the relocated optional fields (basic.contactNotes/preferredContactMethod/groups, system.isExternal).

Contract-cleanup script written but NOT run (scripts/t195-contract-cleanup.ts, gated, dry-run by default). It is the destructive finalizer, run by the owner AFTER deploy + verify. Design: - Full re-derive before delete: per doc it re-computes every map from the current flat fields and writes them ATOMICALLY with the flat-field delete — so correctness does not depend on the reconcile writer having kept maps in sync, and a flat-authoritative reader never sees a deleted flat field before its map exists. - Conflict-gated: reuses the engine's conflict detection; if a person-fact drifted (individual value ≠ the user's current value about to be deleted) the apply aborts until resolved via RESOLUTIONS — never a silent overwrite. - preferredName second-gate: the string→map change (the incident field) is applied only with T195_CONTRACT_PREFERREDNAME=1, and MUST wait until a deploy has made every basic.preferredName reader map-safe. The rest of the contract is safe as soon as this flat-authoritative PR is deployed.

Deploy / run sequence (owner-gated): (1) deploy this PR — app reads flat-first, writes reconcile; (2) run t195-contract-cleanup.ts dry-run, review; (3) T195_CONTRACT_APPLY=1 to atomically refresh maps + delete the flat access/company/chineseName/history/person-facts (preferredName stays a string); (4) later, after deploying preferredName-map-safe readers, run again with T195_CONTRACT_PREFERREDNAME=1 to finish preferredName. Verify checks green: tsc clean, eslint clean, full vitest green except 4 pre-existing failures in workspace/billing/ingest (unrelated — reproduced on a clean tree).

preferredName readers made map-safe (2026-07-30, follow-up — collapses to ONE deploy cycle)¶

Owner: "proceed as rapid as you could so we could get this over with." To finish the whole migration in a single deploy cycle instead of two, the basic.preferredName string→map readers are now safe, so T195_CONTRACT_PREFERREDNAME=1 can run in the SAME contract pass as the rest.

A scout mapped every consumer of the individuals collection's preferredName. Most flow through toIndividual (the four store.server.ts functions) — a single normalizePreferredNameInPlace call there makes them all safe. Three readers bypass toIndividual and were fixed at the source: pages/api/profile/name-preferences.ts (→ ProfileApp, would hard-crash on the map), lib/individuals/payeeOverlay.server.ts (injects the map onto payee rows → several accounting typeaheads), and lib/naming/greetingResolver.server.ts (renderCombination .trim() on a map). The shared helper normalizePreferredNameInPlace (personRecordNormalize.ts) folds the contract-phase {firstName,lastName} map into a composed display string + the split fields, and is a pure no-op while the value is still a string — so it changes nothing during expand and only takes effect once the contract writes the map. The payee directory's OWN preferredName, the naming-token identifiers, and the directors registry are different fields and were left untouched (scout-verified).

Net: T195_CONTRACT_PREFERREDNAME=1 no longer needs its own deploy — one deploy of main carries the whole cutover, then the full contract (maps + all flat deletes + preferredName→map) runs in one pass.

DONE — deployed & contracted, verified in production (2026-07-31)¶

✅ Read AGENTS.md. Attestation of completion of the person-record migration.

Outcome (verdict). The migration is live and complete. On the owner's explicit Deploy command, main @ 3c1bb26 was deployed to Vercel (deploy hook fired via the Secret-Manager VERCEL_DEPLOY_HOOK, fetched with the sandbox Google SA; confirmed live because the production build watermark read 3c1bb26). With the reader-safe code live, scripts/t195-contract-cleanup.ts ran with T195_CONTRACT_APPLY=1 T195_CONTRACT_PREFERREDNAME=1: 34 documents finalized — 29 individuals (123 map-writes incl. the 29 preferredName string→map conversions, 154 flat-deletes) + 5 users (71 map-writes, 87 flat-deletes), zero conflicts. Read-back: 5/5 users and 29/29 individuals clean (no leftover flat fields; access/telegram/ history + company/chineseName + preferredName maps present; role/status resolve off the maps), production served HTTP 200 throughout. Three individuals had empty leftover flat keys (legal.chinese*; one also basic.preferred{First,Last}) that the engine skips — it only deletes flat fields that HELD a value — so they were removed separately after confirming each was empty; re-verify = 0 problems.

Firestore structure — now the LIVE shape. users/{uid} and individuals/{id} in aote-system now carry the grouped maps and no longer carry the old flat fields. The whole-document before→after is the "APPROVED SCHEMA (2026-07-30)" section above; this records it is now in production.

Known follow-up (NOT a bug — durability). The read side is fully migrated, but the WRITE paths still emit the old flat shape and the reconcile writer mirrors it into the maps. Because the dual-read is flat-authoritative, values are always correct — but editing a record re-creates a few of its old flat fields (auto-mirrored to the maps). To make the grouped-map layout permanently flat-free, a later pass should re-point the writers (updateUserProfile / updateUserDirectoryFields / telegram link / individual CRUD) to write the map keys directly. Optional; nothing is broken without it.

Blast radius. users/{uid} + individuals/{id} in aote-system — auth/RBAC reads, Contacts, staff directory, profile name-preferences, payee-directory overlay, email greetings. All routed through the boundary normalizers (normalizeUserProfileRecord, toIndividual) + the 4 raw-read preferredName fixes, so consumers see the same in-memory shapes; only the stored layout changed.

Commits (append-only, oldest→newest): e8c97ef 01e0827 0cef96c b49e902 a618ba0 0f593db 7d190c5 7b7f63f fdf187c 51e261d ed50b26 5d77a4a 71c7e03 3c1bb26 + this close-out commit. Data ops (not commits): backfill applied 2026-07-30; contract applied 2026-07-31.

Source: User Management · https://claude.ai/code/session_01GGT5n9vCxKWoUQSRAfMSiW

Follow-up (2026-07-31): query regression fixed + "no adaptor" writer re-point¶

Two things surfaced after the contract:

Query regression (fixed, PR #969, deployed). Firestore QUERIES read stored field names, which the read-normalizer does NOT cover — so listUsers (orderBy createdAt / where status) returned an empty admin list and the telegram bot's allowlist (telegramId/telegramLinkStatus) matched nobody. Re-pointed the queries to history.createdAt / access.status / telegram.*, fixed countUsersByStatus, and replaced the stale users (status, …) index with (access.status, history.createdAt). Web-app login was never affected (it resolves by email/uid, not these queries). Lesson: re-point QUERIES, not just value-reads, before a contract.
No-adaptor writer re-point (owner, 2026-07-31: "No adaptor"). Convert every writer to persist the grouped shape DIRECTLY and delete the reconcile mirror:
Stage 1 — users (PR #981, merged + deployed db5114b): access/telegram/history writes go native via lib/rbac/userDocShape.ts (toUserDocUpdate); reconcileUserMaps calls removed. Round-trip test locks write-native → read-flat.
Stage 2 — individuals (PR #982, merged): create/updateIndividualServer + swapAbbreviation write legal.chineseName / basic.preferredName / history.* maps natively via a server write-boundary normalizer (lib/individuals/individualWriteShape.ts) so the Contacts/Profile forms need no change; a pass-through test proves it never drops other fields (e.g. bankAccounts). The reconcile FILE (personRecordReconcile.server.ts) is deleted — the mirror is gone.
Stage 3 — person-facts + the rest (PR: this one): every remaining writer is native. (a) Profile person-facts now write to the bridged individual — updateUserAuthProfile routes jobTitle→system.position, phone→basic.phone (parsed; a +-country-code is required at the API, never guessed), whatsapp/contactNotes/preferredContactMethod/groups→basic.*, isExternal→system.isExternal (routeSelfProfileUpdates in lib/rbac/userDocShape.ts; self-heals a missing bridge via ensureIndividualForUser; falls back to a flat user-doc write rather than ever dropping an edit). (b) Fixed a second contract regression: the Profile page reads GET /api/auth/profile, which served the raw user doc — its contact card (job title / phone / …) had been BLANK since the contract; the route now resolves person-facts from the individual (resolvePersonalFields). (c) Org rep writes now maintain company.repOfCompanyIds with a transactional self-healing MERGE (stale flat copy ∪ map ± the org; flat key deleted) instead of arrayUnion on the flat key. (d) payee→individual sync and the director seed persist native sections; updateIndividualServer strips a section echo of the org-managed company links. (e) Contacts editor writes drop any stale flat updated.
The ONLY person-fact still on the user doc: the free-text organization — its Profile-box fate awaits the owner's ruling (drop the box / keep as free text on the individual / turn into an Organization picker). Recommendation on record: drop the box. Everything else is native.