Skip to content

Migrate the document store aote-system/file-archive → tebs-erl

Goal

Move the unified document store (receipts, vendor invoices, contracts) out of aote-system/file-archive/documents/entries into ERL's own database tebs-erl, and drop the redundant file-archive wrapper. Folders move too (tebs-erl/folders/entries). viewerLinks stay in aote-system but record which DB+path their target doc lives in.

Owner decision (2026-06-13): documents are ERL legal records → store in ERL only, NOT fanned out per-subsidiary. Coaching invoices STAY in tebs-mel (domain- coherent with their source sessions; ERL accounting truth is delivered by the journal-derivation layer, not co-location). Project invoices already in tebs-erl. Supersedes the cosmetic-cleanup half of T-012.

Open

  • Confirm exact target path convention: their pattern is group/wrapperDoc/entries (e.g. accounting/transactions/entries); owner wrote "documents/entries" (2 segs) — propose documents/entries as a top-level collection OR documents/<wrapper>/entries.
  • Design the migration to fit T-022's Security-Rules shape (fields Rules match on) so we don't migrate twice. Sequence after/with T-022's claims+rules for documents.

Log

  • 2026-06-13 created from the data-residency discussion. Not started.

Decision log

2026-07-09 — assigned to EOP Local Assistance (fork)

  • ✅ Read AGENTS.md. Owner assignment (2026-07-09, verbatim): "I'll assign T-021 to you". Relevance re-verified 2026-07-09 during the backlog review: aote-system/file-archive is still read live from lib/accounting/billingDisplayTokens.server.ts (×3) + lib/accounting/folders.ts — the migration to tebs-erl remains real work. Queued behind the T-147 close-out; status flips to doing with a dated entry when work starts.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-09 — started: inventory + migration design (one owner confirmation needed)

  • ✅ Read AGENTS.md. Inventory (live, 2026-07-09): aote-system/file-archive/documents/entries = 90 docs, …/folders/entries = 5 docs; target locations in tebs-erl all empty. Code blast radius: 12 files — server feeds (receipts.ts, billingDisplayTokens.server.ts, vendorInvoiceFeed.server.ts, folders.ts), WOPC storage (lib/storage/wopc.ts, wopc.server.ts), lib/rbac/badges.ts, lib/share/shortLinks.server.ts, admin audit/survey endpoints, RecordsApp.tsx.
  • Plan: (1) migration script in the house style — JSON backup → dry-run → copy 95 docs to tebs-erl → per-doc verify; (2) repoint the 12 files in one tsc-clean commit; (3) old location left in place until the code deploys + verifies (deploys are manual), then a separate confirmed cleanup deletes it; (4) share-link safety (I-029 lesson): audit what viewerLinks/shortLinks.server.ts store — if stored refs embed the old DB/path, the resolver gets a dual-path fallback before cutover; (5) security-rules: add tebs-erl rules coverage for the new collections at cutover (T-022-shaped), since T-025 may convert these reads to the client SDK — coordination note to User Management: don't convert file-archive readers until this lands.
  • Owner confirmation needed (the task's own Open item — target path shape): proposal = flat top-level collections tebs-erl/documents/{docId} + tebs-erl/folders/{folderId} (truly drops the redundant wrapper, matches the flat house collections like clients/payees). Alternative = keep the 3-segment group/wrapper/entries house pattern (documents/main/entries). Awaiting the owner's pick before writing data.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-09 — SCOPE WIDENED (owner) + re-designed as residency-then-dissolve; census corrected

  • ✅ Read AGENTS.md. Owner approvals (2026-07-09, verbatim): the dissolve direction — "do you think it's even better for the file path or metadata information to live with the actually document firestore doc…?" — and the widening: "widen the scope please."
  • Census, fully classified (90 cards, all subsidiaryId: erl): 46 workspace_invoice · 20 gcp_statement · 12 "other" = Google Cloud monthly STATEMENTS (statement-as-invoice months, GCLOUD-STMT-*, T-053 model) · 9 gcp_invoice · 3 vendor_invoice · 0 receipts currently stored. Correction on the record: an earlier note here said receipts carry pointers inline with no archive card — wrong. lib/accounting/receipts.ts defines the receipt record AS a file-archive entry (the store's schema is receipt-shaped: telegram fields, match status); the archive simply holds no receipts yet. So the owner's "metadata on the entity's own doc" instinct is satisfied differently: for receipts/vendor uploads the card IS the entity doc — it just lives in the wrong database.
  • Re-design (two phases):
  • Phase 1 — pure residency (the original Jun-13 goal, widened): move ALL 90 cards + 5 folders as-is → tebs-erl/documents/{id} + tebs-erl/folders/{id} (flat, wrapper dropped). Includes the receipt store by construction (receipts land in the same collection going forward). Repoint the 12 reader/writer files + both ingest write-paths; share-link dual-resolution during transition; tebs-erl rules coverage; old location retained until the deploy verifies, then confirmed cleanup. T-025 coordination: User Management must not convert file-archive readers until this lands.
  • Phase 2 — dissolve into entity docs (the owner's design): merge file pointers onto the entity docs where they exist — gcp_invoice cards → Expenses/Subscriptions/Google/GCP/invoices/{number} (entity docs exist); GCP statements → their {monthKey} siblings; workspace_invoice (46) → create the missing Expenses/…/Workspace/invoices entity docs (mirroring the GCP pattern — today the card is the only record) + update the workspace ingest to write there. Standalone docs (receipts, vendor uploads, true orphans) remain entity-in-one-doc in tebs-erl/documents.
  • Phase 1 executes first (simple, reversible, achieves residency); Phase 2 rides on it per type.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-15 — Phase 1 SHIPPED: code repointed (5d17fc56) + data copied & verified

  • ✅ Read AGENTS.md. Owner go (verbatim): "Proceed."
  • Code (5d17fc56, tsc clean): new lib/accounting/documentStore.server.ts = single source of truth for the store's location (flat tebs-erl/documents + tebs-erl/folders); all readers/writers repointed through it — receipts.ts, folders.ts (incl. two cross-database batch fixes: batches now created on the database the refs live in), billingDisplayTokens.server.ts (×3), admin/file-archive-audit, admin/file-storage-survey. Doc IDs preserved (share links + WOPC fileId lookups unaffected). badges.ts "file-archive" is a badge SURFACE KEY — deliberately untouched. scripts/migrate-file-archive-to-erl.ts added (backup → dry-run → idempotent copy → per-doc deep verify).
  • Data: dry-run reviewed → executed: documents 90/90 · folders 5/5 copied, 0 mismatches; JSON backups in scripts/migration-backups/t021-*-2026-07-15*.json. Source left untouched and authoritative until the repointed code deploys.
  • Remaining sequence: (1) owner's one-off production deploy → (2) smoke: Records tabs (receipts / vendor & service invoices), previews, a /v/ share link, folder ops → (3) delta re-run of the script (catches any docs written to the old store during the gap; nightly scrape ingests are the only writers) → (4) CONFIRMED cleanup deletes aote-system/file-archive → (5) Phase 2 (dissolve into entity docs) per the 2026-07-09 design. T-025 flag stands: User Management must not convert these readers until deployed.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-18 — Phase-1 gate CLOSED + Phase 2a EXECUTED (additive dissolve)

  • ✅ Read AGENTS.md. Phase-1 gate: owner deployed (prod 129d0c16, Jul 16, includes 5d17fc56); smoke PASS — live archive-doc share links render off tebs-erl/documents (the one "not found" was broken BEFORE migration: its doc exists in neither store — 488 links accumulated vs 90 live cards); delta pass re-ran clean (source unchanged at 90+5, 0 mismatches). Old aote-system/file-archive is now INERT — deletion still awaits an explicit owner confirm.
  • Phase 2a (data, additive — owner go "Proceed to Phase 2"): scripts/t021-phase2-dissolve.ts dry-run → execute: 9 gcp_invoice cards merged onto their existing Expenses/…/GCP/invoices/{invoiceNumber} entity docs (file map + invoiceCsv + archiveDocId), and 46 Workspace invoice entity docs CREATED at Expenses/…/Workspace/invoices/{invoiceNumber} — Workspace finally has the same entity structure as GCP. 55/55 verified carrying a file map. Statements (32) + vendor invoices (3) + receipts stay entity-in-one-doc in tebs-erl/documents by design. Every touched entity records archiveDocId — the bridge for Phase-2c share-link resolution. (Note: ws invoice 5235091127 is the number behind the long-broken share link found in the smoke — candidates for resurrection in 2c.)
  • Remaining: Phase 2b — reader/writer rewrites (vendorInvoiceFeed workspace branch, billingDisplayTokens maps, ingest write-path writes entity+file directly); Phase 2c — retire the merged cards + archive-doc resolver falls through archiveDocId; old-store cleanup (owner confirm).
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-18 — Phase 2b slice 1 SHIPPED: display-token indexes read entity docs

  • ✅ Read AGENTS.md. Owner go: "Proceed." First 2b slice (self-contained, same output shape): documentStore.server.ts gains the vendor-invoice ENTITY accessors (getGcpInvoiceEntities / getWorkspaceInvoiceEntities — the module stays the single source of truth for record locations), and billingDisplayTokens.server.ts's two indexes (workspace invoiceNumber→monthKey, gcp month→ invoiceNumber) now read the entity collections instead of archive cards. GCP month-summary docs fall out via the existing invoiceCsv guard. GCLOUD-STMT reads unchanged (statements stay entity-in-one-doc). tsc clean.
  • Remaining 2b (one atomic bundle — must ship together): ingest write-path (entity doc instead of card for gcp/workspace invoice types + entity-based idempotency), vendorInvoiceFeed workspace branch
  • GCP pdf-link resolution (entity file map), and the archive-doc share-link resolver fallthrough (documents/{id} miss → GCP entity by id → Workspace entity by archiveDocId). Then 2c: retire the 55 merged cards; and the old-store deletion (owner confirm still pending).
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-21 — old store DELETED (owner-confirmed) + Phase-2b bundle SHIPPED + 2c prepped

  • ✅ Read AGENTS.md. Owner (verbatim): "Delete the inert old aote-system/file-archive and continue to proceed to 2b and 2c."
  • Old store deleted: verify-then-delete — destination re-checked (90+5, per-id presence ✓) before removing all 95 source docs + wrapper docs; residual 0+0. Rollback: dual JSON backups + PITR 7-day window.
  • Phase-2b bundle (this commit, tsc clean): all in the chokepoints so vendorInvoiceFeed needed no changes — receipts.ts: listWorkspaceInvoices/listGcpInvoices read the ENTITY collections; getReceipt gains the dissolved-entity fallthrough (documents/{id} miss → GCP entity by id → Workspace entity by id → by archiveDocId) so every archive-doc share link + WOPC fileId lookup survives card retirement; ingest.ts: gcp/workspace invoices now WRITE entity docs (invoiceCsv + file map, id = invoiceNumber) with entity-first idempotency — statements keep the entity-in-one-doc card path by design.
  • Phase 2c prepped, NOT executed: scripts/t021-phase2c-retire-cards.ts deletes the 55 dissolved cards only after per-card entity-twin verification, and refuses --execute without --deployed — production still reads cards until the 2b bundle deploys. Sequence: owner deploys → I run 2c + smoke → T-021 closes with Phase-3-style cleanup of leftover comments.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b