Skip to content

Consolidate + productionize the NAS billing scraper (schedule · re-auth/alert · decommission Cloud Run)

Why

The NAS GCP/Workspace billing scraper's remaining work was scattered across five items (T-033/T-034/T-035/T-036 + ticket I-024 + T-053's residual cron). This task consolidates them into one productionization effort so the work stops fragmenting (owner request, 2026-07-01).

The scraper's CORE is done + verified on the NAS (Cost-Table backfill, GCP invoice PDFs, Workspace CSV/PDF modes all run via Synology Task Scheduler). What's left is productionization + cleanup.

Consolidated scope (remaining work)

  1. Reliability / re-auth / failure-alert (from T-033 + I-024) — the Google session expires and the scrape fails silently (dark ~5 days, Jun 26 → Jul 1). Alert on scrape rc != 0 via the app's notification channel so a dark scrape is caught same-day; ease the noVNC re-auth.
  2. Multi-invoice capture (from T-033) — months with >1 invoice grab only the primary; enumerate invoice numbers → per-invoice URL-nav.
  3. Schedule all modes (from T-034/T-035/T-036) — confirm cost-table + gcp-invoices + workspace CSV/PDF are all on the Task Scheduler.
  4. Decommission the Cloud Run copy (from T-034) — pages/api/workspace/billing.ts:133 still hard-codes the Cloud Run workspace-billing-service URL as a fallback (service is 0 req/7d); retire it + drop the fallback once the NAS path is sole.
  5. Verify Workspace PDF/CSV operational + decommission (from T-035/T-036).

Decision log

2026-07-01 — opened (consolidation of the scraper cluster)

  • ✅ Read AGENTS.md. Created at the owner's request to unfragment the NAS-billing-scraper work. Absorbs the remaining productionization from T-033/034/035/036, escalates I-024 here, and covers T-053's residual cron; the folded tasks are closed/dropped with pointers here.
  • Proposed: EOP Local Assistance (fork). Approved: owner (2026-07-01, verbatim): "consolidate the scraper cluster please."
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-02 — scope #1 (failure-alert) built, app-side

  • ✅ Read AGENTS.md. Built the app-side half of the scrape failure-alert (scope #1 / I-024): (a) pages/api/workspace/billing/ingest.ts now stamps systemHealth/scrape-gcp-billing.lastIngestAt on every ingest (best-effort); (b) new pages/api/cron/scrape-heartbeat.ts alerts the owner (notifySystemOwner, self-clearing on recovery) if no ingest has landed in 36h; (c) new .github/workflows/scrape-heartbeat.yml pings it every 6h. tsc clean. Note: the alert uses the data_integrity channel (consistent with the reconciliation watchdog), so a dark scrape also flags in the Inspect-Period close gate — owner to confirm that's wanted vs a non-gating type. Remaining T-147: multi-invoice capture, decommission the Cloud Run copy (billing.ts:133), verify schedules; and the operational noVNC re-login is the owner's.
  • Source (edit): EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-02 — re-auth landed + a SECOND break found & fixed (dead p-eop ingest host)

  • ✅ Read AGENTS.md. Ran the scope-#1 recovery + the owner's catch-up, and found the "dark scrape" had two independent causes, not one:
  • Expired Google session (the known one) — fixed by re-login in the scraper's own Chrome profile via its login mode (noVNC :6080); a verify run then authenticated (rc=0, downloaded 8/8).
  • Dead ingest host (new) — all three scraper modes defaulted their ingest POST to p-eop.theestablishers.com, the NAS preview host dropped 2026-06-22 when hosting moved back to Vercel. No override env was set, so every ingest hit a dead host (fetch failed/HTTP 000) — GCP billing data had actually stopped landing ~06-22, days before the auth expiry, even while downloads kept working. Fixed both layers: (a) NAS runtime overrides GCP_INGEST_URL + GCP_COST_TABLE_INGEST_URL → eop.theestablishers.com (ingest.env / gcp-cost-table.env, backups saved); (b) the baked code defaults, commit 7989ef75 on main, so an image rebuild can't reintroduce it.
  • Verified: gcp-invoices now ingests 8/8 → HTTP 200 against eop.theestablishers.com (all "exists" — the 8 invoices May 2026→Oct 2025 were already in Firestore; latest issued is May 31 2026, so the dark window dropped no new invoice; June's lands when GCP issues it).
  • Validates scope #1: the heartbeat I built watches systemHealth/…lastIngestAt, which catches the p-eop failure mode (rc=0 but 0 ingested), not just the rc!=0 auth case. Still needs deploying.
  • New defect → I-026: gcp-cost-table authenticates but can't select the billing-period dropdown ("Could not select invoice month June 2026") — mode-specific DOM drift, unrelated to auth/p-eop.
  • Remaining T-147: deploy the heartbeat; multi-invoice capture; decommission the Cloud Run copy (billing.ts:133); verify schedules; fix cost-table selection (I-026).
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-02 — cost-table BigQuery write gap root-caused + write-identity decided (option 1)

  • ✅ Read AGENTS.md. Beyond the selection bug (I-026), the cost-table ingest was also failing its BigQuery write. Root cause: it writes a derived table (gcp_billing_cost_table_backfill) into aote-pms:gcp_billing_export, but on Vercel prod runs as bq-billing-reader@ (read-only) — GCP_BILLING_CLIENT_EMAIL is blank so it falls back to GOOGLE_CLIENT_EMAIL = the reader (confirmed via BigQuery audit log). Collateral from the pre-task-log SA refactor ([[project_sa_audit_aote_pms]] / the Service Account (Local) agent): moving the billing identity off owner-scoped service@ onto a least-privilege reader was right, but the one write path wasn't carried over. (service@ is NOT stripped — still has Owner; downgrade is deferred.)
  • Decision — option 1 (owner-approved): derived/computed tables move to their own dataset (gcp_billing_derived); raw Google-managed export stays read-only for every app identity. Owner approval (2026-07-02, verbatim): "I have picked option 1 (Best End-State)." Proposed by the Service Account (Local) agent (§5, ranked); seconded by EOP Local Assistance (fork).
  • Landed (code): 4f5e334c — cost-table backfill now targets GCP_BILLING_DERIVED_DATASET_ID, falling back to GCP_BILLING_DATASET_ID until infra exists (no behavior change until set). tsc clean.
  • Remaining (SA/infra → T-148): Service Account (Local) to create gcp_billing_derived, mint a scoped writer (read on gcp_billing_export + dataEditor on gcp_billing_derived), set the Vercel env (GCP_BILLING_DERIVED_DATASET_ID + the writer's GCP_BILLING_CLIENT_EMAIL/_PRIVATE_KEY), redeploy. T-148 also captures the full post-refactor SA→workload map + the deferred service@ strip — this write path is on the critical path to service@ deletion.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-02 — cost-table BigQuery write gap CLOSED end-to-end (verified live)

  • ✅ Read AGENTS.md. The option-1 write-identity fix is live + verified end-to-end. Owner provisioned the infra (created gcp_billing_derived, granted bq-billing-reader@ WRITER on it, pinned GCP_BILLING_DERIVED_DATASET_ID=gcp_billing_derived in prod); the code half (4f5e334c) was already live in production (476968cd) — no redeploy needed.
  • Verified with a real ingest: live cost-table ingest (June 2026, invoice 5626347939) → POST /api/gcp-billing/cost-table-ingest returned HTTP 200, bigQueryRows: 94, deletedCostTableRows: 0. Rows in gcp_billing_derived.gcp_billing_cost_table_backfill: 711 → 805 (202606 = 94, on top of the historical months). Raw gcp_billing_export stays read-only for every app identity — option-1 intent achieved.
  • Blocker cleared for T-130 / I-024: the last BigQuery write path is now off the over-privileged service@ and scoped to the derived dataset. The SA-retirement critical path (fix write identity → strip service@ → soak → delete) can proceed to the strip.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-02 — scrape watchdog FALSE-alarm fixed (cost-table path now stamps the marker)

  • ✅ Read AGENTS.md. The scope-#1 watchdog emailed "GCP billing scraper is not delivering — never delivered data" — a false alarm. Root cause: it reads systemHealth/scrape-gcp-billing.lastIngestAt, stamped ONLY by the invoices ingest (workspace/billing/ingest.ts), not the cost-table ingest — and the marker doc didn't exist (no invoices ingest had stamped it post-deploy). Meanwhile the cost-table ingest was verified delivering (94 rows → gcp_billing_derived) but didn't register.
  • Fix (0e2dd0d5): pages/api/gcp-billing/cost-table-ingest.ts now stamps lastIngestAt on every successful ingest, so any live billing-delivery path keeps the watchdog fresh (not just invoices). tsc clean. Immediate relief: manually stamped the marker to reflect the verified delivery; the watchdog self-clears the alert on its next run (~12:30 UTC). Deploy needed for the code stamp to take effect on future ingests.
  • Adjacent flag (→ "verify schedules" in this task): the marker being 404 means the daily scrape hasn't been stamping it — worth confirming the invoices/cost-table modes are actually scheduled + running on the NAS, else the marker goes stale between manual runs.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-07 — session expired AGAIN (~3.5 days); schedules + watchdog verified working; re-auth prepped

  • ✅ Read AGENTS.md. The scrape-heartbeat emailed the owner ("last ingest 76h ago") — a TRUE positive this time. Diagnosed via the DSM FileStation channel (NAS Tailscale was flapping; SSH worked over DERP): cost-table-cron.log shows clean end-to-end nightly runs Jul 3 + Jul 4 (ingest HTTP 200, 94 rows) then "NOT authenticated — run login mode first" on Jul 5/6/7 — the Google session from the Jul-1 noVNC re-login died after ~3.5 days (previous session lasted ~16; Google's sign-in URL carries passive=1209600 = 14-day nominal window, so early invalidation → likely a Workspace session-control policy or step-up). Confirmed working: the owner's daily cost-table DSM task (scope #3 ✓ for cost-table), the watchdog (fires correctly, self-clears on recovery), and no data loss — June's invoice
  • cost-table are fully ingested; July's invoice isn't due until month-end.
  • Re-auth prepped: gcp-login container started (noVNC :6080) for the owner's re-login.
  • Scope #1 residual sharpened: "ease the noVNC re-auth" is now the critical piece — recurrence is ~days, not weeks. Candidate fixes (owner to choose): (a) Workspace Admin → Security → Google session control: extend/never-expire the web session for the scraper account/OU; (b) dedicated minimal-rights billing-viewer Google account in a never-expire OU; (c) de-browser the pipeline — BigQuery export for cost data + revive the dormant gmail-invoice-fetcher@ (DWD gmail.readonly; GCP emails invoice PDFs monthly) so no interactive Google session is needed at all.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-07 — SMOKING GUN: the Jul-5 session death was a SELF-LOGOUT (concurrent-profile race), not policy

  • ✅ Read AGENTS.md. Owner pulled the Workspace login-audit CSV: Logout at 2026-07-05T00:01:17+08:00 from the NAS egress IP (203.145.89.7) — the exact minute the nightly scrape jobs run. No suspicious flag, no challenge, no policy expiry. The statements job (00:00) and the cost-table job (~00:00/00:01) run concurrently against the SAME Chrome profile (csv-profile); two Chromes racing one cookie DB killed the session (the drafted cost-table task specified 00:30 for exactly this reason; it was scheduled at ~00:00). Worked Jul 3–4 by race luck, died Jul 5.
  • Two confirmed causes, two fixes: (1) Jun-26 death (~16d) = the 14-day Cloud Identity Free cap → owner removing the Cloud Identity Free license so "Session never expires" (already set at 'The Establish Group') governs me@jefferochan.com — license changes are owner-console actions, no API path delegated; (2) Jul-5 death (~3.5d) = the schedule overlap → owner staggering the DSM tasks (statements 00:00 · invoices 00:20 · cost-table 00:40); jobs must never share the profile concurrently.
  • Post-fix expectation: possibly ONE more re-login (new session-duration applies only after the current session ends), then steady state. The watchdog remains the tripwire.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-07 — FIX SHIPPED: profile lock serializes all scrape runs (schedule-independent)

  • ✅ Read AGENTS.md. Instead of relying on the owner re-staggering DSM task times, made overlap impossible at the script layer: all three NAS wrappers (run-gcp-statements.sh, run-gcp-cost-table.sh, run-gcp-invoices.sh) now take an exclusive flock on .scrape-profile.lock before starting Chrome (45-min wait, then abort rc=3). Concurrent jobs serialize regardless of what the Task Scheduler fires — kills the 2026-07-05 self-logout race class entirely, including against future added jobs. Verified: lock semantics proven live (second locker blocks while held, acquires after release); all three scripts syntax-checked; .bak-prelock backups saved beside them.
  • License angle likely moot: Cloud Identity Free can't be cancelled (Google), and the owner appears unassigned (50/50 seats free). Owner to eyeball Directory → Users → me@jefferochan.com → Licenses; if Business Plus, "Session never expires" (already set) governs. Note the June ~16-day death may ALSO have been this race (statements 00:00 + invoices 00:01 both existed then) — the lock may be the whole fix. The watchdog adjudicates over the coming weeks; owner re-logs on next alert (their call, 2026-07-07: "Let's login again when we should"). Jul-5 Japan logins confirmed owner (VPN/travel).
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-07 — NAS Task Scheduler audit (owner-requested): 9 live tasks, 4 findings

  • ✅ Read AGENTS.md. Enumerated every DSM scheduled task (via /etc/crontab + the synoschedule.d store read through the docker mount path — scheduler DB itself is root-only). 9 live tasks; ids 7–17 are deleted remnants. Keep: id=1 DSM Auto Update, id=2 Security Advisor, id=6 GCP Cost Table daily (lock-guarded), id=19 reconciliation-heartbeat, id=20 dmarc-heartbeat. Findings:
  • id=5 t053a-build — footgun, delete: the June one-off image build is armed as a YEARLY cron (every Jun 17) that would rebuild workspace-csv-scraper:latest from the NAS source dir, silently reverting image-level fixes (e.g. the dead-p-eop repoint).
  • id=18 t084-ssh-fix — delete: daily re-assertion of a stale 2026-06-22 cloud-session SSH key onto the Claude account; the channel is superseded (FileStation API needs no SSH; root-exec is T-096-blocked). Owner to also remove the stale authorized_keys line.
  • id=4 t053a-gcp-statements-backfill — repurpose: runs DRY (posts nothing) while NO scheduled LIVE invoices ingest exists — the real schedule gap. Recommended cmd swap → run-gcp-invoices.sh (idempotent, lock-guarded, stamps the watchdog marker).
  • id=3 Workspace CSV scrape (monthly) — broken, disable: raw docker run … scrape with no env-file (no ingest secret → nothing posts) on an image whose baked ingest default is still dead p-eop; also touches the shared profile outside the flock. This empirically settles scope #5 (Workspace CSV is NON-operational) — rehabilitate (wrapper + env + image rebuild) or decommission.
  • Scheduler create/set/delete isn't exposed to agents — the four actions are the owner's DSM-UI clicks.
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-07 — scope #5 REHABILITATED: Workspace billing scrape delivering again (first since Jun 9)

  • ✅ Read AGENTS.md. Owner chose rehabilitate ("Rehabilitate it as it is important and it matters"). Evidence first: Workspace invoices path in tebs-erl = 0 docs ever; the transactions feed (aote-system/workspace-billing/transactions/entries) = 11 docs all written Jun 9 — the monthly task (id=3) had never delivered in its current form (raw image, no env-file → no ingest secret, stale baked p-eop default).
  • Fix: new NAS wrapper run-workspace-scrape.sh — flock-serialized on the shared profile, passes ingest.env + explicit live INGEST_URL, DRY_RUN overridable. Two auth wrinkles surfaced: (a) admin.google.com is a step-up surface — it demanded a password re-challenge even on a same-day session (likely the original silent killer); owner cleared it via noVNC login mode. (b) implication for the dedicated-gmail cutover: the Workspace scrape cannot move to the gmail identity (admin-only surface) → end-state is a two-profile split (gmail profile: GCP billing; Workspace-admin profile: this scrape).
  • Verified live: DRY authenticated + parsed; LIVE run persisted 10 rows (0 new, 10 updated) covering May–Jul 2026 — data current through Jul 6. Bonus finding from the data: the org pays Business Plus × 2 seats (re-check the "everyone on Cloud Identity Free" read — the session-control never-expire may already cover the scraper account).
  • Owner's remaining DSM clicks: swap id=3 cmd → bash /volume1/docker/workspace-billing/run-workspace-scrape.sh >> /volume1/docker/workspace-billing/workspace-cron.log 2>&1 (keep monthly 6th 03:00); swap id=4 cmd → run-gcp-invoices.sh; delete id=5 + id=18 (audit).
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-21 — PHASE 1 SHIPPED: de-browsered billing pipeline (candidate (c) chosen + built)

  • ✅ Read AGENTS.md · checked the board by scope (this IS the scraper task; no duplicate opened) · tracking T-147.
  • Trigger: the scrape-heartbeat emailed the owner again (Jul 18, "last ingest 68h ago") — the Google session died a THIRD time (~Jul 15, ~8 days after the Jul-7 flock fix, so the race wasn't the whole story). Owner (verbatim): "can we find a solution so that things could be automated and that I can hand it off and not needing to repeat this manual step ever again?" → scope-#1 resolved by eliminating the browser dependency (the Jul-7 entry's candidate (c)), not extending the session.
  • Mailbox probe (read-only, gmail-invoice-fetcher@ DWD, owner-authorized): ~120 billing emails from payments-noreply@google.com. Workspace: the monthly "Your invoice is available" email carries the invoice PDF as an attachment (verified: 5609520523.pdf, Jul-1) + number in body → fully email-automatable. GCP: statements are NOT emailed for this self-serve account (exhaustive sender/ subject sweep; payment receipts carry amount only) → statement PDFs become on-demand pulls (a few times a year, records-driven), cost DETAIL comes from the BigQuery export (keyless, already live).
  • Infra staged (agent-executed): minted gmail-invoice-fetcher@ user key 52412817…; owner added the DWD grant (client 115562755728083115762, gmail.readonly) in admin.google.com; enabled gmail.googleapis.com on aote-pms; set GMAIL_FETCHER_CLIENT_EMAIL/_PRIVATE_KEY + GMAIL_USER_EMAIL in Vercel prod+preview. Scoped key, 0 project IAM — per the owner's 2026-07-02 web-app guardrail; no GOOGLE_* fallback in the new code by design.
  • Code shipped (this commit): lib/gcpBilling/emailBillingSync.server.ts (Gmail → parse PDF via pdf-parse → POST the exact NAS body shape to /api/workspace/billing/ingest, so Drive filing + idempotency + mismatch policy are unchanged); pages/api/cron/billing-pipeline-sync.ts (daily: email leg + syncRecentMonths(2) BQ leg; stamps billing-email-sync + scrape-gcp-cost-table/legacy markers); .github/workflows/billing-pipeline-sync.yml (daily 01:30 UTC, dmarc-pattern); watchdog + system-health repointed (cost-table remedy → BQ/cron; gcp-invoices pipeline entry removed — on-demand pulls carry no freshness expectation; new billing-email-sync pipeline, 36h). Heavy tsc --noEmit (8 GB): clean, exit 0.
  • Supersedes (forward): the Jul-7 rec to arm id=4 → run-gcp-invoices.sh — do NOT arm it; GCP statement pulls are on-demand now. Jul-7's two-profile-split end-state note stays relevant only for the Workspace transactions CSV scrape (admin.google.com), which is Phase 2.
  • Interim verify (browser path): owner re-logged in via noVNC (Jul 21) but the follow-up cost-table run STILL hit a Google signin/challenge/pwd step-up on console.cloud.google.com (login default URL is admin.google.com; Cloud Console challenges separately) — a live demonstration of why the browser path can't be productionized. Not chased further: June is fully ingested, July isn't finalized, and the BQ leg covers freshness from first cron run. No data gap.
  • Owner's decommission clicks (DSM UI, after deploy + first green cron run): disable id=6 (GCP Cost Table daily); leave id=4 DRY/unarmed; id=3 workspace monthly STAYS (Phase 2); id=5 + id=18 deletions from the Jul-7 audit still stand. run-gcp-*.sh + login mode stay on disk for on-demand statement pulls.
  • Remaining: deploy (manual, owner command — auto-deploy off) → fire the workflow_dispatch once to arm the billing-email-sync marker + backfill the first-ever Workspace invoice docs (probe showed Jul-1 + Jun-2 in the 45d window) → then the DSM clicks above. Phase 2 (kills the last browser use): Workspace transactions via workspace-admin@ Admin SDK / licensing API + this email fetcher.
  • Source: Service Account (Local) · local session ca159e39-42a3-4569-b1cd-6e006f6639c3

2026-07-07 — watchdog re-architected: per-pipeline + Vercel Cron, ZERO GitHub crons (391a0273)

  • ✅ Read AGENTS.md. Owner policy set this turn (verbatim): "1. I'd like the watchdog to work without existing as a cron job on GitHub 2. I'd like all cron jobs on GitHub to be moved to my NAS as scheduled task 3. I'd still rather for the watchdog to send me emails as thru the system email routing instead of it being from Synology."
  • Placement principle honored both ways: a watchdog must not live on the box it watches — the scrape watchdog (watches the NAS) moved to Vercel Cron (vercel.json crons, 6h; Bearer CRON_SECRET auto-sent); the reconciliation/DMARC heartbeats (watch the app) STAY on the NAS. GitHub is out of both loops; all alert email flows through notifySystemOwner → Resend (never Synology SMTP).
  • Per-pipeline coverage (closes the masking gap): gcp-cost-table 36h (legacy-marker fallback → no false alarm on rollout), gcp-invoices 36h (arms itself on first successful ingest), workspace 40d (freshness read from the delivered data itself). Per-pipe alert entities + self-clear on recovery. Ingest endpoints stamp per-pipeline markers + the legacy shared one.
  • GitHub crons zeroed: scrape-heartbeat.yml deleted; nas-deploy.yml 15-min converger descheduled (NAS app hosting retired — dispatch-only now); rotate-bootstrap-key-reminder.yml descheduled → NEW /api/cron/rotation-reminder (access_expiry_warning/warning: emails, does not gate period close) + NAS script nas-cron/rotation-reminder.sh (invoke_cron family) written on the box.
  • Pending owner actions: (1) one-off production deploy to activate the Vercel cron + new watchdog; (2) create the quarterly DSM task — CRON_ENV_FILE=/volume1/docker/eop/cron.env /bin/sh /volume1/docker/eop/nas-cron/rotation-reminder.sh >> /volume1/docker/eop/logs/eop-cron-rotation-reminder.log 2>&1 (1st of Mar/Jun/Sep/Dec, 09:00); (3) the earlier scheduler clicks (id=3/4 cmd swaps, delete id=5/18).
  • Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-22 — PHASE 2 SHIPPED: Workspace transactions ledger de-browsered (last SCHEDULED browser use gone)

  • ✅ Read AGENTS.md · board checked by scope (same task, extended) · tracking T-147. Owner (verbatim): "Proceed to phase 2 first" — built before the Phase-1 deploy so ONE deploy ships the whole login-free pipeline.
  • What: Leg C added to /api/cron/billing-pipeline-sync — runWorkspaceLedgerEmailSync() in lib/gcpBilling/emailBillingSync.server.ts derives aote-system/workspace-billing/transactions/entries rows from billing emails: charges ← the monthly invoice-email's attached PDF (amount/date via pdf-parse); payments ← the "We've received your payment for theestablishers.com" emails (amount from body, stored NEGATIVE per the ledger convention the reader/matcher expect). Replaces the monthly run-workspace-scrape.sh admin.google.com browser scrape as the ledger's writer.
  • Compatibility: Google's ledger txn id is NOT recoverable from email (probe-verified) → email rows carry googleTransactionId: null; both consumers (billingDisplayTokens.server.ts, workspace-auto-match.ts) already fall back to a ledger-<docId> key. Own wse_<hash> ids from stable email facts (messageId / invoiceNumber) make daily re-runs no-ops before any body fetch; a same-type/date/|amount| guard prevents duplicating the ~11 existing ws_* scraper rows and — per detect-don't-correct — scraper docs are never rewritten. Declines deliberately skipped (owner: known cash-flow timing, ignore). 400-day window backfills ~a year of ledger on first run.
  • Watchdog: zero changes needed — the workspace pipeline already derives freshness from this collection's own updateTime (40d). Jul-7's "two-profile split" end-state note is now moot: there is no scheduled browser use left to split profiles for.
  • Docs/infra alongside: AGENTS.md gains the Programmatic Gmail section (SA identity, DWD client id 115562755728083115762 + gmail.readonly scope, env names, retrieval + gotchas) and a signed strike of the earlier "no Gmail API key exists" claim it supersedes; the full SA key JSON is now in Secret Manager as GMAIL_INVOICE_FETCHER_KEY (v1, uploaded via the agent-reserved service@ — the owner's gcloud login had re-expired) so cloud agents self-serve per the standard credential-discovery order.
  • Heavy tsc --noEmit (8 GB): clean. Scheduling rides T-175's NAS DSM task (nas-cron/billing-pipeline-sync.sh) — no new schedule of its own.
  • Decommission (owner DSM clicks, after deploy + first green run including Leg C): disable id=6 (GCP cost-table nightly) and id=3 (Workspace monthly scrape — now redundant); leave id=4 unarmed; id=5 + id=18 deletions from the Jul-7 audit still stand. The browser stack stays on disk solely for on-demand GCP statement pulls.
  • Remaining: owner's one-off deploy → fire billing-pipeline-sync once → verify legs A/B/C + markers + first-ever Workspace invoice docs → DSM clicks above → T-147 eligible for done.
  • Source: Service Account (Local) · local session ca159e39-42a3-4569-b1cd-6e006f6639c3

2026-07-22 — PHASE 3: pivot to owner-side Apps Script PUSH; mailbox credential fully decommissioned

  • ✅ Read AGENTS.md · board checked by scope (same task, extended) · tracking T-147.
  • Why (owner, verbatim): "I don't feel very safe for read-only access to my mailbox. Instead of a read-only access, is it possible for us to setup an Apps Script that scans thru my mailbox for the specific emails that contains the information that we need in reverse?" → approved ("sure") after an evidence review. The DWD pull architecture (Phases 1–2's transport) inverted the trust model — the app held a key that could read the whole mailbox. The push model keeps ALL mailbox capability inside the owner's own account; the app only ever receives the four declared email families.
  • Evidence review that de-risked the pivot (recorded for posterity): reconciled all 17 browser-scraped ledger rows (the ground truth) against the mailbox — manual payments 3/3 exact (amount+date sentences), declines 5/5, and the invoice PDFs reproduce the admin-ledger's proration granularity cent-for-cent (May-1 invoice 5557036172 line items Apr 1 · 13.78 + Apr 2–30 · 399.81 = the two scraped April charge rows exactly). One true gap: Google sends NO email for a successful AUTOMATIC payment → covered by a deterministic inference (invoice + no decline within 2 days ⇒ auto-payment of invoice total on invoice date; flagged inferred: true; bank-feed cross-check; rule validated against all 4 months of ground truth). Bonus: GCP payment receipts carry the CLOUD xxxxxx reference that mirrors the bank descriptor + card last4 — captured as gcp-payment-events for future 3-point matching evidence.
  • Shipped: pages/api/workspace/billing/email-ingest.ts (push receiver, WORKSPACE_INGEST_SECRET envelope, kinds workspace_invoice/workspace_payment/workspace_decline/gcp_payment/heartbeat); docs/apps-script/billing-email-push.gs (canonical script copy — queries declared at top; install runbook in docs/apps-script/README.md); emailBillingSync.server.ts rewritten pull→push (+ per-LINE-ITEM charge rows, + runAutoPaymentInference); cron slimmed to Leg B (BQ) + Leg C (inference); watchdog + health remedies → Apps-Script surfaces; .env.example GMAIL block → "no Gmail credential, by design". Heavy tsc: clean.
  • Credential teardown (all four stores, 2026-07-22): Vercel GMAIL_FETCHER_*/GMAIL_USER_EMAIL deleted (prod+preview); Secret Manager GMAIL_INVOICE_FETCHER_KEY deleted; SA user key 52412817… deleted (SA stays as a keyless shell); owner's local key file removed. AGENTS.md rewritten accordingly (the "no Gmail API key" claim is true again — now deliberately).
  • Owner actions remaining: (1) install the Apps Script (5-min steps in the .gs header) — first run back-fills ~13 months; (2) remove the now-inert DWD grant for client 115562755728083115762 in admin.google.com; (3) the one-off deploy; (4) the DSM decommission clicks (disable id=6 + id=3; id=5/id=18 deletions). Then T-147 → done.
  • Source: Service Account (Local) · local session ca159e39-42a3-4569-b1cd-6e006f6639c3

2026-07-23 — Phase 3 fix pass: first live run diagnosed; invoice PDF-parse bug fixed

  • Attestation: re-read AGENTS.md + scanned the board; this continues T-147 (no dup).
  • Source: Service Account (Local) · local session e392d97a-fb85-4c88-b556-ec00580de1a8
  • What the first live run showed (5 emails, Jul 1–2, after the deploy): the plumbing works but parsing had one real bug. Ground truth from aote-system:
  • Invoice 5609520523 → FAILED (invoice-events=0, no charge rows). Root cause: pdf-parse (pdfjs-dist) was not in next.config.ts serverExternalPackages, so Next's server bundle mangled it and new PDFParse().getText() threw — the exact T-130 bundler class of failure, and this is the first API route to parse a PDF server-side. The handler returned 422, which the Apps Script treats as "processed", so it self-stranded.
  • Payment HK$413.60 → CORRECT (wrote nothing because it deduped against the legacy scraper row ws_V5796958095 payment −413.6 @ 2026-07-01). Not a bug.
  • 2× declines → events created, date ✓, amountAbs=null (the rigid for HK$… regex missed getPlainBody()'s phrasing). Non-critical: inference uses the decline date, and the scraper itself stored declines as amount 0.
  • GCP payment → event created, amount 340.16 ✓, reference/cardLast4=null (the single optional-group regex caught the amount but never the reference clause).
  • Also found: the script's BOOTSTRAPPED flag was set unconditionally at the end of every run, so the pre-deploy 404-storm (errors=56) flipped it and the window silently dropped 400d→45d, stranding the ~51 emails older than Jun 8.
  • Fixes (this commit):
  • next.config.ts — added pdf-parse + pdfjs-dist to serverExternalPackages. The invoice fix.
  • emailBillingSync.server.ts — PDF-extraction throw now RE-THROWS (→ endpoint 500 → script retries) instead of returning 422 (self-marking processed); decline amount = first-currency match; GCP reference/card split into independent tolerant regexes; new workspace-billing/parse-failures/entries capture (truncated source-text sample) — the only debug ground truth left now that the mailbox credential is gone.
  • billing-email-push.gs — BOOTSTRAPPED set only after a CLEAN pass (errors===0), so an interrupted/pre-deploy run keeps back-fill open; added resetState() to force a full re-scan after a server fix.
  • Recovery path (needs owner): (a) one-off deploy of this commit; (b) in script.google.com, Run ▶ resetState then Run ▶ runOnce — re-scans 400d and re-pushes all ~56 emails against the fixed parsers (idempotent: invoices re-file "exists", rows dedupe, events overwrite). Then verify invoice-events>0 + Drive filing + zero scraper-row duplicates.
  • Blast radius: serverExternalPackages change affects the whole server bundle — but it's additive (two more packages loaded un-transformed, same pattern as Firestore); no other route imports pdf-parse. Server-engine + Apps-Script changes are confined to the billing-email pipeline.
  • Heavy unfiltered tsc --noEmit --incremental false on fresh npm ci: clean (EXIT=0).

2026-07-23 — Phase 3 fix pass #2: pdf-parse → unpdf (serverless worker-file trap)

  • Source: Service Account (Local) · local session e392d97a-fb85-4c88-b556-ec00580de1a8
  • Fix pass #1 (externalize pdf-parse) was the wrong lever. The resetState()+runOnce() back-fill ran the full 400-day window (scanned=56) and 28 non-invoice pushes succeeded (every payment / decline / GCP receipt across 13 months), but all 28 invoices returned 500 with: PDF text extraction failed: Setting up fake worker failed: "Cannot find module '…/pdfjs-dist/legacy/build/pdf.worker.mjs'". Externalizing got pdfjs to load, but pdfjs then dynamically imports a worker .mjs that Vercel never traces into the lambda — a classic pdf-parse-on-serverless trap.
  • The retry design validated itself: those 28 invoices came back 500 → were left UNprocessed (back-fill kept open), so they retry automatically on the next runOnce — no resetState needed after the next deploy.
  • Fix: swapped the invoice PDF reader from pdf-parse to unpdf (extractText + getDocumentProxy) — unpdf ships a serverless-safe pdfjs build with no separate worker file. Reverted the pdf-parse/pdfjs-dist serverExternalPackages additions (unpdf is meant to be bundled). Verified unpdf@1.6.2 extracts text from a real PDF locally before shipping.
  • Recovery after this deploys: owner runs runOnce again (NOT resetState) — the 28 stranded invoices retry against the working extractor. Then verify invoice-events + Drive filing + ledger.
  • Heavy unfiltered tsc --noEmit --incremental false on fresh npm ci: clean (EXIT=0).

2026-08-10 — Watchdog alert investigated: script-side execution death (owner action needed)

✅ Read AGENTS.md · checked the board by scope (no dup — this is the T-147 pipeline's own outage) · tracking T-147 (investigation only, no code change). Source: Records (Infrastructure) · https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

The owner has been receiving the [EOP] Billing email push (owner Apps Script) is not delivering watchdog email (165h stale at 2026-08-09, threshold 36h). Investigated read-only; every server-side layer is healthy, so the failure is isolated to Google executing the trigger in the owner's account:

  • Marker (systemHealth/billing-email-sync, aote-system): last stamp 2026-08-03T01:45:04Z = 09:45 HKT (the trigger hour), lastVendor: apps-script-heartbeat, scanned: 7 — a clean, authenticated, zero-new-mail run. Nothing at all since.
  • Endpoint: POST /api/workspace/billing/email-ingest answers 401 unauthenticated in production — route deployed and reachable. (The 4 known-failing ingest.test.ts cases are the LEGACY scraper route's tests — unrelated.)
  • Secret: Vercel WORKSPACE_INGEST_SECRET created/updated 2026-06-09 — no rotation anywhere near the death date; and a mismatch wouldn't explain zero attempts.
  • Script project (read via the T-177 manager SA, projects.get/getContent, read-only): updateTime 2026-07-23T04:17:31Z — the T-177 verified push, untouched since (the Aug 2–3 substitute-agent window did NOT modify it); live code byte-identical to the repo canonical (docs/apps-script/billing-email-push.gs, trailing newline aside).

Conclusion: the daily time-driven trigger stopped executing after 2026-08-03 09:45 HKT — either the script's Gmail authorization was revoked (password change / security-checkup "remove access" around Aug 2–3 revokes grants and every run then fails "Authorization is required"), or the trigger itself is gone/disabled. Owner's 30-second check (script.google.com as me@jefferochan.com → "EOP billing email push" → Executions): failing rows after Aug 3 → open the editor, Run ▶ runOnce, re-approve — the trigger resumes. NO rows after Aug 3 → Triggers (clock icon) → re-add the daily runOnce timer, then Run ▶ runOnce once. Also check the inbox for Google's "Summary of failures for Google Apps Script" email — it names the exact error.

No data loss either way: the scan window is newer_than:45d and PROCESSED_IDS only marks pushed messages, so the first healthy run auto-catches-up the entire dead week. resetState is NOT needed.

2026-08-10 — ROOT CAUSE CORRECTED: the trigger never died — the marker stamp was fire-and-forget

✅ Read AGENTS.md · checked the board by scope (no dup) · tracking T-147. Source: Records (Infrastructure) · https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

The previous entry's hypothesis (trigger/authorization death) is WRONG — retracted. The owner's manual runs pinned it:

  • 10:18 HKT run: clean (scanned=7 errors=0), heartbeat response invisible (old script) — marker did NOT advance (checked ~2 min later: still Aug 3).
  • Heartbeat-response logging pushed to the live script (T-177 SA, repo-canonical first).
  • 10:35 HKT run: logs heartbeat → HTTP 200 (freshness marker stamped) — yet the marker then read lastIngestAt: 2026-08-10T02:18:09Z — the 10:18 run's timestamp. The marker runs exactly ONE request behind.

Mechanism: stampBillingEmailMarker was fire-and-forget (void …set().catch()) and the handler returned 200 immediately. On Vercel the instance FREEZES once the response is sent; the pending Firestore write flushes only when that same instance thaws for a later request. Steady state pre-Aug-3: each daily trigger run flushed the previous day's stamp (~24h stale — under the 36h threshold, invisible). The #979 (Aug 3 ~23:00 UTC) and #980 deploys retired those instances; each day's cold-start run froze its own stamp and no thaw followed, so the marker stuck at Aug 3 while the trigger, script, auth, and secret were healthy the entire time. The daily 09:45 HKT executions for Aug 4–9 should all show clean in the script's Executions history (owner can confirm).

Fix (this commit): stampBillingEmailMarker returns its write promise (errors still swallowed — a marker hiccup must never fail an ingest) and email-ingest.ts AWAITS it at all five call sites (heartbeat + 4 push kinds) before responding. Gates: unfiltered tsc clean · vitest 842 passed (same 4 known pre-existing legacy ingest.test.ts). The script-side heartbeat-response logging (already live) turns any future rejection loud in the Executions log.

Until this deploys (owner-commanded only) the stamp still races on cold instances — the 36h alert may re-fire; expected, self-clears after deploy + the next trigger run. Two-line status: pipeline was never losing DATA (message pushes are awaited during processing; only the freshness marker raced) — the alert was a false "not delivering" caused by the marker itself.