Skip to content

T 175

uid: T-175 title: Billing-pipeline-sync scheduled on the NAS β€” resolves the scrape-watchdog alert emails status: done area: infrastructure created: 2026-07-22 updated: 2026-07-22 owner: girafeev1 assignee: Records (Infrastructure) related: T-147, T-171, T-170

Goal (owner, 2026-07-22)

"I have received some new alert warnings from the web app, indicating some things that we've previously worked on stopped work?"

Two system:scrape-heartbeat error alerts were emailing every 6h: "GCP cost detail (BigQuery sync) is not delivering" (marker 156h stale, threshold 36h) and "Billing email sync (Workspace invoices) is not delivering" (marker never stamped).

Root cause β€” a scheduling gap, not a regression

  1. ~Jul 15 the old nightly browser-scrape stopped feeding systemHealth/scrape-gcp-cost-table (T-147 Phase 1 decommissioned its scheduled role; its replacement wasn't scheduled yet).
  2. The replacement β€” /api/cron/billing-pipeline-sync (Gmail invoice leg + BigQuery cost leg, built under T-147) β€” shipped with a GitHub Actions daily schedule, which violates owner policy (2026-07-07 "no cron jobs on GitHub") and escaped the schedule sweep. Its first-ever scheduled run (2026-07-22 04:39 UTC) failed 401: the GH repo secret CRON_SECRET predates the T-171 rotation (deliberately not rotated there β€” GH wasn't supposed to hold a schedule).
  3. The alerts only started arriving now because the scrape-heartbeat watchdog itself was dead until T-171 revived it on the NAS (2026-07-21) β€” it surfaced staleness that predates it.

What shipped

  • NAS (live now): /volume1/docker/eop/nas-cron/billing-pipeline-sync.sh (house invoke_cron pattern, 240s timeout) + DSM task id=13 "Daily - EOP Billing Pipeline Sync", daily 09:30 HKT = 01:30 UTC (the slot the GH schedule had chosen: after Google's overnight invoice email + BQ export refresh). Full chain test-run: HTTP 200, both legs ok.
  • Recovery (live now): endpoint fired manually with the rotated secret β†’ both markers stamped (BQ legs synced 202606+202607, 6,229 tx; Gmail leg scanned the mailbox) β†’ scrape-heartbeat re-run β†’ scraperHealthy: true, 24 stale alert notifications self-cleared. The 6-hourly email nag stops.
  • Repo: GH workflow schedule commented out (workflow_dispatch fallback kept, with a note that it 401s until the repo secret is updated); scrape-heartbeat remedy strings + scheduling header and the endpoint's scheduling header now point at the NAS task instead of GitHub/Vercel Cron.
  • AGENTS.md: documented (owner ask, same message) that the owner's Gmail/Calendar/Drive are reachable via claude.ai connectors (discover with ListConnectors; per-chat enablement; OAuth is owner-side only) β€” and that alert emails are mirrored in Firestore notifications (aote-system), which is how this investigation read them without Gmail access.

Log

  • 2026-07-22 β€” created + done (Records Infrastructure). βœ… Read AGENTS.md Β· checked the board by scope (T-147 owns the pipeline itself β€” local agent's domain; T-171 owns the NAS cron estate and closed before this endpoint existed; this task owns only the scheduling gap + alert episode β€” no dup). Source: Records (Infrastructure) Β· https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD
  • SHAs: a8eb2767 (workflow schedule retired + remedy text + AGENTS.md connectors) + the docs commit carrying this record (one PR). Unfiltered NODE_OPTIONS=--max-old-space-size=8192 npx tsc --noEmit clean pre-merge. NAS-side changes are live regardless of deploy; the remedy-string edits ride the next owner-ordered deploy.
  • Verify: NAS task fires 2026-07-23 09:30 HKT; the 18:30 HKT heartbeat today should stay green. Owner's inbox: no new "not delivering" emails after ~12:53 HKT today.
  • Blast-radius: NAS Task Scheduler (+1 task, id=13); GH workflow schedule retired (dispatch kept); alert remedy text. No app logic, schema, or store changes.
  • Handoffs: (1) β†’ Service Account (Local): NAS task id=6 "Daily - GCP Cost Table Ingest" (00:01 HKT) still runs nightly but hasn't stamped its marker since ~Jul 15 β€” now redundant with the BQ leg; retire or repurpose (id=4 "Daily - GCP Invoice PDF Ingest" doubles as the root-exec dispatcher β€” do NOT delete it outright). (2) β†’ Service Account (Local): the endpoint ran 56.7s / 66s on its two proofs against Vercel Hobby's 60s ceiling β€” split the two legs into separate invocations (e.g. ?leg=email|bq) before it starts flapping. (3) GH repo secret CRON_SECRET left stale on purpose; update it only if the workflow_dispatch fallback is ever wanted.

2026-09-24 β€” The alert emails are back, for a different reason β†’ T-212

βœ… Read AGENTS.md. Source: Records (Infrastructure) Β· https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

The "GCP cost detail (BigQuery sync) is not delivering" emails returned from Sep 2. This task's fix β€” scheduling the job on the NAS β€” is still in place and still correct. What changed is that since Aug 29 the NAS fails to look up the web app's address on most runs, so the job never starts. It affects every NAS cron, not just this one. Investigation and proposed fixes are in T-212; this task stays done.

2026-10-06 β€” Cause of the returning alerts found and fixed β†’ T-212

βœ… Read AGENTS.md. Source: Records (Infrastructure) Β· https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

The NAS's own network port was dropping (not the router's DNS, as first thought). It was moved to its second port on Oct 5, and every run of this job since has succeeded. Details in T-212.