Skip to content

T 175

uid: T-175 title: Billing-pipeline-sync scheduled on the NAS β€” resolves the scrape-watchdog alert emails status: done area: infrastructure created: 2026-07-22 updated: 2026-07-22 owner: girafeev1 assignee: Records (Infrastructure) related: T-147, T-171, T-170

Goal (owner, 2026-07-22)

"I have received some new alert warnings from the web app, indicating some things that we've previously worked on stopped work?"

Two system:scrape-heartbeat error alerts were emailing every 6h: "GCP cost detail (BigQuery sync) is not delivering" (marker 156h stale, threshold 36h) and "Billing email sync (Workspace invoices) is not delivering" (marker never stamped).

Root cause β€” a scheduling gap, not a regression

  1. ~Jul 15 the old nightly browser-scrape stopped feeding systemHealth/scrape-gcp-cost-table (T-147 Phase 1 decommissioned its scheduled role; its replacement wasn't scheduled yet).
  2. The replacement β€” /api/cron/billing-pipeline-sync (Gmail invoice leg + BigQuery cost leg, built under T-147) β€” shipped with a GitHub Actions daily schedule, which violates owner policy (2026-07-07 "no cron jobs on GitHub") and escaped the schedule sweep. Its first-ever scheduled run (2026-07-22 04:39 UTC) failed 401: the GH repo secret CRON_SECRET predates the T-171 rotation (deliberately not rotated there β€” GH wasn't supposed to hold a schedule).
  3. The alerts only started arriving now because the scrape-heartbeat watchdog itself was dead until T-171 revived it on the NAS (2026-07-21) β€” it surfaced staleness that predates it.

What shipped

  • NAS (live now): /volume1/docker/eop/nas-cron/billing-pipeline-sync.sh (house invoke_cron pattern, 240s timeout) + DSM task id=13 "Daily - EOP Billing Pipeline Sync", daily 09:30 HKT = 01:30 UTC (the slot the GH schedule had chosen: after Google's overnight invoice email + BQ export refresh). Full chain test-run: HTTP 200, both legs ok.
  • Recovery (live now): endpoint fired manually with the rotated secret β†’ both markers stamped (BQ legs synced 202606+202607, 6,229 tx; Gmail leg scanned the mailbox) β†’ scrape-heartbeat re-run β†’ scraperHealthy: true, 24 stale alert notifications self-cleared. The 6-hourly email nag stops.
  • Repo: GH workflow schedule commented out (workflow_dispatch fallback kept, with a note that it 401s until the repo secret is updated); scrape-heartbeat remedy strings + scheduling header and the endpoint's scheduling header now point at the NAS task instead of GitHub/Vercel Cron.
  • AGENTS.md: documented (owner ask, same message) that the owner's Gmail/Calendar/Drive are reachable via claude.ai connectors (discover with ListConnectors; per-chat enablement; OAuth is owner-side only) β€” and that alert emails are mirrored in Firestore notifications (aote-system), which is how this investigation read them without Gmail access.

Log

  • 2026-07-22 β€” created + done (Records Infrastructure). βœ… Read AGENTS.md Β· checked the board by scope (T-147 owns the pipeline itself β€” local agent's domain; T-171 owns the NAS cron estate and closed before this endpoint existed; this task owns only the scheduling gap + alert episode β€” no dup). Source: Records (Infrastructure) Β· https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD
  • SHAs: a8eb2767 (workflow schedule retired + remedy text + AGENTS.md connectors) + the docs commit carrying this record (one PR). Unfiltered NODE_OPTIONS=--max-old-space-size=8192 npx tsc --noEmit clean pre-merge. NAS-side changes are live regardless of deploy; the remedy-string edits ride the next owner-ordered deploy.
  • Verify: NAS task fires 2026-07-23 09:30 HKT; the 18:30 HKT heartbeat today should stay green. Owner's inbox: no new "not delivering" emails after ~12:53 HKT today.
  • Blast-radius: NAS Task Scheduler (+1 task, id=13); GH workflow schedule retired (dispatch kept); alert remedy text. No app logic, schema, or store changes.
  • Handoffs: (1) β†’ Service Account (Local): NAS task id=6 "Daily - GCP Cost Table Ingest" (00:01 HKT) still runs nightly but hasn't stamped its marker since ~Jul 15 β€” now redundant with the BQ leg; retire or repurpose (id=4 "Daily - GCP Invoice PDF Ingest" doubles as the root-exec dispatcher β€” do NOT delete it outright). (2) β†’ Service Account (Local): the endpoint ran 56.7s / 66s on its two proofs against Vercel Hobby's 60s ceiling β€” split the two legs into separate invocations (e.g. ?leg=email|bq) before it starts flapping. (3) GH repo secret CRON_SECRET left stale on purpose; update it only if the workflow_dispatch fallback is ever wanted.