NAS GCP billing scraper silently fails when its Google session expires — no re-auth + no failure alert (dark since ~2026-06-26)

Symptom¶

The NAS GCP billing scraper (Synology Task Scheduler, daily 00:00 HKT, image workspace-csv-scraper) runs on schedule but has failed every run since ~2026-06-26 — the Google Cloud Console session expired and the scraper can't re-auth itself. No alert fired, so it went dark for ~5 days until a manual health check caught it.

Verbatim from /volume1/docker/workspace-billing/gcp-statements-last-run.log:

[gcp] NOT authenticated — re-run login mode
=== done rc=2 @ Tue Jun 30 16:00:21 UTC 2026 ===

Last successful data: csv-downloads/ mtime 2026-06-26 00:51. Every 00:00 run since = rc=2.

Root cause¶

The scraper drives the GCP billing console via a stored Google session (per T-033). Google's step-up / session timeout invalidates it periodically. The scraper has no re-auth automation and, more importantly, no failure alert — so a broken scrape produces no signal, just stale billing data, until someone notices.

Immediate fix (operational — owner)¶

Re-login to the scraper's Google account via the NAS noVNC browser (http://100.101.13.71:6080/vnc.html, Tailscale-only). The next 00:00 run then resumes. This is the documented re-auth path (the T-033 caveat) and will recur whenever the session expires.

Structural follow-up (→ candidate T-NNN)¶

The silent-failure part is the real gap. Worth a deliberate task: - Alert on scrape rc != 0 — wire the scraper's exit status to the app's notification channel so a dark scrape is caught same-day instead of by chance. - Optionally: a re-auth reminder / longer-lived session so the manual noVNC step is rarer.

Escalate to a T-NNN if/when the alerting is built. Ticket-only for now (symptom + the recurring operational re-auth).

Decision log¶

2026-07-01 — opened (during the NAS health check)¶

✅ Read AGENTS.md. Opened during an owner-requested NAS health check: SSH'd to rooftop (gatekeeper), confirmed the box + Task Scheduler are alive but the GCP billing scrape has failed auth every day since ~06-26 (evidence above). Scope-checked the board — no existing ticket/task covers the scraper's silent-auth-failure (nearest is T-033, which built the scraper and flagged re-auth as a manual caveat, not an alert), so this is new, not a dup.
Proposed: EOP Local Assistance (fork). Approved: the owner (2026-07-01, verbatim): "(2) open a ticket for the scraper auth-expiry (recurring re-auth reminder)."
Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-01 — escalated -> T-147 ¶

The structural fix (alert on scrape rc!=0 + re-auth easing) now lives in the scraper consolidation T-147; this ticket stays the symptom record. - Source (edit): EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b

2026-07-03 — RESOLVED (silent-failure gap closed)¶

✅ Read AGENTS.md. Fixed end-to-end this session under T-147:
Re-auth: re-logged the scraper's Google session via the noVNC login mode; the scrape authenticates (rc=0) + downloads.
Second, hidden break: the ingest was POSTing to the dead p-eop.theestablishers.com host (dropped when hosting moved to Vercel) — repointed to eop.theestablishers.com (commit 7989ef75 + NAS env), so a working scrape lands data again.
The "silent" part — the real gap — is covered: the scrape-heartbeat watchdog (T-147 scope #1) alerts when no billing ingest lands in 36h (proven live — it emailed on 07-02). A wiring false-alarm (the cost-table path didn't stamp the freshness marker) was found + fixed (0e2dd0d5, deploy pending).
Verified: invoices 9/9 → HTTP 200 (marker stamped by the real ingest); cost-table end-to-end (711→805 rows in gcp_billing_derived).
Residual (stays in T-147, not this ticket): the daily schedule had been dark since ~Jun 24 (this auth expiry) — invoices resume now that auth is fixed; cost-table needs a daily DSM task (drafted); deploy 0e2dd0d5.
Status → fixed. Source: EOP Local Assistance (fork) · session edb0abc7-6373-449b-8e31-379fd38d391b