Skip to content

uid: T-208 title: Worker-health light in the receipts sheet + one Telegram message when a link drops or returns status: doing area: telegram-receipts created: 2026-09-05 updated: 2026-09-05 owner: girafeev1 assignee: GCP Billing Helper related: T-207, T-206, T-171

T-208 β€” is every link in the receipt chain answering?

βœ… Attestation: read AGENTS.md Β· checked the board by scope (T-207 is the chain rebuild this watches over; T-206 is the sheet this draws in; the *-heartbeat crons are dead-man's switches for the pipelines, not reachability of the workers β€” no dup) Β· tracking T-208. Source: GCP Billing Helper Β· https://claude.ai/code/session_014NZVz9AJVHdfgLBGKC99oA

Originally opened as a second T-207.md on the PR branch, colliding with the file the EOP Telegram Bot had already put on main. Renumbered to T-208 on merge per the collision rule (merged second renumbers); the #1010 backfill that was in that draft is folded into T-207's log instead.

Why this exists

The receipt chain runs in three places on purpose β€” the app on Vercel, the image workers on the NAS (T-171, for cost), and Gemini / Drive / Firestore on Google Cloud. Any one link can drop on its own: the NAS tunnel "flaps every minute or two" and 502s in bursts (the channel runbook's own words), a tunnel token can be revoked, a Cloud Run service can be deleted (all three image services were, in T-171), a Vercel env var can simply be unset. When a link drops, receipts do not fail loudly; they fall quietly down the crop chain to a rougher tier, and nobody finds out until the next time someone opens a stored image.

Owner, 2026-09-05 (verbatim): "since workers live in different places, some on GCP, some on the NAS, can the Telegram bot check the connection between workers and prompt and warn user if connections to any of these places have dropped at any point, or comes back online at any point, and show a status 'light' at one of the corners?"

What a person gets

  • A status light in the receipts sheet β€” one lamp for the whole chain: green (all up), amber (an image worker is down; receipts still read, just with a rougher crop), red (Gemini or Firestore is down; receipts cannot be read or saved), grey (not checked yet). Tap it for the per-worker list, split by where each runs β€” NAS Β· Google Cloud β€” with how long each has been up or down.
  • One Telegram message when a worker drops, one when it comes back. Which worker, where it lives, what it costs receipts until it is back, the error; on recovery, how long it was gone. Silence in between β€” the check runs every five minutes but only speaks on a change.
πŸ”΄ Receipt cropper (NAS) went down at 14:10 HKT
   Until it is back, receipts get a rougher crop.
   GET /health β†’ HTTP 502

🟒 Receipt cropper (NAS) is back after 12 min

How it is built, and why this way

  • The rules are pure and tested (lib/health/workers.ts, 20 tests): what "down" means, what colour the lamp is, what a message says. Two consecutive misses before "down" β€” one miss is tunnel noise and would page the owner every few minutes. A worker configured for the first time that never answers is announced (that is exactly the moment someone is watching for β€” the segmenter, the week it is wired up); becoming configured, or losing configuration, is a deploy fact and stays silent.
  • One cheap read-only probe per worker (lib/health/probes.ts): GET /health on the cropper and the segmenter; a model-metadata GET on Vertex (proves the credential mints a token and the region answers, spends no generation); drive.about.get; a Firestore read.
  • On Inngest, not on the NAS (lib/inngest/worker-health.ts, every 5 min). The NAS is one of the things being watched, and a watchdog on the box it watches cannot report that box going dark β€” the same reasoning the *-heartbeat crons record. Alerts go to TELEGRAM_HEALTH_CHAT_ID; when that is unset, or Telegram itself is the failure, notifySystemOwner (in-app + email) carries it.
  • The colour is decided server-side and sent by /api/telegram/miniapp/list, so the light and the bot's messages can never disagree about what "down" is. The sheet reads a copy cached for a minute rather than hitting the document on every 4-second poll.

Firestore β€” one new document (whole-document before β†’ after)

aote-system Β· systemHealth/workers. Sibling documents in systemHealth today, whole shape:

billing-email-sync:        { lastInvoiceNumber: string, scanned: number, lastVendor: string, lastIngestAt: timestamp }
scrape-gcp-billing:        { note: string, lastIngestAt: timestamp, lastVendor: string, lastInvoiceNumber: string }
scrape-gcp-cost-table:     { lastVendor: string, lastIngestAt: timestamp, lastInvoiceNumber: string }
scrape-workspace-invoices: { lastVendor: string, lastIngestAt: timestamp, lastInvoiceNumber: string }

Proposed new sibling, grouped per the "maps, not flat" rule:

+ checkedAt: string (ISO)
+ workers: {
+   cropper:   { status: 'up'|'down'|'unconfigured', since: ISO, lastCheckedAt: ISO,
+                lastOkAt: ISO|null, lastError: string|null, latencyMs: number|null, consecutiveFailures: number }
+   segmenter: { …same }   vertex: { …same }   drive: { …same }   firestore: { …same }
+ }

Considered and rejected: the existing scrape-* / billing-email-sync markers β€” those are freshness stamps the pipelines write about themselves ("I ran"); this is an outside-in probe ("can the app reach you"), and folding one into the other would blur a dead pipeline with a dead link. The notifications collection β€” that is where the alert lands, not the state the sheet needs to draw the lamp, and the cron needs the previous state to tell a change from a steady state (or it would alert every 5 minutes).

Status: awaiting the owner's answer. The EOP Telegram Bot session reviewed the PR and said it would approve the document as proposed; that is a recommendation, not the owner's ruling. Nothing merges until the owner answers.

Review round 1 (EOP Telegram Bot session, 2026-09-05) β€” all addressed

  1. Worker names and "what it costs" lines were wrong β€” the cropper is the straightener, and a dead cropper means a rougher crop, not none. β†’ Renamed: "Receipt cropper β€” receipts get a rougher crop"; "Receipt finder β€” receipts fall back to the cropper alone".
  2. The light sat fixed in the corner, on top of the "Add receipts" button. β†’ Moved into the tally strip, in the flow; the panel anchors to it.
  3. A second T-207.md and a task-number collision. β†’ Renumbered to T-208; #1010 folded into T-207's log.
  4. The PR body claimed nothing from #1008–#1010 was live. β†’ Wrong: #1008 and #1009 have been in production since deployment 74c98a5 (2026-09-04, owner-fired via the deploy hook). Corrected.
  5. The health document was read on every 4 s poll. β†’ Cached for 60 s in the list endpoint.
  6. unconfigured β†’ down was silent. β†’ Now announced (rule + test above).

Still open

  • The owner's answer on systemHealth/workers.
  • TELEGRAM_HEALTH_CHAT_ID in Vercel (optional β€” in-app fallback otherwise).
  • RECEIPT_SEGMENTER_URL once the NAS container is up (T-207), or the finder shows "not set up".
  • Deploy: none commanded; nothing here is live.

Commits (append-only, newest last)

ae00dda (opened, as T-207 on the branch)

Log

  • 2026-09-05 β€” opened. βœ… Read AGENTS.md. Source: GCP Billing Helper Β· https://claude.ai/code/session_014NZVz9AJVHdfgLBGKC99oA. Proposed by the agent from the owner's question quoted above; the owner's approval of the shape is pending. Interactive preview of the sheet with the light: https://claude.ai/code/artifact/22218cdb-6fb7-421f-b3c4-7ba1fbb99f91