Skip to content

T 200

uid: T-200 title: August GCP bill investigation β€” "24/7 container" claim refuted; spike = Gemini agent burst status: done area: infra created: 2026-08-06 updated: 2026-08-06 owner: girafeev1 assignee: EOP Local Assistance (Jefito) related: T-015, T-171, T-148

βœ… Read AGENTS.md Β· EOP Local Assistance (Jefito) Β· local session 8ab2d4ce-6acc-4a26-ad72-c5bf1964cad2 Β· checked the board by scope (no dup β€” T-015 was the June cropper fix, T-171 the July Cloud Run retirement; neither covers the AUGUST spike or the fresh always-on-container claim) Β· tracking T-200. Source: EOP Local Assistance (Jefito) Β· local session 8ab2d4ce-6acc-4a26-ad72-c5bf1964cad2

Why this exists

The owner's Google Cloud bill looked oddly expensive again in early August 2026. A separate agent the owner consulted claimed "there's a container running 24/7 on Google Cloud" and the owner asked for verification + downscaling. This task records the verification so the next cost hunt doesn't restart from zero β€” and so the refuted claim doesn't get re-actioned by a future agent.

Findings (verified live 2026-08-06, read-only: Cloud Run Admin API + Compute API + BigQuery billing export)

1. The "24/7 container" claim is FALSE for the web-app project (aote-pms), today. - Both surviving Cloud Run services β€” workspace-billing-service and bank-login-service (asia-east1) β€” have minInstances=0 + CPU throttling on (request-based billing, scale to zero). Configs last modified Jun 8 / Jun 13 by the owner; unchanged since. - Zero Compute Engine instances, zero GKE clusters in aote-pms. - The historical 24/7 container (receipt-cropper, min-instances=1, ~HK$770/mo) was scaled to zero 2026-07-09 (T-015) and deleted outright 2026-07-21 (T-171). The consulted agent was reading stale state.

2. The only literal always-on machine is NOT the web app and costs ~nothing. An e2 VM fortune-teller in the owner's separate project hotbes (us-west1-b) runs 24/7 β€” but it sits in the Always Free tier: net HK$0.01–0.06/month (gross ~HK$49/mo fully offset by free-tier credit lines; verified across Apr–Aug in the billing export). Not worth touching; the aote-pms service account has no access to that project anyway (403).

3. The ACTUAL August spike is Vertex AI Gemini agent usage, not any container. Whole-bill by month (net, HKD, all projects on the billing account): Apr ~21 Β· May ~843 (incl. a one-off HK$99.95 project-less charge) Β· Jun ~880 (peak β€” the cropper) Β· Jul ~313 Β· Aug ~567 in the first 5 days. August's HK$563 of that is Vertex AI "Gemini 3.6 Flash Global" SKUs concentrated in Aug 2–3 (HK$459 on Aug 3 alone; burst hours 09:00–18:00 HKT), with billing labels trajectory_id / last_execution_id / last_step_index (up to ~2280 steps) / used_claude: false β€” the signature of an agentic coding tool (Gemini CLI / Gemini-based agent) authenticated against aote-pms's Vertex AI. The web app itself pins gemini-2.5-flash (+-image) for receipt extraction (lib/telegram/receiptProcessor.ts) β€” its Vertex usage is the small Jun/Jul baseline (HK$25–32/mo) and it never calls Gemini 3.6. No charges Aug 4–5: the burst has stopped. Plausibly the burst IS the consulted agent's own investigation sessions.

4. Remaining steady-state aote-pms baseline is reasonable: Secret Manager ~HK$13/mo + Artifact Registry ~HK$8/mo + Firestore ops (billed under "App Engine") ~HK$3–7/mo + near-zero Cloud Run β‰ˆ ~HK$25–30/mo absent agent bursts.

Verdict / what was (not) done

Nothing was downscaled β€” there is nothing left to downscale. The infra is already in its cheapest sane configuration; the owner's "proceed to downscale" instruction was premised on a 24/7 container that no longer exists. Config changes were deliberately NOT invented to look busy. Read-only verification only; no code, no Firestore documents touched (reads only), no deploys.

Open levers for the owner (each needs an owner decision, none executed): 1. Identify the Gemini agent tool that billed HK$563 on Aug 2–3 and decide whether it should keep using aote-pms as its Vertex billing project (vs. a personal API key/project, vs. accepting the cost per use). It is dormant now but will bill again next time it runs. 2. Optional: a GCP budget alert on the billing account (e.g. HK$300/mo threshold) so the next spike emails the owner within a day instead of surfacing at invoice time. 3. Optional micro-cleanup: Artifact Registry (~HK$8/mo) still stores images incl. those of the deleted Cloud Run services; pruning needs care not to touch the two live services' images.

Blast radius: none β€” no state changed anywhere. Related SHAs: this docs commit only.

2026-08-06 β€” lever 2 (budget alert) attempted; blocked on billing-account IAM β€” owner action needed

  • βœ… Read AGENTS.md Β· owner asked (verbatim): "Can I limit Google Cloud budget to HK$200/month?" β€” clarified in chat that GCP budgets alert, not block (a true hard cap = auto-disabling billing, rejected as too dangerous for a production app), then attempted to create a HK$200/month alert budget (thresholds 50% / 90% / 100% actual + 100% forecasted) on billing account 0102F7-3FFF33-823945 via the Billing Budgets API.
  • State changed: enabled billingbudgets.googleapis.com on aote-pms (harmless, free, reversible; needed for API-based budget management).
  • Blocked: budgets are billing-account-level objects; billing-account IAM is separate from project IAM, and service@aote-pms holds no role there β†’ PERMISSION_DENIED. By design no agent credential can self-grant this.
  • Owner's two paths (either works): (a) Console, no new grants (~2 min): console.cloud.google.com β†’ Billing β†’ Budgets & alerts β†’ Create budget β†’ scope: all projects Β· amount HK$200 Β· thresholds 50/90/100 + forecasted 100. (b) Delegate: grant service@aote-pms.iam.gserviceaccount.com the Billing Account Costs Manager role (roles/billing.costsManager) on billing account 0102F7-3FFF33-823945 (Billing β†’ Account management β†’ Permissions), then tell the agent to re-run creation β€” script already staged. The grant can be removed right after if preferred.
  • Source: EOP Local Assistance (Jefito) Β· local session 8ab2d4ce-6acc-4a26-ad72-c5bf1964cad2

2026-08-06 β€” budget alert LIVE βœ… (owner delegated the billing role; lever 2 closed)

  • βœ… Read AGENTS.md Β· owner granted service@aote-pms the billing-account role and said (verbatim): "SA delegated" β†’ budget created via the Billing Budgets API.
  • Live config (verified from the API's create response): budget billingAccounts/0102F7-3FFF33-823945/budgets/2a154fea-21a1-43d8-aba9-023a4e35472b, display name "Monthly spend watch - HKD200 (T-200)", HK$200 per calendar month, all projects on the account, credits included (net spend); alerts at 50% / 90% / 100% of actual spend + 100% forecasted. No custom notification channel set β†’ Google's default applies: threshold emails go to the billing account's admins/users (the owner).
  • What this does / doesn't do (recorded so nobody assumes a hard cap): it emails the owner as spend crosses HK$100 / HK$180 / HK$200 (or is forecast to) β€” it does not stop charges. The auto-disable-billing kill-switch was considered and rejected (would take Firestore + the app down mid-month automatically).
  • Follow-up for the owner (least-privilege, optional): the delegated billing-account role on service@aote-pms can now be revoked β€” the budget persists without it; re-grant only if a future budget edit should go through an agent. Fits the T-148-era scoped-access model.
  • Source: EOP Local Assistance (Jefito) Β· local session 8ab2d4ce-6acc-4a26-ad72-c5bf1964cad2