Skip to content

I 009

tid: I-009 title: NAS app can't fetch a Google OAuth token ("Premature close" on googleapis.com/oauth2/v4/token) β€” blocks GCS chop read + likely all fresh-token Google calls status: fixed source: IR56M sign-test β†’ chop image / "Next: preview" β†’ HTTP 500 (eop.theestablishers.com, owner report, 2026-06-24) area: infra-nas-egress created: 2026-06-24 updated: 2026-06-24 related: T-047 (IR56M signing β€” the feature that surfaced this), T-073 (NAS deploy) task: T-047

Symptom

On the NAS-hosted prod app (eop.theestablishers.com), the IR56M sign-test chop image and "Next: preview" both return HTTP 500. Read off the wire via an authed browser-console fetch:

GET /api/ir56m-signing/test/chop-image?variant=1  β†’ 500
{"error":"Invalid response body while trying to fetch https://www.googleapis.com/oauth2/v4/token: Premature close"}

Root cause (confirmed β€” NOT the chop feature)

The chop reader (lib/wopc/assets/secureAssets.server.ts) authenticates to Cloud Storage via @google-cloud/storage β†’ google-auth-library, which must first mint a Google OAuth access token by POSTing to https://www.googleapis.com/oauth2/v4/token. On the NAS that HTTPS request fails with "Premature close" β€” the connection is terminated before the response body completes. No token β†’ no GCS read β†’ 500.

Everything else in the chop path is proven correct: - WOPC_ASSETS_BUCKET=aote-system-assets is baked into the deployed image (T-047) and live on the NAS. - Bucket aote-system-assets grants storage.objectViewer to drive-accounting@aote-pms. - All 11 doc/stamps/authorization/erl/22mm/erl-*.png objects exist. - The identical code + credentials read the chop fine from a dev machine (downloaded erl-1.png, 2.25 MB). So the code, env var, creds, and SA permissions are all correct.

Ruled out: missing env var, missing/wrong credentials, GCS 403/permissions, app-level proxy (no HTTP(S)_PROXY anywhere in app code, env templates, services, workflows, or Dockerfile). Runtime is Node 22 (node:22-slim). The failure is specific to the NAS's network egress to googleapis.com.

Why this is escalated (infra, not a one-off)

"Premature close" on the OAuth token endpoint means it blocks any Google call from the NAS that mints a fresh token β€” Cloud Storage (chop + VOID/SAMPLE/IN-CASH stamps) and plausibly Drive uploads (invoice filing, the final IR56M PDF push). Blast radius is the whole NAS↔Google integration, so this is infra, not a chop bug.

Open diagnostic question (blast radius)

Does the NAS prod app successfully do other Google operations — e.g. file an invoice or the signed IR56M PDF to Google Drive? - If Drive also fails → blanket NAS→googleapis egress problem (network/DNS/MTU/firewall on the NAS). - If Drive works → the Drive client is reusing a cached token / different flow while GCS mints a fresh one that hits the failure; points to a token-endpoint-specific or keep-alive/connection issue.

Candidate causes + fixes (to triage next)

  1. NAS network egress (most likely): something between the eop-app container and googleapis.com resets the connection β€” Synology firewall / security pkg, MTU/MSS (e.g. a Tailscale 1280-MTU path without clamping), DNS, or ISP. Owner-side NAS diagnosis: from the container, curl -v https://www.googleapis.com/oauth2/v4/token and a Drive/GCS reachability test.
  2. Hidden proxy in .env.production (can't read it from a dev machine): if HTTPS_PROXY is set on the NAS for bank scraping, gaxios would route the token fetch through it. Fix: NO_PROXY covering googleapis.com (and *.googleapis.com), or unset the proxy for the app.
  3. Code-side mitigation (sidesteps the token endpoint): configure the GCS client to use a self-signed JWT (useJWTAccessWithScope) so it authenticates to storage.googleapis.com directly without the oauth2/v4/token round-trip; and/or add retry-on-Premature close with a fresh connection (helps if it's an undici keep-alive reuse of a half-closed socket). Only helps if the NAS can reach storage.googleapis.com itself.

Fix shipped β€” code-side (candidate #3), pending NAS verification (2026-06-24)

Chosen candidate #3 (self-signed JWT) because it's the one fix that doesn't require NAS shell access (which I couldn't get β€” over Tailscale the NAS rooftop/100.101.13.71 is reachable but no local key is authorized for the Claude/gatekeeper users; the deploy's NAS_SSH_PRIVATE_KEY + CF-Access token are GitHub-Actions-only secrets, not in local env or Secret Manager).

Change (lib/wopc/assets/secureAssets.server.ts, commit 0564ee4e): build the GCS Storage client with a JWT auth client + useJWTAccessWithScope = true, so the bearer token is signed locally with the SA private key instead of being exchanged at …/oauth2/v4/token. That eliminates the failing call entirely. Same SA + credentials, so bucket access is unchanged.

Locally validated before deploy: the produced token is a real JWT (eyJhbGci…, 3 parts β€” not an opaque ya29), proving no token-endpoint round-trip, and it downloads erl-1.png (2.25 MB). tsc + eslint clean.

Residual risk: this only removes the token-endpoint dependency. Object reads still hit storage.googleapis.com; if the NAS's egress problem is broader than the token endpoint, the chop could still fail there (a different error). If so β†’ it's the blanket-egress branch (network fix on the NAS). Owner verifies by re-opening the IR56M sign-test after the deploy lands.

ROOT CAUSE CONFIRMED β€” broken IPv6 egress on the NAS (2026-06-24)

Owner asked to find the why rather than rely on the self-signed-JWT workaround. Found it via DNS topology + the deploy evidence: - www.googleapis.com / storage.googleapis.com publish both A (IPv4) and AAAA (IPv6) records (2404:6800:…). ghcr.io (where the NAS pulls its image on every deploy) publishes IPv4 only β€” no AAAA. - The NAS pulls multi-MB images from GHCR fine (3 green deploys: f749c915/ad9be2dc/89c70e95) β‡’ it is not a bandwidth/MTU problem and the NAS's IPv4 egress is healthy. - So the NAS's IPv6 is enabled-but-not-routable: Node's happy-eyeballs tries Google's AAAA first, the IPv6 connection goes nowhere and dies mid-handshake β†’ "Premature close". IPv4-only services (GHCR, GitHub, etc.) are unaffected β€” which is why only Google calls fail. - This also invalidates the self-signed-JWT fix (0564ee4e) as a standalone cure: it removes the token-endpoint call, but the object download still goes to storage.googleapis.com (has AAAA) β†’ would still die over IPv6. The real fix is forcing IPv4.

Fix (force IPv4): 1. Network-level (root, owner-doable in DSM): disable IPv6 on the NAS network interface (Control Panel β†’ Network β†’ Network Interface β†’ Edit β†’ IPv6 β†’ Off). Fixes the chop, Drive uploads, and every Google call from the NAS; reversible. 2. App-level (alternative, deployable): bake NODE_OPTIONS=--dns-result-order=ipv4first into the image (same mechanism as WOPC_ASSETS_BUCKET) so Node prefers the working IPv4 path for all outbound β€” fixes the app's Google calls without a network change. The self-signed-JWT change can then stay or be reverted; it's harmless either way.

Fix applied (option 2 β€” owner's choice, 2026-06-24): owner opted to keep the NAS broadcasting IPv6 and have the fix live in the app instead ("I'd rather that my NAS continues to broadcast its IPv6 and with you [baking] the use IPv4 first into the web app instead"). Shipped ENV NODE_OPTIONS=--dns-result-order=ipv4first in the Dockerfile runner stage (commit bc2f72f6) β†’ Node prefers the working IPv4 path for all outbound (chop read, Drive uploads, every Google call), IPv6 left enabled on the NAS. The self-signed-JWT change (0564ee4e) was reverted in the same commit β€” it only dodged the token endpoint, not the IPv6-bound object download, so it wasn't a real cure; secureAssets.server.ts is back to the standard Storage({ credentials }) flow (consistent with lib/drive/client.ts). Verified --dns-result-order is a valid NODE_OPTIONS flag and dns.getDefaultResultOrder() returns ipv4first; tsc + eslint clean.

Caveat recorded for the owner: the NAS's IPv6 stays enabled-but-broken for internet, so any other NAS service that reaches the public internet over IPv6 would hit the same wall until the IPv6 path itself is fixed (a separate router/NAS-network task). For the eop app, ipv4first fully covers it.

Why fixing IPv6 isn't pursued (owner Q, 2026-06-24): the NAS sits behind a TP-Link Deco on a cellular SIM (CGNAT β€” "the real IP address is not traceable"). Cellular-SIM + Deco is a notoriously unreliable combo for IPv6: carriers often hand out partial / no routable IPv6, and Deco's IPv6-over-cellular support is limited. CGNAT itself is an IPv4 constraint (it's why inbound uses the Cloudflare Tunnel + Tailscale), not the direct IPv6 blocker, but the overall cellular setup makes routable IPv6 uncertain and a rabbit hole. Since IPv4 works perfectly, the owner confirmed sticking with the ipv4first app fix rather than chasing carrier/Deco IPv6.

The earlier 0564ee4e deploy failure (client_loop: send disconnect: Broken pipe mid-docker pull) was a transient Cloudflare-tunnel drop β€” the re-run (attempt 2) completed green, so the deploy channel is healthy; unrelated to the IPv6 root cause (GHCR is IPv4-only).

Closed β€” fixed (2026-06-24)

Verdict: root cause was the NAS's broken IPv6 egress (Google publishes AAAA; the NAS's IPv6 isn't routable, so Node's happy-eyeballs died on it mid-handshake β†’ "Premature close" on only the Google calls). Fixed app-side by forcing IPv4 (ENV NODE_OPTIONS=--dns-result-order=ipv4first in the Dockerfile runner stage), per the owner's choice to keep IPv6 broadcasting on the NAS. The interim self-signed-JWT change was reverted in the same commit β€” it only dodged the token endpoint, not the IPv6-bound object download, so it wasn't a real cure.

Verified: - Owner confirmed the chop renders on the NAS ("tested it on my phone … things are working now") β€” that was on the interim 0564ee4e build; the ipv4first build is the same chop outcome via the standard auth path plus it fixes Drive and every other fresh-token Google call. - ipv4first validated locally (dns.getDefaultResultOrder() β†’ ipv4first, valid NODE_OPTIONS flag); tsc + eslint clean. - The ipv4first image (bc2f72f6) deployed green to the NAS via nas-image.yml β€” all steps incl. "Deploy to NAS (pre-pull + recreate)" succeeded, so it is the live image now.

Commit: bc2f72f6 (the ipv4first ENV + revert of the self-signed-JWT workaround). Interim, superseded: 0564ee4e. Related (separate fix, same incident): WOPC_ASSETS_BUCKET bake under T-047.

One residual check (non-blocking): the owner hasn't re-exercised the live ipv4first build directly β€” the strongest final confirmation is a Drive write from the NAS (file an invoice or push a signed IR56M PDF), since Drive was the untested half of the blast radius. If anything Google-bound regresses, reopen.

Blast radius for other agents: NODE_OPTIONS=--dns-result-order=ipv4first is process-wide on the eop-app container β†’ every outbound connection from the app now prefers IPv4. Intended and safe (the NAS's IPv4 egress is healthy). The NAS's IPv6 is left enabled-but-broken-for-internet, so any other NAS service that reaches the public internet over IPv6 would hit the same wall until the IPv6 path itself is fixed (a separate router/NAS-network task β€” out of scope here; cellular-SIM + Deco, see above). lib/wopc/assets/secureAssets.server.ts is back to the standard Storage({ credentials }) flow (no special JWT config), consistent with lib/drive/client.ts.

Source: EOP Local Assistance (fork) Β· session edb0abc7-6373-449b-8e31-379fd38d391b (local Claude Code).

Notes

  • Attestation: read AGENTS.md before filing. Source: EOP Local Assistance (fork) Β· session e527df5f-d3ea-4716-8076-b46319d9d830 (local Claude Code).
  • This ticket is the symptom + root-cause record; if the fix turns out structural (network reconfig or a code change to the auth flow), escalate to a T-NNN task and cross-link.