Skip to content

T 105

uid: T-105 title: "Smart" NAS deploy auto-retry β€” self-heal transient tunnel drops, fail-fast on real failures status: dropped area: infra created: 2026-06-24 updated: 2026-07-01 related: T-091 (NAS auto-deploy β€” this hardens it), I-009 (the Cloudflare-tunnel drops that surfaced it)

Why

The Deploy to NAS step in nas-image.yml (built under T-091) SSHes into the NAS over the Cloudflare tunnel to docker pull + recreate eop-app. That tunnel drops transiently β€” twice in one day the step died with client_loop: send disconnect: Broken pipe after the image had already built + pushed to GHCR, leaving the new image un-deployed. Each time a human (the agent) had to notice β€” by manually polling the Actions API β€” and hand-run rerun-failed-jobs. The owner asked for this to self-heal, but "smartly": it must not paper over a real failure, nor fight the concurrency cancellation.

Owner instruction (verbatim, 2026-06-24): "Let's proceed for auto-retry and make it 'smart' so that it doesn't retry on a build that's not suppose to succeed and builds that are cancelled because of new commits"

What shipped

An in-step retry inside the Deploy to NAS step ONLY, keyed on the ssh exit code. The three guardrails fall out of where and how the retry is scoped:

Failure ssh exit Behaviour Why
Cloudflare-tunnel / connection drop (Broken pipe) 255 retry (bounded: 3 attempts, 10s→20s backoff) transient — the thing we keep hitting
Remote pull/recreate errored (bad image, disk, perms, missing env-file) non-zero, β‰  255 fail fast, no retry a deploy that isn't meant to succeed β€” owner guardrail #1
Real build / type error n/a β€” fails the earlier docker/build-push step never reaches the retry the retry wraps only the deploy step β€” owner guardrail #1
Run cancelled by a newer commit (concurrency: cancel-in-progress) step gets SIGTERM loop dies mid-flight, no retry superseded build β€” owner guardrail #2

Implementation: the idempotent NAS script is written to a file once ($RUNNER_TEMP/nas_deploy.sh), then a while loop re-runs ssh … < file and inspects rc β€” 0β†’done, 255β†’retry (bounded), anything elseβ†’exit $rc. The NAS script is idempotent (re-pull, compare OLD vs NEW image digest, skip recreate if already on it), so a retry after a partial attempt converges safely. ssh exit 255 specifically means "ssh/transport failed" (as opposed to the remote command's own exit code passing straight through), which is exactly the transient-vs-real discriminator the owner asked for.

Layer 2 β€” build-step backstop (.github/workflows/nas-deploy-autoretry.yml, added 2026-06-24)

The in-step retry only covers the deploy step. The very next deploy proved transient flakes also hit earlier steps: a real run died at docker/setup-buildx-action (pulling the buildkit image β€” Docker Hub rate-limit / runner network), so the build never reached the deploy step and the in-step retry couldn't help.

Owner approval to add the backstop (verbatim, 2026-06-24): "Sure, integrate the auto-retry please"

A separate on: workflow_run workflow fires when Build NAS app image completes and re-runs the failed job β€” only "smartly", same philosophy: - conclusion gate: acts ONLY on conclusion == 'failure'. A run cancelled by a newer commit is 'cancelled', so a superseded build is never retried (owner guardrail #2). - step gate: inspects (jobs API + jq) which step failed, and re-runs ONLY a known-transient INFRA step β€” setup-buildx-action, Deploy to NAS, login-action, checkout. If it's the actual docker/build-push step (where next build type/code errors surface) β€” or an unidentified step β€” it's left FAILED, never retried (owner guardrail #1). - bounded: run_attempt < 3 β€” at most 3 total attempts; nothing loops.

The two layers compose: a tunnel Broken pipe self-heals fast in-step; a buildx-setup flake (or a deploy still failing after its 3 in-step tries) self-heals via the backstop's job re-run. Validated pre-commit (YAML parses, bash -n clean, jq step-extractor confirmed on a sample failure payload).

Note β€” does NOT fix deploy starvation: separately observed 2026-06-24, rapid cloud-agent PR merges were cancelling in-flight builds back-to-back (cancel-in-progress), leaving the NAS on a stale image with no failure at all. That's a merge-cadence issue (the backstop correctly ignores cancellations) needing a different fix β€” a scheduled "is newest main actually deployed?" check β€” now built as Layer 3 below.

Layer 3 β€” deploy-only "ensurer" (.github/workflows/nas-deploy.yml, added 2026-06-24)

Owner approval (verbatim, 2026-06-24): "Sure" β€” to "build that deploy-ensurer now to get the latest onto the NAS".

A standalone DEPLOY-only workflow (no build) that SSHes to the NAS, pulls the latest eop-app:main, and recreates the container if the digest changed β€” reusing the same smart in-step retry. It runs on its own concurrency group (nas-deploy, cancel-in-progress: false) so a build push can never cancel it, plus a */15 schedule so the NAS converges to the latest pushed image within ~15 min no matter how badly the build pipeline is being starved by rapid merges. workflow_dispatch forces it immediately. (The SSH / pull / recreate block mirrors nas-image.yml's deploy step β€” kept duplicated under time pressure; unify into a composite action later.)

Why this is the right shape: every build's build-push step pushes :main to GHCR before the deploy step runs, so even a build whose deploy got cancelled has already published its image. The ensurer's only job is to make the NAS actually pull that image β€” which it can do independently of the (starved) build runs.

Out of scope (deferred β€” pending owner decision)

The notification half of the broader ask ("a mechanism for the NAS to communicate a successful/failed build to the related party") is NOT in this task. It needs the owner to pick a channel β€” reuse the app's existing admin-email path (same one the dmarc / reconciliation heartbeat jobs use) vs a new Telegram / Slack webhook. Left as a pending decision; not built.

Decision log

  • βœ… Attestation: read AGENTS.md + checked the board by scope (T-091 is done, so this is a new follow-up, not a dup). Source: EOP Local Assistance (fork) Β· session edb0abc7-6373-449b-8e31-379fd38d391b (local Claude Code).
  • Proposed by: EOP Local Assistance (fork). Approved by: the owner (verbatim quote above, 2026-06-24).
  • Verification: the workflow YAML parses and bash -n is clean on the extracted deploy script (pre-commit). The happy path self-tests on the next push to main (this commit triggers a build that runs the new step). The retry path (255 β†’ retry) only exercises on a real tunnel drop, so it's confirmed opportunistically the next time one happens β€” at which point this flips to done with the evidence.

2026-07-01 β€” dropped (superseded by the Vercel move)

  • βœ… Read AGENTS.md. Verdict: DROPPED (superseded). The app moved back to Vercel, retiring the NAS app-deploy pipeline this task optimized. The NAS eop-app container was stopped 2026-07-01 (verified eop.theestablishers.com β†’ Vercel). the smart retry + backstop + deploy-only ensurer (66e43561/5b375b10/377e0b32) already landed and is harmless, but it's no longer exercised. Reopen if NAS app-hosting is ever revisited. (The NAS still runs the billing scraper + home automation β€” unaffected; those don't use this pipeline.)
  • Owner context (2026-07-01, verbatim): "We can hold on NAS web app hosting related tasks as I have moved the web app hosting back to ... Vercel."
  • Source (edit): EOP Local Assistance (fork) Β· session edb0abc7-6373-449b-8e31-379fd38d391b