Skip to content

T 116

uid: T-116 title: Speed up / harden NAS deploys — trim the volatile app layer; docker pull resilience status: done area: infra, deploy created: 2026-06-25 updated: 2026-06-25 renumbered_from: T-113 → T-116 (collision — T-113 taken by Accounting (Diagnostics) receipt→GL; T-114/T-115 by Records (Infrastructure)) related: T-073, T-106, T-091, T-105

Why (owner, 2026-06-25)

The owner flagged a NAS deploy that took unusually long. Diagnosed from the GHA log (Build NAS app image, run 28144763788, commit 12964c1c):

  • Build itself fine (~12 min, normal).
  • Deploy step took 12m 39s vs. the prior run's ~4m 52s. The whole excess was a single image layer (c846f2…, the app-code layer) stalling on the GHCRβ†’NAS pull: attempt #1 hung ~6m55s with zero bytes, docker declared it failed and retried, attempt #2 pulled it in ~4m49s. So docker's own per-layer retry recovered it β€” max-download-attempts (default 5) was enough; the cost was the ~7-min stall-detection.
  • Ruled out: deps-layer cache miss (8 layers incl. the ~2GB node_modules came back Already exists β€” the T-106 registry buildcache is working) and a Cloudflare-tunnel drop (succeeded on ssh attempt 1; the T-105 smart-retry never engaged).

Owner asked for two improvements: #2 trim the app layer, #1 docker pull resilience.

#2 β€” Drop the webpack build cache from the runtime image βœ… (landed)

Dockerfile runner stage does COPY --from=builder /app /app, which carried the freshly-built .next/cache (webpack build cache) into the image. .dockerignore only filters the build context (host .next), not stage→stage COPY, so the cache shipped. That cache:

  • is pure build-time state β€” next start never reads it at runtime (output: standalone omits it for the same reason);
  • cache-busts every build (new content hashes) β†’ it's dead weight on the exact app layer the NAS MUST re-pull every deploy.

Fix: RUN rm -rf .next/cache in the builder after next build / the node_modules hoist, before the runner copies /app. Shrinks per-deploy NAS transfer with zero runtime effect. (Any ISR / next-image runtime caches under .next/cache are recreated on demand β€” they don't need to ship.)

#1 β€” docker pull resilience on the NAS βœ… (landed; owner approved "do it now")

Deeper analysis: the knobs first proposed (max-download-attempts, max-concurrent-downloads) would not have shortened the observed ~7-min stall β€” that's a TCP read-hang, and neither knob adds a no-progress timeout; docker's existing retry already recovered. A naive timeout-wrapper around docker pull is unsafe too (can't distinguish a stalled pull from a legitimately slow large-layer pull β†’ risks killing healthy pulls in a loop). So #2 is the real lever; this is general hardening β€” extra retry headroom + live-restore: true so future dockerd restarts no longer drop running containers.

Applied to /var/packages/Docker/etc/dockerd.json (Synology "Docker" package, docker 20.10.3) via the SSH-over-Cloudflare-tunnel root channel. Merged in (preserving the existing 4 keys):

"live-restore": true,
"max-concurrent-downloads": 6,
"max-download-attempts": 10

How it was applied safely: backup β†’ python-merge (preserves existing keys) β†’ JSON-validate β†’ install, all before any restart (config is only read at daemon start, so writing is reversible). Then a detached, self-healing restart script (setsid nohup, survives the channel drop): synopkg restart Docker β†’ poll for dockerd β†’ auto-rollback to the backup if dockerd doesn't return β†’ explicitly restart MQTT (its restart policy is no, so it wouldn't auto-recover) β†’ verify. Result file /var/packages/Docker/etc/_t116_apply.out; backup kept at /var/packages/Docker/etc/dockerd.json.bak-T116.

Verified (2026-06-25 04:17 UTC): synopkg restart Docker rc=0; docker info β†’ Live Restore Enabled: true; effective config carries all three keys; all 8 containers back up (eop-app, ram-monitor, bank-login, cloudflare-cloudflared, homeassistant, homebridge, Zigbee2MQTT, MQTT). The one-time restart blip was ~50s (04:16:29β†’04:17:19). This was the last container-dropping restart for config changes β€” live-restore makes future ones seamless.

Decision log

2026-06-25 β€” opened; #2 landed, #1 gated

  • βœ… Attestation (Accounting (Infrastructure)): read AGENTS.md; this is a follow-up to my T-106 (minimize per-deploy NAS transfer) / T-091 deploy work. Branch signal main 🟒 Β· nightly πŸ”΄ β†’ main only; mirrored to taskboard.
  • Source: Accounting (Infrastructure) Β· https://claude.ai/code/session_015P6KzVYsQCLgEmUjR9bMwM
  • Proposed by: the owner (deploy-speed investigation β†’ "#1 & #2 please"). Approved by: the owner (same prompt).
  • Renumber: opened as T-113, but T-113/T-114/T-115 were taken by parallel agents (Accounting (Diagnostics), Records (Infrastructure)) before this pushed; renumbered to T-116 per the AGENTS.md "whoever merges second renumbers" rule. The in-code Dockerfile comment uses [T-116].
  • #2 landed: Dockerfile β€” rm -rf .next/cache before the runner copy. Commit to main (triggers a normal build+deploy; this deploy already ships the smaller layer).
  • #1 landed: owner answered "do it now". Applied live-restore: true + max-concurrent-downloads: 6 + max-download-attempts: 10 to the NAS dockerd.json via the SSH root channel; self-healing detached restart with auto-rollback; verified Live Restore Enabled: true and all 8 containers back (incl. MQTT, restart-policy=no, explicitly restarted). One-time ~50s blip; backup at dockerd.json.bak-T116. Both #1 and #2 done β†’ task done.