uid: T-212 title: The NAS can't look up the web app's address about 60% of the time — every NAS cron has been failing silently since Aug 29 status: done area: infrastructure created: 2026-09-24 updated: 2026-10-06 owner: girafeev1 assignee: Records (Infrastructure) related: T-175, T-084, T-147, T-096

T-212 — NAS name lookups are failing, and every scheduled job with them¶

Why this exists (owner, 2026-09-24)¶

The owner forwarded the alert email "[EOP] GCP cost detail (BigQuery sync) is not delivering" — "I've been receiving these emails, can you look into the possible issues?"

What is actually happening, plainly¶

The web app, the code, BigQuery and the credentials are all fine. The NAS — which is what wakes each scheduled job up every day — cannot look up the web app's internet address on most mornings, so the request never leaves the building. On the days the lookup works, everything succeeds.

The alert email only covers one of those jobs. It is happening to every one of them, and only the BigQuery job has a watchdog loud enough to email about it. The rest have been failing silently.

Job on the NAS	Runs since Sep 1	Could not look up the address
Billing pipeline sync (the email's subject)	24	17
Reconciliation	23	13
Workspace auto-match	23	13
Verify attachments	23	13
IR56M distributions	23	15
DMARC heartbeat	23	13
Reconciliation heartbeat	23	13
The watchdog that sends these alerts	92	59

Every hour of the day is affected, so it is not one bad time slot. Before Aug 29 every run succeeded — the billing job ran clean every day from Aug 1 to Aug 31, then began failing on Sep 1. The other jobs started failing Aug 29–31.

How this was established (evidence, not inference)¶

The freshness marker (aote-system / systemHealth / scrape-gcp-cost-table) only landed on Aug 31, Sep 3, 7, 11, 15, 21 and 24 — reconstructed from each alert's "last ingest was Nh ago".
BigQuery's own job history shows the sync's queries ran on exactly those days and no others, and every query that ran succeeded in 1–2 s. That ruled out BigQuery failing and ruled out the stamp being lost after a successful run — the sync simply never started on the missing days.
The NAS cron log (/volume1/docker/eop/logs/eop-cron-billing-pipeline-sync.log, read through DSM File Station, read-only) says the same thing on every missing day: curl: (6) Could not resolve host: eop.theestablishers.com.
The other seven cron logs, read the same way, show the identical error at the rates above.

An initial suspicion worth recording because it was wrong: the endpoint writes its freshness stamp without waiting for it (void ….set() in pages/api/cron/billing-pipeline-sync.ts), the same pattern that caused an earlier billing-email alert. The BigQuery history disproved it here — every run that reached BigQuery also landed its stamp. The pattern is still worth fixing, but it is not this.

Why the NAS can't look addresses up¶

⚠️ Superseded — see the 2026-10-06 entry at the end. The cause was the NAS's own network port dropping, not the router's DNS. The analysis below is kept as the record of what was thought at the time. Edit-in signed: Records (Infrastructure) · https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

Read from DSM (Control Panel → Network, read-only):

The NAS gets its DNS automatically from the router, and relies on one resolver — 192.168.8.1 — with no backup (the secondary is blank).
192.168.8.1 is a Linksys EA8300, which reaches the internet through a link to the TP-Link Deco mesh at 192.168.68.1.
That link was already known to be unreliable: a boot-time script, /volume1/docker/link-keepalive.sh (DSM event task "Link Keepalive"), pings the Deco every 3 seconds to keep it up and logs drops to /var/log/link-keepalive.log.

So when the EA8300's own lookups fail — or its link to the Deco drops — the NAS has nothing to fall back on. Which of those two it is on any given morning is not yet known: the keepalive's drop log sits in /var/log, which File Station cannot read, and the root command channel is broken (T-096).

What was missed, and whether it came back¶

These jobs work through whatever is pending at the moment they run, so a missed day is mostly a delay rather than lost data — the next successful run catches up. GCP cost detail in particular replaces the whole current and previous month on every run, so nothing is permanently missing there. Anything time-sensitive is late, though: an IR56M form waiting to go to an employee waits until the next day the lookup happens to work.

Two further findings¶

The watchdog runs on the same NAS it is watching. When the NAS can't reach the app, the watchdog can't run either — so alerts arrive late or not at all, and recovered alerts don't clear. It failed 59 of its 92 runs since Sep 1.
The NAS and the repo have drifted. The repo's scripts/nas-cron/_common.sh is newer than the copy running on the NAS (it reads the secret out of the env file instead of executing the whole file). And four scripts run on the NAS with no copy in the repo: billing-pipeline-sync.sh, scrape-heartbeat.sh, ir56m-distributions.sh, rotation-reminder.sh.

Fixes, for the owner to choose¶

Give the NAS a backup DNS server (owner, DSM, one minute). Control Panel → Network → General → Manually configure DNS server → keep 192.168.8.1 as Preferred, add 1.1.1.1 as Alternate. Covers the case where the router's lookups fail while the internet link is up. Keeping the router first means nothing is lost if an outside resolver is ever blocked.
~~Retry when the address can't be looked up (code, in scripts/nas-cron/_common.sh). A lookup or connection failure guarantees the request never reached the app, so retrying it is safe — unlike retrying after an error response, which could run a job twice. A few attempts over a couple of minutes would ride out short drops. Needs the file copied onto the NAS, and is the moment to bring the four missing scripts into the repo.~~ — withdrawn as a fix: the outages last 20–48 hours, so a retry measured in minutes rescues ~1% of failed runs (see the 2026-09-24 (later) entry below). Edit-out signed: Records (Infrastructure) · https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD
The link itself (owner, physical). If the EA8300 reaches the Deco wirelessly, a cable — or putting the NAS on the Deco directly — removes the hop that needed a keepalive in the first place.

Nothing has been changed on the NAS, the router or in the code. Every read above was read-only.

Log¶

2026-09-24 — Investigated; opened¶

✅ Read AGENTS.md · checked the board by scope (no task covers NAS network reliability; T-175 is the task that last "resolved" these same alert emails, by scheduling the job on the NAS). Source: Records (Infrastructure) · https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

Owner, verbatim: "I've been receiving these emails, can you look into the possible issues?"

Proposed: the three fixes above. Approved: nothing yet — awaiting the owner's choice.

2026-09-24 (later) — The outages last a day or two, so retrying won't fix it¶

✅ Read AGENTS.md. Source: Records (Infrastructure) · https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

Measured from the same NAS cron logs (read-only), to settle how useful fix 2 would be.

The failures come in long stretches, not blips. Jobs that start in the same minute have never disagreed (21 of 21 pairs: both failed or both worked); jobs 20–60 minutes apart differ only 6% of the time. Grouping consecutive failures across all jobs gives 14 outages since Sep 1, most lasting 20–48 hours, separated by a day or two of lookups working normally:

Outage started (UTC)	Failed runs	Lasted (seen failing → at most)
Sep 03 19:00	18	33.5 h → 42 h
Sep 07 19:00	19	39.5 h → 48 h
Sep 09 19:00	11	21.5 h → 26.5 h
Sep 11 16:15	23	48 h → 56.5 h
Sep 15 04:30	13	30 h → 39 h
Sep 17 19:30	17	33 h → 39.5 h
Sep 21 22:30	18	42 h → 47 h
(seven shorter ones)	1–6 each	0–20 h

What a retry would have rescued — if each failed job kept retrying for this long: 15 min → 1% · 1 h → 1% · 6 h → 16% · 24 h → 71% of the 156 failed runs.

So fix 2 is withdrawn as a solution. My earlier line that fixes 1 and 2 together "cover most of it" was wrong: a retry measured in minutes saves almost nothing, and a job that retries for a whole day is a poor design. It may still be worth having for short blips, but it is not the fix.

The internet link itself also drops. Cloudflare's record of the NAS tunnel (one read-only query of its connection state — the tunnel admin token, used here strictly to diagnose the NAS link, per AGENTS.md) shows all four connections were re-opened today between 01:31 and 01:37 UTC. That happened three hours into a period when lookups were working, so the link has its own drops, separate from the long lookup outages.

Still not established: whether the day-long outages are the router's lookups getting stuck while the link stays up, or the link being down for that long. Two reads that would have settled it did not work — the keepalive's drop log is not in DSM's Log Center, and the Container Manager log API refused every parameter shape tried (code 114). The shape of the outages — a day or two stuck, then recovering on their own — is typical of a router whose lookup service hangs until something resets it.

Revised recommendation:

Backup DNS on the NAS first — one minute, harmless, and if the outages are the router's lookups it ends them. Keep 192.168.8.1 as Preferred and add 1.1.1.1 as Alternate.
Then check the logs again after a few days (read-only; I can do it). Failures stop → it was the router's lookups. Failures continue → it is the link, and fix 3 (a cable, or the NAS on the Deco directly) is the cure.
A quick test the owner can do during the next outage: restart the EA8300. If lookups come straight back, the router's lookup service is what hangs.

2026-10-06 — Real cause found, a fix is in place and holding; handed off¶

✅ Read AGENTS.md. Source: Records (Infrastructure) · https://claude.ai/code/session_018RDB37kCqfouHdygVXTAtD

Owner, verbatim: "Check if what you're working on has been proceeding properly and off your hand, so that you will let it go."

Correction, plainly: the cause was not the router's DNS. DSM's system log (Log Center, read-only) shows the NAS's own network port, "LAN 1", dropping and reconnecting dozens of times — on Sep 20 alone more than thirty times, often seconds apart. After several of those drops the NAS failed to get an address from the router and gave itself a dead fallback address (169.254.7.195, assigned on Sep 20, 22, Oct 1 and Oct 4, among others), which cannot reach anything. It then stayed stuck like that until a later reconnect got a proper address — which is why each outage lasted a day or two, why every job failed together, and why it showed up as "could not resolve host". It also explains why a backup DNS server could never have helped: with no working address there is nobody to ask. The keepalive script pings the Deco, one hop further out, so it was watching the wrong link and never saw this.

What changed, and what happened since:

Backup DNS 8.8.8.8 is now present on the NAS (arrived through the router's automatic settings — dns_manual is still off), but failures continued after Sep 24: an outage Sep 30 → Oct 2 (~42 h) and Oct 3 → Oct 4 (~24 h), plus timeouts Sep 27–28. Consistent with the corrected cause.
Oct 5, ~09:19 HKT: the NAS was moved to its second port, "LAN 2" (now eth1, 192.168.8.24). Every job since has succeeded — billing sync, reconciliation and the watchdog alike.

Verdict: very likely fixed by the port change, not yet proven — outages used to arrive every couple of days, so about a week of clean runs is what proves it. If outages return, look at the cable and the router port before anything else. Note that LAN 1 is still flapping (a drop and reconnect at 01:07 HKT on Oct 6): whatever is plugged into it is still faulty, and if the NAS ever falls back to that port the problem comes straight back. Unplugging that cable, or replacing it, removes the risk.

Closed as done because the investigation is complete and the fix is the owner's physical change, already made. Nothing in the app or the cron scripts needed changing; the watchdog alerts should simply stop. Record: PR #1038 (two investigation entries + this one).

Blast radius for other agents: none in code. Every NAS cron (reconciliation, workspace auto-match, verify attachments, IR56M distributions, DMARC and reconciliation heartbeats, billing sync, the scrape watchdog) was failing on most days between Aug 29 and Oct 4 for this reason. If you see gaps in any of their data across that period, this is why — and they catch up on their next successful run.