Skip to content

Inspect-Period "Data Integrity" stays red on a STALE reconciliation-watchdog alert after the cron recovered

Symptom

The Inspect-Period modal's "Data Integrity" light shows red ("some transactions have data integrity issues") without spelling out what. Owner asked what triggered it.

What actually triggered it (investigated live, 2026-06-23)

Not an invoice/transaction data problem. The Inspect/Close readiness check (lib/periodClose/readiness.server.ts) gates on unread data_integrity notifications for the current user (in aote-system / notifications). Live query found 5 unread data_integrity docs, all one alert fanned out to 4 recipients:

"Reconciliation watchdog is not running" β€” "Last reconciliation report was 72h ago (threshold 36h)…" β€” severity error, created 2026-06-22 07:55 UTC.

The reconciliation reports collection confirms the gap and the recovery: - last report 2026-06-19 08:22 β†’ then nothing until 2026-06-22 08:02 (β‰ˆ72h gap) β†’ watchdog fired, correctly. - reports have resumed; today's ran 2026-06-23 03:00 UTC, exactly on the in-app scheduler's 0 3 * * * (T-084). The cron is healthy again.

Root cause of the gap = the Vercel→NAS cron migration window (the daily reconciliation cron was off-air while the in-app scheduler was being wired up on the NAS). Same underlying event as the watchdog email in the I-006/T-095 thread — that thread fixed who receives the alert; it did not address the stale alert lingering in the books-close gate.

The residual gap (why it's still red)

The alert is now stale (the cron recovered) but the 5 notifications are still read: false, so the Inspect check stays red. Two reasons nothing cleared them: 1. No auto-resolve on recovery. The heartbeat (pages/api/cron/reconciliation-heartbeat.ts) fires a notification when stale but never marks prior stale alerts read once a fresh report lands. There's no "all clear". 2. The cleanup script doesn't cover this type. scripts/cleanup-stale-integrity-notifications.ts only re-checks invoice-tied alerts (relatedEntityType === 'invoice'); a watchdog alert (relatedEntityType β‰  invoice) is skipped, so it's never auto-cleared.

Also note: the Inspect modal shows only a compact 3Γ—2 traffic-light table β€” the per-alert detail (title / message / deep-link) renders in the Close modal, not Inspect β€” which is why it reads "red, no details".

Suggested fixes (owner to pick)

  • Immediate: mark the 5 stale data_integrity watchdog notifications read (one-off). Clears the red light now. (Mutates 4 users' inboxes β€” awaiting owner OK; not done unprompted.)
  • Durable (small): have the heartbeat mark prior unread reconciliation-watchdog alerts read when it observes a fresh report within threshold (emit an implicit "all clear"); and/or extend cleanup-stale-integrity-notifications.ts to also resolve watchdog-type alerts when the latest report is fresh.
  • UX (optional): surface the alert titles in the Inspect modal too (not just Close), so "red" always comes with the reason.

Decision log

2026-06-23 β€” opened (investigated, root cause found)

  • βœ… Attestation (Accounting (Diagnostics)): read AGENTS.md; scope-scanned tasks + tickets β€” the watchdog email recipient scope is I-006/T-095; this ticket is the distinct stale-alert persistence in the period-close gate scope, cross-linked rather than duplicated.
  • Source: Accounting (Diagnostics) Β· https://claude.ai/code/session_01G58Y71noihrYCDEDMexmea

2026-07-02 β€” fixed (gate-hardening)

  • βœ… Read AGENTS.md. Immediate symptom already clear (live Firestore: 0 unread data_integrity docs). Structural fix in this commit: the reconciliation watchdog (pages/api/cron/reconciliation-heartbeat.ts), on a HEALTHY heartbeat, now auto-clears lingering "Reconciliation watchdog is not running" alerts (relatedEntityId: 'reconciliation-watchdog', all recipient copies) by marking them read β€” so a transient watchdog blip that recovers no longer holds the Inspect-Period integrity gate red. The gate (readiness.server.ts) is untouched (stays strict on genuinely-unread alerts).
  • Blast radius: reconciliation heartbeat cron + the period-close readiness gate. No financial-logic change.
  • Approved: owner (2026-07-02): "Proceed to fix I-008 as well."
  • Source (edit): EOP Local Assistance (fork) Β· session edb0abc7-6373-449b-8e31-379fd38d391b