Seen on a live install. A tracked cachekeep target whose prewarm keeps failing is retried every 5 minutes indefinitely — no backoff, no failure budget, and no stop once the real cache TTL has elapsed.
From my plugin log this evening, one session:
16:59:53 WARN [cachekeep] prewarm failed {"session":"ses_018f...","status":529}
17:04:50 WARN [cachekeep] prewarm failed {"session":"ses_018f...","status":529}
17:09:50 WARN [cachekeep] prewarm failed {"session":"ses_018f...","status":529}
... 14 consecutive failures, exactly 5 minutes apart ...
18:04:52 WARN [cachekeep] prewarm failed {"session":"ses_018f...","status":529}
14 retries spanning 65 minutes, across a 60-minute cache TTL. 45 such failures across 6 distinct sessions in the same log.
Mechanism — packages/core/src/cachekeep.ts
The failure branch reschedules by a constant:
target.cacheExpiresAt = now + CACHE_KEEP_PREWARM_LEAD_MS + 5 * 60_000
and tick() treats a target as due when target.cacheExpiresAt <= now + CACHE_KEEP_PREWARM_LEAD_MS. The lead term cancels, so the retry interval is a fixed 5 minutes regardless of how many times it has already failed. pruneTargets() only drops a target more than 2 * CACHE_KEEP_TTL_MS past its expiry, and each failed attempt pushes expiry forward, so the target keeps itself alive.
Two consequences:
- No backoff against an overloaded upstream. 529 means overloaded. Retrying on a fixed cadence, from every process running the plugin, is the opposite of what that status asks for.
- Retries continue past the point of usefulness. Once more than
CACHE_KEEP_TTL_MS has passed since the last success, the cache entry is gone. A later retry that succeeds is no longer refreshing a warm entry — it pays a full cache write. In the log above the last 5+ retries were past that line.
Fix direction: track consecutive failures per target and apply bounded exponential backoff with jitter, and stop retrying (or drop the target) once the elapsed time since the last success exceeds the real cache TTL, since there is no longer a warm entry to keep alive.
Happy to send a PR for this if you want it — I have one open for a separate tick() bug in the same file and can keep them independent.
Seen on a live install. A tracked cachekeep target whose prewarm keeps failing is retried every 5 minutes indefinitely — no backoff, no failure budget, and no stop once the real cache TTL has elapsed.
From my plugin log this evening, one session:
14 retries spanning 65 minutes, across a 60-minute cache TTL. 45 such failures across 6 distinct sessions in the same log.
Mechanism —
packages/core/src/cachekeep.tsThe failure branch reschedules by a constant:
and
tick()treats a target as due whentarget.cacheExpiresAt <= now + CACHE_KEEP_PREWARM_LEAD_MS. The lead term cancels, so the retry interval is a fixed 5 minutes regardless of how many times it has already failed.pruneTargets()only drops a target more than2 * CACHE_KEEP_TTL_MSpast its expiry, and each failed attempt pushes expiry forward, so the target keeps itself alive.Two consequences:
CACHE_KEEP_TTL_MShas passed since the last success, the cache entry is gone. A later retry that succeeds is no longer refreshing a warm entry — it pays a full cache write. In the log above the last 5+ retries were past that line.Fix direction: track consecutive failures per target and apply bounded exponential backoff with jitter, and stop retrying (or drop the target) once the elapsed time since the last success exceeds the real cache TTL, since there is no longer a warm entry to keep alive.
Happy to send a PR for this if you want it — I have one open for a separate
tick()bug in the same file and can keep them independent.