feat: add stale-on-error Redis recovery

[lan17]

Summary

Add opt-in stale-on-error recovery from physically retained Redis values.

• F = ttlSec[CacheLayer.REMOTE] remains the logical freshness age.
• M = staleOnErrorMaxAgeSec is the absolute maximum recovery age.
• Normal reads serve only age < F.
• After a definitive normal Redis miss and a source-of-truth rejection, one independent reread may serve age < M.
• Recovered data is returned without refreshing Redis, populating process-local cache, starting shadow work, or otherwise promoting it to fresh.

Closes #117

Contract and flow

age < F        fresh
F <= age < M   stale; eligible only after the source rejects
age >= M       unavailable

flowchart TD
  A[Redis read with maxAge F] -->|hit| B[Return fresh]
  A -->|definitive miss| C[Call source of truth]
  A -->|error or timeout| D[Call source; recovery forbidden]
  C -->|success| E[Return and publish normally with Redis PX M]
  C -->|rejection| F[Redis reread with maxAge M]
  F -->|eligible| G[Return retained value without publication]
  F -->|miss, error, timeout, or decode failure| H[Throw identical source rejection]
  D -->|rejection| H

Loading

All source rejections qualify in v1, including synchronous throws, arbitrary rejection values, and FallbackTimeoutError. A recovery read gets its own effective remoteReadTimeoutMs budget. Existing single-flight behavior means one leader performs the source attempt and at most one recovery read; process/request followers share the same result. A recovered value may be memoized only inside an already-enabled request-local scope.

Configuration

new DialCacheKeyConfig({
  ttlSec: { [CacheLayer.REMOTE]: 300 },
  ramp: { [CacheLayer.REMOTE]: 100 },
  staleOnErrorMaxAgeSec: 3_600,
});

• Omitted: disabled by default; inherits through a runtime overlay.
• 0: explicitly disables an inherited policy.
• Positive: enables recovery and requires 0 < F < M <= 31,536,000 seconds.
• Static invalid combinations fail fast.
• Invalid runtime M disables only stale recovery for that invocation, preserves valid fresh Redis caching, and records existing configuration-error telemetry.
• DialCacheKeyConfig.disabled() explicitly sets the field to 0.

The invocation's once-resolved F/M snapshot governs the whole flight. Lowering a boundary takes effect immediately. Raising M cannot resurrect or extend a key written with a shorter physical TTL; only a later successful write receives the longer retention.

Redis protocol and adapters

RedisReadRequest.maxAgeMs is now required. Both read Lua scripts:

1. validate the positive integer age bound;
2. load and validate the existing v1 frame;
3. compare its Redis-created timestamp with Redis TIME;
4. return a miss when age >= maxAgeMs;
5. for tracked data, require a valid current watermark and created_at > watermark.

Writes keep the existing key/frame and use physical PX F when recovery is off or PX M when it is on. Tracked watermark retention continues to derive from the physical value retention, so an opted-in write keeps the watermark for approximately M + 60s.

Bundled node-redis and Valkey GLIDE adapters pass the age bound atomically. Custom semantic clients must now declare enforcesMaxAge: true; DialCache checks the marker at construction so an old compiled JavaScript adapter cannot silently ignore logical age. Packed TypeScript, ESM, and CommonJS negative fixtures cover this migration guard.

Failure, invalidation, and shadow safety

• Recovery is attempted only after a definitive initial Redis miss. An initial Redis error or timeout never triggers a second Redis operation.
• Recovery failure always rethrows the exact original source rejection object/value.
• The recovery reread observes the current tracked watermark, so invalidation during the source attempt blocks retained data.
• Missing/malformed watermarks, malformed frames, invalid encoding, and deserialization failures never qualify.
• Recovery never writes Redis, extends TTL, populates process-local cache, or enters a shadow comparison/fill path.
• Shadow/confirmation reads always use F; a successful clean-miss shadow fill uses physical M so future serving reads have the configured reservoir.
• Late source or Redis settlement is consumed and cannot publish.

Observability

Add one bounded optional observer:

staleRecovery?({ outcome })

Outcomes are served, miss, read_error, read_timeout, and deserialization_error. Prometheus exposes dialcache_stale_recovery_counter; Datadog exposes dialcache.stale_recovery.count. Existing fallback errors and duration remain truthful even when stale data ultimately reaches the caller. Observer throws/rejections remain isolated.

Rollout and rollback

The current v1 key/frame is intentionally reused, so rollout must be readers-first:

1. deploy this age-aware library and adapters everywhere while the option remains omitted or 0;
2. upgrade the full reader fleet;
3. enable positive M only for selected use cases;
4. monitor Redis CPU, memory, evictions, source failures, and recovery outcomes.

An old reader would treat retained F..M data as fresh. Setting the policy back to 0 stops new extended-retention writes but does not delete prior ones, so an old binary cannot be restored safely until the largest previously enabled M has elapsed since the final PX M write, or affected keys are isolated/removed. Fleets that require mixed-version or immediate rollback safety must use a new key/frame version instead.

Performance and resource evidence

The checked-in no-threshold harness reports final-script CPU, outage traffic, coalescing, memory, expiration, watermark residency, and opt-in eviction pressure. The following loopback Docker runs used node-redis, 2,000 sequential iterations, F=60s, M=300s, and 100-way coalescing:

Server	JSON payload	Fresh Lua CPU	Logical-stale miss CPU	Outage recoveries/s	Redis wire bytes/recovery in / out	Value / watermark memory
Redis 6.2.22	64 B	4.478 us/op	4.352 us/op	2,953	401 / 77.5 B	248 / 152 B
Redis 6.2.22	4,096 B	7.334 us/op	5.388 us/op	2,709	401 / 4,111.5 B	5,288 / 152 B
Valkey 8.1.8	64 B	4.815 us/op	4.455 us/op	2,504	401 / 77.9 B	240 / 128 B
Valkey 8.1.8	4,096 B	11.761 us/op	7.995 us/op	1,712	401 / 4,112.0 B	5,280 / 128 B

The logical-stale miss still performs a full GET, but does not return the payload, so it was cheaper than a fresh hit in these sequential runs. The design issue contains the direct old-script versus timestamp-aware comparison: roughly +0.6–0.7 us fixed Redis CPU per present read in that separate synthetic setup. Treat all figures as directional rather than production capacity promises.

Every 100-way coalescing run produced one source rejection, two Redis reads, and one stale-recovery metric. Tracked probes observed value PTTL ~= M and watermark PTTL ~= M + 60s; a 1-second tracked value expired while its watermark remained. On isolated Redis 6.2 with 4 MiB maxmemory, allkeys-lru, and 500 benchmark-owned 16 KiB pressure keys, the pressure-local counter recorded 348 evictions and the retained stale read became a safe miss. The harness changes no Redis configuration, cleans only its random namespace, gives all owned keys finite TTLs, and has a hard watchdog plus bounded cleanup.

Validation

• Added regression coverage for retained-key opt-outs, dark-ramp safety, successful source refreshes, exact-M expiry during source failure, per-flight policy snapshots, runtime F changes, and complete served-recovery telemetry.

• corepack pnpm check

  • typecheck
  • 21 unit files / 465 tests with coverage
  • build
  • packed ESM/CJS/TypeScript consumer tests

• corepack pnpm test:integration

  • 2 integration files / 106 tests
  • Redis 6.2 and Valkey 8
  • node-redis and Valkey GLIDE
  • three-primary Redis 7 Cluster

• pnpm benchmark:stale-on-error

  • Redis 6.2 and Valkey 8 at 64 B and 4 KiB
  • isolated eviction-pressure probe

• DIALCACHE_BENCH_ITERATIONS=1000 DIALCACHE_BENCH_FANOUT=100 pnpm benchmark:request-local

• git diff --check

• Independent final core and benchmark reviews: clean