feat: pgdog.min_lsn read-your-writes replica routing floor#1156
feat: pgdog.min_lsn read-your-writes replica routing floor#1156mediprtl wants to merge 1 commit into
Conversation
Add an opt-in read-your-writes floor. A client sets the `pgdog.min_lsn`
connection parameter (startup option or SET) to a commit LSN it must observe;
PgDog then routes the read only to a replica that has replayed at least that far.
- Reads carrying `pgdog.min_lsn` are filtered to replicas whose replayed LSN
(from the existing LSN monitor) is >= the floor.
- When no replica qualifies, PgDog raises NoReplicaCaughtUp (SQLSTATE 58000)
carrying a catch-up ETA ("eta ~Ns"), derived from the replica's replication lag
in time (now() - pg_last_xact_replay_timestamp()), so a client can size a retry.
The rejection logs at debug (expected backpressure), not error.
- Optional `min_lsn_primary_fallback`: an unmet floor falls back to the primary
instead of erroring.
- `pgdog.min_lsn` is registered as an untracked parameter; inert when unset or on
setups without replicas.
Includes unit tests (routing, NoReplicaCaughtUp, ETA-from-lag) and a pgx
integration test.
|
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
| ELSE | ||
| now() | ||
| END AS timestamp | ||
| END AS timestamp, |
There was a problem hiding this comment.
This is not the correct formula for replica lag. This only works if and only if the primary and replicas do not have the same LSN, i.e. replica is behind. When they do, i.e., replica is caught up to the primary and the primary hasn't sent any writes yet, the replica lag should be zero, but this will give it a non-zero lag.
We handle this in here: https://github.com/pgdogdev/pgdog/blob/main/pgdog/src/backend/pool/shard/monitor.rs#L139
It's an edge case and it's not "terminal", i.e., it won't break the app but will cause pgdog to route the query to a primary unnecessarily. Since we have a function to calculate replica lag already, maybe you could re-use it here?
| /// reading — a bytes/apply-rate derivative gets fooled by bursty replay into a | ||
| /// runaway ETA. Safe-biased (true wait <= lag, since `min_lsn` committed after | ||
| /// the frontier). `None` if stats are invalid; `ZERO` once at/past the floor. | ||
| pub fn eta_to(&self, min_lsn: i64) -> Option<Duration> { |
There was a problem hiding this comment.
See comment above about re-using existing replica lag calculator.
| // `SET`) to the commit LSN it must observe; the LB routes the read only to | ||
| // a replica that has replayed at least this far. | ||
| let min_lsn = context | ||
| .params |
There was a problem hiding this comment.
For code quality reasons, maybe we could handle this here: https://github.com/pgdogdev/pgdog/blob/main/pgdog/src/frontend/router/parameter_hints.rs
Caveat is you have to store this on Route and pass it back up, and that the parser has to be "on", but that's the case with your deployment since you're using the load balancer, so I think that should work.
|
Looks great actually - I was wrong about pushing back on this. A few code quality comments to go through and we should be good to merge. |
3 participants
Sticky reads via a per-read LSN floor - #912
Part of #912 (@levkk asked for a PR there). Opt-in read-your-writes that keeps
reads on replicas instead of pinning tables to the primary.
A client sets
pgdog.min_lsn(startupoptions=-corSET pgdog.min_lsn='X/Y'),intercepted like
pgdog.role. The load balancer routes the read only to areplica whose replayed LSN (already tracked by the LSN monitor) is >= that floor.
If none qualify it errors recoverably with
NoReplicaCaughtUp(SQLSTATE 58000)carrying a catch-up ETA from the replica's replication lag, or — opt-in —
falls back to the primary.
Addressing the concerns from #912
rides on the async message that triggers the read — so there's no round-trip to
race. (PG 19's
WAIT FOR LSNmakes client-side LSN a first-class primitive too.)per-driver-plugin direction you described — it's a small primitive those can
build on.
never serves a stale read.
Testing
pgdog.min_lsnhas run continuously in our production and sandbox for 3+ weeks,built on the LSN monitor PgDog already ships (it reads the monitor's tracked
replayed LSN — no new poller).
Production — 2 read replicas
pgdog.min_lsnover the window.caught up before the client stopped retrying and fell back on the client side.
No stale reads — routing is conservative (a read only goes to a replica
whose replayed LSN is ≥ the floor).
NoReplicaCaughtUpresponses (clientsretried): most reads were served immediately or after one short wait; the floor
only holds a read back until the replica catches up.
~50 min over the window. The floor adapts — the reported ETA averaged ~1 min
and tracked actual lag (max ~26 min).
Sandbox — 1 read replica
NoReplicaCaughtUpresponses over ~4 weeks; lower lag / lighter load.Coverage — unit tests (replica-filter routing,
NoReplicaCaughtUp+ETA-from-lag) and a pgx integration test in
integration/load_balancer/pgx/.Notes / happy to adjust
min_lsn_primary_fallbackis currently a bool in[general]— glad to fold itinto an existing routing enum if you'd prefer.
replay_lag_secondslives on the pool'sLsnStatswrapper; can move it intopgdog_stats::LsnStatsif that's the better home.