cdc: massively speed replay with ordered concurrency and batching

[thesyncim]

Summary

• pipeline CDC DML and target transaction control, with per-session relation and prepared-statement caches
• replay independent table lanes concurrently while preserving source order for every table
• remove the global commit gate: workers commit DML plus a durable receipt range immediately, while canonical progress advances only across the contiguous source-order prefix
• use a bounded WAL read pump so decoding a large transaction cannot starve worker completion/checkpoint events
• adaptively batch up to 128 contiguous dependent source transactions into one durable target commit, capped at 16 MiB
• use CPU-scaled defaults of 8–32 replay sessions and an 8× scheduling window, with CLI overrides

Reproduced deadlock and fix

The 2,048-transaction / 16-table reproduction filled the 512-transaction window and hung. The goroutine dump showed all workers idle and the scheduler unable to read: a later same-table job had been linked to a predecessor that had already emitted its completion event, so that dependency could never be released.

The scheduler now treats already-durable table tails as satisfied, workers never wait inside an open target transaction for coordinator permission, and WAL decoding runs through a one-item bounded read pump. Apply sessions also have a 2-minute idle_in_transaction_session_timeout fuse so any future stall rolls back and reconnects instead of pinning transactions indefinitely.

Sustained performance

• 10,000 tiny transactions across 16 independent tables with production defaults: 688–692 ms, or 14.45–14.52k events/s; serialized replay was 5.39–5.71 s (7.8–8.3× faster)
• the same 10,000-event independent-lane test under -race: 877 ms / 11.4k events/s / 6.6×; CI enforces at least 3×
• 10,000 same-table transactions with batch 128: 146 ms / 68.7k events/s / 27.8×; under -race: 305 ms / 32.7k events/s / 16.2×
• the batch sweep measured 59.7k/s at 64, 68.7k/s at 128, and 77.4k/s at 256; 128 is the default to balance throughput against transaction/rollback scope and lock lifetime
• 32 independent workers reached 14.75k/s, only ~1.6% above 16 workers on this 16-CPU host, confirming that larger pools are not the main remaining lever
• five 300 ms target transactions: ~662–694 ms concurrent vs a 1.5 s serial floor

Durability

synchronous_commit remains enabled for CDC. Each parallel target transaction atomically commits DML with one receipt range for its contiguous source batch. A coordinator atomically advances authoritative progress and removes only the contiguous receipt prefix. Therefore:

• DML committed but checkpoint not advanced: restart sees the receipt and skips exactly that DML
• checkpoint advanced: receipt cleanup and progress are atomic
• DML rolled back: no receipt exists, so restart replays it
• changing batch size or reducing to one worker on restart is safe because receipts describe source-LSN ranges

The forced recovery test commits a later two-transaction batch first, leaves canonical progress absent, restarts with one worker, and verifies exact-once convergence plus receipt cleanup.

Validation

• go test ./...
• make vet
• make race
• focused PostgreSQL concurrency, sustained 10,000-event throughput, batch, ordering, and crash-gap tests under -race
• make e2e with batch 128
• make crash-e2e (kills and resumes copy, indexes, catchup, and follow; exact final hashes)