fix(insight): buffer on-change export bursts

[wangyb-A]

Summary

Stacked on #702 (feat/insight-async-export). This preserves ordinary on-change bursts without weakening the base scheduler's timeout, retained-memory, or exporter-ownership bounds.

What changed

• keep a bounded FIFO of 16 pending snapshots per execution ARN
• keep at most 1,024 pending records and 16 MB of estimated retained memory per lane
• serve ARNs round-robin while evicting by true record age
• split pre- and post-flush work with generation-specific queue tokens
• preserve generation order after a timed-out flush barrier is canceled
• preserve exporter shaping for custom-renderable, safe opaque, and non-JSON records
• reject an individually over-budget record without evicting accepted same-ARN FIFO snapshots or unrelated backlog
• contain retained-size inspection failures as ordinary over-budget rejection
• disable failed-start lanes while clearing FIFO tokens, records, bytes, and barriers
• coalesce timed-out flushes to one detached marker after the bounded caller wait
• document lane-wide flush behavior for batching exporters

Why

Tight Python hook bursts could overwrite every intermediate RUNNING snapshot before the daemon worker received a turn. A single latest-pending slot therefore made on-change behave like on-complete during parallel/map fan-in and warm replay.

The FIFO keeps normal bursts observable. Record-count and retained-memory caps keep blocked lanes bounded. Generation tokens keep invocation-end barriers between the records they must flush and later records they must not flush.

Validation

• full Insight suite: 153 passed, including local-runner e2e
• real DurableFunctionTestRunner parallel fan-in verifies four RUNNING snapshots followed by terminal completion
• fan-in e2e passed 10 repeated runs and was proven to fail the old latest-only scheduler
• exact-age eviction, pre/post-barrier, canceled-barrier recovery, pending-plus-in-flight accounting, slice-referent sizing, non-dispatched sizing, two-barrier-race, accepted-FIFO preservation, safe-opaque-rendering, inspection-failure, startup-failure, cleanup, and ownership regressions passed
• focused scheduler/plugin/fan-in suite: 61 passed
• focused ordering regressions repeated 20 times
• mypy passed
• Ruff lint and format passed
• wheel and sdist built
• full repository collection completed without errors
• prior Workflow Insight conformance remains CloudWatch 18/18 and S3 18/18
• exact-SHA local correctness review at d876e48: all blocker classes closed; no remaining or new Critical/High findings

no linked issue: #687 tracks broader Workflow Insight follow-up work

[wangyb-A]

/ai review

[github-actions]

Codex AI review · Finding arf_v1_dmyq7d5h3r3s5adl2jgaghrhgf

[P2] Avoid assuming exactly three operation-change batches. The checkpoint batcher closes batches using an idle timeout, so six concurrent branches can legitimately produce a different number of OperationChangeInfo hooks depending on thread scheduling. The pending count may skip past 4, causing this wait and the fixed five-record assertion below to fail despite correct FIFO behavior. Record the actual hook count with a probe plugin, signal after invocation end, and assert one exported snapshot per observed start/change/end hook instead of hard-coding the count.

[github-actions]

Codex AI review

One P2 test-stability issue: the new fan-in e2e assumes a fixed number of timing-dependent checkpoint batches.

Reviewed commit d876e48a82b3b04bac18689b965fcf561f007b74. Workflow run