Disconnect non-consuming IPC subscribers to fix publisher memory leak (#68114)

[dwoz]

What does this PR do?

IPCMessagePublisher now disconnects a subscriber that does not consume
messages within a configurable timeout (ipc_write_timeout, default 30s).
This stops the master event publisher from accumulating pending
stream.write() coroutines (and their referenced payloads) on a stalled
IPC client, which was a real source of unbounded RSS growth.

What issues does this PR fix or reference?

Fixes #68114

Previous Behavior

IPCMessagePublisher.publish() did
self.io_loop.spawn_callback(self._write, stream, pack) for every
subscriber on every published message. _write then awaited
stream.write(pack) with no timeout. A subscriber that stopped reading
caused every following publish to queue another _write coroutine that
awaited a write that never completed; the pending writes and their
payloads piled up in the ioloop and the publisher's RSS grew without
bound.

New Behavior

_write wraps stream.write(pack) in
salt.ext.tornado.gen.with_timeout(...) using the new ipc_write_timeout
opt (seconds; default 30, 0 disables the timeout for legacy behavior).
When a single write does not complete in time, the publisher logs a
warning, closes the offending stream, and removes it from self.streams.
publish() also skips streams that still have a pending write buffer so
we no longer pile up more _write coroutines on a slow client while the
existing one is waiting to time out.

The opt is documented on the master and minion configuration pages and a
changelog fragment was added.

Merge requirements satisfied?

• Used a Forge-PR Label (test:full)
• Tests written/updated (tests/pytests/unit/transport/test_ipc.py::test_ipc_publisher_drops_non_consuming_client_68114)
• Documentation updated (master and minion config refs)
• Changelog updated (changelog/68114.fixed.md)

Commits signed with GPG?

Yes

[dwoz]

Context on prior work in this area (none of it surfaced in PR-list / timeline searches because it landed as direct-to-branch commits rather than as PRs):

• 4c0e5529f5d Ipc leak fix (May 24 2025) shipped the same ipc_write_timeout/with_timeout approach this PR uses, on 3006.x / 3007.x / 3008.x / master.
• 088b33bc635 Add regression test for ipc leak issue (same day) added tests/pytests/functional/master/test_event_publisher.py — a memory-growth regression test against EventPublisher.
• 17257ebe1f3 Revert "Ipc leak fix" (Aug 3 2025) reverted only the fix commit, not the regression test. The reason is visible in 4c0e552 itself: it left literal merge-conflict markers in salt/config/__init__.py (<<<<<<< HEAD ... >>>>>>> f1de671598e (Ipc leak fix)), so the release branch was broken until the revert. The revert appears to have been tactical ("this commit is broken, get the branch building"), not a rejection of the approach.
• 07a6a0788c4 Remove legacy salt.transport.ipc module removed salt/transport/ipc.py from 3008.x / master. That is why this PR targets 3006.x only — the merge-forward branches don't need the same patch because the module is gone.
• No clean re-attempt has landed on 3006.x / 3007.x in the ~10 months since the revert. No open or closed PR references [BUG] Non consuming ipc clients #68114 or names ipc_write_timeout / IPCMessagePublisher / non-consuming.

This PR is a clean re-implementation of 4c0e552, without the merge-conflict damage, plus:

• adds the opt to DEFAULT_MINION_OPTS as well as DEFAULT_MASTER_OPTS (the original only added it to the master),
• short-circuits publish() for streams that still have a pending write so we don't pile up spawn_callback(self._write, ...) coroutines before the timeout fires,
• raises the log level from trace to warning so operators can see when a non-consuming subscriber is being dropped,
• documents the new opt on the master and minion config pages,
• adds a small unit-level regression test (test_ipc_publisher_drops_non_consuming_client_68114) that fails on unmodified 3006.x and passes after this patch. The original test_event_publisher.py is preserved as the functional/RSS-based regression test.

Default timeout bumped from 15s (original) to 30s — more headroom for legitimately slow consumers under load — but is configurable and 0 disables (legacy behavior).

[dwoz]

Superseded by #69227 (merged as fa7e51f). #69227 converted IPCMessagePublisher._write from a @gen.coroutine (with spawn_callback) to a plain function that returns immediately after stream.write() enqueues into Tornado's IOStream buffer, and attaches a future.add_done_callback for the disconnect path. The docstring on the new _write (lines 56-73 in salt/transport/ipc.py on 3006.x) explicitly identifies the gen.Runner pile-up as the same root cause as #68114.

That structural change removes the surface this PR was patching:

• The "skip on _write_buffer_size > 0" predicate that dropped test_event_many_backlog event Make docstrings consistend with apt and file states implementations. #279 is gone — there's no longer a coroutine queue to skip.
• ipc_publisher_pending_writes HWM is moot — no spawn_callback Runners to bound.
• ipc_write_timeout is moot — stalled writes now accumulate only in Tornado's IOStream buffer, which ipc_write_buffer already bounds via max_write_buffer_size / StreamBufferFullError.

Verified tests/pytests/unit/utils/event/test_event.py::test_event_many_backlog PASSES on bare origin/3006.x at f2c62b12a7c (3/3 reruns, 9s each) with no changes from this PR applied.

Closing as superseded. #68114 remains addressed by #69227.