feat: always count native allocations and log executor native memory usage

[andygrove]

Which issue does this PR close?

Part of #4576.

Rationale for this change

Comet's memory pools only see what operators reserve. When an executor is OOM-killed, nothing in its logs says how much native memory Comet was actually holding. #5934 added an allocator wrapper that counts what the Rust allocator hands out, but it was an opt-in cargo feature that only reported through tracing, so the number was never there when needed.

This PR makes the count part of every build and logs it next to the pool reservations. Every executor log then shows the gap between the two, which is what spark.executor.memoryOverhead has to cover. The executor also warns when that gap looks larger than its container allows.

What changes are included in this PR?

Allocation accounting is always on. The alloc-accounting cargo feature is removed, and the accounting allocator now wraps whichever backend the build selects (system, jemalloc or mimalloc). alloc_overhead measures the wrapper against the bare backend in one binary, since there is no longer a build without it. The rust-test CI step that exercised the feature combinations now lints and tests the jemalloc backend instead.

Executor memory usage log. The first CometExecIterator on an executor starts one daemon thread. While native plans are running, it logs one line per interval for the whole executor, plus one more after the last plan finishes:

Comet native memory usage: allocated 2381.9 MiB, reserved 2350.9 MiB (8 native plans, 8 memory pools)

• It samples on a timer rather than between batches, because a plan can spend its whole run inside one executePlan call. For example, a plan rooted at a native shuffle writer consumes all of its input before returning.
• The figures come from a new JNI call, Native.getMemoryUsage, which reads only the allocation counter and the pool registry, never an execution context. It counts a pool shared by several plans once.
• The interval is the new spark.comet.memory.logInterval (default 10s, 0 disables). One log serves every session on the executor, so the setting is read from the executor's SparkConf.
• A value that does not parse disables the log with a warning rather than failing every Comet task. A value set only in the session is reported as ignored.
• The line after the last plan is best-effort at application shutdown, because Spark stops the executor shortly after its last task.

Container warning. The log thread warns, once each time the threshold is crossed, when untracked native memory (allocated - reserved) plus Spark's off-heap memory in use (which includes Comet's reservations) exceeds spark.memory.offHeap.size plus the executor's memory overhead. The overhead is computed the way Spark sizes the default container: spark.executor.memoryOverhead if set, otherwise spark.executor.memoryOverheadFactor of spark.executor.memory with a minimum of spark.executor.minMemoryOverhead. The warning is skipped in local mode and in on-heap mode.

Pool registry lock fix. releasePlan read the remaining pools' reservations while holding the pool registry lock on every plan, not only when tracing was on. CometFairMemoryPool holds its own lock across the JNI acquire from Spark, which Spark can park until another task frees memory. A finishing task frees its reservations only after releasePlan has taken the registry lock to unregister, so this could deadlock under fair_unified. Every reader now copies the pools out under the lock and reads their reservations after releasing it: the memory log, tracing, and the per-thread total. Unregistering no longer reads any reservation, and releasePlan reads the remaining total only when tracing.

Docs. The tuning guide gets a section on sizing spark.executor.memoryOverhead from the log. The tracing and memory management contributor guides are updated now that the counter is always present.

How are these changes tested?

Native unit tests cover:

• the plan and pool counts, including a pool shared by two plans on different threads being counted once;
• a probe pool that records whether the registry lock was held when its reservation was read, exercised through the memory log, tracing, the per-thread total and unregistering. Putting a reservation read back into unregistering makes it fail;
• the accounting allocator installed as the global allocator, and a thread's un-flushed delta being settled when its thread-local is dropped.

CometExecIteratorLifecycleSuite covers:

• a created plan being counted through JNI until it is released, with a nonzero allocation reported;
• the log line format, and the rule that the log goes quiet after reporting once past the last plan;
• interval parsing, including invalid and negative values;
• the overhead calculation, the container limit, and the warning threshold.

TPC-H SF100 on a 32-core Linux box (Spark 4.1.1, 2 executors x 8 cores, glibc allocator, no jemalloc), 3 runs of each build alternated against the base commit:

• All 22 queries return the same results as base on every run.
• The sum of per-query medians is 200.45 s base vs 201.97 s with this PR (+0.8%).
• Q21 is about 5% slower (29.7 s vs 31.2 s), in every run. A perf profile attributes this to the wrapper's thread-local accesses, which go through __tls_get_addr because libcomet.so is loaded with dlopen. This is tracked in perf: allocation accounting wrapper costs ~5% on TPC-H Q21 from thread-local lookups in the dlopened library #6165 for a separate PR. No other query is outside noise.
• The log printed one line per executor every 10 s, with allocated peaking at 2.0 to 2.4 GiB per executor. No container warning fired, since the largest untracked amount was 419 MiB against an 18 GiB limit.

The self-review for this revision was done with the review-comet-pr skill.

[sunchao]

Summary

Reviewed f6526ba759761848d7d2faac93221c9e284cae31 using the Comet FFI and memory review guides. No reproducible introduced or materially worsened P1/P2 findings were identified. Full integration and workload-performance validation are still incomplete.

• Prior state and problem: Native allocation accounting was opt-in, and observing it required tracing. Pool reservations alone do not describe all native allocations, so executor OOM investigations lacked a readily available allocation measurement. Registry readers also held the registry lock while querying pool reservations.
• Design approach: The selected Rust allocator is always wrapped for accounting. A new JNI snapshot reports allocation, distinct-pool reservations, pool count, and plan count. A single executor daemon periodically logs those figures and evaluates an advisory native-memory threshold.
• Correctness / compatibility analysis: The snapshot reads process-wide state rather than execution-context pointers. Pool references are cloned under the registry lock, and reservations are read after releasing that lock. Unregistering no longer reads reservations outside tracing. The new JNI entry point uses the existing exception/panic boundary. This change does not alter expression support levels, query semantics, or Spark memory acquisition rules. The allocator module's five unit tests passed in an isolated harness against the reviewed source, and git diff --check passed. Full native/JVM integration and workload benchmarks were not run locally. Native-library and Rust CI jobs were still pending at review completion. Completed checks were successful, while macOS, benchmark, Spark SQL, and Iceberg jobs were skipped.
• Key design decisions: Accounting applies to every allocator backend, even when memory logging is disabled. Logging uses executor configuration and a timer so long native calls can still be sampled. Shared pools are counted once, and reporting does not enforce a new memory budget.
• Implementation sketch: AccountingAllocator updates the allocation balance. The pool registry produces a deduplicated snapshot for Native.getMemoryUsage. CometExecIterator starts the scheduler and formats the sampled values, with focused native and Scala tests covering accounting, counts, lock scope, configuration, and message construction.
• Behavioral changes worth calling out: The alloc-accounting Cargo feature is removed because accounting is unconditional. spark.comet.memory.logInterval defaults to 10 seconds and is read from the executor configuration. Setting it to zero disables periodic logging. The reported balance is an allocation measure, not RSS, and the warning remains advisory.
• Suggested improvements: No P1/P2 review comments proposed.