chore: deprecate spark.comet.exec.memoryPool.fraction

[andygrove]

Which issue does this PR close?

Part of #4576.

Rationale for this change

The tuning guide recommends setting spark.comet.exec.memoryPool.fraction below 1.0 to leave room in the off-heap pool for native memory that Comet does not reserve. It cannot do that:

• greedy_unified ignores the fraction. It is built with a pool size of 0 and asks Spark for every byte it reserves.
• fair_unified applies it to each task's pool, limiting each memory consumer in the task to offHeap.size * fraction / num_consumers. Spark's execution pool already limits each of N running tasks to offHeap.size / N, which is tighter whenever more than one task is running. So on a busy executor the fraction never binds, and the tasks together can still acquire the whole pool.
• Spark's own off-heap consumers, non-Comet operators and off-heap storage, draw on the same pool with no Comet limit.

The only room Spark leaves for memory outside the pool is spark.executor.memoryOverhead, and the executor memory usage log added in #6162 measures how much of it Comet needs.

What changes are included in this PR?

• spark.comet.exec.memoryPool.fraction is documented as deprecated, with what it actually does. Its behaviour is unchanged: several tests set it very low to force spills through the fair_unified per-task cap.
• The driver plugin logs a deprecation warning when the setting is in the SparkConf, pointing at spark.executor.memoryOverhead.
• The tuning guide and the memory management contributor guide stop recommending it and explain why it does not create headroom.

How are these changes tested?

New CometPluginsMemoryPoolFractionWarningSuite covers the warning being logged when the setting is present and not otherwise. It is registered in both the Linux and macOS PR workflows. The rest is documentation.

[sunchao]

Summary

Reviewed the full 19-file, four-commit PR diff against main at 97d1c5e4c3d55b3ae6f8c489b37fc717e434b2ed, including the three commits shared with #6162. No reproducible introduced or materially worsened P1/P2 finding was established. Source review is complete. Full integration and workload-performance validation remain incomplete.

• Prior state and problem: Native allocation accounting was opt-in and required tracing to observe. Pool reservations alone do not describe all native allocations, and registry readers held the registry lock while reading pool reservations. The tuning guidance also presented spark.comet.exec.memoryPool.fraction as an executor-wide headroom control, although its behavior depends on the pool type and cannot guarantee that headroom across all consumers.
• Design approach: The selected Rust allocator is always wrapped for accounting. A JNI snapshot reports allocations, distinct-pool reservations, pool count, and plan count. One executor daemon logs these figures and evaluates an advisory memory threshold. The final commit documents the fraction as deprecated and adds a driver warning when it is explicitly present in SparkConf.
• Correctness / compatibility analysis: The snapshot uses process-wide state and clones pool references under the registry lock, then reads reservations after releasing it. The JNI entry point uses the existing exception/panic boundary. The fraction's default, parsing, budget arithmetic, and reservation behavior remain unchanged. The warning reads its configured value without parsing or mutating it. The cumulative change does not alter expression support levels or Spark memory-acquisition rules. The five isolated allocator tests passed in the shared-commit review, and the tested module is byte-identical at this head. Suite registration and git diff --check passed for this PR. Full native/JVM integration and workload benchmarks were not run locally. The Linux native build and Rust tests were running at the last check. Completed visible checks passed, while macOS, Spark SQL, Iceberg, and benchmark jobs were skipped.
• Key design decisions: Accounting applies to every allocator backend even when logging is disabled. Logging reads executor configuration and uses a timer so long native calls can be sampled. Shared pools are counted once. Reporting does not enforce a new memory budget, and retaining the fraction avoids changing existing spill thresholds.
• Implementation sketch: AccountingAllocator updates the allocation balance. The pool registry supplies a deduplicated snapshot to Native.getMemoryUsage, and CometExecIterator schedules and formats the log. CometDriverPlugin.init also calls warnIfMemoryPoolFractionSet, which emits the deprecation message when the setting is present. Configuration, tracing, tuning, and contributor documentation describe these changes.
• Behavioral changes worth calling out: The alloc-accounting Cargo feature is removed because accounting is unconditional. spark.comet.memory.logInterval defaults to 10 seconds and is read from executor configuration. Zero disables periodic logging. The reported balance measures allocations, not RSS, and warnings remain advisory. Applications explicitly setting the fraction receive a deprecation warning, while its existing behavior is retained.
• Suggested improvements: No additional change met the requested P1/P2 review bar.