Report resident memory (RssAnon) in the executor memory usage log and warn on it against the container size

[andygrove]

What is the problem the feature request solves?

#6162 adds a periodic executor log line that reports allocated (bytes live through Rust's global allocator) next to reserved (bytes in Comet's memory pools), and warns when the estimated native footprint exceeds what the container allows. Neither figure is what gets an executor killed. The kernel and the cluster manager act on dirty resident pages, and allocated bytes differ from those in both directions:

• Allocated but not resident. Capacity that is reserved and never written counts in full: Vec::with_capacity, pre-sized hash tables, and large zeroed allocations that calloc serves from fresh mmap pages. Pages are only charged once they are touched, so this is memory that might become resident on the next write loop, not memory that is resident now.
• Resident but not allocated. This includes allocator fragmentation, freed pages the allocator retains rather than returning to the OS, allocations made by C dependencies such as zstd, and Arrow buffers on the JVM side.

The second group can dominate. A measurement in #4576 found jemalloc's resident at 6x to 8x allocated at the median, and about 4.3 GB resident whether spark.memory.offHeap.size was 2g or 16g.

So the log can show a comfortable gap while the executor is close to its container limit. The container warning in #6162 is an estimate built from allocated bytes plus Spark's off-heap usage, and it can be wrong in either direction.

Describe the potential solution

Add the process's resident anonymous memory to the memory usage log, and base the warning on it.

1. Report resident memory. Read RssAnon from /proc/self/status on the log's timer thread and add it to the line, for example allocated 2381.9 MiB, reserved 2350.9 MiB, resident 9120.0 MiB (...). RssAnon counts dirty anonymous pages for the whole process, including the JVM heap. It excludes page cache, which is what made the memory.current threshold in feat: add optional container memory guard based on cgroup usage #5993 fire on reclaimable memory.
2. Warn on resident memory against the container size. Compare against spark.executor.memory + spark.memory.offHeap.size + the memory overhead (+ spark.executor.pyspark.memory when set), using the overhead calculation from feat: always count native allocations and log executor native memory usage #6162, and warn when RssAnon passes a high fraction of it, for example 90%. This replaces the estimate in feat: always count native allocations and log executor native memory usage #6162 with the quantity the cluster manager enforces.
3. Keep allocated and reserved for attribution. They remain the only view of how much of Comet's memory the pools do not track. The tuning guide should say that allocated is what Rust code has asked for, which can be higher or lower than what is resident, and that resident is what counts towards the container.

This is observability only. Acting on the number, as the circuit breaker in #4576 proposes, is separate work.

Additional context

• RssAnon is Linux-only. On other platforms the line would omit it and the warning would be skipped. With jemalloc, stats.resident is a native-only alternative, but it excludes the JVM heap, which also counts towards the container.
• On cgroup v2, memory.stat's anon field is the container-wide equivalent and includes child processes such as Python workers. It is worth considering if Python UDF workers should count.
• Related: Investigate adopting DataFusion's allocator-level memory accounting to replace manual memory tuning #4576 (allocator-level accounting and the RSS circuit breaker), Improve memory accounting: bound the JVM Arrow allocator and make native reclaim work #5997 (bounding the JVM Arrow allocator and native reclaim), feat: add optional container memory guard based on cgroup usage #5993 (closed cgroup memory.current guard), feat: always count native allocations and log executor native memory usage #6162 (the memory usage log).