Skip to content

[spark] Expose Paimon metrics through JMX - #10131

Open
hutiefang76 wants to merge 1 commit into
apache:masterfrom
hutiefang76:frank/paimon-7646-spark-jmx
Open

hutiefang76 wants to merge 1 commit into
apache:masterfrom
hutiefang76:frank/paimon-7646-spark-jmx

Conversation

@hutiefang76

Copy link
Copy Markdown
Contributor

Purpose

Fixes #7646. Paimon metrics already reach the Spark UI, but commit metrics are not visible to external JMX/Prometheus scrapers. This adds a Spark metrics source with an eagerly started JMX reporter, mirrors Paimon gauges, counters, and histograms into its Codahale registry, and wires the Spark V1 commit path to use the same metric registry. Existing Spark UI metrics remain available.

Metric names are deterministic for each group/table/metric. A later commit replaces the previous value for that key so repeated commits do not add new MBeans. Entries remain available for scraping during the Spark application lifetime. Source registration is retried when a Spark environment first appears or changes in the same JVM.

Tests

  • A new JMX regression failed before the change because no Paimon MBean was registered; the Spark 3 PaimonMetricTest suite now passes 11/11, including a real V1 INSERT commit check.
  • Spark 4 shared-module compile passed; Spotless and Scalastyle passed.
  • For the focused Spark test, paimon-codegen-loader must be packaged first so its paimon-codegen/ runtime resource is present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Paimon commit metrics are not exposed via Spark's MetricsSystem for external monitoring (JMX/Prometheus)

1 participant