[SPARK-58203][K8S] Support configuring spark-managed UI service to work with spark.ui.port=0 - #57350
[SPARK-58203][K8S] Support configuring spark-managed UI service to work with spark.ui.port=0#57350zhengchenyu wants to merge 11 commits into
Conversation
…rk with spark.ui.port=0 Signed-off-by: zhengchenyu <zhengchenyu16@163.com>
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
Summary
This PR adds an opt-in, Spark-managed Kubernetes Service for the driver Web UI and patches its targetPort after Jetty binds. The direction addresses the host-network/random-port problem, but the current head does not yet make the stated spark.ui.port=0 path valid and misses one supported backend path.
Prior state and problem
The existing headless driver Service is built from the configured spark.ui.port. In host-network mode, multiple drivers can collide on a fixed UI port, while an external operator cannot know the random port selected by spark.ui.port=0.
Design approach
The patch adds enable/type/name configurations, creates a dedicated UI Service with a stable Service port and placeholder target port, passes the Service name to the driver, and patches targetPort from SparkUI.boundPort during Kubernetes scheduler-backend startup.
Correctness / compatibility analysis
Default-off behavior remains backward compatible, and keeping a stable Service port while patching only targetPort is appropriate for ClusterIP, NodePort, and LoadBalancer. The enabled path is currently blocked because the existing driver Pod and headless Service still contain port zero. In addition, driver-only local[*] applications never call the patcher, and the documented RBAC grant omits the GET verb used by the implementation.
Key design decisions
- Keep the externally visible Service port stable while patching only
targetPort. - Ensure no generated Kubernetes manifest contains a numeric port of zero.
- Run patching from a driver lifecycle shared by Kubernetes and local driver-only backends, or explicitly scope the feature away from driver-only mode.
- Treat
getandpatchas required Service permissions if retaining the read-before-write implementation.
Implementation sketch
- When the dedicated UI Service is enabled and
spark.ui.port=0, omit the UI container-port declaration and omit or substitute the UI entry on the existing headless Service. - Move the patch hook to a shared driver initialization point after UI binding, with Kubernetes-client access, or reject this configuration for driver-only mode.
- Update the RBAC/configuration documentation and add full-builder plus patcher/backend tests.
Behavioral changes worth calling out
Enabling the feature creates an additional Kubernetes Service and currently makes patch failure fatal during SparkContext initialization. LoadBalancer can expose the UI externally. The behavior for supported spark.kubernetes.driver.master=local[*] applications therefore needs to be defined explicitly.
Suggested improvements
Add a full KubernetesDriverBuilder test for spark.ui.port=0, focused patcher tests for matching/missing/updated targets, integration coverage for driver-only mode, and the three new settings plus their RBAC requirements to docs/running-on-kubernetes.md.
| * [[org.apache.spark.scheduler.cluster.k8s.K8sDriverUIServicePatcher]] updates the Service's | ||
| * `targetPort` to the real bound port. | ||
| */ | ||
| private lazy val servicePort: Int = if (configuredUIPort == 0) { |
There was a problem hiding this comment.
[P1] Handle port zero in the existing driver resources too
This placeholder only fixes the new dedicated Service. With spark.ui.port=0, BasicDriverFeatureStep still emits containerPort=0, and DriverServiceFeatureStep still emits port=0 / targetPort=0 for the mandatory headless Service. Kubernetes requires these numeric ports to be in 1..65535, so the driver Pod/resources are rejected before this runtime patch can run. Please omit or substitute both existing UI port declarations and cover the complete KubernetesDriverBuilder output in a test.
There was a problem hiding this comment.
@sunchao In fact, Because I think the internal service do not need to expose ports. So I use spark.kubernetes.executor.useDriverPodIP=true and add org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep to spark.kubernetes.driver.pod.excludedFeatureSteps.
If the driver service's port is configured to 0, then this is indeed necessary. However, it's unclear whether it's appropriate to do this in this PR, since this is a PR for adding a UI service. I'll add it to this PR if you think it's suitable.
There was a problem hiding this comment.
[P1] This remains reproducible on 44b952a19a85a35e9ecad237dde9cfb604183e9b. BasicDriverFeatureStep.scala:100,117-120 still copies spark.ui.port=0 into the driver's containerPort, and DriverServiceFeatureStep.scala:52,89-93 independently emits Service port=0 and targetPort=0. The proposed spark.kubernetes.executor.useDriverPodIP=true plus excluding DriverServiceFeatureStep only removes the second invalid resource; BasicDriverFeatureStep still makes the Pod invalid before the runtime patcher can execute. Because this PR explicitly promises spark.ui.port=0, both existing declarations need to be omitted/replaced, with a full KubernetesDriverBuilder regression test.
| if (!conf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)) { | ||
| setUpExecutorConfigMap(podAllocator.driverPod) | ||
| } | ||
| maybePatchDriverUIServiceTargetPort() |
There was a problem hiding this comment.
[P2] Patch the Service in driver-only mode too
spark.kubernetes.driver.master=local[*] is a supported driver-only mode, but KubernetesClusterManager returns LocalSchedulerBackend for it, so this method never runs even though DriverUIServiceFeatureStep still creates the Service. After the zero-port manifest issue is fixed, the UI binds randomly while targetPort remains at the placeholder; a nonzero bind collision produces the same stale target. Please move this hook to a driver lifecycle shared by both backends, or explicitly reject/disable the feature in driver-only mode.
There was a problem hiding this comment.
Local mode generally doesn't require exposing the UI port, so there's no need to patch the port; simply keep spark.kubernetes.driver.ui.service.enabled set to false.
There was a problem hiding this comment.
[P2] This still fails on 44b952a19a85a35e9ecad237dde9cfb604183e9b, and the latest endpointless-Service change makes it affect even a normal fixed port with no bind collision. For spark.kubernetes.driver.master=local[*], KubernetesClusterManager.scala:56-72 returns LocalSchedulerBackend, but only KubernetesClusterSchedulerBackend.start() invokes the patcher. DriverUIServiceFeatureStep now creates every enabled UI Service without a selector, so spark.ui.port=4040 plus spark.kubernetes.driver.ui.service.enabled=true leaves the Service permanently endpointless. Setting the feature to false avoids the configuration rather than handling the explicitly enabled one; please patch from a shared driver lifecycle or reject/disable this incompatible combination.
| "Web UI (separate from the headless driver service). When enabled, after the driver " + | ||
| "Web UI starts, Spark will patch the Service's targetPort to match the actual bound " + | ||
| "UI port, which allows using `spark.ui.port=0` (random port). Requires the driver's " + | ||
| "ServiceAccount to have `patch services` permission.") |
There was a problem hiding this comment.
[P2] Include GET in the required RBAC permissions
The patcher calls get() before patch(), and Kubernetes RBAC verbs are independent. A least-privilege role granting only the documented patch verb receives a 403 on the GET; the exception is rethrown from backend startup and aborts SparkContext initialization. Please document both get and patch, or issue a targeted patch that does not require the preliminary read.
There was a problem hiding this comment.
I have updated the document.
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
[P2] Document these public Kubernetes settings in the configuration table
Config.scala:104 adds three public user-facing settings, but none is listed in the Kubernetes configuration table in docs/running-on-kubernetes.md. Users therefore cannot discover the enable flag, service type/name controls and defaults, supported service types, or the required get + patch RBAC permissions from the primary Kubernetes documentation. Please add table entries for all three settings.
@sunchao I have updated the document! |
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
Incremental findings on exact head e9497be15df10c121a9f1d8db5a09ea9b56f77ec.
| .addToLabels(kubernetesConf.serviceLabels.asJava) | ||
| .endMetadata() | ||
| .withNewSpec() | ||
| .withType(serviceType) |
There was a problem hiding this comment.
[P2] Preserve the configured IP family on the UI Service
This Service does not set ipFamilyPolicy or ipFamilies, even though the existing driver Service honors spark.kubernetes.driver.service.ipFamilyPolicy / .ipFamilies. On a dual-stack cluster, Kubernetes defaults an unspecified Service to SingleStack on the first Service CIDR, so applications configured for IPv6 or RequireDualStack get a compatible RPC Service but an incompatible UI Service. Please apply the driver Service family settings here (or add equivalent UI-service settings) and cover the dual-stack case.
There was a problem hiding this comment.
@sunchao I have updated the code. UI service will reuse Driver Service's ip family to keep the same IP family.
| <td><code>false</code></td> | ||
| <td> | ||
| If true, Spark will create a dedicated Kubernetes Service for the Spark driver Web UI. | ||
| When enabled, after the driver Web UI starts, Spark will patch the Service's |
There was a problem hiding this comment.
[P2] Remove the trailing whitespace from this line
This line ends with a trailing space, and git diff --check c9c45a6..e9497be exits 2 on it. Please remove the whitespace so the patch passes the standard diff check.
There was a problem hiding this comment.
OK, I have removed the space.
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
Incremental findings on exact head 7cd1b9647603ac655b7e925647f8bcaa2ee011cd.
| "Web UI starts, Spark will patch the Service's targetPort to match the actual bound " + | ||
| "UI port, which allows using `spark.ui.port=0` (random port). Requires the driver's " + | ||
| "ServiceAccount to have `get` and `patch` verbs on `services`.") | ||
| .version("4.3.0") |
There was a problem hiding this comment.
[P2] Declare binding policies for all three new configs
These three ConfigBuilder entries do not call .withBindingPolicy(...), so each registers with bindingPolicy=None. SparkConfigBindingPolicySuite requires every new config to declare a policy and treats the old exceptions list as frozen. These are deployment-only settings, so please add .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) to the enabled, type, and name entries. The current audit suite stays green only because this lazily loaded Kubernetes Config object is not initialized in that test JVM.
There was a problem hiding this comment.
OK, I have fixed it.
| client.services() | ||
| .inNamespace(namespace) | ||
| .withName(serviceName) | ||
| .patch(updated) |
There was a problem hiding this comment.
[P2] Retry resourceVersion conflicts while patching the UI Service
updated retains the first GET's metadata.resourceVersion. In Fabric8 7.8, patch(T) uses JSON Patch and performs a second GET; its diff logic clears only the second-GET base's resourceVersion, specifically causing the first GET's value to be sent in the patch. If a controller updates the Service between those GETs—for example while publishing LoadBalancer status—the API returns 409 for the stale version. This path has no conflict retry, and the catch below rethrows, aborting backend startup. Please retry 409s boundedly with a fresh GET/rebuild, or send a narrow targetPort patch that does not carry the stale full-object version.
There was a problem hiding this comment.
Great catch! Instead of using get, we used patch directly to solve this problem, and the documentation was also updated.
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
One additional issue found on the latest head; the existing unresolved findings remain applicable.
| .withType(serviceType) | ||
| .withIpFamilyPolicy(ipFamilyPolicy) | ||
| .withIpFamilies(ipFamilies) | ||
| .withSelector(kubernetesConf.labels.asJava) |
There was a problem hiding this comment.
[P1] Keep the placeholder Service endpointless until the UI port is known
This installs the real driver selector while targetPort is still the 4040 placeholder. In the hostNetwork setup motivating this change, the selected Pod endpoint is the node IP; if another driver already owns 4040, this Service routes to that other driver's UI until KubernetesClusterSchedulerBackend.start() patches it. With NodePort or LoadBalancer, this can expose the wrong application's UI, and the window can last until user code creates SparkContext. The existing zero-port manifest blocker currently masks this path, but it becomes reachable once that blocker is fixed by omitting the zero-valued port declarations. Please initially create the Service without a matching selector, then patch the selector and actual targetPort together after boundPort is known.
There was a problem hiding this comment.
A selector has already been added during patching, and patching only occurs when spark.ui port=0.
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
Incremental findings on exact head cfdb584a865a056ea5065343996ffd726d77f391.
| override def getAdditionalPodSystemProperties(): Map[String, String] = { | ||
| // These properties exist solely to drive the runtime patch, which only happens for a random | ||
| // port. A fixed-port Service is already complete, so nothing needs to reach the driver. | ||
| if (enabled && configuredUIPort == 0) { |
There was a problem hiding this comment.
[P1] Reconcile fixed UI ports after bind collisions too
A nonzero spark.ui.port is not guaranteed to be the actual bound port: JettyUtils.startJettyServer delegates to Utils.startServiceOnPort, which retries 4040 at 4041 and higher after a collision. With two hostNetwork drivers requesting 4040, the second driver can bind 4041, but this condition omits all runtime patch metadata and the new Service permanently selects nodeIP:4040—the first application's UI. NodePort/LoadBalancer therefore exposes the wrong application's UI, while ordinary pod networking leaves the second UI unreachable. Please emit reconciliation metadata for fixed ports as well and install the selector only after SparkUI.boundPort is known.
There was a problem hiding this comment.
@sunchao
This issue stems from my previous commit. In that commit, I only patched the UI service for ports configured as 0. I've now changed it to patch the UI service for all configured task ports.
The problem arises because the bound port might change when we configure hostNetwork or set it to 0. Therefore, I personally suggest setting a switch to enable hostNetwork, similar to FLINK-24947. If this switch is enabled, the relevant port will be configured as 0, and the patch UI/driver service will be started. If you agree, I think we should create a new issue to address this problem.
| extends KubernetesFeatureConfigStep with Logging { | ||
| import DriverUIServiceFeatureStep._ | ||
|
|
||
| private val enabled = kubernetesConf.get(KUBERNETES_DRIVER_UI_SERVICE_ENABLED) |
There was a problem hiding this comment.
[P1] Do not expose a UI Service when the Spark UI is disabled
This only checks spark.kubernetes.driver.ui.service.enabled; it never checks spark.ui.enabled. With spark.ui.enabled=false, spark.ui.port=4040, this feature enabled, and hostNetwork=true, Spark still creates an immediately selectable Service targeting the node's port 4040 even though SparkContext never starts a UI. If another application on that node owns 4040, this application's Service routes to that unrelated UI, and NodePort/LoadBalancer can publish it externally. The runtime cannot repair this because sc.ui is None. Please suppress the Service or reject this configuration whenever the Spark UI is disabled.
There was a problem hiding this comment.
OK, I have fixed it.
| configuredUIPort | ||
| } | ||
|
|
||
| private lazy val serviceName: String = kubernetesConf.get(KUBERNETES_DRIVER_UI_SERVICE_NAME) |
There was a problem hiding this comment.
[P1] Reject custom Service names already owned by another application
A static spark.kubernetes.driver.ui.service.name can be reused by two live Spark applications, but this override is accepted without checking whether the Service already exists or who owns it. During submission, KubernetesClientApplication calls addOwnerReference(createdDriverPod, ...) and then .forceConflicts().serverSideApply(); addOwnerReference replaces the owner references with the new driver Pod. Consequently, the second submission can forcibly transfer the existing UI Service's selector, application labels, and controller ownership to itself, disrupting the first application and causing the shared Service to be garbage-collected with the wrong Pod. Please use create-only semantics or verify existing ownership before applying, reject collisions with this application's mandatory driver Service, and add cross-application collision coverage.
There was a problem hiding this comment.
Theoretically, all Kubernetes resources have this issue; it wasn't introduced by this PR. If configured, users need to ensure no duplicates. However, without configuration, the default random value makes duplicates almost impossible.
There was a problem hiding this comment.
[P1] This collision surface is introduced by the new arbitrary spark.kubernetes.driver.ui.service.name override. The existing mandatory driver Service derives its name from the application-specific randomized resourceNamePrefix; existing driver Pods use create-only semantics. By contrast, two otherwise distinct applications can now both set shared-ui: KubernetesUtils.addOwnerReference replaces the controller owner UID, and KubernetesClientApplication.scala:175-177 uses forceConflicts().serverSideApply() on that same Service; the second application's runtime patch then replaces the selector. The first application silently loses its Service, and the Service can be garbage-collected with the wrong Pod. Choosing this application's own mandatory driver-Service name also makes both feature steps emit the same Service identity. Please reject name collisions or verify existing ownership before applying; unique defaults do not protect explicitly configured names.
There was a problem hiding this comment.
I will change it to be the same as the driver, and no longer use separate configuration.
| .addNewPort() | ||
| .withName(UI_PORT_NAME) | ||
| .withPort(servicePort) | ||
| .withNewTargetPort(servicePort) |
There was a problem hiding this comment.
[P2] Target the HTTPS connector when UI TLS is enabled
For spark.ui.port=4040 with spark.ssl.ui.enabled=true, Jetty binds HTTPS on spark.ssl.ui.port or 4440 by default, and port 4040 becomes an HTTP-to-HTTPS redirect connector. This Service nevertheless targets and exposes only 4040; TLS clients therefore reach a plaintext connector, and HTTP clients are redirected to HTTPS port 4440/8443, which the Service does not expose. SparkUI.boundPort already resolves to the actual secure connector, but the current implementation only consults it when the configured UI port equals zero. Please reconcile fixed-port Services against SparkUI.boundPort too, and cover both default and explicitly configured TLS ports.
There was a problem hiding this comment.
Same with "[P1] Reconcile fixed UI ports after bind collisions too", have fix it.
|
@sunchao |
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
Incremental findings on exact head 44b952a19a85a35e9ecad237dde9cfb604183e9b.
| try { | ||
| val patch = new ServiceBuilder() | ||
| .withNewSpec() | ||
| .withSelector(selector.asJava) |
There was a problem hiding this comment.
[P1] Withdraw the UI Service selector before stopping Jetty
This installs the real selector but never withdraws it. With hostNetwork=true, spark.kubernetes.driver.ui.service.type=NodePort (or LoadBalancer), and spark.kubernetes.driver.service.deleteOnTermination=false, SparkContext.stop() closes the UI at SparkContext.scala:2350 before scheduler/backend shutdown at SparkContext.scala:2360, while KubernetesClusterSchedulerBackend.stop() leaves this Service and its selector intact. The driver JVM can continue running after sc.stop(), so its Pod remains a Ready endpoint at the shared node IP. If another application binds the released port on that node, the first application's externally reachable Service now routes to the other application's UI. This is distinct from the already-fixed startup placeholder window. Please clear the selector or delete the dedicated UI Service before releasing the UI port, regardless of the headless driver Service's retention setting.
There was a problem hiding this comment.
The driver has the same issue. Furthermore, by default, spark.kubernetes.driver.service.deleteOnTermination=true will delete the UI service when stops. I think this could be modified in a separate PR: if spark.kubernetes.driver.service.deleteOnTermination=false is configured, then the selector will be dropped. Actually, I don't think it's necessary to change it since the default value is already true.
| .addToLabels(kubernetesConf.serviceLabels.asJava) | ||
| .endMetadata() | ||
| .withNewSpec() | ||
| .withType(serviceType) |
There was a problem hiding this comment.
[P2] Clean up external UI Services when the driver exits before SparkContext
A LoadBalancer or NodePort Service is created by KubernetesClientApplication before the driver application constructs SparkContext. If application startup exits or throws before a context exists, KubernetesClusterSchedulerBackend.stop() never runs, so no Service cleanup occurs even when spark.kubernetes.driver.service.deleteOnTermination=true. Owner references do not help until the driver Pod is deleted; Spark's Kubernetes documentation says completed driver Pods remain until eventual garbage collection or manual cleanup. Every early failure can therefore retain a cloud load balancer/IP or an allocated NodePort for the Pod retention period. The existing headless driver Service did not allocate either external resource. Please defer external Service provisioning until the driver initializes, or clean it up when its Pod reaches a terminal phase.
There was a problem hiding this comment.
The driver service also has the same issue, which can lead to resource leaks. However, compared to that, using LoadBalancer/NodePort might result in wasted resources. Perhaps we could move the service creation to after the driver starts, but that's outside the scope of this PR discussion.
@zhengchenyu Yes, I think we should continue. This PR is useful on its own because it provides a dedicated UI Service, while SPARK-58499 can address the broader driver-Service reconciliation problem. One thing to clarify is that |
| <td><code>spark.kubernetes.driver.ui.service.enabled</code></td> | ||
| <td><code>false</code></td> | ||
| <td> | ||
| If true, Spark will create a dedicated Kubernetes Service for the Spark driver Web UI. |
There was a problem hiding this comment.
[P3] Document that UI Service creation requires cluster deploy mode
Spark also supports Kubernetes client-mode drivers running inside Pods, but --deploy-mode client does not run KubernetesDriverBuilder, so enabling spark.kubernetes.driver.ui.service.enabled=true silently creates no Service. Supporting client mode can remain out of scope; please document here that this setting only applies to cluster deploy mode.
@sunchao In fact, I want to solve the invalid port 0 in #57349. This PR aim to add UI service. |
2 participants
What changes were proposed in this pull request?
Add config
spark.kubernetes.driver.ui.service.enabled(default false). When enabled:DriverUIServiceFeatureStepcreates a dedicated ClusterIP Service exposing the driver's Web UI port.SparkContextinitializes and the Web UI binds,K8sDriverUIServicePatcher(invoked fromKubernetesClusterSchedulerBackend.start()) patches the Service'stargetPortto the actual bound port.When
spark.ui.port=0, the Service is initially built with the default UI port as placeholder to satisfy Kubernetes port validation; the runtime patch then rewritestargetPort.Two other configs:
spark.kubernetes.driver.ui.service.type(ClusterIP / NodePort / LoadBalancer, default ClusterIP) andspark.kubernetes.driver.ui.service.name(optional explicit name).Why are the changes needed?
In hostNetwork mode,
spark.ui.port=0is needed to avoid UI port collisions between drivers on the same host. Today the driver UI Service is created by spark-operator, not by Spark itself, and spark-operator has no way to know the actual UI port the driver ends up binding to.Letting Spark create and manage the UI Service resolves this because Spark knows the actual bound port and patches the Service accordingly.
Does this PR introduce any user-facing change?
Yes. Three new opt-in configs, default off.
How was this patch tested?
DriverUIServiceFeatureStepSuite:Was this patch authored or co-authored using generative AI tooling?