[SPARK-58202][K8S] Skip driver containerPort declaration when the por…

[zhengchenyu]

What changes were proposed in this pull request?

BasicDriverFeatureStep skips containerPort entries whose port is 0.

Why are the changes needed?

When we use host network, we want to resolve port conflicts by binding port 0. But setting spark.driver.port / spark.driver.blockManager.port / spark.ui.port / spark.connect.grpc.binding.port to 0 makes driver Pod creation fail. The error log is as follows:

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://XX.XX.XX.XX:443/api/v1/namespaces/default/pods. Message: Pod "xxx-driver" is invalid: [spec.containers[0].ports[0].containerPort: Required value, spec.containers[0].ports[1].containerPort: Required value ...

BasicExecutorFeatureStep already skips the entry when the port is 0. The driver side should follow the same pattern.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Unit tests in BasicDriverFeatureStepSuite.
• Test in kubernetes cluster.

[zhengchenyu]

@sunchao @dongjoon-hyun Can you please review this PR?

[uros-b]

@zhengchenyu PTAL at resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala:79-99

getAdditionalKubernetesResources() unconditionally builds four ServicePort entries (driver RPC / block manager / UI / Connect) with the same port values and no port != 0 filter. DriverServiceFeatureStep is always in the driver feature-step chain (KubernetesDriverBuilder line 79), and K8s rejects a ServicePort with port: 0. So the same port=0/hostNetwork scenario this PR enables for the container will still fail end-to-end at the headless driver Service creation. Please address or acknowledge the gap in the PR body and file/track a follow-up (or extend scope) so the use case is functional end-to-end.

[zhengchenyu]

@uros-b Thanks for your reply!

For hostNetwork, I prefer this configuration to avoid using DriverServiceFeatureStep:

spark.kubernetes.driver.pod.excludedFeatureSteps=org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep
spark.kubernetes.executor.useDriverPodIP=true

Additionally, when the service is configured to 0, it also uses a random port, which is meaningless. Because when the service is configured to 0, I expect the port to be dynamically patched based on the actual port. I solved this problem at #57350.