You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2026/07/18 13:40:48 UTC
[PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the por… [spark]
zhengchenyu opened a new pull request, #57349:
URL: https://github.com/apache/spark/pull/57349
### What changes were proposed in this pull request?
`BasicDriverFeatureStep` skips `containerPort` entries whose port is 0.
### Why are the changes needed?
When we use host network, we want to resolve port conflicts by binding port 0. But setting `spark.driver.port` / `spark.driver.blockManager.port` / `spark.ui.port` / `spark.connect.grpc.binding.port` to 0 makes driver Pod creation fail. The error log is as follows:
```
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://XX.XX.XX.XX:443/api/v1/namespaces/default/pods. Message: Pod "xxx-driver" is invalid: [spec.containers[0].ports[0].containerPort: Required value, spec.containers[0].ports[1].containerPort: Required value ...
```
BasicExecutorFeatureStep already skips the entry when the port is 0. The driver side should follow the same pattern.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
* Unit tests in `BasicDriverFeatureStepSuite`.
* Test in kubernetes cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the por… [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #57349:
URL: https://github.com/apache/spark/pull/57349#issuecomment-5333749907
cc @uros-b @dongjoon-hyun to take another look before I merge it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the port is 0. [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed pull request #57349: [SPARK-58202][K8S] Skip driver containerPort declaration when the port is 0.
URL: https://github.com/apache/spark/pull/57349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the port is 0. [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #57349:
URL: https://github.com/apache/spark/pull/57349#issuecomment-5337787375
Thanks! merged to `master` / `branch-4.x`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the por… [spark]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #57349:
URL: https://github.com/apache/spark/pull/57349#issuecomment-5130263914
@uros-b Thanks for your reply!
For hostNetwork, I prefer this configuration to avoid using DriverServiceFeatureStep:
```
spark.kubernetes.driver.pod.excludedFeatureSteps=org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep
spark.kubernetes.executor.useDriverPodIP=true
```
Additionally, when the service is configured to 0, it also uses a random port, which is meaningless. Because when the service is configured to 0, I expect the port to be dynamically patched based on the actual port. I solved this problem at https://github.com/apache/spark/pull/57350.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the por… [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57349:
URL: https://github.com/apache/spark/pull/57349#issuecomment-5333765958
The title seems to be broken, @sunchao . What is the original title?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the por… [spark]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #57349:
URL: https://github.com/apache/spark/pull/57349#issuecomment-5018503508
@sunchao @dongjoon-hyun Can you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58202][K8S] Skip driver containerPort declaration when the port is 0. [spark]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #57349:
URL: https://github.com/apache/spark/pull/57349#issuecomment-5336482259
@sunchao @dongjoon-hyun Thanks for your review! I have updated the title and added notes to the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org