You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2023/03/13 06:23:38 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request, #40392: [SPARK-42769][K8S] Add `ENV_DRIVER_POD_IP` env variable to executor pods
dongjoon-hyun opened a new pull request, #40392:
URL: https://github.com/apache/spark/pull/40392
### What changes were proposed in this pull request?
This PR aims to add a new environment variable `ENV_DRIVER_POD_IP` to all executor pods.
### Why are the changes needed?
This is helpful to connect driver pods via IP.
### Does this PR introduce _any_ user-facing change?
No, this is a new environment variable.
### How was this patch tested?
Pass the CIs with the newly added test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #40392:
URL: https://github.com/apache/spark/pull/40392#issuecomment-1465812124
> ... for some executor pods to connect driver pods via IP.
Hi @dongjoon-hyun, I think it's quite useful, but in https://github.com/apache/spark/pull/39160#pullrequestreview-1229691638, you left a concern
> ... I have a concern. Currently, Apache Spark uses K8s Service entity via [DriverServiceFeatureStep](https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala) to access Spark driver pod in K8s environment.
do you still concern that now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40392:
URL: https://github.com/apache/spark/pull/40392#issuecomment-1465870752
@pan3793 .
The goal of PR is different from your PR's goal.
- Your PR tried to add `SPARK_DRIVER_POD_NAME` to `Driver Pod` to expose it to **3rd party pods**.
- This PR aims to add `SPARK_DRIVER_POD_IP` to `Executor Pod` in order to help **internal communications between Spark executors and Spark driver**.
In addition, this is a kind of propagation of the information from the driver pod to the executor pods instead of exposing the executor pods' internal information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #40392:
URL: https://github.com/apache/spark/pull/40392#discussion_r1134323523
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala:
##########
@@ -54,6 +54,7 @@ private[spark] object Constants {
val UI_PORT_NAME = "spark-ui"
// Environment Variables
+ val ENV_DRIVER_POD_IP = "SPARK_DRIVER_POD_IP"
Review Comment:
Thank you for review, @viirya .
In K8s, `DRIVER_HOST_ADDRESS` is protected by `DriverServiceFeatureStep` here.
https://github.com/apache/spark/blob/7fccd676a27540b88fbcf951e53b6e43fdc26562/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala#L109
https://github.com/apache/spark/blob/7fccd676a27540b88fbcf951e53b6e43fdc26562/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala#L38-L40
It's because we inject like this systematically.
https://github.com/apache/spark/blob/7fccd676a27540b88fbcf951e53b6e43fdc26562/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala#L67-L68
However, when DNS doesn't work, we need IP which is unknown from the executor pods so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40392:
URL: https://github.com/apache/spark/pull/40392#issuecomment-1466547407
Yes, correct, @viirya ! Thank you for the approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
URL: https://github.com/apache/spark/pull/40392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #40392: [SPARK-42769][K8S] Add `ENV_DRIVER_POD_IP` env variable to executor pods
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40392:
URL: https://github.com/apache/spark/pull/40392#issuecomment-1465741894
Hi, @HyukjinKwon . Could you review this PR when you have some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #40392:
URL: https://github.com/apache/spark/pull/40392#discussion_r1134259551
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala:
##########
@@ -54,6 +54,7 @@ private[spark] object Constants {
val UI_PORT_NAME = "spark-ui"
// Environment Variables
+ val ENV_DRIVER_POD_IP = "SPARK_DRIVER_POD_IP"
Review Comment:
Is it different to `DRIVER_HOST_ADDRESS`? I saw K8s uses `DRIVER_HOST_ADDRESS` to derive driver url for env var `ENV_DRIVER_URL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40392:
URL: https://github.com/apache/spark/pull/40392#issuecomment-1466385779
Hi, @viirya . Could you review this PR when you have some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #40392: [SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40392:
URL: https://github.com/apache/spark/pull/40392#issuecomment-1466559161
Merged to master for Apache Spark 3.5.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org