You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gedeh (via GitHub)" <gi...@apache.org> on 2024/08/27 21:08:22 UTC
[PR] [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5 [spark]
gedeh opened a new pull request, #47896:
URL: https://github.com/apache/spark/pull/47896
### What changes were proposed in this pull request?
Uptake `--bind-address` parameter in `YarnCoarseGrainedExecutorBackend` when launching new container in Yarn cluster. This PR also ensure `YarnAllocator` uses default hostname when its not configured.
### Why are the changes needed?
We've came across issue with Spark running on Yarn in Istio enabled Kubernetes cluster. Previous PR https://github.com/apache/spark/pull/32633 is not merged because Spark 2.4 was EOL and 3.x branch didnt get enough traction.
### Does this PR introduce any user-facing change?
Yes, new config specifically for Yarn cluster mode is added and relevant doc is updated.
### How was this patch tested?
Tested in Kubenetes with Istio and added tests to `YarnAllocatorSuite`
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5 [spark]
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #47896:
URL: https://github.com/apache/spark/pull/47896#issuecomment-2316020543
Hi @dongjoon-hyun looking back again, is this still a feature improvement? It seems like a bug though that bind address which is exposed in configuration is not adhered by Yarn?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47896:
URL: https://github.com/apache/spark/pull/47896#issuecomment-2316076675
Adding a new configuration is definitely a new feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47896:
URL: https://github.com/apache/spark/pull/47896#issuecomment-2313624165
Let me close this in order to prevent accidental merging. We can still continue to discuss on the closed PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #47896: [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5
URL: https://github.com/apache/spark/pull/47896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5 [spark]
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #47896:
URL: https://github.com/apache/spark/pull/47896#issuecomment-2313704188
Gotcha, thanks @dongjoon-hyun OK I understand its new feature/improvement, its just that I have same PR during Spark 3.4.0-SNAPSHOT and sadly it was closed due to not much activity on that one.
Would be great to take a look for the upcoming Spark 4.0.0 version, although Spark 4.0.0 changes a lot compared to 3.5.0 branch. Thanks in advance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5 [spark]
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #47896:
URL: https://github.com/apache/spark/pull/47896#issuecomment-2313551738
Same with https://github.com/apache/spark/pull/47892, but this one specifically for 3.5 branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org