You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gedeh (via GitHub)" <gi...@apache.org> on 2023/09/11 13:03:23 UTC
[GitHub] [spark] gedeh opened a new pull request, #42870: SPARK-29670][CORE] Allow configuring Yarn executor bind address
gedeh opened a new pull request, #42870:
URL: https://github.com/apache/spark/pull/42870
### What changes were proposed in this pull request?
Uptake `--bind-address` parameter in `YarnCoarseGrainedExecutorBackend` when launching new container in Yarn cluster. This PR also ensure `YarnAllocator` uses default hostname when its not configured.
### Why are the changes needed?
We've came across https://github.com/istio/istio/issues/27900 with Spark running on Yarn in Istio enabled Kubernetes cluster. Previous PR https://github.com/apache/spark/pull/32633 is not merged because Spark 2.4 was EOL.
### Does this PR introduce any user-facing change?
Yes, new config specifically for Yarn cluster mode is added and relevant doc is updated.
### How was this patch tested?
Tested in Kubenetes with Istio and added tests to `YarnAllocatorSuite`
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #42870: [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn
URL: https://github.com/apache/spark/pull/42870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1951500559
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gedeh commented on pull request #42870: [YARN] Allow configuring Yarn executor bind address in Yarn
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1724240588
Thank you for the prompt reply @srowen
> Can you remove the unrelated changes?
Which changes these will be? Are you suggesting the changes in `docs/configuration.md`?
> Do you have a JIRA account? You can request one on the JIRA site.
I requested an account few days ago, but unfortunately the request rejected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn [spark]
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1981727449
Hello, I noticed this PR is closed, this is blocking Spark with Yarn in kubernetes. I dont understand whats left missing for this PR. If anyone in Spark project can shed a light what required to merge this for next release of Spark that would be great. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn [spark]
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on code in PR #42870:
URL: https://github.com/apache/spark/pull/42870#discussion_r1389285109
##########
docs/running-on-yarn.md:
##########
@@ -528,6 +519,25 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.executor.bindAddress</code></td>
+ <td><code>(executor hostname)</code></td>
+ <td>
+ Hostname or IP address where to bind listening sockets in YARN cluster mode.
Review Comment:
Hmm true, will benefit `client` mode as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gedeh commented on pull request #42870: [SPARK-29670][CORE] Allow configuring Yarn executor bind address
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1713839779
Apologies using old Jira because I am unable to raise new Spark Jira in ASF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #42870: [YARN] Allow configuring Yarn executor bind address in Yarn
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1723618227
Can you remove the unrelated changes?
Do you have a JIRA account? You can request one on the JIRA site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn [spark]
Posted by "laocoder2014 (via GitHub)" <gi...@apache.org>.
laocoder2014 commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-2017210634
I think this PR is needed for Spark to work under Istio/service mesh. I am just wondering if anyone has run Spark/YARN successfully under Istio/Service Mesh.
The PR [Spark-26331](https://github.com/apache/spark/pull/26331) was merged and made the 'executor's bindAddress' configurable. But without the above PR, how can user configure the executor's bindAddress?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gedeh commented on pull request #42870: [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1731143447
@srowen got my ASF account created and raised this Jira https://issues.apache.org/jira/browse/SPARK-45278. Removed the change in `configuration.md` file too. Thanks in advance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gedeh commented on pull request #42870: [YARN] Allow configuring Yarn executor bind address in Yarn
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1723227194
@srowen and @dongjoon-hyun you were on previous PR https://github.com/apache/spark/pull/32633
Apologies I can't raise new Jira in Spark project for this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gedeh commented on pull request #42870: [SPARK-24203][CORE] Allow configuring Yarn executor bind address in Yarn
Posted by "gedeh (via GitHub)" <gi...@apache.org>.
gedeh commented on PR #42870:
URL: https://github.com/apache/spark/pull/42870#issuecomment-1714673329
Latest full test run via Github Action https://github.com/gedeh/spark/actions/runs/6143764745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn [spark]
Posted by "tokoko (via GitHub)" <gi...@apache.org>.
tokoko commented on code in PR #42870:
URL: https://github.com/apache/spark/pull/42870#discussion_r1387673722
##########
docs/running-on-yarn.md:
##########
@@ -528,6 +519,25 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.executor.bindAddress</code></td>
+ <td><code>(executor hostname)</code></td>
+ <td>
+ Hostname or IP address where to bind listening sockets in YARN cluster mode.
Review Comment:
Why `YARN cluster mode`? The change affects client mode as well, doesn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org