You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "beliefer (via GitHub)" <gi...@apache.org> on 2024/04/10 09:39:29 UTC
[PR] [SPARK-47795][DOC] Supplement the doc of job schedule for K8S [spark]
beliefer opened a new pull request, #45982:
URL: https://github.com/apache/spark/pull/45982
### What changes were proposed in this pull request?
This PR propose to supplement the doc of job schedule for K8S.
### Why are the changes needed?
Spark document missing the description of job schedule for K8S.
### Does this PR introduce _any_ user-facing change?
'No'.
Just update document.
### How was this patch tested?
Manual tests.
### Was this patch authored or co-authored using generative AI tooling?
'No'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][KUBERNETES][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1560535092
##########
docs/job-scheduling.md:
##########
@@ -54,6 +54,10 @@ Resource allocation can be configured as follows, based on the cluster type:
(`spark.executor.memory` configuration property) and `--executor-cores` (`spark.executor.cores` configuration
property) control the resources per executor. For more information, see the
[YARN Spark Properties](running-on-yarn.html#spark-properties).
+* **K8s:** The same as the situation with Yarn, please refer to the description of Yarn above. Furthermore,
+ Spark on K8s offers higher priority versions of spark.kubernetes.executor.limit.core and
Review Comment:
```suggestion
Spark on K8s offers higher priority versions of spark.kubernetes.executor.limit.cores and
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1560434354
##########
docs/job-scheduling.md:
##########
@@ -92,6 +96,8 @@ In standalone mode, simply start your workers with `spark.shuffle.service.enable
In YARN mode, follow the instructions [here](running-on-yarn.html#configuring-the-external-shuffle-service).
+In K8s mode, K8s doesn't support external shuffle service yet.
Review Comment:
Given the content, `### Caveats` section would be better than here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][KUBERNETES][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1560534838
##########
docs/job-scheduling.md:
##########
@@ -54,6 +54,10 @@ Resource allocation can be configured as follows, based on the cluster type:
(`spark.executor.memory` configuration property) and `--executor-cores` (`spark.executor.cores` configuration
property) control the resources per executor. For more information, see the
[YARN Spark Properties](running-on-yarn.html#spark-properties).
+* **K8s:** The same as the situation with Yarn, please refer to the description of Yarn above. Furthermore,
+ Spark on K8s offers higher priority versions of spark.kubernetes.executor.limit.core and
+ spark.kubernetes.executor.request.core than spark.executor.core. For more information, see the
Review Comment:
```suggestion
spark.kubernetes.executor.request.cores than spark.executor.cores. For more information, see the
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2049959930
Let's wait for @yaooqinn 's final sign-off, @beliefer .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1559997742
##########
docs/job-scheduling.md:
##########
@@ -53,7 +53,11 @@ Resource allocation can be configured as follows, based on the cluster type:
on the cluster (`spark.executor.instances` as configuration property), while `--executor-memory`
(`spark.executor.memory` configuration property) and `--executor-cores` (`spark.executor.cores` configuration
property) control the resources per executor. For more information, see the
- [YARN Spark Properties](running-on-yarn.html).
+ [YARN Spark Properties](running-on-yarn.html#spark-properties).
Review Comment:
Yes, please handle this separately, @beliefer .
As you know, I'm the release manager of Apache Spark 3.4.3 (for next Monday).
I can backport this fix to branch-3.4 as a part of Apache Spark 3.4.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOC] Supplement the doc of job schedule for K8S [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1559154695
##########
docs/job-scheduling.md:
##########
@@ -53,7 +53,11 @@ Resource allocation can be configured as follows, based on the cluster type:
on the cluster (`spark.executor.instances` as configuration property), while `--executor-memory`
(`spark.executor.memory` configuration property) and `--executor-cores` (`spark.executor.cores` configuration
property) control the resources per executor. For more information, see the
- [YARN Spark Properties](running-on-yarn.html).
+ [YARN Spark Properties](running-on-yarn.html#spark-properties).
Review Comment:
This line change is irrelevant to K8s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2048826987
> Do we support scheduling jobs across applications? It's odd to me.
This section is about scheduling across applications.
`Scheduling Within an Application` section is related to jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOC] Supplement the doc of job schedule for K8S [spark]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2047129492
> Is this a required change, @beliefer?
I think we should add these document so as follows the other cluster mangers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2047412733
I didn't even notice this page or section. The dropdown from the Navi Bar is enough for me.
Do we support scheduling jobs across applications? It's odd to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1561282169
##########
docs/job-scheduling.md:
##########
@@ -96,6 +96,12 @@ All other relevant configurations are optional and under the `spark.dynamicAlloc
`spark.shuffle.service.*` namespaces. For more detail, see the
[configurations page](configuration.html#dynamic-allocation).
+
+### Caveats
+
+- In [standalone mode](spark-standalone.html), without explicitly setting `spark.executor.cores`, each executor will get all the available cores of a worker. In this case, when dynamic allocation enabled, spark will possibly acquire much more executors than expected. When you want to use dynamic allocation in [standalone mode](spark-standalone.html), you are recommended to explicitly set cores for each executor before the issue [SPARK-30299](https://issues.apache.org/jira/browse/SPARK-30299) got fixed.
+- In [K8s mode](running-on-kubernetes.html), we can't using this feature by set `spark.shuffle.service.enabled` to `true` due to Spark on K8s doesn't support external shuffle service yet.
Review Comment:
In [K8s mode](running-on-kubernetes.html), we cannot use this feature by setting `spark.shuffle.service.enabled` to `true` because Spark on K8s does not yet support the external shuffle service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2047422813
Nit: Use K8s instead of K8S, the former is the official abbreviation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2050974804
@yaooqinn @dongjoon-hyun @bjornjorgensen Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOC] Supplement the doc of job schedule for K8S [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1559159692
##########
docs/job-scheduling.md:
##########
@@ -92,6 +96,8 @@ In standalone mode, simply start your workers with `spark.shuffle.service.enable
In YARN mode, follow the instructions [here](running-on-yarn.html#configuring-the-external-shuffle-service).
+In K8S mode, follow the future work [here](running-on-kubernetes.html#future-work).
Review Comment:
This looks wrong to me because there is actually nothing to **follow** in the future work section. May I ask what do you want to say here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOC] Supplement the doc of job schedule for K8S [spark]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1559190807
##########
docs/job-scheduling.md:
##########
@@ -92,6 +96,8 @@ In standalone mode, simply start your workers with `spark.shuffle.service.enable
In YARN mode, follow the instructions [here](running-on-yarn.html#configuring-the-external-shuffle-service).
+In K8S mode, follow the future work [here](running-on-kubernetes.html#future-work).
Review Comment:
How about `In K8S mode, K8S doesn't support external shuffle service yet.` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1560363139
##########
docs/job-scheduling.md:
##########
@@ -53,7 +53,11 @@ Resource allocation can be configured as follows, based on the cluster type:
on the cluster (`spark.executor.instances` as configuration property), while `--executor-memory`
(`spark.executor.memory` configuration property) and `--executor-cores` (`spark.executor.cores` configuration
property) control the resources per executor. For more information, see the
- [YARN Spark Properties](running-on-yarn.html).
+ [YARN Spark Properties](running-on-yarn.html#spark-properties).
Review Comment:
reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #45982: [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S
URL: https://github.com/apache/spark/pull/45982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1561290153
##########
docs/job-scheduling.md:
##########
@@ -54,6 +54,10 @@ Resource allocation can be configured as follows, based on the cluster type:
(`spark.executor.memory` configuration property) and `--executor-cores` (`spark.executor.cores` configuration
property) control the resources per executor. For more information, see the
[YARN Spark Properties](running-on-yarn.html#spark-properties).
+* **K8s:** The same as the situation with Yarn, please refer to the description of Yarn above. Furthermore,
+ Spark on K8s offers higher priority versions of spark.kubernetes.executor.limit.cores and
+ spark.kubernetes.executor.request.cores than spark.executor.cores. For more information, see the
+ [K8s Spark Properties](running-on-kubernetes.html#spark-properties).
Review Comment:
spark.kubernetes.executor.limit.cores and
spark.kubernetes.executor.request.cores than spark.executor.cores -> `spark.kubernetes.executor.limit.cores` and
`spark.kubernetes.executor.request.cores` than `spark.executor.cores`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2050828484
Merged to master
Thank you @beliefer @dongjoon-hyun @bjornjorgensen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOC] Supplement the doc of job schedule for K8S [spark]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #45982:
URL: https://github.com/apache/spark/pull/45982#discussion_r1559169283
##########
docs/job-scheduling.md:
##########
@@ -53,7 +53,11 @@ Resource allocation can be configured as follows, based on the cluster type:
on the cluster (`spark.executor.instances` as configuration property), while `--executor-memory`
(`spark.executor.memory` configuration property) and `--executor-cores` (`spark.executor.cores` configuration
property) control the resources per executor. For more information, see the
- [YARN Spark Properties](running-on-yarn.html).
+ [YARN Spark Properties](running-on-yarn.html#spark-properties).
Review Comment:
Yes.
Because the link is not accurate even if it was not related to K8S.
Do you want me revert it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47795][DOC] Supplement the doc of job schedule for K8S [spark]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #45982:
URL: https://github.com/apache/spark/pull/45982#issuecomment-2047130102
cc @yaooqinn @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org