You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/07 09:10:03 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
AngersZhuuuu opened a new pull request #32804:
URL: https://github.com/apache/spark/pull/32804
### What changes were proposed in this pull request?
Support YARN Placement Constraint, In this pr:
- Add `LocalityPreferredSchedulingRequestContainerPlacementStrategy` for compute locality for SchedulingRequest
- Add `ContainerImpl` for UT, since the interface Container's common API can't support `setAllocationRequestId` and `getAllocationRequestId`
- Add `YarnSchedulingRequestAllocator ` as the implement of allocator when use Placement Constraint.
### Why are the changes needed?
Spark can allow users to configure the Placement Constraint so that users will have more control on where the executors will get placed. For example:
1. Spark job wants to be run on machines where Python version is x or Java version is y (Node Attributes)
2. Spark job needs / does not need executors to be placed on machine where Hbase RegionServer / Zookeeper / Or any other Service is running. (Affinity / Anti Affinity)
3. Spark job wants no more than 2 of it's executors on same node (Cardinality)
4. Spark Job A executors wants / does not want to be run on where Spark Job / Any Other Job B containers runs (Application_Tag NameSpace)
### Does this PR introduce _any_ user-facing change?
Use can set `spark.yarn.schedulingRequestEnabled` to use YARN placement constraint and set `spark.yarn.executor.nodeAttributes` to set constraint.
### How was this patch tested?
Added UT and manuel tested this in yarn cluser:
**set node attribute**
```
/usr/share/yarn-3/bin/yarn nodeattributes -attributestonodes -attributes attr1
Hostname Attribute-value rm.yarn.io/attr1 :ip-xxxx 1
/usr/share/yarn-3/bin/yarn nodeattributes -attributestonodes -attributes attr2
Hostname Attribute-value rm.yarn.io/attr2 :ip-xxx
spark command
export SPARK_CONF_DIR=/tmp/spark-conf-3.1.1
export SPARK_HOME=/tmp/spark-3.1.1-sdi-002-bin-3.2.0-sdi-001
/tmp/spark-3.1.1-sdi-002-bin-3.2.0-sdi-001/bin/spark-sql --queue infra \
--executor-memory 1g --executor-cores 1 \
--conf spark.yarn.schedulingRequestEnabled=true \
--conf spark.executor.instances=1 \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.initialExecutors=1 \
--conf spark.dynamicAllocation.maxExecutors=2 \
--conf spark.dynamicAllocation.minExecutors=0 \
--conf spark.dynamicAllocation.executorIdleTimeout=10s \
--conf spark.yarn.executor.nodeAttribute='attr1=1'
```
Under dynamic allocation, can work as expect, all allocated container is under NM `ip-xxx`,
if we set node attribute as `attr1=2`, we can't allocate any executor container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855785891
**[Test build #139414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139414/testReport)** for PR 32804 at commit [`4ef8f35`](https://github.com/apache/spark/commit/4ef8f35011a02dc965f027ddc85589423b59d4a9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855781696
**[Test build #139410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139410/testReport)** for PR 32804 at commit [`c80b66f`](https://github.com/apache/spark/commit/c80b66fadb3e6b84cabe94f31e34aa99027b962f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857456767
**[Test build #139553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139553/testReport)** for PR 32804 at commit [`2cd9e69`](https://github.com/apache/spark/commit/2cd9e69926a0900eec85d4424b9cdce919ba056c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857494783
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139553/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-866006388
thanks for working on this, it looks very interesting
> Add LocalityPreferredSchedulingRequestContainerPlacementStrategy for compute locality for SchedulingRequest
Can you add more description about this? This seems like a lot of changes and not what I expected from the description. I was expecting us just to pass the node attributes along to yarn but this is much more then that so please describe in detail how this is working. How does this work exactly with constraint vs data locality? Is there some sort of timed wait or error handling if it never becomes available or cluster doesn't have node with that attribute? You had like 4 examples of things you could do, but are there more, what dependencies does it need. How does an attribute allow you to get cardinality or affinity?
is there any constraints on Hadoop version? (I thought it was just introduced in 3.2.0)
eventually the documentation .md file would need to be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855768767
**[Test build #139414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139414/testReport)** for PR 32804 at commit [`4ef8f35`](https://github.com/apache/spark/commit/4ef8f35011a02dc965f027ddc85589423b59d4a9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857494783
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139553/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857474980
**[Test build #139553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139553/testReport)** for PR 32804 at commit [`2cd9e69`](https://github.com/apache/spark/commit/2cd9e69926a0900eec85d4424b9cdce919ba056c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856450037
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139445/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855762920
**[Test build #139410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139410/testReport)** for PR 32804 at commit [`c80b66f`](https://github.com/apache/spark/commit/c80b66fadb3e6b84cabe94f31e34aa99027b962f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856436747
**[Test build #139445 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139445/testReport)** for PR 32804 at commit [`ec9e127`](https://github.com/apache/spark/commit/ec9e127caa335ae1512714b61f2a1a8e5e67392a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855806538
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43936/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855839353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856450037
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139445/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855839353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855818351
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43932/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857577822
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857577822
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #32804:
URL: https://github.com/apache/spark/pull/32804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855768767
**[Test build #139414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139414/testReport)** for PR 32804 at commit [`4ef8f35`](https://github.com/apache/spark/commit/4ef8f35011a02dc965f027ddc85589423b59d4a9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855827410
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43936/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857572425
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856456977
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43968/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855802446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856456977
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43968/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-931793085
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #32804:
URL: https://github.com/apache/spark/pull/32804#discussion_r656210527
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -27,6 +27,24 @@ import org.apache.spark.network.util.ByteUnit
package object config extends Logging {
/* Common app configuration. */
+ private[spark] val SCHEDULING_REQUEST_ENABLED =
+ ConfigBuilder("spark.yarn.schedulingRequestEnabled")
Review comment:
I think we need to come up with better name and more description. From just reading this, I have no idea what it does. spark always requests containers from yarn to schedule on, so why is it off...(ie that is what I read from the config name)
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/LocalityPreferredSchedulingRequestContainerPlacementStrategy.scala
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn
+
+import scala.collection.mutable
+import scala.collection.mutable.{ArrayBuffer, HashMap, Set}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.yarn.api.records.{ContainerId, SchedulingRequest}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.resource.ResourceProfile
+
+
+/**
+ * This strategy is calculating the optimal locality preferences of YARN containers by considering
Review comment:
this looks like a direct copy of the ContainerLocalityPreferences javadoc, I'm assuming this is different so please update description, is I missed the difference, I think we need to pull it up to the top to be clear how its different
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/LocalityPreferredSchedulingRequestContainerPlacementStrategy.scala
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn
+
+import scala.collection.mutable
+import scala.collection.mutable.{ArrayBuffer, HashMap, Set}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.yarn.api.records.{ContainerId, SchedulingRequest}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.resource.ResourceProfile
+
+
+/**
+ * This strategy is calculating the optimal locality preferences of YARN containers by considering
+ * the node ratio of pending tasks, number of required cores/containers and locality of current
+ * existing and pending allocated containers. The target of this algorithm is to maximize the number
+ * of tasks that would run locally.
+ *
+ * Consider a situation in which we have 20 tasks that require (host1, host2, host3)
+ * and 10 tasks that require (host1, host2, host4), besides each container has 2 cores
+ * and cpus per task is 1, so the required container number is 15,
+ * and host ratio is (host1: 30, host2: 30, host3: 20, host4: 10).
+ *
+ * 1. If the requested container number (18) is more than the required container number (15):
+ *
+ * requests for 5 containers with nodes: (host1, host2, host3, host4)
+ * requests for 5 containers with nodes: (host1, host2, host3)
+ * requests for 5 containers with nodes: (host1, host2)
+ * requests for 3 containers with no locality preferences.
+ *
+ * The placement ratio is 3 : 3 : 2 : 1, and set the additional containers with no locality
+ * preferences.
+ *
+ * 2. If requested container number (10) is less than or equal to the required container number
+ * (15):
+ *
+ * requests for 4 containers with nodes: (host1, host2, host3, host4)
+ * requests for 3 containers with nodes: (host1, host2, host3)
+ * requests for 3 containers with nodes: (host1, host2)
+ *
+ * The placement ratio is 10 : 10 : 7 : 4, close to expected ratio (3 : 3 : 2 : 1)
+ *
+ * 3. If containers exist but none of them can match the requested localities,
+ * follow the method of 1 and 2.
+ *
+ * 4. If containers exist and some of them can match the requested localities.
+ * For example if we have 1 container on each node (host1: 1, host2: 1: host3: 1, host4: 1),
+ * and the expected containers on each node would be (host1: 5, host2: 5, host3: 4, host4: 2),
+ * so the newly requested containers on each node would be updated to (host1: 4, host2: 4,
+ * host3: 3, host4: 1), 12 containers by total.
+ *
+ * 4.1 If requested container number (18) is more than newly required containers (12). Follow
+ * method 1 with an updated ratio 4 : 4 : 3 : 1.
+ *
+ * 4.2 If request container number (10) is less than newly required containers (12). Follow
+ * method 2 with an updated ratio 4 : 4 : 3 : 1.
+ *
+ * 5. If containers exist and existing localities can fully cover the requested localities.
+ * For example if we have 5 containers on each node (host1: 5, host2: 5, host3: 5, host4: 5),
+ * which could cover the current requested localities. This algorithm will allocate all the
+ * requested containers with no localities.
+ */
+private[yarn] class LocalityPreferredSchedulingRequestContainerPlacementStrategy(
+ val sparkConf: SparkConf,
+ val yarnConf: Configuration,
+ resolver: SparkRackResolver) {
+
+ /**
+ * Calculate each container's node locality and rack locality
+ * @param numContainer number of containers to calculate
+ * @param numLocalityAwareTasks number of locality required tasks
+ * @param hostToLocalTaskCount a map to store the preferred hostname and possible task
+ * numbers running on it, used as hints for container allocation
+ * @param allocatedHostToContainersMap host to allocated containers map, used to calculate the
+ * expected locality preference by considering the existing
+ * containers
+ * @param localityMatchedPendingAllocations A sequence of pending container request which
+ * matches the localities of current required tasks.
+ * @param rp The ResourceProfile associated with this container.
+ * @return node localities and rack localities, each locality is an array of string,
+ * the length of localities is the same as number of containers
+ */
+ def localityOfRequestedContainers(
+ numContainer: Int,
+ numLocalityAwareTasks: Int,
+ hostToLocalTaskCount: Map[String, Int],
+ allocatedHostToContainersMap: HashMap[String, Set[ContainerId]],
+ localityMatchedPendingAllocations: Seq[SchedulingRequest],
+ schedulingRequestToNodes: mutable.HashMap[Long, Array[String]],
Review comment:
not documented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855762920
**[Test build #139410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139410/testReport)** for PR 32804 at commit [`c80b66f`](https://github.com/apache/spark/commit/c80b66fadb3e6b84cabe94f31e34aa99027b962f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856075198
+CC @otterc, @venkata91
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856427821
**[Test build #139445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139445/testReport)** for PR 32804 at commit [`ec9e127`](https://github.com/apache/spark/commit/ec9e127caa335ae1512714b61f2a1a8e5e67392a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856427821
**[Test build #139445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139445/testReport)** for PR 32804 at commit [`ec9e127`](https://github.com/apache/spark/commit/ec9e127caa335ae1512714b61f2a1a8e5e67392a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855802446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32804:
URL: https://github.com/apache/spark/pull/32804#discussion_r648021094
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSchedulingRequestAllocator.scala
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn
+
+import java.util.{List => JList, Map => JMap, Set => JSet}
+import java.util.concurrent.atomic.AtomicLong
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hadoop.yarn.api.records._
+import org.apache.hadoop.yarn.api.records.NodeAttributeOpCode._
+import org.apache.hadoop.yarn.api.resource.PlacementConstraints._
+import org.apache.hadoop.yarn.api.resource.PlacementConstraints.PlacementTargets._
+import org.apache.hadoop.yarn.client.api.AMRMClient
+import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
+import org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+import org.apache.hadoop.yarn.util.constraint.PlacementConstraintParser
+
+import org.apache.spark.{SecurityManager, SparkConf}
+import org.apache.spark.deploy.yarn.config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.util.{Clock, SystemClock}
+
+
+/**
+ * YarnSchedulingRequestAllocator is charged with requesting containers from the
+ * YARN ResourceManager using Placement Constraint and deciding what to do with
+ * containers when YARN fulfills these requests.
+ *
+ * This class makes use of YARN's AMRMClient APIs. We interact with the AMRMClient in three ways:
+ * * Making our resource needs known, which updates local bookkeeping about containers requested.
+ * * Calling "allocate", which syncs our local container requests with the RM, and returns any
+ * containers that YARN has granted to us. This also functions as a heartbeat.
+ * * Processing the containers granted to us to possibly launch executors inside of them.
+ *
+ * The public methods of this class are thread-safe. All methods that mutate state are
+ * synchronized.
+ */
+private[yarn] class YarnSchedulingRequestAllocator(
+ driverUrl: String,
+ driverRef: RpcEndpointRef,
+ conf: YarnConfiguration,
+ sparkConf: SparkConf,
+ amClient: AMRMClient[ContainerRequest],
+ appAttemptId: ApplicationAttemptId,
+ securityMgr: SecurityManager,
+ localResources: Map[String, LocalResource],
+ resolver: SparkRackResolver,
+ clock: Clock = new SystemClock)
+ extends YarnAllocator(driverUrl, driverRef, conf, sparkConf, amClient,
+ appAttemptId, securityMgr, localResources, resolver, clock) with Logging {
+
+ private[yarn] lazy val nextAllocationRequestId = new AtomicLong(0)
+ private[yarn] val allocationRequestIdToNodes = new mutable.HashMap[Long, Array[String]]
+ private[yarn] val allocationRequestIdToRacks = new mutable.HashMap[Long, Array[String]]
+ // A container placement strategy based on pending tasks' locality preference
+ private[yarn] val schedulingRequestContainerPlacementStrategy =
+ new LocalityPreferredSchedulingRequestContainerPlacementStrategy(sparkConf, conf, resolver)
+ private[yarn] val nodeAttributes =
+ sparkConf.get(NODE_ATTRIBUTE).map(PlacementConstraintParser.parseExpression)
+
+ private[yarn] val outstandingSchedRequests: JMap[JSet[String], JList[SchedulingRequest]] = {
+ val field = classOf[AMRMClientImpl[ContainerRequest]]
+ .getDeclaredField("outstandingSchedRequests")
+ field.setAccessible(true)
+ val outstandingSchedRequests =
+ field.get(amClient.asInstanceOf[AMRMClientImpl[ContainerRequest]])
+ .asInstanceOf[JMap[JSet[String], JList[SchedulingRequest]]]
+ // For pass UT, since mock AMRMClientImpl use reflection will got null.
+ Option(outstandingSchedRequests)
+ .getOrElse(Map.empty[JSet[String], JList[SchedulingRequest]].asJava)
+ }
+
+ /**
+ * A sequence of pending container requests that have not yet been fulfilled.
+ * ResourceProfile id -> pendingAllocate container request
+ */
+ def getPendingSchedAllocate: Map[Int, Seq[SchedulingRequest]] = {
+ outstandingSchedRequests.asScala.values.flatMap(_.asScala.map { request =>
+ request.getPriority.getPriority -> request
+ }).groupBy(_._1).mapValues(_.map(_._2).toSet.toSeq)
+ }
+
+ override def getNumContainersPendingAllocate: Int = synchronized {
+ getPendingSchedAllocate.values.flatten.size
+ }
+
+ /**
+ * Update the set of SchedulingRequest that we will sync with the RM based on the number of
+ * executors we have currently running and our target number of executors for each
+ * ResourceProfile.
+ *
+ * Visible for testing.
+ */
+ override def updateResourceRequests(): Unit = synchronized {
+ val pendingAllocatePerResourceProfileId = getPendingSchedAllocate
+
+ // Here we lack pending size for each profile
+ val missingPerProfile = targetNumExecutorsPerResourceProfileId.map { case (rpId, targetNum) =>
+ val starting = getOrUpdateNumExecutorsStartingForRPId(rpId).get
+ val pending = pendingAllocatePerResourceProfileId.getOrElse(rpId, Seq.empty).size
+ val running = getOrUpdateRunningExecutorForRPId(rpId).size
+ logDebug(s"Updating resource requests for ResourceProfile id: $rpId, target: " +
+ s"$targetNum, pending: $pending, running: $running, executorsStarting: $starting")
+ (rpId, targetNum - pending - running - starting)
+ }.toMap
+
+ missingPerProfile.foreach { case (rpId, missing) =>
+ val hostToLocalTaskCount =
+ hostToLocalTaskCountPerResourceProfileId.getOrElse(rpId, Map.empty)
+ val pendingAllocate = pendingAllocatePerResourceProfileId.getOrElse(rpId, Seq.empty)
+ val numPendingAllocate = pendingAllocate.size
+ // Split the pending container request into three groups: locality matched list, locality
+ // unmatched list and non-locality list. Take the locality matched container request into
+ // consideration of container placement, treat as allocated containers.
+ // For locality unmatched and locality free container requests, cancel these container
+ // requests, since required locality preference has been changed, recalculating using
+ // container placement strategy.
+ val (localRequests, staleRequests, anyHostRequests) =
+ splitPendingSchedulingRequestAllocationsByLocality(hostToLocalTaskCount, pendingAllocate)
+
+ if (missing > 0) {
+ val resource = rpIdToYarnResource.get(rpId)
+ if (log.isInfoEnabled()) {
+ var requestContainerMessage = s"Will request $missing executor container(s) for " +
+ s" ResourceProfile Id: $rpId, each with " +
+ s"${resource.getVirtualCores} core(s) and " +
+ s"${resource.getMemory} MB memory."
+ if (ResourceRequestHelper.isYarnResourceTypesAvailable() &&
+ ResourceRequestHelper.isYarnCustomResourcesNonEmpty(resource)) {
+ requestContainerMessage ++= s" with custom resources: " + resource.toString
+ }
+ logInfo(requestContainerMessage)
+ }
+
+ // cancel "stale" requests for locations that are no longer needed
+ staleRequests.foreach { stale =>
+ removeFromOutstandingSchedulingRequests(stale)
+ }
+ val cancelledContainers = staleRequests.size
+ if (cancelledContainers > 0) {
+ logInfo(s"Canceled $cancelledContainers container request(s) (locality no longer needed)")
+ }
+
+ // consider the number of new containers and cancelled stale containers available
+ val availableContainers = missing + cancelledContainers
+
+ // to maximize locality, include requests with no locality preference
+ // that can be cancelled
+ val potentialContainers = availableContainers + anyHostRequests.size
+
+ val allocatedHostToContainer = getOrUpdateAllocatedHostToContainersMapForRPId(rpId)
+ val numLocalityAwareTasks = numLocalityAwareTasksPerResourceProfileId.getOrElse(rpId, 0)
+ val containerLocalityPreferences =
+ schedulingRequestContainerPlacementStrategy.localityOfRequestedContainers(
+ potentialContainers, numLocalityAwareTasks, hostToLocalTaskCount,
+ allocatedHostToContainer, localRequests, allocationRequestIdToNodes,
+ rpIdToResourceProfile(rpId))
+
+ val newLocalityRequests = new mutable.ArrayBuffer[SchedulingRequest]
+ containerLocalityPreferences.foreach {
+ case ContainerLocalityPreferences(nodes, racks) if nodes != null =>
+ newLocalityRequests += createSchedulingRequest(resource, nodes, racks, rpId)
+ case _ =>
+ }
+
+ if (availableContainers >= newLocalityRequests.size) {
+ // more containers are available than needed for locality, fill in requests for any host
+ for (i <- 0 until (availableContainers - newLocalityRequests.size)) {
+ newLocalityRequests += createSchedulingRequest(resource, rpId = rpId)
+ }
+ } else {
+ val numToCancel = newLocalityRequests.size - availableContainers
+ // cancel some requests without locality preferences to schedule more local containers
+ anyHostRequests.slice(0, numToCancel).foreach { nonLocal =>
+ removeFromOutstandingSchedulingRequests(nonLocal)
+ }
+ if (numToCancel > 0) {
+ logInfo(s"Canceled $numToCancel unlocalized container requests to " +
+ s"resubmit with locality")
+ }
+ }
+ amClient.addSchedulingRequests(newLocalityRequests.asJava)
+ } else if (numPendingAllocate > 0 && missing < 0) {
+ val numToCancel = math.min(numPendingAllocate, -missing)
+ logInfo(s"Canceling requests for $numToCancel executor container(s) to have a new " +
+ s"desired total ${getOrUpdateTargetNumExecutorsForRPId(rpId)} executors.")
+ // cancel pending allocate requests by taking locality preference into account
+ val cancelRequests = (staleRequests ++ anyHostRequests ++ localRequests).take(numToCancel)
+ cancelRequests.foreach(removeFromOutstandingSchedulingRequests)
+ }
+ }
+ }
+
+ def createSchedulingRequest(
+ resource: Resource,
+ nodes: Array[String] = Array.empty,
+ racks: Array[String] = Array.empty,
+ rpId: Int): SchedulingRequest = {
+ val allocationRequestId = nextAllocationRequestId.getAndIncrement()
+ val nodesLocality = if (nodes.nonEmpty) {
+ allocationRequestIdToNodes.put(allocationRequestId, nodes)
+ val nodesConstraints = nodes.map { node =>
+ targetNodeAttribute(NODE, EQ, nodeAttribute("host", node))
+ }
+ if (nodesConstraints.size > 1) {
+ Some(or(nodesConstraints: _*))
+ } else {
+ Some(nodesConstraints.head)
+ }
+ } else {
+ None
+ }
+
+ val racksLocality = if (racks.nonEmpty) {
+ allocationRequestIdToRacks.put(allocationRequestId, racks)
+ val rackConstraints = racks.map { rack =>
+ targetNodeAttribute(NODE, EQ, nodeAttribute("rack", rack))
+ }
+ if (rackConstraints.size > 1) {
+ Some(or(rackConstraints: _*))
+ } else {
+ Some(rackConstraints.head)
+ }
+ } else {
+ None
+ }
+
+ val locality = (nodesLocality, racksLocality) match {
+ case (Some(nodes), Some(racks)) => Some(or(nodes, racks))
+ case (Some(nodes), None) => Some(nodes)
+ case (None, Some(racks)) => Some(racks)
+ case _ => None
+ }
+
+ val expression = (nodeAttributes.isDefined, locality.isDefined) match {
+ case (true, true) => Some(or(and(nodeAttributes.get, locality.get), nodeAttributes.get))
Review comment:
Here should use below constraint is better.
```
delayedOr(
timedClockConstraint(and(nodeAttributes, locality), delayedOrIntervalMilliseconds, TimeUnit.MILLISECONDS),
timedClockConstraint(nodeAttributes, delayedOrIntervalMilliseconds * 2, TimeUnit.MILLISECONDS)))
```
But it always stuck when test in yarn-3.3.0 cluster. Hope for help at this part. It's hard to balance node attribute constraint and locality requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-857456767
**[Test build #139553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139553/testReport)** for PR 32804 at commit [`2cd9e69`](https://github.com/apache/spark/commit/2cd9e69926a0900eec85d4424b9cdce919ba056c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-855796036
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43932/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856441509
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43968/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32804: [SPARK-26867][YARN] Spark Support of YARN Placement Constraint
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32804:
URL: https://github.com/apache/spark/pull/32804#issuecomment-856456954
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43968/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org