You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/03 06:10:09 UTC
[GitHub] [spark] Ngone51 opened a new pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Ngone51 opened a new pull request #29332:
URL: https://github.com/apache/spark/pull/29332
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
1. Make `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` considers all kinds of resources when calculating the max concurrent tasks
2. Refactor `calculateAvailableSlots()` to make it be able to be used for both `CoarseGrainedSchedulerBackend` and `TaskSchedulerImpl`
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently, `CoarseGrainedSchedulerBackend.maxNumConcurrentTasks()` only considers the CPU for the max concurrent tasks. This can cause the application to hang when a barrier stage requires extra custom resources but the cluster doesn't have enough corresponding resources. Because, without the checking for other custom resources in `maxNumConcurrentTasks`, the barrier stage can be submitted to the `TaskSchedulerImpl`. But the `TaskSchedulerImpl` won't launch tasks for the barrier stage due to the insufficient task slots calculated by `TaskSchedulerImpl.calculateAvailableSlots` (which does check all kinds of resources).
The application hang issue can be reproduced by the added unit test.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. In case of a barrier stage requires more custom resources than the cluster has, the application can get hang before this PR but can fail due to insufficient resources at the end after this PR.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added a unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668602100
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29332:
URL: https://github.com/apache/spark/pull/29332#discussion_r464778287
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -633,9 +633,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
}
override def maxNumConcurrentTasks(rp: ResourceProfile): Int = synchronized {
- val cpusPerTask = ResourceProfile.getTaskCpusOrDefaultForProfile(rp, conf)
- val executorsWithResourceProfile = executorDataMap.values.filter(_.resourceProfileId == rp.id)
- executorsWithResourceProfile.map(_.totalCores / cpusPerTask).sum
+ val (rpIds, cpus, resources) = executorDataMap.values.toArray.map { executor =>
Review comment:
make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668959724
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-669954242
branch 3 doesn't need it because cores are forced to be limiting resource.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667845751
**[Test build #126958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126958/testReport)** for PR 29332 at commit [`3af932b`](https://github.com/apache/spark/commit/3af932b2fe3db30bcca5a83bf6f783c98a591304).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667824281
ping @tgravescs @jiangxb1987 @mridulm Please take a look, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668601283
**[Test build #127053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127053/testReport)** for PR 29332 at commit [`83a5dff`](https://github.com/apache/spark/commit/83a5dff5d659cd6931e27bcce4419b212d2130db).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668845074
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29332:
URL: https://github.com/apache/spark/pull/29332
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-669995749
That's not true when we don't need to check cores in some clusters? for example, Standalone.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668959450
**[Test build #127074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127074/testReport)** for PR 29332 at commit [`786b145`](https://github.com/apache/spark/commit/786b145771a12004668ea35afdc6e4054924bfed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667827496
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668360534
**[Test build #127022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127022/testReport)** for PR 29332 at commit [`83a5dff`](https://github.com/apache/spark/commit/83a5dff5d659cd6931e27bcce4419b212d2130db).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667826711
**[Test build #126958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126958/testReport)** for PR 29332 at commit [`3af932b`](https://github.com/apache/spark/commit/3af932b2fe3db30bcca5a83bf6f783c98a591304).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668602100
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668094805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126961/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667847279
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668959450
**[Test build #127074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127074/testReport)** for PR 29332 at commit [`786b145`](https://github.com/apache/spark/commit/786b145771a12004668ea35afdc6e4054924bfed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668421017
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127022/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-670010625
Oh right, if you didn't specify the executor cores in standalone mode - you get them all by default. Which honestly causes lots of issues - I filed a jira for this for other things. that is left up to the user to properly configure their cluster and job.
There are other things in spark 3.0 that expect cores to be the limiting resource. Dynamic allocation is a big one, which I know you don't care about, but I'm pretty sure there are other things throughout the code that does as well.
We should remove the isDynamicAllocationCheck here so that the warning gets printed for standalone as well:
https://github.com/apache/spark/blob/branch-3.0/core/src/main/scala/org/apache/spark/SparkContext.scala#L2836
If you want to go through and make sure everything is updated to schedule based on resources, I'm fine with it but I'm pretty sure needs to be more then this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-669985549
This change it checking to see what your executors have. Spark will not let you start the application if your task requirements are such that cpus is not your limiting resource. so the only way you can get an executor with 2 cores and 1 gpu is if each task requires 2 cores and 1 gpu and at that point gpu is not your limiting resource, they are equivalent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29332:
URL: https://github.com/apache/spark/pull/29332#discussion_r464396088
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -468,51 +468,6 @@ private[spark] class TaskSchedulerImpl(
Some(localTaskReqAssign.toMap)
}
- // Use the resource that the resourceProfile has as the limiting resource to calculate the
- // total number of slots available based on the current offers.
- private def calculateAvailableSlots(
- resourceProfileIds: Array[Int],
- availableCpus: Array[Int],
- availableResources: Array[Map[String, Buffer[String]]],
- taskSet: TaskSetManager): Int = {
- val resourceProfile = sc.resourceProfileManager.resourceProfileFromId(
- taskSet.taskSet.resourceProfileId)
- val offersForResourceProfile = resourceProfileIds.zipWithIndex.filter { case (id, _) =>
- (id == resourceProfile.id)
- }
- val coresKnown = resourceProfile.isCoresLimitKnown
- var limitingResource = resourceProfile.limitingResource(conf)
- val taskCpus = ResourceProfile.getTaskCpusOrDefaultForProfile(resourceProfile, conf)
-
- offersForResourceProfile.map { case (o, index) =>
- val numTasksPerExecCores = availableCpus(index) / taskCpus
- // if limiting resource is empty then we have no other resources, so it has to be CPU
- if (limitingResource == ResourceProfile.CPUS || limitingResource.isEmpty) {
- numTasksPerExecCores
- } else {
- val taskLimit = resourceProfile.taskResources.get(limitingResource).map(_.amount)
- .getOrElse {
- val errorMsg = "limitingResource returns from ResourceProfile " +
- s"$resourceProfile doesn't actually contain that task resource!"
Review comment:
@tgravescs This actually should not happen, right? According to:
https://github.com/apache/spark/blob/998086c9a179692b2687bc9a104dbbb35f5a44e2/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala#L194-L197
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668996964
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667845890
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668094799
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668996964
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668600089
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667826711
**[Test build #126958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126958/testReport)** for PR 29332 at commit [`3af932b`](https://github.com/apache/spark/commit/3af932b2fe3db30bcca5a83bf6f783c98a591304).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667847279
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668421014
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668360880
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #29332:
URL: https://github.com/apache/spark/pull/29332#discussion_r464427498
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -633,9 +633,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
}
override def maxNumConcurrentTasks(rp: ResourceProfile): Int = synchronized {
- val cpusPerTask = ResourceProfile.getTaskCpusOrDefaultForProfile(rp, conf)
- val executorsWithResourceProfile = executorDataMap.values.filter(_.resourceProfileId == rp.id)
- executorsWithResourceProfile.map(_.totalCores / cpusPerTask).sum
+ val (rpIds, cpus, resources) = executorDataMap.values.toArray.map { executor =>
Review comment:
I think this should first filter executorDataMap for active executors.
We are doing a lot of calculation here even when not in a barrier stage. I think out in DagScheduler.checkBarrierStageWithNumSlots we should move the call to this after check for rdd.isBarrier (or make it las) so everyone doesn't have to pay for it. This is definitely doing more work than it was before and we want to keep scheduler code as fast as possible.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -468,51 +468,6 @@ private[spark] class TaskSchedulerImpl(
Some(localTaskReqAssign.toMap)
}
- // Use the resource that the resourceProfile has as the limiting resource to calculate the
- // total number of slots available based on the current offers.
- private def calculateAvailableSlots(
- resourceProfileIds: Array[Int],
- availableCpus: Array[Int],
- availableResources: Array[Map[String, Buffer[String]]],
- taskSet: TaskSetManager): Int = {
- val resourceProfile = sc.resourceProfileManager.resourceProfileFromId(
- taskSet.taskSet.resourceProfileId)
- val offersForResourceProfile = resourceProfileIds.zipWithIndex.filter { case (id, _) =>
- (id == resourceProfile.id)
- }
- val coresKnown = resourceProfile.isCoresLimitKnown
- var limitingResource = resourceProfile.limitingResource(conf)
- val taskCpus = ResourceProfile.getTaskCpusOrDefaultForProfile(resourceProfile, conf)
-
- offersForResourceProfile.map { case (o, index) =>
- val numTasksPerExecCores = availableCpus(index) / taskCpus
- // if limiting resource is empty then we have no other resources, so it has to be CPU
- if (limitingResource == ResourceProfile.CPUS || limitingResource.isEmpty) {
- numTasksPerExecCores
- } else {
- val taskLimit = resourceProfile.taskResources.get(limitingResource).map(_.amount)
- .getOrElse {
- val errorMsg = "limitingResource returns from ResourceProfile " +
- s"$resourceProfile doesn't actually contain that task resource!"
Review comment:
correct, I believe it was there just a double check and make sure nothing broke in the future
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -633,9 +633,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
}
override def maxNumConcurrentTasks(rp: ResourceProfile): Int = synchronized {
- val cpusPerTask = ResourceProfile.getTaskCpusOrDefaultForProfile(rp, conf)
- val executorsWithResourceProfile = executorDataMap.values.filter(_.resourceProfileId == rp.id)
- executorsWithResourceProfile.map(_.totalCores / cpusPerTask).sum
+ val (rpIds, cpus, resources) = executorDataMap.values.toArray.map { executor =>
+ (
+ executor.resourceProfileId,
+ executor.totalCores,
+ executor.resourcesInfo.map { case (name, rInfo) =>
+ (name, rInfo.resourceAddresses.length * rInfo.slotsPerAddress)
Review comment:
To be clear, here we are trying to calculate the total concurrent tasks that could run, even if some of them are busy at the moment so I think we should add some documentation there just to clarify.
Also can we add in a function to ExecutorResourceInfo to get this (rInfo.resourceAddresses.length * rInfo.slotsPerAddress).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667827496
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668845081
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127053/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668420208
**[Test build #127022 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127022/testReport)** for PR 29332 at commit [`83a5dff`](https://github.com/apache/spark/commit/83a5dff5d659cd6931e27bcce4419b212d2130db).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668421014
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-671190320
I am not sure about the other things(probably, you could link the specific JIRA?) At least, I'd like to backport the fix to schedule based on the resources: https://github.com/apache/spark/pull/29395.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667846056
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668996210
**[Test build #127074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127074/testReport)** for PR 29332 at commit [`786b145`](https://github.com/apache/spark/commit/786b145771a12004668ea35afdc6e4054924bfed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668959724
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-669981103
I think that's the problem. Think about the case where a barrier stage requires 2 CPUs and 2 GPUs but the cluster only has 2 CPUs and 1 GPU. In 3.0, since the limiting resource is cores/CPUs, the barrier stage would have a chance to launch tasks. However, it could only launch one task because the real limiting resources should be GPU. In this case, the barrier stage will fail because of partial task launch. But the error message is quite confusing for users as it suggests to disable delay scheduling, while the real cause should be insufficient (custom) resources. If we backport this fix to 3.0, the barrier stage should fail early before it was able to launch the task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667845890
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-669795413
Yea let's create a new a PR for 3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668094799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668845074
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668360880
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-669713324
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-669783766
thanks all! BTW, shall we backport this to branch-3.0? Though, it would be a little different fix since branch-3.0 doesn't have `ResourceProfile`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-668360534
**[Test build #127022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127022/testReport)** for PR 29332 at commit [`83a5dff`](https://github.com/apache/spark/commit/83a5dff5d659cd6931e27bcce4419b212d2130db).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29332: [SPARK-32518][CORE] CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29332:
URL: https://github.com/apache/spark/pull/29332#issuecomment-667853878
**[Test build #126961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126961/testReport)** for PR 29332 at commit [`3af932b`](https://github.com/apache/spark/commit/3af932b2fe3db30bcca5a83bf6f783c98a591304).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org