You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2017/06/29 12:54:02 UTC
spark git commit: [SPARK-21225][CORE] Considering CPUS_PER_TASK when
allocating task slots for each WorkerOffer
Repository: spark
Updated Branches:
refs/heads/master d7da2b94d -> 29bd251dd
[SPARK-21225][CORE] Considering CPUS_PER_TASK when allocating task slots for each WorkerOffer
JIRA Issue:https://issues.apache.org/jira/browse/SPARK-21225
In the function "resourceOffers", It declare a variable "tasks" for storage the tasks which have allocated a executor. It declared like this:
`val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores))`
But, I think this code only conside a situation for that one task per core. If the user set "spark.task.cpus" as 2 or 3, It really don't need so much Mem. I think It can motify as follow:
val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores / CPUS_PER_TASK))
to instead.
Motify like this the other earning is that it's more easy to understand the way how the tasks allocate offers.
Author: 杨治国10192065 <ya...@zte.com.cn>
Closes #18435 from JackYangzg/motifyTaskCoreDisp.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/29bd251d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/29bd251d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/29bd251d
Branch: refs/heads/master
Commit: 29bd251dd5914fc3b6146eb4fe0b45f1c84dba62
Parents: d7da2b9
Author: 杨治国10192065 <ya...@zte.com.cn>
Authored: Thu Jun 29 20:53:48 2017 +0800
Committer: Wenchen Fan <we...@databricks.com>
Committed: Thu Jun 29 20:53:48 2017 +0800
----------------------------------------------------------------------
.../main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/29bd251d/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
index 91ec172..737b383 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
@@ -345,7 +345,7 @@ private[spark] class TaskSchedulerImpl(
val shuffledOffers = shuffleOffers(filteredOffers)
// Build a list of tasks to assign to each worker.
- val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores))
+ val tasks = shuffledOffers.map(o => new ArrayBuffer[TaskDescription](o.cores / CPUS_PER_TASK))
val availableCpus = shuffledOffers.map(o => o.cores).toArray
val sortedTaskSets = rootPool.getSortedTaskSetQueue
for (taskSet <- sortedTaskSets) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org