You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2013/11/14 19:32:14 UTC
[1/2] git commit: Don't ignore spark.cores.max when using Mesos
Coarse mode
Updated Branches:
refs/heads/master 5a4f48365 -> 1a4cfbea3
Don't ignore spark.cores.max when using Mesos Coarse mode
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/5125cd34
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/5125cd34
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/5125cd34
Branch: refs/heads/master
Commit: 5125cd34663b83edceaa40deaf5f7f48a12138e5
Parents: 2054c61
Author: Kay Ousterhout <ka...@gmail.com>
Authored: Wed Nov 13 23:06:17 2013 -0800
Committer: Kay Ousterhout <ka...@gmail.com>
Committed: Wed Nov 13 23:06:17 2013 -0800
----------------------------------------------------------------------
.../spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/5125cd34/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
index 300fe69..cd521e0 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
@@ -181,6 +181,7 @@ private[spark] class CoarseMesosSchedulerBackend(
!slaveIdsWithExecutors.contains(slaveId)) {
// Launch an executor on the slave
val cpusToUse = math.min(cpus, maxCores - totalCoresAcquired)
+ totalCoresAcquired += cpusToUse
val taskId = newMesosTaskId()
taskIdToSlaveId(taskId) = slaveId
slaveIdsWithExecutors += slaveId
[2/2] git commit: Merge pull request #169 from kayousterhout/mesos_fix
Posted by rx...@apache.org.
Merge pull request #169 from kayousterhout/mesos_fix
Don't ignore spark.cores.max when using Mesos Coarse mode
totalCoresAcquired is decremented but never incremented, causing Spark to effectively ignore spark.cores.max in coarse grained Mesos mode.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/1a4cfbea
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/1a4cfbea
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/1a4cfbea
Branch: refs/heads/master
Commit: 1a4cfbea334c7b0dae287eab4c3131c8f4b8a992
Parents: 5a4f483 5125cd3
Author: Reynold Xin <rx...@apache.org>
Authored: Thu Nov 14 10:32:11 2013 -0800
Committer: Reynold Xin <rx...@apache.org>
Committed: Thu Nov 14 10:32:11 2013 -0800
----------------------------------------------------------------------
.../spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------