You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by brad-kaiser <gi...@git.apache.org> on 2017/08/24 15:00:10 UTC
[GitHub] spark pull request #19041: [SPARK-21097][CORE][WIP] Add option to recover ca...
GitHub user brad-kaiser opened a pull request:
https://github.com/apache/spark/pull/19041
[SPARK-21097][CORE][WIP] Add option to recover cached data during dynamic a…
…llocation
## What changes were proposed in this pull request?
Add an option to to recover cached data when using dynamic allocation. Currently when dynamic allocation decommissions an idle executor, any cached RDD blocks on that executor will be lost. With this change those blocks will be replicated to other executors. If there is not enough space for the cached blocks, or any error is encountered, we will fall back to the original behavior and drop the blocks.
Added the following config options
```
spark.dynamicAllocation.recoverCachedData
spark.dynamicAllocation.recoverCachedData.timeout
```
## How was this patch tested?
There are unit tests in RecoverCacheShutdownSuite and integration tests in RecoverCachedDataSuite. I have also run benchmarks on a test cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brad-kaiser/spark recover-cached
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19041.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19041
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156744076
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE][WIP] Add option to recover cached da...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/19041
Welcome to the Apache Spark project @brad-kaiser, GitHub tells me this is your first time contributing code :)
While this is a bit outside of my area of review as we've discussed I've been doing some similar WIP work in the area along with some others so I'm hoping we can all find a way to collaborate.
In the meantime I can maybe help out with Jenkins ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179585326
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
--- End diff --
I don't have a strong opinion, but it sounds like your original approach was motivated by a small set of just the in-memory cached blocks. Lets stick to in-memory only blocks for this version (`!useDisk`) as that limits the blocks we try to replicated, and we can consider relaxing that in the future.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143816789
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -76,6 +76,14 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("3s")
+ private[spark] val DYN_ALLOCATION_RECOVER_CACHE =
+ ConfigBuilder("spark.dynamicAllocation.recoverCachedData").booleanConf.createWithDefault(false)
--- End diff --
`.cacheRecovery.enabled` to follow pattern of other feature flags.
I also kinda prefer multiple lines in these constant declarations, for readability.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155387940
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.executor.memory", "512m")
+ .set("spark.shuffle.service.enabled", "true")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.cacheRecovery.enabled", "true")
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s")
+ .set("spark.executor.instances", "1")
+ .set("spark.dynamicAllocation.initialExecutors", "4")
+ .set("spark.dynamicAllocation.minExecutors", "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ type BInfo = Map[BlockId, Map[BlockManagerId, BlockStatus]]
+ private def getLocations(sc: SparkContext, rdd: RDD[_]): BInfo = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+
+ Thread.sleep(3000)
+ sc.getExecutorIds().size shouldBe 3
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+ }
+
+ test("dont fail if a bunch of executors are shut down at once") {
+ conf.set("spark.dynamicAllocation.minExecutors", "1")
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+
+ Thread.sleep(3000)
--- End diff --
Same comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155378488
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @param killTimer The runnable scheduled to kill this executor. Cancel it if we finish before
+ * it does.
+ */
+ private def replicateUntilDone(execId: String, killTimer: ScheduledFuture[_]): Unit = {
+ recoverLatestBlock(execId).onComplete {
+ case Succ(Some(blockId)) =>
+ logTrace(s"Replicated block $blockId on executor $execId")
+ replicateUntilDone(execId, killTimer)
+ case Succ(None) =>
+ killTimer.cancel(false)
+ kill(execId)
+ case Failure(e) =>
+ logWarning("Failure recovering cached data before executor $execId shutdown", e)
+ killTimer.cancel(false)
+ kill(execId)
--- End diff --
Call this only if the timer was successfully stopped? (Same above.)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155387718
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.executor.memory", "512m")
+ .set("spark.shuffle.service.enabled", "true")
+ .set("spark.dynamicAllocation.enabled", "true")
--- End diff --
Use config constants where possible.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163052079
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
--- End diff --
What does the boolean the future returns mean?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163042297
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
--- End diff --
This fits in one line if you drop the type, which is kinda redundant in this context anyway.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156417466
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
--- End diff --
added comment
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84782 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84782/testReport)** for PR 19041 at commit [`47d8eea`](https://github.com/apache/spark/commit/47d8eea0bcc3a68b4df47cb4e6aa8a224a3b0e4a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r154987566
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155377523
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
--- End diff --
I find this more readable:
```
canBeRecovered.foreach { execId =>
replicateUntilDone(execId, startKillTimer(execId))
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r139004878
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
--- End diff --
There was an issue with scala.util.Success getting shadowed by the success object in TaskEndReason.scala, but that's fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138755966
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
+
+ private val forceKillAfterS =
+ conf.getTimeAsSeconds("spark.dynamicAllocation.recoverCachedData.timeout", "120s")
--- End diff --
Please add config constants.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88471/testReport)** for PR 19041 at commit [`668dd82`](https://github.com/apache/spark/commit/668dd827d6bca7201a81be960874523fbfb9bff7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88603/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82478/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155378175
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
--- End diff --
nit: `TrySuccess` (more verbose but less cryptic).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138761658
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala ---
@@ -32,8 +32,13 @@ private[spark] object BlockManagerMessages {
// blocks that the master knows about.
case class RemoveBlock(blockId: BlockId) extends ToBlockManagerSlave
+ // Replicate block excluding certain executors for graceful shutdown.
--- End diff --
Adding this comment makes the whole comment block a little bit confusing. (As in, which one is correct?)
If you want to document the `excluding` parameter you can create a scaladoc comment instead. Mentioning "graceful shutdown" here doesn't really add a lot of information.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155386687
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ test("GracefulShutdown will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1010)
--- End diff --
Same comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143813871
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ *
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
+ logDebug(s"Replicate block on executor $execId.")
+ blocksToSave.get(execId) match {
+ case Some(p) if p.nonEmpty => doReplication(execId, p.dequeue())
+ case _ => (Future.successful(false), None)
+ }
+ }
+
+ private def doReplication(
+ execId: String,
+ blockId: RDDBlockId): (Future[Boolean], Option[RDDBlockId]) = {
+ val savedBlocksForExecutor = savedBlocks.getOrElseUpdate(execId, new mutable.HashSet)
+ savedBlocksForExecutor += blockId
+ val replicateMessage = ReplicateOneBlock(execId, blockId, blocksToSave.keys.toSeq)
+ logTrace(s"Started replicating block $blockId on exec $execId.")
+ val future = blockManagerMasterEndpoint.ask[Boolean](replicateMessage)
+ (future, Some(blockId))
+ }
+
+ /**
+ * Ask ExecutorAllocationManager to kill executor and clean up state
+ * Note executorAllocationManger.killExecutors is blocking, this function should be fixed
+ * to use a non blocking version
+ * @param execId the executor to kill
--- End diff --
nit: empty line before param list
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163050134
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
--- End diff --
These calls need to be in a `finally` block (or in an `after` block if you don't mind moving some state to the class).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r166507738
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Seq[Future[Boolean]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ canBeRecovered.foreach { execId => recoveringExecutors.put(execId, true)}
+ canBeRecovered.map { execId =>
+ val killTimer = startKillTimer(execId)
+ replicateUntilDone(execId)
+ .andThen {
+ // don't log if kill timer is done, because it would be redundant
+ case Failure(e) if !killTimer.isDone =>
+ logWarning(s"Failure recovering cached data before executor $execId shutdown", e)
+ }
+ .andThen {
+ case _ =>
+ if (killTimer.cancel(false)) { kill(execId) }
+ }
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @return a Future of Boolean that returns true if a block was copied and false if there were
+ * no more blocks
+ */
+ private def replicateUntilDone(execId: String): Future[Boolean] = {
+ recoverLatestBlock(execId).flatMap { moreBlocks =>
--- End diff --
oops fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88716/testReport)** for PR 19041 at commit [`c8f7ad0`](https://github.com/apache/spark/commit/c8f7ad04ecff60df6700d53b9d23a3d6888b049a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Lol, no worries. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r177130883
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class CacheRecoveryIntegrationSuite
--- End diff --
This class is meant to be an integration test that shows cache recovery working in a real cluster, unlike the unit tests in CacheRecoveryManagerSuite, where much of the cluster machinery is mocked. That is why i do not shut down executors directly and why I have to inspect the results in a roundabout way. I added some comments explaining this.
If you don't think this kind of testing is necessary, we can take it out, but I thought it was a nice sanity check while developing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
@squito
I updated BlockManagerMasterEndpoint.recoverLatestRDDBlock so that we proactively remove the block from blockManagerInfo when we ask the slave to remove the block. Thanks for finding this!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88471/testReport)** for PR 19041 at commit [`668dd82`](https://github.com/apache/spark/commit/668dd827d6bca7201a81be960874523fbfb9bff7).
* This patch **fails PySpark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19041
Thanks @brad-kaiser -- want to re-iterate my comment from Feb 2nd, I think that is really the most important part to address before getting into the details of the current implementation:
> Thought some more about the race between RemoveBlock getting sent back from the executor vs when the CacheRecoveryManager tries to replicate the next block -- actually why is there the back-and-forth with the driver for every block? Why isn't there just one message from the CacheRecoveryManager to the executor, saying "Drain all RDD blocks" and then one message from the executor back to the driver when its done?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r178968393
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future, Promise}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.CacheRecoveryManager.{DoneRecovering, KillReason, Timeout}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
--- End diff --
I find `recoveringExecutors` pretty confusing, I think its executors that are recovering from some problem, but are going to be OK -- not executors that are about to die, which we are recovering data from. how about `drainingExecutors`? (though I have a feeling this name may have been discussed in earlier rounds of comments and this is what we settled on ... if so, thats fine.)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156744373
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ test("GracefulShutdown will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1010)
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r140841617
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
+
+ private val forceKillAfterS =
+ conf.getTimeAsSeconds("spark.dynamicAllocation.recoverCachedData.timeout", "120s")
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave: ExecMap[mutable.PriorityQueue[RDDBlockId]] = new mutable.HashMap
+ private val savedBlocks: ExecMap[mutable.HashSet[RDDBlockId]] = new mutable.HashMap
+ private val killTimers: ExecMap[ScheduledFuture[_]] = new mutable.HashMap
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ logDebug(s"Get all RDD blocks for executors: ${execIds.mkString(", ")}.")
+ execIds.map { id =>
+ if (isThereEnoughMemory(id)) {
--- End diff --
I rewrote this code and simplified it. I don't call GetMemoryStatus for every executor anymore.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88716/testReport)** for PR 19041 at commit [`c8f7ad0`](https://github.com/apache/spark/commit/c8f7ad04ecff60df6700d53b9d23a3d6888b049a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class ReplicateBlock(`
* ` case class RecoverLatestRDDBlock(executorId: String, excludingExecs: Seq[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143814841
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -88,6 +88,8 @@ private[spark] class ExecutorAllocationManager(
import ExecutorAllocationManager._
+ var cacheRecoveryManager: CacheRecoveryManager = _
--- End diff --
private?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143819863
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -237,6 +246,43 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateOneBlock(
+ execId: String,
+ blockId: BlockId,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"replicating block $blockId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ replicaSet <- blockLocations.asScala.get(blockId)
+ replicas = replicaSet.toSeq
+ maxReps = replicaSet.size + 2
+ } yield info.slaveEndpoint.ask[Boolean](ReplicateBlock(blockId, replicas, excluded, maxReps))
+
+ response.getOrElse(Future.successful(false)).foreach(context.reply)
+ }
+
+ private def getCachedBlocks(executorId: String): collection.Set[BlockId] = {
--- End diff --
just `Set[BlockId]`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Thanks, I will address these shortly
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156744499
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ test("GracefulShutdown will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1010)
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1100)
+ // should be killed once not twice
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ }
+
+
+ test("Blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(1, blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ Thread.sleep(100)
+ bmme.replicated.size shouldBe 2
+ }
+
+ test("Blocks won't replicate if we are stopping all executors") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((2L, 1L)),
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156421510
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -682,6 +689,15 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
}
protected def fetchHadoopDelegationTokens(): Option[Array[Byte]] = { None }
+
+ override def markPendingToRemove(executorIds: Seq[String]): Unit = synchronized {
+ logDebug(s"marking $executorIds pending to remove")
+ executorIds.foreach { id =>
+ if (!executorsPendingToRemove.contains(id)) {
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143802667
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
--- End diff --
indent should follow same style as the class at the top of this file
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155043906
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -698,6 +696,11 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
None
}
}
+
+ override def markPendingToRemove(executorIds: Seq[String]): Unit = synchronized {
+ logDebug(s"marking $executorIds pending to remove")
+ executorIds.foreach(id => executorsPendingToRemove.put(id, true))
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r165488466
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.max[RDDBlockId](Ordering.by(_.rddId)))
+ replicaSet <- blockLocations.asScala.get(firstBlock)
+ // Add 2 below because you need the number of replicas, plus one for the original, plus one
+ // for the new replica.
+ maxReps = replicaSet.size + 2
+ } yield info.slaveEndpoint
+ .ask[Boolean](ReplicateBlock(firstBlock, replicaSet.toSeq, excluded, maxReps))
+ .flatMap { success =>
+ if (success) {
+ logTrace(s"Replicated block $firstBlock on executor $execId")
+ replicaSet -= blockManagerId
+ info.slaveEndpoint.ask[Boolean](RemoveBlock(firstBlock))
--- End diff --
if I understand right, in order for the next iteration to avoid trying to remove the same block over again, you need this call to update the `info.cachedBlocks` used above. But I think that update is async -- even after this future has completed, it doesn't mean `info.cachedBlocks` has been updated.
I think what will happen in that case is, on the next time through, on the next iteration you'll try to remove the exact same block, which will fail on the executor because its already been removed, and then back on the driver you'll decide to stop trying to replicate the rest of the blocks because of this apparent failure.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #85070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85070/testReport)** for PR 19041 at commit [`f637f41`](https://github.com/apache/spark/commit/f637f4164ac341c56c163d9051889b61aea6255b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r139005783
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
+
+ private val forceKillAfterS =
+ conf.getTimeAsSeconds("spark.dynamicAllocation.recoverCachedData.timeout", "120s")
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave: ExecMap[mutable.PriorityQueue[RDDBlockId]] = new mutable.HashMap
+ private val savedBlocks: ExecMap[mutable.HashSet[RDDBlockId]] = new mutable.HashMap
+ private val killTimers: ExecMap[ScheduledFuture[_]] = new mutable.HashMap
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ logDebug(s"Get all RDD blocks for executors: ${execIds.mkString(", ")}.")
+ execIds.map { id =>
+ if (isThereEnoughMemory(id)) {
+ updateBlockState(id)
+ val blocksRemain = blocksToSave.get(id).exists(_.nonEmpty)
+ id -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ } else {
+ id -> NotEnoughMemory
+ }
+ }(collection.breakOut)
+ }
+
+ /**
+ * Is there enough memory on the cluster to replicate the cached data on execId before we delete
+ * it. Take into account all the blocks that are currently on track to be deleted.
+ * @param execId the id of the executor we want to delete.
+ * @return true if there is room
+ */
+ private def isThereEnoughMemory(execId: String): Boolean = {
+ val currentMemStatus = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+ val remaining = currentMemStatus - execId -- blocksToSave.keys
+
+ val thisExecBytes = if (!blocksToSave.contains(execId) && currentMemStatus.contains(execId)) {
+ currentMemStatus(execId)._1 - currentMemStatus(execId)._2
+ } else {
+ 0
+ }
+
+ remaining.nonEmpty && everyExecutorHasEnoughMemory(remaining, thisExecBytes)
+ }
+
+ private def everyExecutorHasEnoughMemory(execMem: Map[String, (Long, Long)], bytes: Long) = {
+ val blocksInQueueBytes = blockManagerMasterEndpoint.askSync[Long](GetSizeOfBlocks(blockList))
+ val bytesPerExec = (blocksInQueueBytes + bytes) / execMem.size.toFloat
+ execMem.forall { case (_, (_, remaining)) => remaining - bytesPerExec >= 0 }
+ }
+
+ private def blockList = for {
+ (id, queue) <- blocksToSave.toSeq
+ block <- queue
+ } yield (id, block)
+
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
+ logDebug(s"Replicate block on executor $execId.")
+ blocksToSave.get(execId) match {
+ case Some(p) if p.nonEmpty => doReplication(execId, p.dequeue())
+ case _ => (Future.successful(false), None)
+ }
+ }
+
+ private def doReplication(execId: String,
+ blockId: RDDBlockId): (Future[Boolean], Option[RDDBlockId]) = {
--- End diff --
Fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143817365
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -646,18 +648,14 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
Future.successful(true)
}
- val killExecutors: Boolean => Future[Boolean] =
- if (!executorsToKill.isEmpty) {
- _ => doKillExecutors(executorsToKill)
- } else {
- _ => Future.successful(false)
- }
-
- val killResponse = adjustTotalExecutors.flatMap(killExecutors)(ThreadUtils.sameThread)
+ val killResponse = if (executorsToKill.nonEmpty) {
+ adjustTotalExecutors.flatMap(_ => doKillExecutors(executorsToKill))(ThreadUtils.sameThread)
--- End diff --
`.flatMap { _ => ... }`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138753970
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -108,6 +110,10 @@ private[spark] class ExecutorAllocationManager(
private val cachedExecutorIdleTimeoutS = conf.getTimeAsSeconds(
"spark.dynamicAllocation.cachedExecutorIdleTimeout", s"${Integer.MAX_VALUE}s")
+ // whether or not to try and save cached data when executors are deallocated
+ private val replicateCachedData =
+ conf.getBoolean("spark.dynamicAllocation.recoverCachedData", defaultValue = false)
--- End diff --
Please add a config constant in `core/src/main/scala/org/apache/spark/internal/config/package.scala`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138755830
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
--- End diff --
Could you import the types instead? Also the code after the `=>` would look better in the next line, given that's how the other cases do it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Thanks for taking a look @vanzin. I'll address these points.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143799104
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
--- End diff --
executor's
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE][WIP] Add option to recover cached da...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138756535
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
--- End diff --
nit: we generally add a blank line before the parameter list.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179180337
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkException, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+/**
+ * This is an integration test for the cache recovery feature using a local spark cluster. It
+ * extends the unit tests in CacheRecoveryManagerSuite which mocks a lot of cluster infrastructure.
+ */
+class CacheRecoveryIntegrationSuite extends SparkFunSuite
+ with Matchers
+ with BeforeAndAfterEach
+ with BeforeAndAfterAll
+ with Eventually {
+
+ private var conf: SparkConf = makeBaseConf()
+ private val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ private val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ private val transportContext = new TransportContext(transportConf, rpcHandler)
+ private val shuffleService = transportContext.createServer()
+ private var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s") // always
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set(EXECUTOR_MEMORY.key, "512m")
+ .set(SHUFFLE_SERVICE_ENABLED.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "500s")
+ .set(EXECUTOR_INSTANCES.key, "1")
+ .set(DYN_ALLOCATION_INITIAL_EXECUTORS.key, "4")
+ .set(DYN_ALLOCATION_MIN_EXECUTORS.key, "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ }
+
+ override def afterAll(): Unit = {
+ shuffleService.close()
+ }
+
+ private def getLocations(
+ sc: SparkContext,
+ rdd: RDD[_]): Map[BlockId, Map[BlockManagerId, BlockStatus]] = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall { case (_, map) => map.nonEmpty } shouldBe true
+
+ eventually(timeout(Span(5, Seconds)), interval(Span(1, Seconds))) {
+ sc.getExecutorIds().size shouldBe 3
+ getLocations(sc, rdd).forall { case (_, map) => map.nonEmpty } shouldBe true
+ }
+ }
+
+ test("dont fail if a bunch of executors are shut down at once") {
--- End diff --
I think you can probably just use this one test, I don't see much value in splitting it into more tests, and you can even rename this one to "cached data is replicated before dynamic de-allocation".
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r178966943
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future, Promise}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.CacheRecoveryManager.{DoneRecovering, KillReason, Timeout}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, String]()
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures of Unit that will complete once the executor has been killed.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Future[Seq[KillReason]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered: Future[Seq[String]] = checkMem(execIds)
--- End diff --
shouldn't you immediately kill those executors which dont' pass `checkMem`? are they getting killed somewhere else?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163045368
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Seq[Future[Boolean]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ canBeRecovered.foreach { execId => recoveringExecutors.put(execId, true)}
+ canBeRecovered.map { execId =>
+ val killTimer = startKillTimer(execId)
+ replicateUntilDone(execId)
+ .andThen {
+ // don't log if kill timer is done, because it would be redundant
+ case Failure(e) if !killTimer.isDone =>
+ logWarning(s"Failure recovering cached data before executor $execId shutdown", e)
+ }
+ .andThen {
+ case _ =>
+ if (killTimer.cancel(false)) { kill(execId) }
+ }
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @return a Future of Boolean that returns true if a block was copied and false if there were
+ * no more blocks
+ */
+ private def replicateUntilDone(execId: String): Future[Boolean] = {
+ recoverLatestBlock(execId).flatMap { moreBlocks =>
--- End diff --
Block is indented too far.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156421454
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -607,16 +614,17 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
val response = synchronized {
val (knownExecutors, unknownExecutors) = executorIds.partition(executorDataMap.contains)
- unknownExecutors.foreach { id =>
- logWarning(s"Executor to kill $id does not exist!")
- }
+ unknownExecutors.foreach(id => logWarning(s"Executor to kill $id does not exist!"))
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
@squito
I've added a line in ExecutorAllocationManager.validateSettings to ensure that the cached executor timeout is set if cache recovery is enabled. I imagine most people would want to set the regular executor timeout to be a little lower than the cached executor timeout when using cache recovery.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88673/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179201817
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mockito.MockitoSugar
+
+import org.apache.spark.CacheRecoveryManager._
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.network.util.ByteUnit
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers {
+
+ val oneGB: Long = ByteUnit.GiB.toBytes(1).toLong
+
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe DoneRecovering
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem, pauseIndefinitely = true)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe Timeout
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+
+ results.head shouldBe DoneRecovering
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.foreach { _ shouldBe DoneRecovering }
+ bmme.replicated.size shouldBe 2
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("blocks won't replicate if we are stopping all executors") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((2L, 1L)))
+ val bmme = FakeBMM(blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3", "4"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.foreach { _ shouldBe DoneRecovering }
+ bmme.replicated.size shouldBe 0
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+}
+
+private case class FakeBMM(
+ blocks: Iterator[BlockId],
+ memStatus: Map[BlockManagerId, (Long, Long)],
+ sizeOfBlock: Long = 1,
+ pauseIndefinitely: Boolean = false
+ ) extends ThreadSafeRpcEndpoint {
+
+ val rpcEnv: RpcEnv = null
+ val replicated: ConcurrentHashMap[String, AtomicInteger] =
--- End diff --
I didn't understand the meaning of this variable at all on my first read, I had assumed it was blockId -> replicationCount, and was really confused by that. Perhaps renaming to `executorToDrainedBlockCount`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155031844
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -234,12 +239,14 @@ private[spark] class ExecutorAllocationManager(
executor.scheduleWithFixedDelay(scheduleTask, 0, intervalMillis, TimeUnit.MILLISECONDS)
client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
+ cacheRecoveryManager = CacheRecoveryManager(this, conf)
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88635/testReport)** for PR 19041 at commit [`faf2b10`](https://github.com/apache/spark/commit/faf2b10784855c4ec45cbbea57f74bf37595fea2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Thanks @vanzin . I've addressed all the comments. Please let me know if there is anything else you would like me to change.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88673/testReport)** for PR 19041 at commit [`9e8e68c`](https://github.com/apache/spark/commit/9e8e68c606a56227e3c124264efaf4346d04d68a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156420611
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @param killTimer The runnable scheduled to kill this executor. Cancel it if we finish before
+ * it does.
+ */
+ private def replicateUntilDone(execId: String, killTimer: ScheduledFuture[_]): Unit = {
+ recoverLatestBlock(execId).onComplete {
+ case Succ(Some(blockId)) =>
+ logTrace(s"Replicated block $blockId on executor $execId")
+ replicateUntilDone(execId, killTimer)
+ case Succ(None) =>
+ killTimer.cancel(false)
+ kill(execId)
+ case Failure(e) =>
+ logWarning("Failure recovering cached data before executor $execId shutdown", e)
+ killTimer.cancel(false)
+ kill(execId)
+ }
+ }
+
+ /**
+ * Replicate the latest cached rdd block off of this executor on to a surviving executor, and then
+ * remove the block from this executor
+ *
+ * @param execId the executor to recover a block from
+ * @return A future holding the id of the block that was recovered or None if there were no blocks
+ * to recover.
+ */
+ private def recoverLatestBlock(execId: String): Future[Option[RDDBlockId]] = {
+ blockManagerMasterEndpoint
+ .ask[Option[RDDBlockId]](ReplicateLatestRDDBlock(execId, recoveringExecutors.toSeq))
+ .andThen {
+ case Succ(Some(blockId)) =>
+ blockManagerMasterEndpoint.ask[Boolean](RemoveBlockFromExecutor(execId, blockId))
--- End diff --
I think this would be fine, the block removal would always happen after the replicate was done, but I think I'm just going to merge those two messages based on your later comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r139223409
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala ---
@@ -32,8 +32,13 @@ private[spark] object BlockManagerMessages {
// blocks that the master knows about.
case class RemoveBlock(blockId: BlockId) extends ToBlockManagerSlave
+ // Replicate block excluding certain executors for graceful shutdown.
--- End diff --
Removed both comments because the old one isn't accurate anymore.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88603/testReport)** for PR 19041 at commit [`c4e9a80`](https://github.com/apache/spark/commit/c4e9a80b5fba5bcd06c63c71bbfeee080c8580dc).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155381360
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -320,15 +326,16 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
case Some(executorInfo) =>
// This must be synchronized because variables mutated
// in this block are read when requesting executors
- val killed = CoarseGrainedSchedulerBackend.this.synchronized {
+ val removeStatus = CoarseGrainedSchedulerBackend.this.synchronized {
addressToExecutorId -= executorInfo.executorAddress
executorDataMap -= executorId
executorsPendingLossReason -= executorId
- executorsPendingToRemove.remove(executorId).getOrElse(false)
+ executorsPendingToRemove.remove(executorId).getOrElse(Died)
}
totalCoreCount.addAndGet(-executorInfo.totalCores)
totalRegisteredExecutors.addAndGet(-1)
- scheduler.executorLost(executorId, if (killed) ExecutorKilled else reason)
+ scheduler.executorLost(executorId,
+ if (removeStatus == KilledByDriver) ExecutorKilled else reason)
--- End diff --
nit: indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138756115
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
--- End diff --
`Map[String, blah]` is actually more readable than `ExecMap[blah]`, since nobody really knows what an `ExecMap` is without reading the rest of the code. In fact you can just avoid any of this by doing:
private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]()
Which is perfectly fine.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r166520104
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
--- End diff --
I added this cache because I found a bug where there was a slight gap between when CacheRecoveryManager finished recovering an executor and asked it to be removed and when that executor actually stops showing up as a peer in the BlockManagerMaster. During this gap we might try to replicate blocks to that executor and they would get lost.
The simplest way I saw to fix this was to let executors linger in "recoveringExecutors" for a while to cover that gap. I chose a guava cache because it has lightweight time-based eviction. It just cleans up expired cache entries on writes, there isn't a big backing thread or anything doing the cleanup.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163052910
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("CacheRecoveryManager will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually(timeout(Span(1010, Millis)), interval(Span(500, Millis))) {
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val minimumTime = System.currentTimeMillis() + 1000
+
+ eventually(timeout(Span(1500, Millis)), interval(Span(500, Millis))) {
+ // should be killed once not twice
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ // wait at least a second to be sure we don't kill executor twice
+ System.currentTimeMillis() should be > minimumTime
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("Blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(1, blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ eventually {
+ bmme.replicated.size shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("Blocks won't replicate if we are stopping all executors") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((2L, 1L)))
+ val bmme = FakeBMM(1, blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3", "4"))
+
+ eventually {
--- End diff --
Same suggestion about checking the futures instead of this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #87137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87137/testReport)** for PR 19041 at commit [`beb2288`](https://github.com/apache/spark/commit/beb22881f85419b164b83430d74ed0ffc40410bd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84875/testReport)** for PR 19041 at commit [`036fea4`](https://github.com/apache/spark/commit/036fea44be5bbd3ab0d33b11a98ab17962cb91fa).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138757166
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
+
+ private val forceKillAfterS =
+ conf.getTimeAsSeconds("spark.dynamicAllocation.recoverCachedData.timeout", "120s")
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave: ExecMap[mutable.PriorityQueue[RDDBlockId]] = new mutable.HashMap
+ private val savedBlocks: ExecMap[mutable.HashSet[RDDBlockId]] = new mutable.HashMap
+ private val killTimers: ExecMap[ScheduledFuture[_]] = new mutable.HashMap
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ logDebug(s"Get all RDD blocks for executors: ${execIds.mkString(", ")}.")
+ execIds.map { id =>
+ if (isThereEnoughMemory(id)) {
+ updateBlockState(id)
+ val blocksRemain = blocksToSave.get(id).exists(_.nonEmpty)
+ id -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ } else {
+ id -> NotEnoughMemory
+ }
+ }(collection.breakOut)
+ }
+
+ /**
+ * Is there enough memory on the cluster to replicate the cached data on execId before we delete
+ * it. Take into account all the blocks that are currently on track to be deleted.
+ * @param execId the id of the executor we want to delete.
+ * @return true if there is room
+ */
+ private def isThereEnoughMemory(execId: String): Boolean = {
+ val currentMemStatus = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+ val remaining = currentMemStatus - execId -- blocksToSave.keys
+
+ val thisExecBytes = if (!blocksToSave.contains(execId) && currentMemStatus.contains(execId)) {
+ currentMemStatus(execId)._1 - currentMemStatus(execId)._2
+ } else {
+ 0
+ }
+
+ remaining.nonEmpty && everyExecutorHasEnoughMemory(remaining, thisExecBytes)
+ }
+
+ private def everyExecutorHasEnoughMemory(execMem: Map[String, (Long, Long)], bytes: Long) = {
+ val blocksInQueueBytes = blockManagerMasterEndpoint.askSync[Long](GetSizeOfBlocks(blockList))
+ val bytesPerExec = (blocksInQueueBytes + bytes) / execMem.size.toFloat
+ execMem.forall { case (_, (_, remaining)) => remaining - bytesPerExec >= 0 }
+ }
+
+ private def blockList = for {
+ (id, queue) <- blocksToSave.toSeq
+ block <- queue
+ } yield (id, block)
+
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
+ logDebug(s"Replicate block on executor $execId.")
+ blocksToSave.get(execId) match {
+ case Some(p) if p.nonEmpty => doReplication(execId, p.dequeue())
+ case _ => (Future.successful(false), None)
+ }
+ }
+
+ private def doReplication(execId: String,
+ blockId: RDDBlockId): (Future[Boolean], Option[RDDBlockId]) = {
--- End diff --
nit: this is not how we do multi-line arg lists.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r178964472
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -250,6 +255,44 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // As a heuristic, prioritize replicating the latest rdd. If this succeeds,
+ // CacheRecoveryManager will try to replicate the remaining rdds.
--- End diff --
rather than the latest rdd, it would actually make more sense to take advantage of the LRU already in the MemoryStore: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L91
but maybe that is not easy to expose.
But I think that also means that the *receiving* end will put the replicated block at the back of that LinkedHashMap, even though it really hasn't been accessed at all.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155386987
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ test("GracefulShutdown will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1010)
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1100)
+ // should be killed once not twice
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ }
+
+
+ test("Blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(1, blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ Thread.sleep(100)
+ bmme.replicated.size shouldBe 2
+ }
+
+ test("Blocks won't replicate if we are stopping all executors") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((2L, 1L)),
--- End diff --
Indentation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155044070
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -237,6 +246,43 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateOneBlock(
+ execId: String,
+ blockId: BlockId,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"replicating block $blockId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ replicaSet <- blockLocations.asScala.get(blockId)
+ replicas = replicaSet.toSeq
+ maxReps = replicaSet.size + 2
+ } yield info.slaveEndpoint.ask[Boolean](ReplicateBlock(blockId, replicas, excluded, maxReps))
+
+ response.getOrElse(Future.successful(false)).foreach(context.reply)
+ }
+
+ private def getCachedBlocks(executorId: String): collection.Set[BlockId] = {
--- End diff --
removed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r177549827
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.max[RDDBlockId](Ordering.by(_.rddId)))
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179200666
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mockito.MockitoSugar
+
+import org.apache.spark.CacheRecoveryManager._
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.network.util.ByteUnit
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers {
+
+ val oneGB: Long = ByteUnit.GiB.toBytes(1).toLong
+
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe DoneRecovering
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem, pauseIndefinitely = true)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe Timeout
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+
+ results.head shouldBe DoneRecovering
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.foreach { _ shouldBe DoneRecovering }
+ bmme.replicated.size shouldBe 2
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("blocks won't replicate if we are stopping all executors") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((2L, 1L)))
+ val bmme = FakeBMM(blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3", "4"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.foreach { _ shouldBe DoneRecovering }
+ bmme.replicated.size shouldBe 0
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+}
+
+private case class FakeBMM(
+ blocks: Iterator[BlockId],
+ memStatus: Map[BlockManagerId, (Long, Long)],
+ sizeOfBlock: Long = 1,
+ pauseIndefinitely: Boolean = false
+ ) extends ThreadSafeRpcEndpoint {
+
+ val rpcEnv: RpcEnv = null
+ val replicated: ConcurrentHashMap[String, AtomicInteger] =
+ new ConcurrentHashMap[String, AtomicInteger]()
+
+ override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
+ case RecoverLatestRDDBlock(execId, _) =>
+ val future = Future {
+ if (blocks.hasNext) {
--- End diff --
I don't totally understand what you're simulating here with the `FakeBMM`. Which exec is each block stored on? If I understand correctly, you're not trying to tie each block to a particular exec, and just saying whatever the next exec we get, say the next block is on that. If so, I'd at least have a comment indicating that. Though I actually think this would be easier to follow if you tied blocks to specific executors, otherwise when there is a failure its hard to follow exactly what is going on.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88899/testReport)** for PR 19041 at commit [`03ed8a2`](https://github.com/apache/spark/commit/03ed8a2f597a4d42566693a63c1860bd5a68d314).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class ReplicateBlock(`
* ` case class RecoverLatestRDDBlock(executorId: String, excludingExecs: Seq[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163052829
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("CacheRecoveryManager will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually(timeout(Span(1010, Millis)), interval(Span(500, Millis))) {
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val minimumTime = System.currentTimeMillis() + 1000
+
+ eventually(timeout(Span(1500, Millis)), interval(Span(500, Millis))) {
+ // should be killed once not twice
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ // wait at least a second to be sure we don't kill executor twice
+ System.currentTimeMillis() should be > minimumTime
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("Blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(1, blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ eventually {
--- End diff --
Should probably have an explicit timeout here somewhere? Or maybe check the futures returned by the call above instead.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143817781
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -646,18 +648,14 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
Future.successful(true)
}
- val killExecutors: Boolean => Future[Boolean] =
- if (!executorsToKill.isEmpty) {
- _ => doKillExecutors(executorsToKill)
- } else {
- _ => Future.successful(false)
- }
-
- val killResponse = adjustTotalExecutors.flatMap(killExecutors)(ThreadUtils.sameThread)
+ val killResponse = if (executorsToKill.nonEmpty) {
+ adjustTotalExecutors.flatMap(_ => doKillExecutors(executorsToKill))(ThreadUtils.sameThread)
+ } else {
+ Future.successful(false)
+ }
- killResponse.flatMap(killSuccessful =>
- Future.successful (if (killSuccessful) executorsToKill else Seq.empty[String])
- )(ThreadUtils.sameThread)
+ killResponse.map { successful => if (successful) executorsToKill else Seq.empty
--- End diff --
```
killResponse.map { successful =>
if (successful) executorsToKill else Seq.empty
}(ThreadUtils.sameThread)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by erikerlandson <gi...@git.apache.org>.
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/19041
I have been thinking about a different but related [use case](https://github.com/apache-spark-on-k8s/spark/issues/261); supporting the ability to operate in dynamic allocation mode without requiring a separate shuffle service. The motivation is to make it more frictionless for a spark driver to operate using D.A. in a container platform (such as k8s), where standing up the shuffle service adds an extra step that may require cluster admin intervention.
Some of this logic seems to overlap - putting executors into a "draining" state, and attempting to reduce data loss from executor scale-down so that the application doesn't start to thrash.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r166505712
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
--- End diff --
removed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19041
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88603/testReport)** for PR 19041 at commit [`c4e9a80`](https://github.com/apache/spark/commit/c4e9a80b5fba5bcd06c63c71bbfeee080c8580dc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155032171
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -76,6 +76,14 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("3s")
+ private[spark] val DYN_ALLOCATION_RECOVER_CACHE =
+ ConfigBuilder("spark.dynamicAllocation.recoverCachedData").booleanConf.createWithDefault(false)
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143818025
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -698,6 +696,11 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
None
}
}
+
+ override def markPendingToRemove(executorIds: Seq[String]): Unit = synchronized {
+ logDebug(s"marking $executorIds pending to remove")
+ executorIds.foreach(id => executorsPendingToRemove.put(id, true))
--- End diff --
`.foreach { id => ... }`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143819718
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -237,6 +246,43 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateOneBlock(
+ execId: String,
+ blockId: BlockId,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"replicating block $blockId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ replicaSet <- blockLocations.asScala.get(blockId)
+ replicas = replicaSet.toSeq
+ maxReps = replicaSet.size + 2
--- End diff --
Why `+ 2`? Good to add comment explaining.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143817162
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -601,40 +602,41 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
*
* @param executorIds identifiers of executors to kill
* @param replace whether to replace the killed executors with new ones, default false
- * @param force whether to force kill busy executors, default false
+ * @param forceIfBusy whether to force kill busy executors, default false
+ * @param forceIfPending whether to force kill executors that are pending to die, default false
* @return the ids of the executors acknowledged by the cluster manager to be removed.
*/
final override def killExecutors(
executorIds: Seq[String],
replace: Boolean,
- force: Boolean): Seq[String] = {
+ forceIfBusy: Boolean,
+ forceIfPending: Boolean): Seq[String] = {
logInfo(s"Requesting to kill executor(s) ${executorIds.mkString(", ")}")
- val response = synchronized {
+ val response: Future[Seq[String]] = synchronized {
val (knownExecutors, unknownExecutors) = executorIds.partition(executorDataMap.contains)
- unknownExecutors.foreach { id =>
- logWarning(s"Executor to kill $id does not exist!")
- }
+ unknownExecutors.foreach(id => logWarning(s"Executor to kill $id does not exist!"))
// If an executor is already pending to be removed, do not kill it again (SPARK-9795)
// If this executor is busy, do not kill it unless we are told to force kill it (SPARK-9552)
val executorsToKill = knownExecutors
- .filter { id => !executorsPendingToRemove.contains(id) }
- .filter { id => force || !scheduler.isExecutorBusy(id) }
- executorsToKill.foreach { id => executorsPendingToRemove(id) = !replace }
+ .filter { id => forceIfPending || !executorsPendingToRemove.contains(id) }
+ .filter { id => forceIfBusy || !scheduler.isExecutorBusy(id) }
+ executorsToKill.foreach(id => executorsPendingToRemove(id) = !replace)
--- End diff --
Previous style was correct.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163053606
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class CacheRecoveryIntegrationSuite
+ extends SparkFunSuite with Matchers with BeforeAndAfterEach with Eventually {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s") // always
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set(EXECUTOR_MEMORY.key, "512m")
+ .set(SHUFFLE_SERVICE_ENABLED.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "500s")
+ .set(EXECUTOR_INSTANCES.key, "1")
+ .set(DYN_ALLOCATION_INITIAL_EXECUTORS.key, "4")
+ .set(DYN_ALLOCATION_MIN_EXECUTORS.key, "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
--- End diff --
Do you need to start a new shuffle service for each test?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143802485
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
--- End diff --
one more indent level
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143813032
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ *
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
--- End diff --
This method is only called in one place. It may be simpler to just inline it there so you don't have to explain this odd return type.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143807978
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
--- End diff --
nit: `case (_, v)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r157586843
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.executor.memory", "512m")
+ .set("spark.shuffle.service.enabled", "true")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.cacheRecovery.enabled", "true")
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s")
+ .set("spark.executor.instances", "1")
+ .set("spark.dynamicAllocation.initialExecutors", "4")
+ .set("spark.dynamicAllocation.minExecutors", "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ type BInfo = Map[BlockId, Map[BlockManagerId, BlockStatus]]
+ private def getLocations(sc: SparkContext, rdd: RDD[_]): BInfo = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+
+ Thread.sleep(3000)
+ sc.getExecutorIds().size shouldBe 3
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+ }
+
+ test("dont fail if a bunch of executors are shut down at once") {
+ conf.set("spark.dynamicAllocation.minExecutors", "1")
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+
+ Thread.sleep(3000)
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84935/testReport)** for PR 19041 at commit [`036fea4`](https://github.com/apache/spark/commit/036fea44be5bbd3ab0d33b11a98ab17962cb91fa).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #82427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82427/testReport)** for PR 19041 at commit [`95d5eb5`](https://github.com/apache/spark/commit/95d5eb5c792f51903db139972129d24710e2738c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179191524
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mockito.MockitoSugar
+
+import org.apache.spark.CacheRecoveryManager._
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.network.util.ByteUnit
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers {
+
+ val oneGB: Long = ByteUnit.GiB.toBytes(1).toLong
+
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe DoneRecovering
--- End diff --
rather than just checking head, you can check the whole result as easily. and also here, prefer assert over shouldBe
```
assert(results === Seq(DoneRecovering))
```
throughout this file
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r177544111
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Seq[Future[Boolean]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ canBeRecovered.foreach { execId => recoveringExecutors.put(execId, true)}
+ canBeRecovered.map { execId =>
+ val killTimer = startKillTimer(execId)
+ replicateUntilDone(execId)
+ .andThen {
+ // don't log if kill timer is done, because it would be redundant
+ case Failure(e) if !killTimer.isDone =>
+ logWarning(s"Failure recovering cached data before executor $execId shutdown", e)
+ }
+ .andThen {
+ case _ =>
+ if (killTimer.cancel(false)) { kill(execId) }
+ }
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
--- End diff --
made this asynchronous
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179579886
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
--- End diff --
Thanks for the clarification @squito . So my original vision for this feature was that we would just do in memory blocks, and that this would be more lightweight than all blocks, and still useful for spark notebook users who had persisted the results of some long calculation. However, maybe replicating all blocks would be more useful and then I could get rid of that .checkMem function and simplify the code a bit.
Do you have any preference for which approach to take here? I'm open to input.
Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84585/testReport)** for PR 19041 at commit [`6a54b86`](https://github.com/apache/spark/commit/6a54b8652888d5e214b7941773644868091fe5f0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88673/testReport)** for PR 19041 at commit [`9e8e68c`](https://github.com/apache/spark/commit/9e8e68c606a56227e3c124264efaf4346d04d68a).
* This patch **fails PySpark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #87138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87138/testReport)** for PR 19041 at commit [`b3baeb5`](https://github.com/apache/spark/commit/b3baeb5a5a4a59d619eef045e3339227f0f763f0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88588/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155381466
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -607,16 +614,17 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
val response = synchronized {
val (knownExecutors, unknownExecutors) = executorIds.partition(executorDataMap.contains)
- unknownExecutors.foreach { id =>
- logWarning(s"Executor to kill $id does not exist!")
- }
+ unknownExecutors.foreach(id => logWarning(s"Executor to kill $id does not exist!"))
--- End diff --
nit: `.foreach { id =>`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #82389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82389/testReport)** for PR 19041 at commit [`59d593f`](https://github.com/apache/spark/commit/59d593f8109b4de851490f58f03cbcb83ce7b46c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155383283
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -240,6 +248,30 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Option[RDDBlockId]]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.max[RDDBlockId](Ordering.by(_.rddId)))
+ replicaSet <- blockLocations.asScala.get(firstBlock)
+ replicas = replicaSet.toSeq
+ // Add 2 below because you need the number of replicas, plus one for the original, plus one
+ // for the new replica.
+ maxReps = replicaSet.size + 2
+ } yield info.slaveEndpoint
+ .ask[Boolean](ReplicateBlock(firstBlock, replicas, excluded, maxReps))
+ .map( success => if (success) Some(firstBlock) else None)
--- End diff --
`.map { success =>`
Instead of making the driver make an explicit call to `removeBlockFromExecutor`, couldn't you chain that action here, and have a single RPC from the driver do replication + drop the block?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r165490041
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("CacheRecoveryManager will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually(timeout(Span(1010, Millis)), interval(Span(500, Millis))) {
--- End diff --
Also you can skip the `pause` and instead tell your FakeBMM to just stop responding after N blocks -- eg., make it only respond to the first 2 calls to `RecoverLatestRDDBlock` and just ignore the rest. (You're still waiting on the kill timer, though)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #82427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82427/testReport)** for PR 19041 at commit [`95d5eb5`](https://github.com/apache/spark/commit/95d5eb5c792f51903db139972129d24710e2738c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155386205
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
--- End diff --
Don't you need to stop these things to avoid leaking thread pools?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155387894
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.executor.memory", "512m")
+ .set("spark.shuffle.service.enabled", "true")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.cacheRecovery.enabled", "true")
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s")
+ .set("spark.executor.instances", "1")
+ .set("spark.dynamicAllocation.initialExecutors", "4")
+ .set("spark.dynamicAllocation.minExecutors", "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ type BInfo = Map[BlockId, Map[BlockManagerId, BlockStatus]]
+ private def getLocations(sc: SparkContext, rdd: RDD[_]): BInfo = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+
+ Thread.sleep(3000)
--- End diff --
Same comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143816598
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -416,63 +423,52 @@ private[spark] class ExecutorAllocationManager(
* Request the cluster manager to remove the given executors.
* Returns the list of executors which are removed.
*/
- private def removeExecutors(executors: Seq[String]): Seq[String] = synchronized {
- val executorIdsToBeRemoved = new ArrayBuffer[String]
-
+ private def removeExecutors(executors: Seq[String]): Unit = synchronized {
logInfo("Request to remove executorIds: " + executors.mkString(", "))
- val numExistingExecutors = allocationManager.executorIds.size - executorsPendingToRemove.size
-
- var newExecutorTotal = numExistingExecutors
- executors.foreach { executorIdToBeRemoved =>
- if (newExecutorTotal - 1 < minNumExecutors) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (minimum number of executor limit $minNumExecutors)")
- } else if (newExecutorTotal - 1 < numExecutorsTarget) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (number of executor target $numExecutorsTarget)")
- } else if (canBeKilled(executorIdToBeRemoved)) {
- executorIdsToBeRemoved += executorIdToBeRemoved
- newExecutorTotal -= 1
- }
- }
+ val numExistingExecs = allocationManager.executorIds.size - executorsPendingToRemove.size
+ val execCountFloor = math.max(minNumExecutors, numExecutorsTarget)
+ val (executorIdsToBeRemoved, dontRemove) = executors
+ .filter(canBeKilled)
+ .splitAt(numExistingExecs - execCountFloor)
- if (executorIdsToBeRemoved.isEmpty) {
- return Seq.empty[String]
+ dontRemove.foreach { execId =>
+ logDebug(s"Not removing idle executor $execId because it " +
+ s"would put us below the minimum limit of $minNumExecutors executors" +
+ s"or number of target executors $numExecutorsTarget")
}
- // Send a request to the backend to kill this executor(s)
- val executorsRemoved = if (testing) {
- executorIdsToBeRemoved
+ if (executorIdsToBeRemoved.isEmpty) {
+ Seq.empty[String]
+ } else if (testing) {
+ recordExecutorKill(executorIdsToBeRemoved)
+ } else if (recoverCachedData) {
+ logDebug(s"Starting replicate process for $executorIdsToBeRemoved")
+ client.markPendingToRemove(executorIdsToBeRemoved)
+ recordExecutorKill(executorIdsToBeRemoved)
+ cacheRecoveryManager.startExecutorKill(executorIdsToBeRemoved)
} else {
- client.killExecutors(executorIdsToBeRemoved)
+ val killed = killExecutors(executorIdsToBeRemoved)
+ recordExecutorKill(killed)
}
- // [SPARK-21834] killExecutors api reduces the target number of executors.
- // So we need to update the target with desired value.
- client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
- // reset the newExecutorTotal to the existing number of executors
- newExecutorTotal = numExistingExecutors
- if (testing || executorsRemoved.nonEmpty) {
- executorsRemoved.foreach { removedExecutorId =>
- newExecutorTotal -= 1
- logInfo(s"Removing executor $removedExecutorId because it has been idle for " +
- s"$executorIdleTimeoutS seconds (new desired total will be $newExecutorTotal)")
- executorsPendingToRemove.add(removedExecutorId)
- }
- executorsRemoved
- } else {
+ }
+
+ def killExecutors(executorIds: Seq[String], forceIfPending: Boolean = false): Seq[String] = {
+ logDebug(s"Starting kill process for $executorIds")
+ val result = client.killExecutors(executorIds, forceIfPending = forceIfPending)
+ if (result.isEmpty) {
logWarning(s"Unable to reach the cluster manager to kill executor/s " +
- s"${executorIdsToBeRemoved.mkString(",")} or no executor eligible to kill!")
- Seq.empty[String]
+ s"${executorIds.mkString(",")} or no executor eligible to kill!")
}
+ result
}
- /**
- * Request the cluster manager to remove the given executor.
- * Return whether the request is acknowledged.
- */
- private def removeExecutor(executorId: String): Boolean = synchronized {
- val executorsRemoved = removeExecutors(Seq(executorId))
- executorsRemoved.nonEmpty && executorsRemoved(0) == executorId
+ private def recordExecutorKill(executorsRemoved: Seq[String]): Unit = synchronized {
+ // [SPARK-21834] killExecutors api reduces the target number of executors.
+ // So we need to update the target with desired value.
+ client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
+ executorsPendingToRemove ++= executorsRemoved
+ logInfo(s"Removing executor $executorsRemoved because it has been idle for " +
--- End diff --
message looks off when you have multiple executors.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84777 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84777/testReport)** for PR 19041 at commit [`43bda6c`](https://github.com/apache/spark/commit/43bda6c80328de1ad8f5c491fcc00efa965b4509).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19041
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r177222641
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.max[RDDBlockId](Ordering.by(_.rddId)))
+ replicaSet <- blockLocations.asScala.get(firstBlock)
+ // Add 2 below because you need the number of replicas, plus one for the original, plus one
+ // for the new replica.
+ maxReps = replicaSet.size + 2
--- End diff --
Oh, you're right it does already include the original. TBH blockManager.replicate subtracts one from the number i put here and I'm not sure what the logic is behind that. I have updated the comment to reflect that we have to add 2 to get a replica on one new executor.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163045008
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Seq[Future[Boolean]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ canBeRecovered.foreach { execId => recoveringExecutors.put(execId, true)}
+ canBeRecovered.map { execId =>
+ val killTimer = startKillTimer(execId)
+ replicateUntilDone(execId)
+ .andThen {
+ // don't log if kill timer is done, because it would be redundant
+ case Failure(e) if !killTimer.isDone =>
+ logWarning(s"Failure recovering cached data before executor $execId shutdown", e)
+ }
+ .andThen {
+ case _ =>
+ if (killTimer.cancel(false)) { kill(execId) }
+ }
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
--- End diff --
I'm a little worried about this call. It's called from a synchronized block in the executor allocation manager, which means you'll block that object until the RPC finishes.
It seems to be a process-local RPC, so it's faster, but this is still kind of a sketchy thing to do (can it deadlock? can it take a while to finish? etc etc).
Is there an easy way you could make the `startCacheRecovery` method asynchronous?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155385928
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
--- End diff --
nit: indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87137/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88550/testReport)** for PR 19041 at commit [`c79b68f`](https://github.com/apache/spark/commit/c79b68f8b22e5f0137f5c3431dfc1b124bad3d77).
* This patch **fails PySpark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r166517817
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("CacheRecoveryManager will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually(timeout(Span(1010, Millis)), interval(Span(500, Millis))) {
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val minimumTime = System.currentTimeMillis() + 1000
+
+ eventually(timeout(Span(1500, Millis)), interval(Span(500, Millis))) {
+ // should be killed once not twice
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ // wait at least a second to be sure we don't kill executor twice
+ System.currentTimeMillis() should be > minimumTime
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("Blocks won't replicate if we are running out of space") {
--- End diff --
fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r139004286
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
--- End diff --
That does work.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138758727
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
+
+ private val forceKillAfterS =
+ conf.getTimeAsSeconds("spark.dynamicAllocation.recoverCachedData.timeout", "120s")
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave: ExecMap[mutable.PriorityQueue[RDDBlockId]] = new mutable.HashMap
+ private val savedBlocks: ExecMap[mutable.HashSet[RDDBlockId]] = new mutable.HashMap
+ private val killTimers: ExecMap[ScheduledFuture[_]] = new mutable.HashMap
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ logDebug(s"Get all RDD blocks for executors: ${execIds.mkString(", ")}.")
+ execIds.map { id =>
+ if (isThereEnoughMemory(id)) {
+ updateBlockState(id)
+ val blocksRemain = blocksToSave.get(id).exists(_.nonEmpty)
+ id -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ } else {
+ id -> NotEnoughMemory
+ }
+ }(collection.breakOut)
+ }
+
+ /**
+ * Is there enough memory on the cluster to replicate the cached data on execId before we delete
+ * it. Take into account all the blocks that are currently on track to be deleted.
+ * @param execId the id of the executor we want to delete.
+ * @return true if there is room
+ */
+ private def isThereEnoughMemory(execId: String): Boolean = {
+ val currentMemStatus = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+ val remaining = currentMemStatus - execId -- blocksToSave.keys
+
+ val thisExecBytes = if (!blocksToSave.contains(execId) && currentMemStatus.contains(execId)) {
+ currentMemStatus(execId)._1 - currentMemStatus(execId)._2
+ } else {
+ 0
+ }
+
+ remaining.nonEmpty && everyExecutorHasEnoughMemory(remaining, thisExecBytes)
+ }
+
+ private def everyExecutorHasEnoughMemory(execMem: Map[String, (Long, Long)], bytes: Long) = {
+ val blocksInQueueBytes = blockManagerMasterEndpoint.askSync[Long](GetSizeOfBlocks(blockList))
+ val bytesPerExec = (blocksInQueueBytes + bytes) / execMem.size.toFloat
+ execMem.forall { case (_, (_, remaining)) => remaining - bytesPerExec >= 0 }
+ }
+
+ private def blockList = for {
+ (id, queue) <- blocksToSave.toSeq
+ block <- queue
+ } yield (id, block)
+
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
+ logDebug(s"Replicate block on executor $execId.")
+ blocksToSave.get(execId) match {
+ case Some(p) if p.nonEmpty => doReplication(execId, p.dequeue())
+ case _ => (Future.successful(false), None)
+ }
+ }
+
+ private def doReplication(execId: String,
+ blockId: RDDBlockId): (Future[Boolean], Option[RDDBlockId]) = {
+ val savedBlocksForExecutor = savedBlocks.getOrElseUpdate(execId, new mutable.HashSet)
+ savedBlocksForExecutor += blockId
+ val replicateMessage = ReplicateOneBlock(execId, blockId, blocksToSave.keys.toSeq)
+ logTrace(s"Started replicating block $blockId on exec $execId.")
+ val future = blockManagerMasterEndpoint.askSync[Future[Boolean]](replicateMessage)
+ (future, Some(blockId))
+ }
+
+ /**
+ * Ask ExecutorAllocationManager to kill executor and clean up state
+ * Note executorAllocationManger.killExecutors is blocking, this function should be fixed
+ * to use a non blocking version
+ * @param execId the executor to kill
+ */
+ def killExecutor(execId: String): Unit = synchronized {
+ logDebug(s"Send request to kill executor $execId.")
+ killTimers.get(execId).foreach(_.cancel(false))
--- End diff --
`killTimers.remove(execId).foreach { ... }`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88588/testReport)** for PR 19041 at commit [`ca985c7`](https://github.com/apache/spark/commit/ca985c71979c1b71be7a911386e9554d07569e35).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19041
hey @brad-kaiser lemme temper what I said in my previous comments a bit -- I understand what you're doing here now and I think it makes sense, i don't see any serious design issues. But this is adding something new to a pretty core area of spark, so expect some time still on reviews etc. I also think you should probably go through the SPIP process -- though its not huge, I think its better to increase visibility a bit: https://spark.apache.org/improvement-proposals.html
anyway still think this is looking good, but want to set expectations on what is left to do here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163047551
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -123,6 +123,16 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("3s")
+ private[spark] val DYN_ALLOCATION_CACHE_RECOVERY =
+ ConfigBuilder("spark.dynamicAllocation.cacheRecovery.enabled")
--- End diff --
Your PR summary doesn't match these anymore. I'd just remove the actual name of the configs from the summary.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155376516
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
--- End diff --
This is very minor, but because you don't have the full picture here, it could happen that some of the blocks in the "chosen" executors cannot be replicated (e.g. because a block is too large for any of the remaining executors to hold), and you could potentially replicate blocks from these "loser" executors in that case.
Ok not to handle this, but it'd be nice to add a comment (for a future enhancement, maybe).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19041
Thought some more about the race between `RemoveBlock` getting sent back from the executor vs when the `CacheRecoveryManager` tries to replicate the next block -- actually why is there the back-and-forth with the driver for every block? Why isn't there just one message from the `CacheRecoveryManager` to the executor, saying "Drain all RDD blocks" and then one message from the executor back to the driver when its done?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138758096
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
+
+ private val forceKillAfterS =
+ conf.getTimeAsSeconds("spark.dynamicAllocation.recoverCachedData.timeout", "120s")
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave: ExecMap[mutable.PriorityQueue[RDDBlockId]] = new mutable.HashMap
+ private val savedBlocks: ExecMap[mutable.HashSet[RDDBlockId]] = new mutable.HashMap
+ private val killTimers: ExecMap[ScheduledFuture[_]] = new mutable.HashMap
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ logDebug(s"Get all RDD blocks for executors: ${execIds.mkString(", ")}.")
+ execIds.map { id =>
+ if (isThereEnoughMemory(id)) {
--- End diff --
If I understand things correctly, shouldn't you be considering all executors being killed in this call? Otherwise won't you be counting the memory available at executors that soon will be killed?
e.g. if you're killing 5 executors, when you call this for the first one, it seems to me like that would still consider the memory of the other 4 as available.
Feels like you should calculate the available memory before iterating over the executors; that could help with fixing the above issue, and would also be cheaper (single BM call vs. one per executor being killed).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r139004156
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
--- End diff --
I like that better, renamed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r157587520
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
--- End diff --
Renamed to CacheRecoveryIntegrationSuite because I already have CacheReconveryManagerSuite. I would combine them, but they don't go together cleanly because one uses BeforeAndAfterEach
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #85070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85070/testReport)** for PR 19041 at commit [`f637f41`](https://github.com/apache/spark/commit/f637f4164ac341c56c163d9051889b61aea6255b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #82450 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82450/testReport)** for PR 19041 at commit [`a3af4bd`](https://github.com/apache/spark/commit/a3af4bd17a14e1f3f6fe543a44e977841fd72a93).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179206187
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mockito.MockitoSugar
+
+import org.apache.spark.CacheRecoveryManager._
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.network.util.ByteUnit
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers {
+
+ val oneGB: Long = ByteUnit.GiB.toBytes(1).toLong
+
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe DoneRecovering
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem, pauseIndefinitely = true)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe Timeout
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+
+ results.head shouldBe DoneRecovering
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.foreach { _ shouldBe DoneRecovering }
+ bmme.replicated.size shouldBe 2
--- End diff --
to go along with my comment about never killing executors that fail the check for having enough memory, if you add these checks it will fail:
```
assert(results.size === 3)
execs.foreach { exec =>
verify(eam, times(1)).killExecutors(Seq(exec))
}
```
also to go along with my comment below about tying specific blocks to specific executors in FakeBMM, it would let you make more specific checks here, eg. `bmme.replicated.keySet === Set("1", "2")`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r177129258
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class CacheRecoveryIntegrationSuite
+ extends SparkFunSuite with Matchers with BeforeAndAfterEach with Eventually {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s") // always
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set(EXECUTOR_MEMORY.key, "512m")
+ .set(SHUFFLE_SERVICE_ENABLED.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "500s")
+ .set(EXECUTOR_INSTANCES.key, "1")
+ .set(DYN_ALLOCATION_INITIAL_EXECUTORS.key, "4")
+ .set(DYN_ALLOCATION_MIN_EXECUTORS.key, "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ private def getLocations(
+ sc: SparkContext,
+ rdd: RDD[_]): Map[BlockId, Map[BlockManagerId, BlockStatus]] = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (_, map) => map.nonEmpty } shouldBe true
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179183962
--- Diff: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala ---
@@ -449,18 +457,26 @@ class ExecutorAllocationManagerSuite
assert(executorIds(manager).size === 12)
assert(numExecutorsTarget(manager) === 8)
- assert(removeExecutor(manager, "1"))
- assert(removeExecutors(manager, Seq("2", "3", "4")) === Seq("2", "3", "4"))
- assert(!removeExecutor(manager, "5")) // lower limit reached
- assert(!removeExecutor(manager, "6"))
+ removeExecutor(manager, "1")
+ assert(executorsPendingToRemove(manager).contains("1"))
+ removeExecutors(manager, Seq("2", "3", "4"))
+ assert(executorsPendingToRemove(manager).contains("2"))
+ assert(executorsPendingToRemove(manager).contains("3"))
+ assert(executorsPendingToRemove(manager).contains("4"))
+ removeExecutor(manager, "5") // lower limit reached
+ assert(!executorsPendingToRemove(manager).contains("5"))
+ removeExecutor(manager, "6")
+ assert(!executorsPendingToRemove(manager).contains("6"))
+ removeExecutor(manager, "5") // lower limit reached
--- End diff --
I don't think this is supposed to be here twice (also on L466)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155384053
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -240,6 +248,30 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Option[RDDBlockId]]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.max[RDDBlockId](Ordering.by(_.rddId)))
+ replicaSet <- blockLocations.asScala.get(firstBlock)
+ replicas = replicaSet.toSeq
--- End diff --
Merge with previous line.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179206819
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mockito.MockitoSugar
+
+import org.apache.spark.CacheRecoveryManager._
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.network.util.ByteUnit
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers {
+
+ val oneGB: Long = ByteUnit.GiB.toBytes(1).toLong
+
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe DoneRecovering
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem, pauseIndefinitely = true)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe Timeout
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+
+ results.head shouldBe DoneRecovering
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
--- End diff --
you've got the same block 4 times -- while that is possible with replication, probably not what you're trying to test here. Easiest to fix with
```scala
val blocks = (1 to 4).map(RDDBlockId(_, 1))
```
might as well do that in other places too
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84782 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84782/testReport)** for PR 19041 at commit [`47d8eea`](https://github.com/apache/spark/commit/47d8eea0bcc3a68b4df47cb4e6aa8a224a3b0e4a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class RecoverLatestRDDBlock(executorId: String, excludingExecs: Seq[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163042811
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
--- End diff --
> // Cache value never used and must extend Object
Just use `Any` then? Another more common approach is to map the key to itself in these cases.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84777/testReport)** for PR 19041 at commit [`43bda6c`](https://github.com/apache/spark/commit/43bda6c80328de1ad8f5c491fcc00efa965b4509).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r154988634
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
--- End diff --
changed this and renamed Success => Succ
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #82478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82478/testReport)** for PR 19041 at commit [`985874d`](https://github.com/apache/spark/commit/985874da9f72a942d1a28f413167ab3b7fcc64e6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143800553
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
--- End diff --
You never use the return value, why not `Unit`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163046590
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Seq[Future[Boolean]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ canBeRecovered.foreach { execId => recoveringExecutors.put(execId, true)}
+ canBeRecovered.map { execId =>
+ val killTimer = startKillTimer(execId)
+ replicateUntilDone(execId)
+ .andThen {
+ // don't log if kill timer is done, because it would be redundant
+ case Failure(e) if !killTimer.isDone =>
+ logWarning(s"Failure recovering cached data before executor $execId shutdown", e)
+ }
+ .andThen {
+ case _ =>
+ if (killTimer.cancel(false)) { kill(execId) }
+ }
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @return a Future of Boolean that returns true if a block was copied and false if there were
+ * no more blocks
+ */
+ private def replicateUntilDone(execId: String): Future[Boolean] = {
+ recoverLatestBlock(execId).flatMap { moreBlocks =>
+ if (moreBlocks) replicateUntilDone(execId) else Future.successful(false)
+ }
+ }
+
+ /**
+ * Replicate the latest cached rdd block off of this executor on to a surviving executor, and then
+ * remove the block from this executor
+ *
+ * @param execId the executor to recover a block from
+ * @return A future holding the id of the block that was recovered or None if there were no blocks
--- End diff --
Return value doesn't seem to match the code.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82450/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138761246
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -120,6 +124,10 @@ class BlockManagerMasterEndpoint(
removeExecutor(execId)
context.reply(true)
+ case ReplicateOneBlock(execId, blockId, exclude) =>
+ logDebug(s"Replicating first block on $execId")
+ context.reply(replicateOneBlock(execId, blockId, exclude))
--- End diff --
It's weird for an RPC (even if in this case it's actually a local call) to return a `Future`. Why not pass the `context` to `replicateOneBlock` and have it reply with the correct value when it's done?
That would simplify the code for the sender of this message, too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143815711
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -416,63 +423,52 @@ private[spark] class ExecutorAllocationManager(
* Request the cluster manager to remove the given executors.
* Returns the list of executors which are removed.
*/
- private def removeExecutors(executors: Seq[String]): Seq[String] = synchronized {
- val executorIdsToBeRemoved = new ArrayBuffer[String]
-
+ private def removeExecutors(executors: Seq[String]): Unit = synchronized {
logInfo("Request to remove executorIds: " + executors.mkString(", "))
- val numExistingExecutors = allocationManager.executorIds.size - executorsPendingToRemove.size
-
- var newExecutorTotal = numExistingExecutors
- executors.foreach { executorIdToBeRemoved =>
- if (newExecutorTotal - 1 < minNumExecutors) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (minimum number of executor limit $minNumExecutors)")
- } else if (newExecutorTotal - 1 < numExecutorsTarget) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (number of executor target $numExecutorsTarget)")
- } else if (canBeKilled(executorIdToBeRemoved)) {
- executorIdsToBeRemoved += executorIdToBeRemoved
- newExecutorTotal -= 1
- }
- }
+ val numExistingExecs = allocationManager.executorIds.size - executorsPendingToRemove.size
+ val execCountFloor = math.max(minNumExecutors, numExecutorsTarget)
+ val (executorIdsToBeRemoved, dontRemove) = executors
+ .filter(canBeKilled)
+ .splitAt(numExistingExecs - execCountFloor)
- if (executorIdsToBeRemoved.isEmpty) {
- return Seq.empty[String]
+ dontRemove.foreach { execId =>
--- End diff --
minor: add `log.isDebugEnabled` check.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155031927
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -416,63 +423,52 @@ private[spark] class ExecutorAllocationManager(
* Request the cluster manager to remove the given executors.
* Returns the list of executors which are removed.
*/
- private def removeExecutors(executors: Seq[String]): Seq[String] = synchronized {
- val executorIdsToBeRemoved = new ArrayBuffer[String]
-
+ private def removeExecutors(executors: Seq[String]): Unit = synchronized {
logInfo("Request to remove executorIds: " + executors.mkString(", "))
- val numExistingExecutors = allocationManager.executorIds.size - executorsPendingToRemove.size
-
- var newExecutorTotal = numExistingExecutors
- executors.foreach { executorIdToBeRemoved =>
- if (newExecutorTotal - 1 < minNumExecutors) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (minimum number of executor limit $minNumExecutors)")
- } else if (newExecutorTotal - 1 < numExecutorsTarget) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (number of executor target $numExecutorsTarget)")
- } else if (canBeKilled(executorIdToBeRemoved)) {
- executorIdsToBeRemoved += executorIdToBeRemoved
- newExecutorTotal -= 1
- }
- }
+ val numExistingExecs = allocationManager.executorIds.size - executorsPendingToRemove.size
+ val execCountFloor = math.max(minNumExecutors, numExecutorsTarget)
+ val (executorIdsToBeRemoved, dontRemove) = executors
+ .filter(canBeKilled)
+ .splitAt(numExistingExecs - execCountFloor)
- if (executorIdsToBeRemoved.isEmpty) {
- return Seq.empty[String]
+ dontRemove.foreach { execId =>
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155025857
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ *
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19041
It might take some time for me to find time to review this, but in the meantime others who might be interested: @cloud-fan @JoshRosen
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156417274
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
--- End diff --
thats much better! fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87142/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82160/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156744442
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ test("GracefulShutdown will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1010)
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Hey @vanzin, I just wanted to follow up and see if you've had a chance to look at this. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155382339
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -682,6 +689,15 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
}
protected def fetchHadoopDelegationTokens(): Option[Array[Byte]] = { None }
+
+ override def markPendingToRemove(executorIds: Seq[String]): Unit = synchronized {
+ logDebug(s"marking $executorIds pending to remove")
+ executorIds.foreach { id =>
+ if (!executorsPendingToRemove.contains(id)) {
--- End diff --
`getOrElseUpdate`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138754158
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -416,63 +424,52 @@ private[spark] class ExecutorAllocationManager(
* Request the cluster manager to remove the given executors.
* Returns the list of executors which are removed.
*/
- private def removeExecutors(executors: Seq[String]): Seq[String] = synchronized {
- val executorIdsToBeRemoved = new ArrayBuffer[String]
-
+ private def removeExecutors(executors: Seq[String]): Unit = synchronized {
logInfo("Request to remove executorIds: " + executors.mkString(", "))
- val numExistingExecutors = allocationManager.executorIds.size - executorsPendingToRemove.size
-
- var newExecutorTotal = numExistingExecutors
- executors.foreach { executorIdToBeRemoved =>
- if (newExecutorTotal - 1 < minNumExecutors) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (minimum number of executor limit $minNumExecutors)")
- } else if (newExecutorTotal - 1 < numExecutorsTarget) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (number of executor target $numExecutorsTarget)")
- } else if (canBeKilled(executorIdToBeRemoved)) {
- executorIdsToBeRemoved += executorIdToBeRemoved
- newExecutorTotal -= 1
- }
- }
+ val numExistingExecs = allocationManager.executorIds.size - executorsPendingToRemove.size
+ val execCountFloor = Math.max(minNumExecutors, numExecutorsTarget)
--- End diff --
nit: `math.max` is more Scala-y.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #82160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82160/testReport)** for PR 19041 at commit [`d174761`](https://github.com/apache/spark/commit/d1747617ccd3f5a8c5c58d954ba19ec9829e6f01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class GetSizeOfBlocks(blocks: Map[String, Set[RDDBlockId]]) extends ToBlockManagerMaster`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138761473
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala ---
@@ -116,4 +121,11 @@ private[spark] object BlockManagerMessages {
case class BlockManagerHeartbeat(blockManagerId: BlockManagerId) extends ToBlockManagerMaster
case class HasCachedBlocks(executorId: String) extends ToBlockManagerMaster
+
+ case class GetCachedBlocks(executorId: String) extends ToBlockManagerMaster
+
+ case class GetSizeOfBlocks(blocks: Seq[(String, BlockId)]) extends ToBlockManagerMaster
+
+ case class ReplicateOneBlock(executorId: String, blockId: BlockId, exclude: Seq[String])
--- End diff --
`exclude` should at least be consistent with the name of the corresponding argument in `ReplicateBlock`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r176157894
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("CacheRecoveryManager will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually(timeout(Span(1010, Millis)), interval(Span(500, Millis))) {
--- End diff --
I gave the test 3 s to do the killing and made it so the FakeBMM will stop after so many blocks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r165207476
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
--- End diff --
On first read I thought you were _only_ replicating the latest rdd. Maybe expand this to
As a heuristic, prioritize replicating the latest rdd. If this succeeds, CacheRecoveryManager will try to replicate the remaining rdds.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155029623
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ *
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
+ logDebug(s"Replicate block on executor $execId.")
+ blocksToSave.get(execId) match {
+ case Some(p) if p.nonEmpty => doReplication(execId, p.dequeue())
+ case _ => (Future.successful(false), None)
+ }
+ }
+
+ private def doReplication(
+ execId: String,
+ blockId: RDDBlockId): (Future[Boolean], Option[RDDBlockId]) = {
+ val savedBlocksForExecutor = savedBlocks.getOrElseUpdate(execId, new mutable.HashSet)
+ savedBlocksForExecutor += blockId
+ val replicateMessage = ReplicateOneBlock(execId, blockId, blocksToSave.keys.toSeq)
+ logTrace(s"Started replicating block $blockId on exec $execId.")
+ val future = blockManagerMasterEndpoint.ask[Boolean](replicateMessage)
+ (future, Some(blockId))
--- End diff --
this is gone
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #82160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82160/testReport)** for PR 19041 at commit [`d174761`](https://github.com/apache/spark/commit/d1747617ccd3f5a8c5c58d954ba19ec9829e6f01).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r165207798
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.max[RDDBlockId](Ordering.by(_.rddId)))
+ replicaSet <- blockLocations.asScala.get(firstBlock)
+ // Add 2 below because you need the number of replicas, plus one for the original, plus one
+ // for the new replica.
+ maxReps = replicaSet.size + 2
--- End diff --
doesn't blockLocations already include the original?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r174831270
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
--- End diff --
Hey @squito thanks again for the comments, sorry it took me a minute to get back to you.
So, what you describe is basically what I'm doing. I start replicating the latest block and work back to the earliest blocks. If there are any problems I give up. I use the rdd block id as a proxy for the order in which blocks were created.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r178967087
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future, Promise}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.CacheRecoveryManager.{DoneRecovering, KillReason, Timeout}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, String]()
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures of Unit that will complete once the executor has been killed.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Future[Seq[KillReason]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered: Future[Seq[String]] = checkMem(execIds)
+
+ canBeRecovered.flatMap { execIds =>
+ execIds.foreach { execId => recoveringExecutors.put(execId, execId) }
+ Future.sequence(execIds.map { replicateUntilTimeoutThenKill })
+ }
+ }
+
+ def replicateUntilTimeoutThenKill(execId: String): Future[KillReason] = {
+ val timeoutFuture = returnAfterTimeout(Timeout, forceKillAfterS)
+ val replicationFuture = replicateUntilDone(execId)
+
+ Future.firstCompletedOf(List(timeoutFuture, replicationFuture)).andThen {
+ case scala.util.Success(DoneRecovering) =>
+ logTrace(s"Done recovering blocks on $execId, killing now")
+ case scala.util.Success(Timeout) =>
+ logWarning(s"Couldn't recover cache on $execId before $forceKillAfterS second timeout")
+ case Failure(ex) =>
+ logWarning(s"Error recovering cache on $execId", ex)
+ }.andThen {
+ case _ =>
+ kill(execId)
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Future[Seq[String]] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val futureMemStatusByBlockManager =
+ blockManagerMasterEndpoint.ask[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+
+ val futureMemStatusByExecutor = futureMemStatusByBlockManager.map { memStat =>
+ memStat.map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+ }
+
+ futureMemStatusByExecutor.map { memStatusByExecutor =>
+ val (expiringMemStatus, remainingMemStatus) = memStatusByExecutor.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+ }
+
+ /**
+ * Given a value and a timeout in seconds, complete the future with the value when time is up.
+ *
+ * @param value The value to be returned after timeout period
+ * @param seconds the number of seconds to wait
+ * @return a future that will hold the value given after a timeout
+ */
+ private def returnAfterTimeout[T](value: T, seconds: Long): Future[T] = {
+ val p = Promise[T]()
+ val runnable = new Runnable {
+ def run(): Unit = { p.success(value) }
+ }
+ scheduler.schedule(runnable, seconds, TimeUnit.SECONDS)
+ p.future
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @return a Future of Unit that will complete once all blocks have been replicated.
+ */
+ private def replicateUntilDone(execId: String): Future[KillReason] = {
--- End diff --
return type in doc is wrong
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179196510
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mockito.MockitoSugar
+
+import org.apache.spark.CacheRecoveryManager._
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.network.util.ByteUnit
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers {
+
+ val oneGB: Long = ByteUnit.GiB.toBytes(1).toLong
+
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe DoneRecovering
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(blocks.iterator, plentyOfMem, pauseIndefinitely = true)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ try {
+ val future = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val results = ThreadUtils.awaitResult(future, Duration(3, TimeUnit.SECONDS))
+ results.head shouldBe Timeout
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ } finally {
+ cacheRecoveryManager.stop()
+ }
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
--- End diff --
I don't see any meaningful difference between this and "replicate blocks until empty and then kill executor". This isn't actually checking the timer is canceled
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179266769
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -252,6 +257,44 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // As a heuristic, prioritize replicating the latest rdd. If this succeeds,
+ // CacheRecoveryManager will try to replicate the remaining rdds.
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.maxBy(_.rddId))
+ replicaSet <- blockLocations.asScala.get(firstBlock)
+ // Add 2 to force this block to be replicated to one new executor.
+ maxReps = replicaSet.size + 2
--- End diff --
I figured out why you need +2 instead of +1. The existing code wants you to explicitly *remove* id of the blockManager you're trying to replicate from in `replicaSet`. See:
https://github.com/apache/spark/blob/cccaaa14ad775fb981e501452ba2cc06ff5c0f0a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L236-L239
While the existing code is confusing, I definitely don't like using +2 here as a workaround, as it gets pretty confusing. I'd at least update the comments on `BlockManager.replicate()` etc., or maybe just change its behavior and update the callsites.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88716/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r154988069
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
ok @squito , thanks for the heads up, I will start on the SPIP process.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r142478432
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -120,6 +124,10 @@ class BlockManagerMasterEndpoint(
removeExecutor(execId)
context.reply(true)
+ case ReplicateOneBlock(execId, blockId, exclude) =>
+ logDebug(s"Replicating first block on $execId")
+ context.reply(replicateOneBlock(execId, blockId, exclude))
--- End diff --
Fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88634/testReport)** for PR 19041 at commit [`e68ad5d`](https://github.com/apache/spark/commit/e68ad5d5c04381b76ac881dcc1ff21c5cd66f8b6).
* This patch **fails PySpark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163053874
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class CacheRecoveryIntegrationSuite
+ extends SparkFunSuite with Matchers with BeforeAndAfterEach with Eventually {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s") // always
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set(EXECUTOR_MEMORY.key, "512m")
+ .set(SHUFFLE_SERVICE_ENABLED.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "500s")
+ .set(EXECUTOR_INSTANCES.key, "1")
+ .set(DYN_ALLOCATION_INITIAL_EXECUTORS.key, "4")
+ .set(DYN_ALLOCATION_MIN_EXECUTORS.key, "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ private def getLocations(
+ sc: SparkContext,
+ rdd: RDD[_]): Map[BlockId, Map[BlockManagerId, BlockStatus]] = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (_, map) => map.nonEmpty } shouldBe true
--- End diff --
space before `{`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84935/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163054043
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class CacheRecoveryIntegrationSuite
+ extends SparkFunSuite with Matchers with BeforeAndAfterEach with Eventually {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s") // always
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set(EXECUTOR_MEMORY.key, "512m")
+ .set(SHUFFLE_SERVICE_ENABLED.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "500s")
+ .set(EXECUTOR_INSTANCES.key, "1")
+ .set(DYN_ALLOCATION_INITIAL_EXECUTORS.key, "4")
+ .set(DYN_ALLOCATION_MIN_EXECUTORS.key, "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ private def getLocations(
+ sc: SparkContext,
+ rdd: RDD[_]): Map[BlockId, Map[BlockManagerId, BlockStatus]] = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (_, map) => map.nonEmpty } shouldBe true
+
+ eventually(timeout(Span(5, Seconds)), interval(Span(1, Seconds))) {
+ sc.getExecutorIds().size shouldBe 3
+ getLocations(sc, rdd).forall { case (_, map) => map.nonEmpty } shouldBe true
+ }
+ }
+
+ test("dont fail if a bunch of executors are shut down at once") {
+ conf.set("spark.dynamicAllocation.minExecutors", "1")
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (_, map) => map.nonEmpty } shouldBe true
--- End diff --
space before `{`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88899/testReport)** for PR 19041 at commit [`03ed8a2`](https://github.com/apache/spark/commit/03ed8a2f597a4d42566693a63c1860bd5a68d314).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r157586696
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.executor.memory", "512m")
+ .set("spark.shuffle.service.enabled", "true")
+ .set("spark.dynamicAllocation.enabled", "true")
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88588/testReport)** for PR 19041 at commit [`ca985c7`](https://github.com/apache/spark/commit/ca985c71979c1b71be7a911386e9554d07569e35).
* This patch **fails PySpark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179586148
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def recoverLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
--- End diff --
ok cool thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Thats great thanks @squito. I will start addressing these comments now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Hi @squito,
The back and forth communication between CacheRecoveryManager and the BlockManagerMasterEndpoint is so that we always have an up to date view of what executors are undergoing cache recovery and we don't replicate blocks to those executors. If you look at recoverLatestBlock, we include the contents of the recoveringExecutors cache.
We could conceivably move that cache into the block manager master endpoint, but I think that would end up being messier. I wanted to keep all the cache recovery code localized and not clutter up Block Manager Master Endpoint. CacheRecoveryManager and BlockManagerMaster Endpoint will also be local to the same process so rpc calls between them should be cheap, especially compared to the time it will take to copy blocks around.
I will look into the race between removing the block and replicating the next block.
Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88899/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #88635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88635/testReport)** for PR 19041 at commit [`faf2b10`](https://github.com/apache/spark/commit/faf2b10784855c4ec45cbbea57f74bf37595fea2).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138760176
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
--- End diff --
Does this need to be a separate class? Seems like the code could easily live inside `RecoverCacheShutdown`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155382903
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -240,6 +248,30 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Option[RDDBlockId]]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
--- End diff --
`.filter(_.isInstanceOf...)`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on the issue:
https://github.com/apache/spark/pull/19041
Hi @andrewor14 @vanzin @tdas , would you mind taking a look at this change? Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138760634
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ---
@@ -88,6 +89,12 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
@GuardedBy("CoarseGrainedSchedulerBackend.this")
private val executorsPendingToRemove = new HashMap[String, Boolean]
+ // Mark executors that we will request to kill in the near future.
+ // This is different from executors in executorsPendingToRemove, which have already asked to be
+ // killed.
+ @GuardedBy("CoarseGrainedSchedulerBackend.this")
+ private val executorsToBeKilled = mutable.Set.empty[String]
--- End diff --
Instead of introducing yet another state for executors, would it be possible to re-use the `disableExecutor` logic for this? Sounds like that mostly maps to what you want - an executor that is not available for scheduling but still hasn't fully been decommissioned.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155386009
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
--- End diff --
What is `GracefulShutdown`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r139005809
--- Diff: core/src/main/scala/org/apache/spark/RecoverCacheShutdown.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class RecoverCacheShutdown(
+ state: RecoverCacheShutdownState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, NoMoreBlocks) => state.killExecutor(executorId)
+ case (executorId, NotEnoughMemory) => state.killExecutor(executorId)
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case scala.util.Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case scala.util.Success(false) => checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object RecoverCacheShutdown {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): RecoverCacheShutdown = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new RecoverCacheShutdownState(bmme, eam, conf)
+ new RecoverCacheShutdown(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class RecoverCacheShutdownState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ type ExecMap[T] = mutable.Map[String, T]
+
+ private val forceKillAfterS =
+ conf.getTimeAsSeconds("spark.dynamicAllocation.recoverCachedData.timeout", "120s")
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave: ExecMap[mutable.PriorityQueue[RDDBlockId]] = new mutable.HashMap
+ private val savedBlocks: ExecMap[mutable.HashSet[RDDBlockId]] = new mutable.HashMap
+ private val killTimers: ExecMap[ScheduledFuture[_]] = new mutable.HashMap
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ logDebug(s"Get all RDD blocks for executors: ${execIds.mkString(", ")}.")
+ execIds.map { id =>
+ if (isThereEnoughMemory(id)) {
+ updateBlockState(id)
+ val blocksRemain = blocksToSave.get(id).exists(_.nonEmpty)
+ id -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ } else {
+ id -> NotEnoughMemory
+ }
+ }(collection.breakOut)
+ }
+
+ /**
+ * Is there enough memory on the cluster to replicate the cached data on execId before we delete
+ * it. Take into account all the blocks that are currently on track to be deleted.
+ * @param execId the id of the executor we want to delete.
+ * @return true if there is room
+ */
+ private def isThereEnoughMemory(execId: String): Boolean = {
+ val currentMemStatus = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+ val remaining = currentMemStatus - execId -- blocksToSave.keys
+
+ val thisExecBytes = if (!blocksToSave.contains(execId) && currentMemStatus.contains(execId)) {
+ currentMemStatus(execId)._1 - currentMemStatus(execId)._2
+ } else {
+ 0
+ }
+
+ remaining.nonEmpty && everyExecutorHasEnoughMemory(remaining, thisExecBytes)
+ }
+
+ private def everyExecutorHasEnoughMemory(execMem: Map[String, (Long, Long)], bytes: Long) = {
+ val blocksInQueueBytes = blockManagerMasterEndpoint.askSync[Long](GetSizeOfBlocks(blockList))
+ val bytesPerExec = (blocksInQueueBytes + bytes) / execMem.size.toFloat
+ execMem.forall { case (_, (_, remaining)) => remaining - bytesPerExec >= 0 }
+ }
+
+ private def blockList = for {
+ (id, queue) <- blocksToSave.toSeq
+ block <- queue
+ } yield (id, block)
+
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
+ logDebug(s"Replicate block on executor $execId.")
+ blocksToSave.get(execId) match {
+ case Some(p) if p.nonEmpty => doReplication(execId, p.dequeue())
+ case _ => (Future.successful(false), None)
+ }
+ }
+
+ private def doReplication(execId: String,
+ blockId: RDDBlockId): (Future[Boolean], Option[RDDBlockId]) = {
+ val savedBlocksForExecutor = savedBlocks.getOrElseUpdate(execId, new mutable.HashSet)
+ savedBlocksForExecutor += blockId
+ val replicateMessage = ReplicateOneBlock(execId, blockId, blocksToSave.keys.toSeq)
+ logTrace(s"Started replicating block $blockId on exec $execId.")
+ val future = blockManagerMasterEndpoint.askSync[Future[Boolean]](replicateMessage)
+ (future, Some(blockId))
+ }
+
+ /**
+ * Ask ExecutorAllocationManager to kill executor and clean up state
+ * Note executorAllocationManger.killExecutors is blocking, this function should be fixed
+ * to use a non blocking version
+ * @param execId the executor to kill
+ */
+ def killExecutor(execId: String): Unit = synchronized {
+ logDebug(s"Send request to kill executor $execId.")
+ killTimers.get(execId).foreach(_.cancel(false))
--- End diff --
Fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156744473
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ test("GracefulShutdown will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1010)
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1100)
+ // should be killed once not twice
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ }
+
+
+ test("Blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(1, blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ Thread.sleep(100)
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r178967925
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future, Promise}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.CacheRecoveryManager.{DoneRecovering, KillReason, Timeout}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, String]()
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures of Unit that will complete once the executor has been killed.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Future[Seq[KillReason]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered: Future[Seq[String]] = checkMem(execIds)
+
+ canBeRecovered.flatMap { execIds =>
+ execIds.foreach { execId => recoveringExecutors.put(execId, execId) }
+ Future.sequence(execIds.map { replicateUntilTimeoutThenKill })
+ }
+ }
+
+ def replicateUntilTimeoutThenKill(execId: String): Future[KillReason] = {
+ val timeoutFuture = returnAfterTimeout(Timeout, forceKillAfterS)
+ val replicationFuture = replicateUntilDone(execId)
+
+ Future.firstCompletedOf(List(timeoutFuture, replicationFuture)).andThen {
+ case scala.util.Success(DoneRecovering) =>
+ logTrace(s"Done recovering blocks on $execId, killing now")
+ case scala.util.Success(Timeout) =>
+ logWarning(s"Couldn't recover cache on $execId before $forceKillAfterS second timeout")
+ case Failure(ex) =>
+ logWarning(s"Error recovering cache on $execId", ex)
+ }.andThen {
+ case _ =>
+ kill(execId)
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Future[Seq[String]] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val futureMemStatusByBlockManager =
+ blockManagerMasterEndpoint.ask[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+
+ val futureMemStatusByExecutor = futureMemStatusByBlockManager.map { memStat =>
+ memStat.map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+ }
+
+ futureMemStatusByExecutor.map { memStatusByExecutor =>
+ val (expiringMemStatus, remainingMemStatus) = memStatusByExecutor.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+ }
+
+ /**
+ * Given a value and a timeout in seconds, complete the future with the value when time is up.
+ *
+ * @param value The value to be returned after timeout period
+ * @param seconds the number of seconds to wait
+ * @return a future that will hold the value given after a timeout
+ */
+ private def returnAfterTimeout[T](value: T, seconds: Long): Future[T] = {
+ val p = Promise[T]()
+ val runnable = new Runnable {
+ def run(): Unit = { p.success(value) }
+ }
+ scheduler.schedule(runnable, seconds, TimeUnit.SECONDS)
+ p.future
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @return a Future of Unit that will complete once all blocks have been replicated.
+ */
+ private def replicateUntilDone(execId: String): Future[KillReason] = {
+ recoverLatestBlock(execId).flatMap { moreBlocks =>
+ if (moreBlocks) replicateUntilDone(execId) else Future.successful(DoneRecovering)
+ }
+ }
+
+ /**
+ * Ask the BlockManagerMaster to replicate the latest cached rdd block off of this executor on to
+ * a surviving executor, and then remove the block from this executor
+ *
+ * @param execId the executor to recover a block from
+ * @return A future that will hold true if a block was recovered, false otherwise.
+ */
+ private def recoverLatestBlock(execId: String): Future[Boolean] = {
+ blockManagerMasterEndpoint
+ .ask[Boolean](RecoverLatestRDDBlock(execId, recoveringExecutors.asMap.keySet.asScala.toSeq))
--- End diff --
so the reason you do this one block at a time is because you update `recoveringExecutors` between each call, right? To help avoid replicating to another executor which will start draining its blocks after this starts replicating the first block, but before it starts replicating later blocks?
if so, that explanation should go in a comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r176854025
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("CacheRecoveryManager will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1), RDDBlockId(2, 1), RDDBlockId(3, 1), RDDBlockId(4, 1))
+ val bmme = FakeBMM(600, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually(timeout(Span(1010, Millis)), interval(Span(500, Millis))) {
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 1
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("shutdown timer will get cancelled if replication finishes") {
+ val conf = new SparkConf().set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "1s")
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Set(RDDBlockId(1, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ val minimumTime = System.currentTimeMillis() + 1000
+
+ eventually(timeout(Span(1500, Millis)), interval(Span(500, Millis))) {
+ // should be killed once not twice
+ verify(eam, times(1)).killExecutors(Seq("1"))
+ // wait at least a second to be sure we don't kill executor twice
+ System.currentTimeMillis() should be > minimumTime
+ }
+
+ cleanup(result, cacheRecoveryManager)
+ }
+
+ test("Blocks won't replicate if we are running out of space") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1), RDDBlockId(1, 1))
+ val memStatus = Map(BlockManagerId("1", "host", 12, None) -> ((2L, 1L)),
+ BlockManagerId("2", "host", 12, None) -> ((3L, 1L)),
+ BlockManagerId("3", "host", 12, None) -> ((4L, 1L)),
+ BlockManagerId("4", "host", 12, None) -> ((4L, 4L)))
+ val bmme = FakeBMM(1, blocks.iterator, memStatus)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1", "2", "3"))
+ eventually {
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r166507761
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Seq[Future[Boolean]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ canBeRecovered.foreach { execId => recoveringExecutors.put(execId, true)}
+ canBeRecovered.map { execId =>
+ val killTimer = startKillTimer(execId)
+ replicateUntilDone(execId)
+ .andThen {
+ // don't log if kill timer is done, because it would be redundant
+ case Failure(e) if !killTimer.isDone =>
+ logWarning(s"Failure recovering cached data before executor $execId shutdown", e)
+ }
+ .andThen {
+ case _ =>
+ if (killTimer.cancel(false)) { kill(execId) }
+ }
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @return a Future of Boolean that returns true if a block was copied and false if there were
+ * no more blocks
+ */
+ private def replicateUntilDone(execId: String): Future[Boolean] = {
+ recoverLatestBlock(execId).flatMap { moreBlocks =>
+ if (moreBlocks) replicateUntilDone(execId) else Future.successful(false)
+ }
+ }
+
+ /**
+ * Replicate the latest cached rdd block off of this executor on to a surviving executor, and then
--- End diff --
updated comment
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155379914
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -234,12 +239,19 @@ private[spark] class ExecutorAllocationManager(
executor.scheduleWithFixedDelay(scheduleTask, 0, intervalMillis, TimeUnit.MILLISECONDS)
client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
+
+ if(recoverCachedData) {
--- End diff --
nit: space before '('
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179508180
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.TimeUnit
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future, Promise}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.CacheRecoveryManager.{DoneRecovering, KillReason, Timeout}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, String]()
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures of Unit that will complete once the executor has been killed.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Future[Seq[KillReason]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered: Future[Seq[String]] = checkMem(execIds)
+
+ canBeRecovered.flatMap { execIds =>
+ execIds.foreach { execId => recoveringExecutors.put(execId, execId) }
+ Future.sequence(execIds.map { replicateUntilTimeoutThenKill })
+ }
+ }
+
+ def replicateUntilTimeoutThenKill(execId: String): Future[KillReason] = {
+ val timeoutFuture = returnAfterTimeout(Timeout, forceKillAfterS)
+ val replicationFuture = replicateUntilDone(execId)
+
+ Future.firstCompletedOf(List(timeoutFuture, replicationFuture)).andThen {
--- End diff --
I dont' think this will do what you want. Try this code (if you do it in the scala repl, be sure to use )
```scala
import scala.concurrent._
import scala.concurrent.duration._
import java.util.concurrent.{TimeUnit, Executors, ExecutorService}
val scheduler = Executors.newSingleThreadScheduledExecutor()
val pool = Executors.newSingleThreadExecutor()
implicit val asyncExecutionContext = ExecutionContext.fromExecutorService(pool)
def returnAfterTimeout[T](value: T, seconds: Long): Future[T] = {
val p = Promise[T]()
val runnable = new Runnable {
def run(): Unit = { println("time's up"); p.success(value) }
}
scheduler.schedule(runnable, seconds, TimeUnit.SECONDS)
p.future
}
def printStuff(x: Int): String = {
(0 until x).foreach{ i => println(i); Thread.sleep(1000)}
"done"
}
// The *value* of the future is correct here, but you'll notice the timer keeps going, we still see "time's up".
// Not so bad in this case ...
Future.firstCompletedOf(Seq(Future(printStuff(2)), returnAfterTimeout("timeout", 5)))
// The final *value* of the future is correct again here -- we get the timeout. But (a) you'll see that printStuff keeps
// running anyway and (b) the future isn't actually ready until printStuff completes, even though it should be ready
// as soon as the timer is up
Future.firstCompletedOf(Seq(Future(printStuff(10)), returnAfterTimeout("timeout", 1)))
// Slightly better, the TimeoutException is thrown as soon as the timer is up, but printStuff keeps running anyway
Await.result(Future(printStuff(10)), 1 second)
```
So I'd change this to use ThreadUtils.awaitResult *and* also you need your timer to set some condition which the recovery thread is checking, so it knows to stop trying to replicate more blocks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156518100
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
--- End diff --
Also this should be `CacheRecoveryManagerSuite`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143811908
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
--- End diff --
`: Unit =`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r175124143
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
--- End diff --
That's a good point. I was really concerned with not hurting the stability of the cluster. So in the case where we are running out of cluster memory, I thought it would be safer to fail fast and give up on replication.
Your right it would probably be more useful to get as many complete RDDs recovered as possible, and this method is complicated. I can test taking out this check and see what happens.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163055034
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class CacheRecoveryIntegrationSuite
--- End diff --
I'm not too sure about what this suite is testing. You seem to be relying on the very short idle timeout to have the executors killed; but it doesn't seem like there a strong relation between your test code and which / how many executors will be killed as idle.
Instead, shouldn't you explicitly be killing executors to verify that the data is being replicated?
Or at least explain in a comment how this test is actually testing the functionality being added.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r163045943
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ * @return a sequence of futures representing the kill process.
+ */
+ def startCacheRecovery(execIds: Seq[String]): Seq[Future[Boolean]] = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ canBeRecovered.foreach { execId => recoveringExecutors.put(execId, true)}
+ canBeRecovered.map { execId =>
+ val killTimer = startKillTimer(execId)
+ replicateUntilDone(execId)
+ .andThen {
+ // don't log if kill timer is done, because it would be redundant
+ case Failure(e) if !killTimer.isDone =>
+ logWarning(s"Failure recovering cached data before executor $execId shutdown", e)
+ }
+ .andThen {
+ case _ =>
+ if (killTimer.cancel(false)) { kill(execId) }
+ }
+ }
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * This is a best guess implementation and it is not guaranteed that all returned executors
+ * will succeed. For example a block might be too big to fit on any one specific executor.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @return a Future of Boolean that returns true if a block was copied and false if there were
--- End diff --
The return value description sounds weird (two different unrelated things for true and false).
Maybe just: A future that returns whether there are more blocks to replicate.
Although you don't really use that return value outside of this method. So you could just return a `Future[Unit]` here, and handle the boolean internally.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r166505845
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import com.google.common.cache.CacheBuilder
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors = CacheBuilder.newBuilder()
+ .expireAfterWrite(forceKillAfterS * 2, TimeUnit.SECONDS)
+ .build[String, java.lang.Boolean]() // Cache value never used and must extend Object
--- End diff --
Good idea about making the key map to itself. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84782/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156744333
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ test("GracefulShutdown will kill executor if it takes too long to replicate") {
+ val conf = new SparkConf().set("spark.dynamicAllocation.cacheRecovery.timeout", "1s")
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r165482549
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -420,63 +432,53 @@ private[spark] class ExecutorAllocationManager(
* Request the cluster manager to remove the given executors.
* Returns the list of executors which are removed.
*/
- private def removeExecutors(executors: Seq[String]): Seq[String] = synchronized {
- val executorIdsToBeRemoved = new ArrayBuffer[String]
-
+ private def removeExecutors(executors: Seq[String]): Unit = synchronized {
--- End diff --
return in the doc is wrong
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84489/testReport)** for PR 19041 at commit [`fc84aa7`](https://github.com/apache/spark/commit/fc84aa7b3709202c19f961306d9f3ffb0b7764e4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156458851
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -240,6 +248,30 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateLatestRDDBlock(
+ execId: String,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"Replicating first cached block on $execId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Option[RDDBlockId]]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ blocks = info.cachedBlocks.collect { case r: RDDBlockId => r }
+ // we assume blocks from the latest rdd are most relevant
+ firstBlock <- if (blocks.isEmpty) None else Some(blocks.max[RDDBlockId](Ordering.by(_.rddId)))
+ replicaSet <- blockLocations.asScala.get(firstBlock)
+ replicas = replicaSet.toSeq
+ // Add 2 below because you need the number of replicas, plus one for the original, plus one
+ // for the new replica.
+ maxReps = replicaSet.size + 2
+ } yield info.slaveEndpoint
+ .ask[Boolean](ReplicateBlock(firstBlock, replicas, excluded, maxReps))
+ .map( success => if (success) Some(firstBlock) else None)
--- End diff --
good point, I've merged replicateLatestRDDBlock and removeBlockFromExecutor
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r154988373
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
--- End diff --
This method is gone now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155044004
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -237,6 +246,43 @@ class BlockManagerMasterEndpoint(
blockManagerIdByExecutor.get(execId).foreach(removeBlockManager)
}
+ private def replicateOneBlock(
+ execId: String,
+ blockId: BlockId,
+ excludeExecutors: Seq[String],
+ context: RpcCallContext): Unit = {
+ logDebug(s"replicating block $blockId")
+ val excluded = excludeExecutors.flatMap(blockManagerIdByExecutor.get)
+ val response: Option[Future[Boolean]] = for {
+ blockManagerId <- blockManagerIdByExecutor.get(execId)
+ info <- blockManagerInfo.get(blockManagerId)
+ replicaSet <- blockLocations.asScala.get(blockId)
+ replicas = replicaSet.toSeq
+ maxReps = replicaSet.size + 2
--- End diff --
commented
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155029739
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
+
+ /**
+ * Ask block manager master to replicate one cached block.
+ *
+ * @param execId the executor to replicate a block from
+ * @return false if there are no more blocks or if there is an issue
+ */
+ def replicateFirstBlock(execId: String): (Future[Boolean], Option[RDDBlockId]) = synchronized {
+ logDebug(s"Replicate block on executor $execId.")
+ blocksToSave.get(execId) match {
+ case Some(p) if p.nonEmpty => doReplication(execId, p.dequeue())
+ case _ => (Future.successful(false), None)
+ }
+ }
+
+ private def doReplication(
+ execId: String,
+ blockId: RDDBlockId): (Future[Boolean], Option[RDDBlockId]) = {
+ val savedBlocksForExecutor = savedBlocks.getOrElseUpdate(execId, new mutable.HashSet)
+ savedBlocksForExecutor += blockId
+ val replicateMessage = ReplicateOneBlock(execId, blockId, blocksToSave.keys.toSeq)
+ logTrace(s"Started replicating block $blockId on exec $execId.")
+ val future = blockManagerMasterEndpoint.ask[Boolean](replicateMessage)
+ (future, Some(blockId))
+ }
+
+ /**
+ * Ask ExecutorAllocationManager to kill executor and clean up state
+ * Note executorAllocationManger.killExecutors is blocking, this function should be fixed
+ * to use a non blocking version
+ * @param execId the executor to kill
--- End diff --
this is gone
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143816714
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -76,6 +76,14 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("3s")
+ private[spark] val DYN_ALLOCATION_RECOVER_CACHE =
--- End diff --
as with other suggestion, `CACHE_RECOVERY`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r166508110
--- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -123,6 +123,16 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("3s")
+ private[spark] val DYN_ALLOCATION_CACHE_RECOVERY =
+ ConfigBuilder("spark.dynamicAllocation.cacheRecovery.enabled")
--- End diff --
Removed reference to the configs in the PR summary and JIRA ticket.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19041
**[Test build #84585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84585/testReport)** for PR 19041 at commit [`6a54b86`](https://github.com/apache/spark/commit/6a54b8652888d5e214b7941773644868091fe5f0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155043964
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ---
@@ -97,6 +97,10 @@ class BlockManagerMasterEndpoint(
case GetStorageStatus =>
context.reply(storageStatus)
+ case GetCachedBlocks(executorId) => context.reply(getCachedBlocks(executorId))
--- End diff --
this is removed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155378634
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @param killTimer The runnable scheduled to kill this executor. Cancel it if we finish before
+ * it does.
+ */
+ private def replicateUntilDone(execId: String, killTimer: ScheduledFuture[_]): Unit = {
+ recoverLatestBlock(execId).onComplete {
+ case Succ(Some(blockId)) =>
+ logTrace(s"Replicated block $blockId on executor $execId")
+ replicateUntilDone(execId, killTimer)
+ case Succ(None) =>
+ killTimer.cancel(false)
+ kill(execId)
+ case Failure(e) =>
+ logWarning("Failure recovering cached data before executor $execId shutdown", e)
+ killTimer.cancel(false)
+ kill(execId)
+ }
+ }
+
+ /**
+ * Replicate the latest cached rdd block off of this executor on to a surviving executor, and then
+ * remove the block from this executor
+ *
+ * @param execId the executor to recover a block from
+ * @return A future holding the id of the block that was recovered or None if there were no blocks
+ * to recover.
+ */
+ private def recoverLatestBlock(execId: String): Future[Option[RDDBlockId]] = {
+ blockManagerMasterEndpoint
+ .ask[Option[RDDBlockId]](ReplicateLatestRDDBlock(execId, recoveringExecutors.toSeq))
+ .andThen {
+ case Succ(Some(blockId)) =>
+ blockManagerMasterEndpoint.ask[Boolean](RemoveBlockFromExecutor(execId, blockId))
+ case _ => // do nothing
+ }
+ }
+
+ /**
+ * Remove the executor from the list of currently recovering executors and then kill it.
+ *
+ * @param execId the id of the executor to be killed
+ */
+ private def kill(execId: String): Unit = {
+ recoveringExecutors.remove(execId)
+ executorAllocationManager.killExecutors(Seq(execId))
--- End diff --
You can avoid this call if the executor has already been removed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155023851
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
+ blocksToSave(execId) = mutable.PriorityQueue[RDDBlockId](blocks.toSeq: _*)(Ordering.by(_.rddId))
+ killTimers.getOrElseUpdate(execId, scheduleKill(new KillRunner(execId)))
+ }
+
+ class KillRunner(execId: String) extends Runnable { def run(): Unit = killExecutor(execId) }
+ private def scheduleKill(k: KillRunner) = scheduler.schedule(k, forceKillAfterS, TimeUnit.SECONDS)
--- End diff --
removed this class
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156744222
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
--- End diff --
Replaced all of these with eventually
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143803799
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
--- End diff --
"...enough memory in the remaining executors to hold the cached blocks on the given executors."
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143811393
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
+ }
+
+ /**
+ * Gets a list of cached blocks from the block manager master and updates.
+ * CacheRecoveryManagerState.
+ *
+ * @param execId an executor Id
+ */
+ private def updateBlockState(execId: String): Unit = {
+ val blocks: mutable.Set[RDDBlockId] = blockManagerMasterEndpoint
+ .askSync[collection.Set[BlockId]](GetCachedBlocks(execId))
+ .flatMap(_.asRDDId)(collection.breakOut)
+
+ blocks --= savedBlocks.getOrElse(execId, new mutable.HashSet)
--- End diff --
```
savedBlocks.get(execId).foreach { s => blocks --= s }
```
Or something.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143799387
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
--- End diff --
Same thing as before with the weird name. "cache-recovery-manager-pool" is a much better name.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155386486
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.mock.MockitoSugar
+
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+
+class CacheRecoveryManagerSuite extends SparkFunSuite with MockitoSugar with Matchers {
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("GracefulShutdown will take blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+
+ cacheRecoveryManager.startCacheRecovery(Seq("1"))
+ Thread.sleep(1000)
+ verify(eam).killExecutors(Seq("1"))
+
+
--- End diff --
nit: too many empty lines
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r155387980
--- Diff: core/src/test/scala/org/apache/spark/scheduler/RecoverCachedDataSuite.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class RecoverCachedDataSuite extends SparkFunSuite with Matchers with BeforeAndAfterEach {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.executor.memory", "512m")
+ .set("spark.shuffle.service.enabled", "true")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.cacheRecovery.enabled", "true")
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s")
+ .set("spark.executor.instances", "1")
+ .set("spark.dynamicAllocation.initialExecutors", "4")
+ .set("spark.dynamicAllocation.minExecutors", "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ type BInfo = Map[BlockId, Map[BlockManagerId, BlockStatus]]
+ private def getLocations(sc: SparkContext, rdd: RDD[_]): BInfo = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+
+ Thread.sleep(3000)
+ sc.getExecutorIds().size shouldBe 3
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+ }
+
+ test("dont fail if a bunch of executors are shut down at once") {
+ conf.set("spark.dynamicAllocation.minExecutors", "1")
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+
+ Thread.sleep(3000)
+ sc.getExecutorIds().size shouldBe 1
+ getLocations(sc, rdd).forall{ case (id, map) => map.nonEmpty } shouldBe true
+ }
+
+ test("Executors should not accept new work while replicating away data before deallocation") {
+ conf.set("spark.dynamicAllocation.minExecutors", "1")
+
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 100000, 4).map(_ * 4L).cache() // cache on all 4 executors
+ rdd.reduce(_ + _) shouldBe 20000200000L // realize the cache
+
+ Thread.sleep(1102) // sleep long enough to trigger deallocation
--- End diff --
Same comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143810600
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
+ case ((_, remainingMem), (execId, mem)) => (execId, remainingMem - mem)
+ }.drop(1)
+ .toMap
+ .mapValues(freeMem => freeMem >= 0)
--- End diff --
This should be `> 0` right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r143810176
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ScheduledFuture, TimeUnit}
+
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages.{GetCachedBlocks, GetMemoryStatus, GetSizeOfBlocks, ReplicateOneBlock}
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executors cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ state: CacheRecoveryManagerState,
+ conf: SparkConf)
+ extends Logging {
+
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("recover-cache-shutdown-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startExecutorKill(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ checkForReplicableBlocks(execIds)
+ }
+
+ /**
+ * Stops all thread pools
+ *
+ * @return
+ */
+ def stop(): java.util.List[Runnable] = {
+ threadPool.shutdownNow()
+ state.stop()
+ }
+
+ /**
+ * Get list of cached blocks from BlockManagerMaster. If there are cached blocks, replicate them,
+ * otherwise kill the executors
+ *
+ * @param execIds the executors to check
+ */
+ private def checkForReplicableBlocks(execIds: Seq[String]) = state.getBlocks(execIds).foreach {
+ case (executorId, HasCachedBlocks) => replicateBlocks(executorId)
+ case (executorId, NoMoreBlocks | NotEnoughMemory) => state.killExecutor(executorId)
+ }
+
+ /**
+ * Replicate one cached block on an executor. If there are more, repeat. If there are none, check
+ * with the block manager master again. If there is an error, go ahead and kill executor.
+ *
+ * @param execId the executor to save a block one
+ */
+ private def replicateBlocks(execId: String): Unit = {
+ import scala.util.Success
+ val (response, blockId) = state.replicateFirstBlock(execId)
+ response.onComplete {
+ case Success(true) =>
+ logTrace(s"Finished replicating block ${blockId.getOrElse("unknown")} on exec $execId.")
+ replicateBlocks(execId)
+ case Success(false) =>
+ checkForReplicableBlocks(Seq(execId))
+ case Failure(f) =>
+ logWarning(s"Error trying to replicate block ${blockId.getOrElse("unknown")}.", f)
+ state.killExecutor(execId)
+ }
+ }
+}
+
+private object CacheRecoveryManager {
+ def apply(eam: ExecutorAllocationManager, conf: SparkConf): CacheRecoveryManager = {
+ val bmme = SparkEnv.get.blockManager.master.driverEndpoint
+ val state = new CacheRecoveryManagerState(bmme, eam, conf)
+ new CacheRecoveryManager(state, conf)
+ }
+}
+
+/**
+ * Private class that holds state for all the executors being shutdown.
+ *
+ * @param blockManagerMasterEndpoint blockManagerMasterEndpoint
+ * @param executorAllocationManager ExecutorAllocationManager
+ * @param conf spark conf
+ */
+final private class CacheRecoveryManagerState(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf
+ ) extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_RECOVER_CACHE_TIMEOUT)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("recover-cache-shutdown-timers")
+
+ private val blocksToSave = new mutable.HashMap[String, mutable.PriorityQueue[RDDBlockId]]
+ private val savedBlocks = new mutable.HashMap[String, mutable.HashSet[RDDBlockId]]
+ private val killTimers = new mutable.HashMap[String, ScheduledFuture[_]]
+
+ /**
+ * Query Block Manager Master for cached blocks.
+ *
+ * @param execIds the executors to query
+ * @return a map of executorId to its replication state.
+ */
+ def getBlocks(execIds: Seq[String]): Map[String, ExecutorReplicationState] = synchronized {
+ val isThereEnoughMemory = checkFreeMem(execIds)
+
+ isThereEnoughMemory.map {
+ case (execId, true) =>
+ updateBlockState(execId)
+ val blocksRemain = blocksToSave.get(execId).exists(_.nonEmpty)
+ execId -> (if (blocksRemain) HasCachedBlocks else NoMoreBlocks)
+ case (execId, false) =>
+ execId -> NotEnoughMemory
+ }
+ }
+
+ /**
+ * Checks to see if there is enough memory on the cluster to hold the cached blocks on these
+ * executors. Checks on a per executor basis from the smallest to the largest.
+ *
+ * @param execIds the executors to query
+ * @return a map of executor ids to whether or not there is enough free memory for them.
+ */
+ private def checkFreeMem(execIds: Seq[String]): Map[String, Boolean] = {
+ val execIdsToShutDown = execIds.toSet
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (k, v) => execIdsToShutDown.contains(k)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ val alreadyReplicatedBlocks: Map[String, Set[RDDBlockId]] =
+ savedBlocks.filterKeys(execIdsToShutDown).map { case (k, v) => k -> v.toSet }.toMap
+
+ val getSizes = GetSizeOfBlocks(alreadyReplicatedBlocks)
+ val alreadyReplicatedBytes = blockManagerMasterEndpoint.askSync[Map[String, Long]](getSizes)
+
+ val bytesToReplicate: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val alreadyReplicated = alreadyReplicatedBytes.getOrElse(execId, 0L)
+ val usedMem = maxMem - remainingMem
+ val toBeReplicatedMem = usedMem - alreadyReplicated
+ execId -> toBeReplicatedMem
+ }.toSeq.sortBy { case (k, v) => v }
+
+ bytesToReplicate.scan(("start", freeMemOnRemaining)) {
--- End diff --
I prefer to indent these a little bit differently:
```
bytesToReplicate
.scan { ... =>
}
.drop(1)
...
```
Also, probably not a big issue here, but I'd rather avoid `mapValues`. It causes stack overflows when you have lots of elements, so better to just avoid it and use `map` instead.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r156417594
--- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, ScheduledFuture, TimeUnit}
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+import scala.concurrent.{ExecutionContext, Future}
+import scala.util.{Success => Succ}
+import scala.util.Failure
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.storage.{BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Responsible for asynchronously replicating all of an executor's cached blocks, and then shutting
+ * it down.
+ */
+final private class CacheRecoveryManager(
+ blockManagerMasterEndpoint: RpcEndpointRef,
+ executorAllocationManager: ExecutorAllocationManager,
+ conf: SparkConf)
+ extends Logging {
+
+ private val forceKillAfterS = conf.get(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT)
+ private val threadPool = ThreadUtils.newDaemonCachedThreadPool("cache-recovery-manager-pool")
+ private implicit val asyncExecutionContext: ExecutionContext =
+ ExecutionContext.fromExecutorService(threadPool)
+ private val scheduler =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("cache-recovery-shutdown-timers")
+ private val recoveringExecutors: mutable.Set[String] =
+ ConcurrentHashMap.newKeySet[String]().asScala
+
+ /**
+ * Start the recover cache shutdown process for these executors
+ *
+ * @param execIds the executors to start shutting down
+ */
+ def startCacheRecovery(execIds: Seq[String]): Unit = {
+ logDebug(s"Recover cached data before shutting down executors ${execIds.mkString(", ")}.")
+ val canBeRecovered = checkMem(execIds)
+ recoveringExecutors ++= canBeRecovered
+ val executorsWithKillTimers = canBeRecovered.zip(canBeRecovered.map(startKillTimer))
+ executorsWithKillTimers.foreach((replicateUntilDone _).tupled)
+ }
+
+ /**
+ * Given a list of executors that will be shut down, check if there is enough free memory on the
+ * rest of the cluster to hold their data. Return a list of just the executors for which there
+ * will be enough space. Executors are included smallest first.
+ *
+ * @param execIds executors which will be shut down
+ * @return a Seq of the executors we do have room for
+ */
+ private def checkMem(execIds: Seq[String]): Seq[String] = {
+ val execsToShutDown = execIds.toSet
+ // Memory Status is a map of executor Id to a tuple of Max Memory and remaining memory on that
+ // executor.
+ val allExecMemStatus: Map[String, (Long, Long)] = blockManagerMasterEndpoint
+ .askSync[Map[BlockManagerId, (Long, Long)]](GetMemoryStatus)
+ .map { case (blockManagerId, mem) => blockManagerId.executorId -> mem }
+
+ val (expiringMemStatus, remainingMemStatus) = allExecMemStatus.partition {
+ case (execId, _) => execsToShutDown.contains(execId)
+ }
+ val freeMemOnRemaining = remainingMemStatus.values.map(_._2).sum
+
+ // The used mem on each executor sorted from least used mem to greatest
+ val executorAndUsedMem: Seq[(String, Long)] =
+ expiringMemStatus.map { case (execId, (maxMem, remainingMem)) =>
+ val usedMem = maxMem - remainingMem
+ execId -> usedMem
+ }.toSeq.sortBy { case (_, usedMem) => usedMem }
+
+ executorAndUsedMem
+ .scan(("start", freeMemOnRemaining)) {
+ case ((_, freeMem), (execId, usedMem)) => (execId, freeMem - usedMem)
+ }
+ .drop(1)
+ .filter { case (_, freeMem) => freeMem > 0 }
+ .map(_._1)
+ }
+
+ /**
+ * Given an executor id, start a timer that will kill the given executor after the configured
+ * timeout
+ *
+ * @param execId The id of the executor to be killed
+ * @return a future representing the timer
+ */
+ private def startKillTimer(execId: String): ScheduledFuture[_] = {
+ val killer = new Runnable {
+ def run(): Unit = {
+ logDebug(s"Killing $execId because timeout for recovering cached data has expired")
+ kill(execId)
+ }
+ }
+ scheduler.schedule(killer, forceKillAfterS, TimeUnit.SECONDS)
+ }
+
+ /**
+ * Recover cached RDD blocks off of an executor until there are no more, or until
+ * there is an error
+ *
+ * @param execId the id of the executor to be killed
+ * @param killTimer The runnable scheduled to kill this executor. Cancel it if we finish before
+ * it does.
+ */
+ private def replicateUntilDone(execId: String, killTimer: ScheduledFuture[_]): Unit = {
+ recoverLatestBlock(execId).onComplete {
+ case Succ(Some(blockId)) =>
+ logTrace(s"Replicated block $blockId on executor $execId")
+ replicateUntilDone(execId, killTimer)
+ case Succ(None) =>
+ killTimer.cancel(false)
+ kill(execId)
+ case Failure(e) =>
+ logWarning("Failure recovering cached data before executor $execId shutdown", e)
+ killTimer.cancel(false)
+ kill(execId)
--- End diff --
good idea, fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82427/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r176157509
--- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala ---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.concurrent.{Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration.Duration
+import scala.reflect.ClassTag
+
+import org.mockito.Mockito._
+import org.scalatest.Matchers
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mockito.MockitoSugar
+import org.scalatest.time.{Millis, Span}
+
+import org.apache.spark.internal.config.DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT
+import org.apache.spark.rpc._
+import org.apache.spark.storage.{BlockId, BlockManagerId, RDDBlockId}
+import org.apache.spark.storage.BlockManagerMessages._
+import org.apache.spark.util.ThreadUtils
+
+class CacheRecoveryManagerSuite
+ extends SparkFunSuite with MockitoSugar with Matchers with Eventually {
+
+ val oneGB: Long = 1024L * 1024L * 1024L * 1024L
+ val plentyOfMem = Map(
+ BlockManagerId("1", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("2", "host", 12, None) -> ((oneGB, oneGB)),
+ BlockManagerId("3", "host", 12, None) -> ((oneGB, oneGB)))
+
+ test("CacheRecoveryManager will replicate blocks until empty and then kill executor") {
+ val conf = new SparkConf()
+ val eam = mock[ExecutorAllocationManager]
+ val blocks = Seq(RDDBlockId(1, 1), RDDBlockId(2, 1))
+ val bmme = FakeBMM(1, blocks.iterator, plentyOfMem)
+ val bmmeRef = DummyRef(bmme)
+ val cacheRecoveryManager = new CacheRecoveryManager(bmmeRef, eam, conf)
+ when(eam.killExecutors(Seq("1"))).thenReturn(Seq("1"))
+ val result = cacheRecoveryManager.startCacheRecovery(Seq("1"))
+
+ eventually {
+ verify(eam).killExecutors(Seq("1"))
+ bmme.replicated.get("1").get shouldBe 2
+ }
+
+ cleanup(result, cacheRecoveryManager)
--- End diff --
I rearranged the code here to put cacheRecoveryManager.stop() in a finally block
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r177129287
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.server.TransportServer
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+class CacheRecoveryIntegrationSuite
+ extends SparkFunSuite with Matchers with BeforeAndAfterEach with Eventually {
+
+ var shuffleService: TransportServer = _
+ var conf: SparkConf = _
+ var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s") // always
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set(EXECUTOR_MEMORY.key, "512m")
+ .set(SHUFFLE_SERVICE_ENABLED.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "500s")
+ .set(EXECUTOR_INSTANCES.key, "1")
+ .set(DYN_ALLOCATION_INITIAL_EXECUTORS.key, "4")
+ .set(DYN_ALLOCATION_MIN_EXECUTORS.key, "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ val transportContext = new TransportContext(transportConf, rpcHandler)
+ shuffleService = transportContext.createServer()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ shuffleService.close()
+ }
+
+ private def getLocations(
+ sc: SparkContext,
+ rdd: RDD[_]): Map[BlockId, Map[BlockManagerId, BlockStatus]] = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (_, map) => map.nonEmpty } shouldBe true
+
+ eventually(timeout(Span(5, Seconds)), interval(Span(1, Seconds))) {
+ sc.getExecutorIds().size shouldBe 3
+ getLocations(sc, rdd).forall { case (_, map) => map.nonEmpty } shouldBe true
+ }
+ }
+
+ test("dont fail if a bunch of executors are shut down at once") {
+ conf.set("spark.dynamicAllocation.minExecutors", "1")
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall{ case (_, map) => map.nonEmpty } shouldBe true
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88471/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by brad-kaiser <gi...@git.apache.org>.
Github user brad-kaiser commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r138935220
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -416,63 +424,52 @@ private[spark] class ExecutorAllocationManager(
* Request the cluster manager to remove the given executors.
* Returns the list of executors which are removed.
*/
- private def removeExecutors(executors: Seq[String]): Seq[String] = synchronized {
- val executorIdsToBeRemoved = new ArrayBuffer[String]
-
+ private def removeExecutors(executors: Seq[String]): Unit = synchronized {
logInfo("Request to remove executorIds: " + executors.mkString(", "))
- val numExistingExecutors = allocationManager.executorIds.size - executorsPendingToRemove.size
-
- var newExecutorTotal = numExistingExecutors
- executors.foreach { executorIdToBeRemoved =>
- if (newExecutorTotal - 1 < minNumExecutors) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (minimum number of executor limit $minNumExecutors)")
- } else if (newExecutorTotal - 1 < numExecutorsTarget) {
- logDebug(s"Not removing idle executor $executorIdToBeRemoved because there are only " +
- s"$newExecutorTotal executor(s) left (number of executor target $numExecutorsTarget)")
- } else if (canBeKilled(executorIdToBeRemoved)) {
- executorIdsToBeRemoved += executorIdToBeRemoved
- newExecutorTotal -= 1
- }
- }
+ val numExistingExecs = allocationManager.executorIds.size - executorsPendingToRemove.size
+ val execCountFloor = Math.max(minNumExecutors, numExecutorsTarget)
+ val (executorIdsToBeRemoved, dontRemove) = executors
+ .filter(canBeKilled)
+ .splitAt(numExistingExecs - execCountFloor)
- if (executorIdsToBeRemoved.isEmpty) {
- return Seq.empty[String]
+ dontRemove.foreach { execId =>
+ logDebug(s"Not removing idle executor $execId because it " +
+ s"would put us below the minimum limit of $minNumExecutors executors" +
+ s"or number of target executors $numExecutorsTarget")
}
- // Send a request to the backend to kill this executor(s)
- val executorsRemoved = if (testing) {
- executorIdsToBeRemoved
+ if (executorIdsToBeRemoved.isEmpty) {
+ Seq.empty[String]
+ } else if (testing) {
+ recordExecutorKill(executorIdsToBeRemoved)
+ } else if (replicateCachedData) {
+ logDebug(s"Starting replicate process for $executorIdsToBeRemoved")
+ client.markForDeath(executorIdsToBeRemoved)
+ recordExecutorKill(executorIdsToBeRemoved)
+ recoverShutdown.startExecutorKill(executorIdsToBeRemoved)
} else {
- client.killExecutors(executorIdsToBeRemoved)
+ val killed = killExecutors(executorIdsToBeRemoved)
+ recordExecutorKill(killed)
}
- // [SPARK-21834] killExecutors api reduces the target number of executors.
- // So we need to update the target with desired value.
- client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount)
- // reset the newExecutorTotal to the existing number of executors
- newExecutorTotal = numExistingExecutors
- if (testing || executorsRemoved.nonEmpty) {
- executorsRemoved.foreach { removedExecutorId =>
- newExecutorTotal -= 1
- logInfo(s"Removing executor $removedExecutorId because it has been idle for " +
- s"$executorIdleTimeoutS seconds (new desired total will be $newExecutorTotal)")
- executorsPendingToRemove.add(removedExecutorId)
- }
- executorsRemoved
- } else {
+ }
+
+ def killExecutors(executorIds: Seq[String]): Seq[String] = {
+ logDebug(s"Starting kill process for $executorIds")
+ val result = client.killExecutors(executorIds)
+ if (result.isEmpty) {
logWarning(s"Unable to reach the cluster manager to kill executor/s " +
- s"${executorIdsToBeRemoved.mkString(",")} or no executor eligible to kill!")
- Seq.empty[String]
+ s"${executorIds.mkString(",")} or no executor eligible to kill!")
}
+ result
}
- /**
- * Request the cluster manager to remove the given executor.
- * Return whether the request is acknowledged.
- */
- private def removeExecutor(executorId: String): Boolean = synchronized {
- val executorsRemoved = removeExecutors(Seq(executorId))
- executorsRemoved.nonEmpty && executorsRemoved(0) == executorId
+ private def recordExecutorKill(executorsRemoved: Seq[String]): Unit = synchronized {
+ // [SPARK-21834] killExecutors api reduces the target number of executors.
--- End diff --
fixed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19041
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19041#discussion_r179179616
--- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala ---
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.util.Try
+
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach, Matchers}
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Seconds, Span}
+
+import org.apache.spark.{SparkConf, SparkContext, SparkException, SparkFunSuite, TestUtils}
+import org.apache.spark.internal.config._
+import org.apache.spark.network.TransportContext
+import org.apache.spark.network.netty.SparkTransportConf
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage._
+
+/**
+ * This is an integration test for the cache recovery feature using a local spark cluster. It
+ * extends the unit tests in CacheRecoveryManagerSuite which mocks a lot of cluster infrastructure.
+ */
+class CacheRecoveryIntegrationSuite extends SparkFunSuite
+ with Matchers
+ with BeforeAndAfterEach
+ with BeforeAndAfterAll
+ with Eventually {
+
+ private var conf: SparkConf = makeBaseConf()
+ private val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores = 4)
+ private val rpcHandler = new ExternalShuffleBlockHandler(transportConf, null)
+ private val transportContext = new TransportContext(transportConf, rpcHandler)
+ private val shuffleService = transportContext.createServer()
+ private var sc: SparkContext = _
+
+ private def makeBaseConf() = new SparkConf()
+ .setAppName("test")
+ .setMaster("local-cluster[4, 1, 512]")
+ .set("spark.dynamicAllocation.enabled", "true")
+ .set("spark.dynamicAllocation.executorIdleTimeout", "1s") // always
+ .set("spark.dynamicAllocation.cachedExecutorIdleTimeout", "1s")
+ .set(EXECUTOR_MEMORY.key, "512m")
+ .set(SHUFFLE_SERVICE_ENABLED.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY.key, "true")
+ .set(DYN_ALLOCATION_CACHE_RECOVERY_TIMEOUT.key, "500s")
+ .set(EXECUTOR_INSTANCES.key, "1")
+ .set(DYN_ALLOCATION_INITIAL_EXECUTORS.key, "4")
+ .set(DYN_ALLOCATION_MIN_EXECUTORS.key, "3")
+
+ override def beforeEach(): Unit = {
+ conf = makeBaseConf()
+ conf.set("spark.shuffle.service.port", shuffleService.getPort.toString)
+ }
+
+ override def afterEach(): Unit = {
+ sc.stop()
+ conf = null
+ }
+
+ override def afterAll(): Unit = {
+ shuffleService.close()
+ }
+
+ private def getLocations(
+ sc: SparkContext,
+ rdd: RDD[_]): Map[BlockId, Map[BlockManagerId, BlockStatus]] = {
+ import scala.collection.breakOut
+ val blockIds: Array[BlockId] = rdd.partitions.map(p => RDDBlockId(rdd.id, p.index))
+ blockIds.map { id =>
+ id -> Try(sc.env.blockManager.master.getBlockStatus(id)).getOrElse(Map.empty)
+ }(breakOut)
+ }
+
+ test("cached data is replicated before dynamic de-allocation") {
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 4, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
+ getLocations(sc, rdd).forall { case (_, map) => map.nonEmpty } shouldBe true
+
+ eventually(timeout(Span(5, Seconds)), interval(Span(1, Seconds))) {
+ sc.getExecutorIds().size shouldBe 3
+ getLocations(sc, rdd).forall { case (_, map) => map.nonEmpty } shouldBe true
+ }
+ }
+
+ test("dont fail if a bunch of executors are shut down at once") {
+ conf.set("spark.dynamicAllocation.minExecutors", "1")
+ sc = new SparkContext(conf)
+ TestUtils.waitUntilExecutorsUp(sc, 2, 60000)
+
+ val rdd = sc.parallelize(1 to 1000, 4).map(_ * 4).cache()
+ rdd.reduce(_ + _) shouldBe 2002000
+ sc.getExecutorIds().size shouldBe 4
--- End diff --
though there are 4 tasks, they could run on fewer executors before dynamic allocation spins them all up. This could become a flaky tests. I think you should change the `waitUntilExecutorsUp` to be 4
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org