You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2018/07/16 06:53:24 UTC
spark git commit: [SPARK-24558][CORE] wrong Idle Timeout value is
used in case of the cacheBlock.
Repository: spark
Updated Branches:
refs/heads/master 9f929458f -> 2603ae30b
[SPARK-24558][CORE] wrong Idle Timeout value is used in case of the cacheBlock.
It is corrected as per the configuration.
## What changes were proposed in this pull request?
IdleTimeout info used to print in the logs is taken based on the cacheBlock. If it is cacheBlock then cachedExecutorIdleTimeoutS is considered else executorIdleTimeoutS
## How was this patch tested?
Manual Test
spark-sql> cache table sample;
2018-05-15 14:44:02 INFO DAGScheduler:54 - Submitting 3 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[8] at processCmd at CliDriver.java:376) (first 15 tasks are for partitions Vector(0, 1, 2))
2018-05-15 14:44:02 INFO YarnScheduler:54 - Adding task set 0.0 with 3 tasks
2018-05-15 14:44:03 INFO ExecutorAllocationManager:54 - Requesting 1 new executor because tasks are backlogged (new desired total will be 1)
...
...
2018-05-15 14:46:10 INFO YarnClientSchedulerBackend:54 - Actual list of executor(s) to be killed is 1
2018-05-15 14:46:10 INFO **ExecutorAllocationManager:54 - Removing executor 1 because it has been idle for 120 seconds (new desired total will be 0)**
2018-05-15 14:46:11 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Disabling executor 1.
2018-05-15 14:46:11 INFO DAGScheduler:54 - Executor lost: 1 (epoch 1)
Author: sandeep-katta <sa...@gmail.com>
Closes #21565 from sandeep-katta/loginfoBug.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2603ae30
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2603ae30
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2603ae30
Branch: refs/heads/master
Commit: 2603ae30be78c6cb24a67c26fb781fae8763f229
Parents: 9f92945
Author: sandeep-katta <sa...@gmail.com>
Authored: Mon Jul 16 14:52:49 2018 +0800
Committer: Wenchen Fan <we...@databricks.com>
Committed: Mon Jul 16 14:52:49 2018 +0800
----------------------------------------------------------------------
.../scala/org/apache/spark/ExecutorAllocationManager.scala | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/2603ae30/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
index aa363ee..17b8863 100644
--- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
+++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
@@ -488,9 +488,15 @@ private[spark] class ExecutorAllocationManager(
newExecutorTotal = numExistingExecutors
if (testing || executorsRemoved.nonEmpty) {
executorsRemoved.foreach { removedExecutorId =>
+ // If it is a cached block, it uses cachedExecutorIdleTimeoutS for timeout
+ val idleTimeout = if (blockManagerMaster.hasCachedBlocks(removedExecutorId)) {
+ cachedExecutorIdleTimeoutS
+ } else {
+ executorIdleTimeoutS
+ }
newExecutorTotal -= 1
logInfo(s"Removing executor $removedExecutorId because it has been idle for " +
- s"$executorIdleTimeoutS seconds (new desired total will be $newExecutorTotal)")
+ s"$idleTimeout seconds (new desired total will be $newExecutorTotal)")
executorsPendingToRemove.add(removedExecutorId)
}
executorsRemoved
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org