You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2017/10/23 08:07:26 UTC
[GitHub] spark pull request #18492: [SPARK-19326] Speculated task attempts do not get...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18492#discussion_r146190420
--- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -373,8 +373,14 @@ private[spark] class ExecutorAllocationManager(
// If our target has not changed, do not send a message
// to the cluster manager and reset our exponential growth
if (delta == 0) {
- numExecutorsToAdd = 1
- return 0
+ // Check if there is any speculative jobs pending
+ if (listener.pendingTasks == 0 && listener.pendingSpeculativeTasks > 0) {
+ numExecutorsTarget =
+ math.max(math.min(maxNumExecutorsNeeded + 1, maxNumExecutors), minNumExecutors)
--- End diff --
@janewangfb Would you please explain why here `+ 1` if there's pending speculativeTasks, should the number of executors be calculated based on the number of pending tasks? Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org