[GitHub] spark pull request #20604: [SPARK-23365][CORE] Do not adjust num executors w...

[Ngone51 <gi...@git.apache.org>]

Github user Ngone51 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20604#discussion_r182086337
  
    --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
    @@ -1643,7 +1646,10 @@ class SparkContext(config: SparkConf) extends Logging {
       def killExecutors(executorIds: Seq[String]): Boolean = {
         schedulerBackend match {
           case b: ExecutorAllocationClient =>
    -        b.killExecutors(executorIds, replace = false, force = true).nonEmpty
    +        require(executorAllocationManager.isEmpty,
    --- End diff --
    
    Hi, @squito , I'm quite questioned about the cases:
    >  If you've got just one executor, and then you kill it, should your app sit with 0 executors?
    
    if app sit with 0 executors, then pending tasks increase, which lead to `ExecutorAllocationManager` increases target number of executors. So, app will not always sit with 0 executors.
    
    > Or even if you've got 10 executors, and you kill one -- when is dynamic allocation allowed to bump the total back up?
    
    for this case, to be honest, I really do not get your point. But, it must blame my poor English.
    
    And, what will happens if we use this method without `ExecutorAllocationManager `? Or do we really need adjust TargetNumExecutors (set `adjustTargetNumExecutors  = true` below) if we are not using `ExecutorAllocationManager `?
    
    see these several lines in `killExecutors()`:
    ```
    if (adjustTargetNumExecutors) {
      requestedTotalExecutors = math.max(requestedTotalExecutors - executorsToKill.size, 0)
      ...
      doRequestTotalExecutors(requestedTotalExecutors)
    }
    ```
    Set `adjustTargetNumExecutors  = true` will change `requestedTotalExecutors` . And IIUC, `requestedTotalExecutors ` is only used in dynamic allocation mode. So, if we are not  using `ExecutorAllocationManager `, allocation client will request `requestedTotalExecutors = 0`  number of executors to cluster manager (this is really terrible). But, actually, app without `ExecutorAllocationManager ` do not have a limit requesting executors (in default).
    
    Actually, I think this series methods, including `killAndReplaceExecutor `,  `requestExecutors`, etc, are designed with dynamic allocation mode. And if we still want use these methods while app do not use `ExecutorAllocationManager`, we should not change `requestedTotalExecutors `, or even not request cluster manager with a specific number.
    
    WDYT?
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org