You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Zhenyu Hu <cs...@gmail.com> on 2021/08/11 16:35:03 UTC

Dynamic Allocation& ExecutorMonitor Shuffle Timeout & CacheTimeout

In private class Tracker of
org.apache.spark.scheduler.dynalloc.ExecutorMonitor, the method `
updateTimeout
<https://github.com/apache/spark/blob/e1a5d9411733437b5a18045bbd18b48f7aa40f46/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L567>`
will take the min of `_cacheTimeout` and `_shuffleTimeout`.
I think is more reasonable to take the max of `_cacheTimeout` and
`_shuffleTimeout`, since we need to wait both of cache and shuffle to be
timeout if they all be set.
Any one known about that?