You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by John Fang <hu...@gmail.com> on 2018/06/06 01:50:37 UTC

Why can per task‘s max memory only reach 1 / numTasks at most , not greater than 1 / numTasks in ExecutionMemoryPool ?

[image: 1.png]

In fact not all tasks belong to the same stage. Thus, per task may be is
deferent for the dependence of memory. For example, the executor
 are running two tasks(A and B), and the ExecutionMemoryPool own 1000M. We
can hope the task-A occupy 900M, and task-B occupy 100M due to the task-A
need much more memory when it is running. But per task's max memory is
limited, so the task-A can't occupy 900M memory.



-- 

Regards

John Fang