You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by John Fang <hu...@gmail.com> on 2018/06/05 17:13:03 UTC

Why can per task‘s memory only reach 1 / numTasks , not greater than 1 / numTasks in ExecutionMemoryPool ?

In fact not all tasks belong to the same stage. Thus, per task may be is
deferent for the dependence of memory. For example, the executor
 are running two tasks(A and B), and the ExecutionMemoryPool own 1000M. We
can hope the task-A occupy 900M, and task-B occupy 100M due to the task-A
need much more memory when it is running.

-- 

Regards

John Fang