You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tez.apache.org by "Jonathan Eagles (JIRA)" <ji...@apache.org> on 2017/05/17 21:09:04 UTC

[jira] [Created] (TEZ-3733) Auto-Reduce Parallelism needs to consider shuffle overhead to avoid OOM

Jonathan Eagles created TEZ-3733:
------------------------------------

             Summary: Auto-Reduce Parallelism needs to consider shuffle overhead to avoid OOM
                 Key: TEZ-3733
                 URL: https://issues.apache.org/jira/browse/TEZ-3733
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Jonathan Eagles
            Assignee: Jonathan Eagles


For each fetch there is associated memory overhead per partition which can be enough to affect the number of tasks needed to process the data. In some cases the reduction can be so severe the tasks can OOM due to overhead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)