You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2015/02/25 20:42:04 UTC

[jira] [Created] (PIG-4434) Improve auto-parallelism for tez

Daniel Dai created PIG-4434:
-------------------------------

             Summary: Improve auto-parallelism for tez
                 Key: PIG-4434
                 URL: https://issues.apache.org/jira/browse/PIG-4434
             Project: Pig
          Issue Type: Improvement
          Components: tez
            Reporter: Daniel Dai
            Assignee: Daniel Dai
             Fix For: 0.15.0


Tez auto-parallelism currently has some limitation:
1. ShuffledVertexManager only decrease parallelism not increase
2. Pig currently exaggerate parallelism at frontend, ShuffledVertexManager might get initial parallelism way large than actual, that would be costly

Instead of that, we can gradually adjust initial vertex parallelism at runtime once upstream vertexes finishes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)