You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Zhiyuan Yang (JIRA)" <ji...@apache.org> on 2017/05/03 15:54:04 UTC

[jira] [Created] (TEZ-3708) Arbitrary parallelism for unpartitioned cartesian product regardless of # src tasks

Zhiyuan Yang created TEZ-3708:
---------------------------------

             Summary: Arbitrary parallelism for unpartitioned cartesian product regardless of # src tasks
                 Key: TEZ-3708
                 URL: https://issues.apache.org/jira/browse/TEZ-3708
             Project: Apache Tez
          Issue Type: Sub-task
            Reporter: Zhiyuan Yang
            Assignee: Zhiyuan Yang


Current unpartitioned cartesian product has a few limitations
1. parallelism can be not enough in case of large split and small # src task
2. parallelism can be too much in in case of large # src task
3. workload is not ideally distributed across the worker. Even with auto grouping, grouping by size may not be accurate because same size can means different #record and different cartesian product ops.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)