You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Kuhu Shukla (JIRA)" <ji...@apache.org> on 2017/05/04 19:21:04 UTC

[jira] [Updated] (TEZ-3605) Detect and prune empty partitions for the Ordered case

     [ https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kuhu Shukla updated TEZ-3605:
-----------------------------
    Description: 
Analogous to the Unordered case we should not have empty partition entries/segments in the Ordered/DefaultSorter case. This will save writing unnecessary data.
Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced job, this change would allow not fetching empty partitions and then throwing them away.

  was:Analogous to the Unordered case we should not have empty partition entries/segments in the Ordered/DefaultSorter case. This will save writing unnecessary data.


> Detect and prune empty partitions for the Ordered case
> ------------------------------------------------------
>
>                 Key: TEZ-3605
>                 URL: https://issues.apache.org/jira/browse/TEZ-3605
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch, TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch, TEZ-3605.006.patch
>
>
> Analogous to the Unordered case we should not have empty partition entries/segments in the Ordered/DefaultSorter case. This will save writing unnecessary data.
> Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced job, this change would allow not fetching empty partitions and then throwing them away.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)