You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Zhiyuan Yang (JIRA)" <ji...@apache.org> on 2016/12/02 21:28:58 UTC

[jira] [Commented] (TEZ-3552) Shuffle split array when size-based sorting is turned off

    [ https://issues.apache.org/jira/browse/TEZ-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716570#comment-15716570 ] 

Zhiyuan Yang commented on TEZ-3552:
-----------------------------------

[~mingma] Are you going to work on this? If not, I'll take it over. Do you think it's necessary to add a another knob for whether to shuffle splits? I prefer simply to do randomization if sorting is turned off.

> Shuffle split array when size-based sorting is turned off
> ---------------------------------------------------------
>
>                 Key: TEZ-3552
>                 URL: https://issues.apache.org/jira/browse/TEZ-3552
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Ming Ma
>
> TEZ-3430 adds the functionality to skip size-based split sorting to help with job runtime. During further testing, the original split array for certain inputs before sorting aren't randomly distributed in size. So when the spit sorting is turned off, we should shuffle the split instead of doing nothing. That will make the size distribution more even.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)