You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by Kuhu Shukla <ks...@yahoo-inc.com.INVALID> on 2017/02/02 23:57:35 UTC

DefaultSorter/OrderedWriter and Empty Partitions

Hello all,

In DefaultSorter, for the OrderedPartitioned case, is the inclusion of empty partition headers and segments in the file.out.index and file.out needed?

In the Unordered case, we prune out the empty partitions in the writer and it might be worth doing the same in the Ordered case unless this behavior was intentional. Would appreciate some comments on this.

Thanks and Regards,
Kuhu

Re: DefaultSorter/OrderedWriter and Empty Partitions

Posted by Kuhu Shukla <ks...@yahoo-inc.com.INVALID>.
Thank you Siddharth for your inputs. I have opened https://issues.apache.org/jira/browse/TEZ-3605 to track this change.
Regards,Kuhu

 

    On Friday, February 3, 2017 12:52 PM, Siddharth Seth <ss...@apache.org> wrote:
 

 Makes sense to not write out unnecessary data. There's also a case where DefaultSorter writes out empty files when there's no output. Have to be careful about the configuration which enables empty partitions via events.
On Thu, Feb 2, 2017 at 3:57 PM, Kuhu Shukla <ks...@yahoo-inc.com.invalid> wrote:

Hello all,

In DefaultSorter, for the OrderedPartitioned case, is the inclusion of empty partition headers and segments in the file.out.index and file.out needed?

In the Unordered case, we prune out the empty partitions in the writer and it might be worth doing the same in the Ordered case unless this behavior was intentional. Would appreciate some comments on this.

Thanks and Regards,
Kuhu




   

Re: DefaultSorter/OrderedWriter and Empty Partitions

Posted by Siddharth Seth <ss...@apache.org>.
Makes sense to not write out unnecessary data. There's also a case where
DefaultSorter writes out empty files when there's no output. Have to be
careful about the configuration which enables empty partitions via events.

On Thu, Feb 2, 2017 at 3:57 PM, Kuhu Shukla <ks...@yahoo-inc.com.invalid>
wrote:

> Hello all,
>
> In DefaultSorter, for the OrderedPartitioned case, is the inclusion of
> empty partition headers and segments in the file.out.index and file.out
> needed?
>
> In the Unordered case, we prune out the empty partitions in the writer and
> it might be worth doing the same in the Ordered case unless this behavior
> was intentional. Would appreciate some comments on this.
>
> Thanks and Regards,
> Kuhu
>