[jira] [Comment Edited] (TEZ-2950) Poor performance of UnorderedPartitionedKVWriter

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290330#comment-16290330 ] 

Rohini Palaniswamy edited comment on TEZ-2950 at 12/14/17 4:59 AM:
-------------------------------------------------------------------

Here is a simpler suggestion to try speed it up a bit. Can probably be addressed in a separate jira as a short term solution and leave this one for long term solution.

https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/writers/UnorderedPartitionedKVWriter.java#L1010-L1022 - For each partition, each spill file is opened once.  For parallelism of 1000 and 8500 spills, it will be making 8500000 file open calls. We can try keeping the first N file handles open always (will need a new IFile.Reader method that does not close the underlying input stream but does rest of close() like freeing up decompressor and buffers).  Let us say we keep first 100 spill files always open, it will cut down number of file open calls to  8400100. For parallelism of 1000 and 100 spills, it will cut down file open calls from 100000 to 100.


was (Author: rohini):
Here is a simpler suggestion to try speed it up a bit. Can probably be addressed in a separate jira as a short term solution and leave this one for long term solution.

https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/writers/UnorderedPartitionedKVWriter.java#L1010-L1022 - For each partition, each spill file is opened once.  For parallelism of 1000 and 8500 spills, it will be making 8500000 file open calls. This can be cut down by batching of spill file reads and partition writes. Let's say for a batch size of 10, 10 writers (partitions) and 10 spill file readers are kept open in parallel and merging is done. It will cut down file open by 10x to 850000.

> Poor performance of UnorderedPartitionedKVWriter
> ------------------------------------------------
>
>                 Key: TEZ-2950
>                 URL: https://issues.apache.org/jira/browse/TEZ-2950
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Kuhu Shukla
>         Attachments: TEZ-2950.001_prelim.patch
>
>
> Came across a job which was taking a long time in UnorderedPartitionedKVWriter.mergeAll. It was decompressing and reading data from spill files (8500 spills) and then writing the final compressed merge file. Why do we need spill files for UnorderedPartitionedKVWriter? Why not just buffer and keep directly writing to the final file which will save a lot of time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)