[jira] [Comment Edited] (TEZ-3709) TezMerger is slow for high number of segments

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995850#comment-15995850 ] 

Jonathan Eagles edited comment on TEZ-3709 at 5/4/17 1:32 AM:
--------------------------------------------------------------

Example performance gains

| task attempt | duration w/o patch | duration w/ patch | NUM_SHUFFLE_INPUTS |  NUM_SKIPPED_INPUTS | SHUFFLE_BYTES |
| 0 | 22m 30s 458ms | 7m 34s 752ms | 1238533 | 118987 | 138894095 | 


was (Author: jeagles):
Example performance gains

| task attempt | duration w/o patch | duration w/ patch | NUM_SHUFFLE_INPUTS | SHUFFLE_BYTES |
| 0 | 22m 30s 458ms | 7m 34s 752ms | 1238533 | 138894095 | 

> TezMerger is slow for high number of segments
> ---------------------------------------------
>
>                 Key: TEZ-3709
>                 URL: https://issues.apache.org/jira/browse/TEZ-3709
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>              Labels: performance
>         Attachments: TEZ-3709.1.patch
>
>
> The below code is a bad performer at scale since it has to memcpy the whole list of segments for each item in the batch instead of of just once per batch.
> This is true for both computeBytesInMerges and getSegmentDescriptors.
> {code}
> for (int i = 0; i < batch; i++) {
>   ArrayList#remove(0)
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)