You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tez.apache.org by "Jonathan Eagles (JIRA)" <ji...@apache.org> on 2017/05/03 22:51:04 UTC

[jira] [Created] (TEZ-3709) TezMerger is slow for high number of segments

Jonathan Eagles created TEZ-3709:
------------------------------------

             Summary: TezMerger is slow for high number of segments
                 Key: TEZ-3709
                 URL: https://issues.apache.org/jira/browse/TEZ-3709
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Jonathan Eagles
            Assignee: Jonathan Eagles


The below code is a bad performer at scale since it has to memcpy the whole list of segments for each item in the batch instead of of just once per batch.
This is true for both computeBytesInMerges and getSegmentDescriptors.
{code}
for (int i = 0; i < batch; i++) {
  ArrayList#remove(0)
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)