You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2016/02/26 00:02:18 UTC

[jira] [Created] (TEZ-3142) Intermediate mem-to-mem enhancements

Rajesh Balamohan created TEZ-3142:
-------------------------------------

             Summary: Intermediate mem-to-mem enhancements
                 Key: TEZ-3142
                 URL: https://issues.apache.org/jira/browse/TEZ-3142
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Rajesh Balamohan


-InMemoryMapOutputs is already sorted by size. This information will be retained in the generic startMerge arrayCopy - so the inputs list should already be sorted. We could walk the list starting from the lowest to the highest to try and merge as many segments as possible, and then terminate, instead of trying to look for more segments.

-InMem merged segments make it back into inMemoryMergedMapOutputs. That makes them ineligible for subsequent memToMem merges. These could be considered as well - especially when handling extremely small segments.

Ref: TEZ-1911



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)