[jira] [Commented] (TEZ-1448) Make WeightedScalingMemoryDistributor as the default memory distributor

["Siddharth Seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104721#comment-14104721 ] 

Siddharth Seth commented on TEZ-1448:
-------------------------------------

{code} static final double MAX_ADDITIONAL_RESERVATION_FRACTION_PER_IO = 0.03d; {code}
This value seems far too low. Inputs typically have an overhead beyond the standard buffers they allocate. Should this be closer to 0.1 ?

{code}static String[] generateWeightStrings(int broadcastIn, int sortedOut, int scatterGatherShuffleIn,{code}
Please change the unit test to make use of this method as well; it has the same code otherwise.

RequestType for UNSORTED_OUTPUT (without partitioning is missing). Can you please add that as part of this patch.

Otherwise, looks good.

> Make WeightedScalingMemoryDistributor as the default memory distributor
> -----------------------------------------------------------------------
>
>                 Key: TEZ-1448
>                 URL: https://issues.apache.org/jira/browse/TEZ-1448
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>         Attachments: TEZ-1448.1.patch
>
>
> Currently Tez uses ScalingAllocator; WeightedScalingMemoryDistributor provides better control over ScalingAllocator.  
> In certain DAG executions, response time improvements of over 6% have been observed at 10 TB scale and ~26% runtime improvement in memory intensive reducers.



--
This message was sent by Atlassian JIRA
(v6.2#6252)