[jira] [Commented] (TEZ-932) Add a type aware Initial memory distributor

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948912#comment-13948912 ] 

Hitesh Shah commented on TEZ-932:
---------------------------------

Overall comment
  - there should be a big enough paragraph at the start explaining what this class does and how it does it.
  - at this point, for a user trying to use this, there is hardly any information on how to configure and use it. From that point of view, it seems a new user will never try to use this. 
  - it should be okay to go once documentation is addressed given that it can be considered somewhat "experimental" and also as its not the default option in use - have not looked too deeply into correctness though

{code}
+  public static final String TEZ_TASK_SCALE_MEMORY_ADDITIONAL_RESERVATION_FRACTION_PER_IO = TEZ_TASK_PREFIX
+      + "scale.memory.additional.reservation.fraction.per-io";
+
+  /**
+   * Max cumulative total reservation for additional IOs.
+   */
+  public static final String TEZ_TASK_SCALE_MEMORY_ADDITIONAL_RESERVATION_FRACTION_MAX_TOTAL_LIMIT = TEZ_TASK_PREFIX
+      + "scale.memory.additional reservation.fraction.max-total-limit";
{code}
  - maybe slightly shorter properties if possible? 

s/RuntimeLibraryInitialMemoryAllocator/InitialMemoryAllocatorImpl/ ? or just BasicInitialMemoryAllocator ? 

{code}
+      // Fall back to regular scaling. e.g. BROADCAST : SHUFFLE = 0:1 .
+      // i.e. if Shuffle present, Broadcast gets nothing, but otherwise it
+      // should get an allocation
{code}
   - wasnt there recent code that would start throwing an error if received 0 for available memory ? 

bq. if (reserveFraction >= 1.0d) {
   - why is this not an error where the task is failed due to misconfiguration? 

bq. TezJobConfig.TEZ_TASK_SCALE_MEMORY_WEIGHTED_RATIOS
   - this does not seem right. What is the format? Seems to have 5 numbers which are set without any clue as to what they map to. 


> Add a type aware Initial memory distributor
> -------------------------------------------
>
>                 Key: TEZ-932
>                 URL: https://issues.apache.org/jira/browse/TEZ-932
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: TEZ-932.1.txt, TEZ-932.2.txt
>
>
> The current memory scaler has not idea about the requirements of the underlying components (broadcast, shfufle etc). This jira is to add a scaler which can take the types into consideration.



--
This message was sent by Atlassian JIRA
(v6.2#6252)