[GitHub] Ben-Zvi commented on a change in pull request #1351: DRILL-6543: Disable Hash Join fallback, add percent_reserved_allowance_from_direct

[GitBox <gi...@apache.org>]

Ben-Zvi commented on a change in pull request #1351: DRILL-6543: Disable Hash Join fallback, add percent_reserved_allowance_from_direct
URL: https://github.com/apache/drill/pull/1351#discussion_r199985944
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/util/MemoryAllocationUtilities.java
 ##########
 @@ -138,16 +139,36 @@ public static long computeOperatorMemory(OptionSet optionManager, long maxAllocP
   @VisibleForTesting
   public static long computeQueryMemory(DrillConfig config, OptionSet optionManager, long directMemory) {
 
+    // Get the options
+    double percent_per_query = optionManager.getOption(ExecConstants.PERCENT_MEMORY_PER_QUERY);
+    long max_query_per_node = optionManager.getOption(ExecConstants.MAX_QUERY_MEMORY_PER_NODE);
+    double percent_allowance = optionManager.getOption(ExecConstants.PERCENT_RESERVED_ALLOWANCE_FROM_DIRECT);
+
+    // verify that the allowance is kept
+    if ( percent_per_query + percent_allowance > 1.0 ) {
 
 Review comment:
   The code in the PR "enforces" keeping the allowance (out of the direct memory); any user settings that violates this limit returns an error (unfortunately can only be done when a query (using buffered ops) is executed).
     This mainly serves as a reminder (for the common user that does not read the documentation). It does not help with concurrent query execution (though users using concurrent are usually more sophisticated, so may know about the allowance). 
        The example suggested above applies the *allowance* onto the final computed memory, which is not the intention. For example, Direct Memory of 20GB, and Mem Per Query of 4GB -- then the code about would subtract 25% and the Mem Per Query would be only 3GB (and confuse the user).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services