[jira] [Commented] (DRILL-7372) MethodAnalyzer consumes too much memory

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964033#comment-16964033 ] 

ASF GitHub Bot commented on DRILL-7372:
---------------------------------------

vvysotskyi commented on pull request #1887: DRILL-7372: MethodAnalyzer consumes too much memory
URL: https://github.com/apache/drill/pull/1887#discussion_r341122302
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/compile/bytecode/MethodAnalyzer.java
 ##########
 @@ -174,14 +171,22 @@ public void initJumpTarget(int opcode, LabelNode target) {
           case IF_ICMPLE:
           case IF_ACMPEQ:
           case IF_ACMPNE:
+          case IFNULL:
           case IFNONNULL:
             // for the case when conditional block is handled, creates new variables set
             // to store local variables declared inside current conditional block and
             // stores its target LabelNode to restore previous variables set after conditional block is ended
-            localVariablesSet.push(new HashSet<>());
+            localVariablesSet.push(new BitSet());
             labelsStack.push(target);
         }
       }
     }
+
+    @Override
+    public void clearStack() {
+      super.clearStack();
 
 Review comment:
   Thanks, done.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> MethodAnalyzer consumes too much memory
> ---------------------------------------
>
>                 Key: DRILL-7372
>                 URL: https://issues.apache.org/jira/browse/DRILL-7372
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.16.0
>            Reporter: Vova Vysotskyi
>            Assignee: Vova Vysotskyi
>            Priority: Major
>             Fix For: 1.17.0
>
>
> In the scope of DRILL-6524 was added logic for determining whether a variable is assigned in conditional block to prevent incorrect scalar replacement for such cases.
> But for some queries, this logic consumes too many memory, for example, for the following query:
> {code:sql}
> SELECT *
> FROM cp.`employee.json`
> WHERE employee_id+0 < employee_id
>   OR employee_id+1 < employee_id
>   AND employee_id+2 < employee_id
>   OR employee_id+3 < employee_id
>   AND employee_id+4 < employee_id
>   OR employee_id+5 < employee_id
>   AND employee_id+6 < employee_id
>   OR employee_id+7 < employee_id
>   AND employee_id+8 < employee_id
>   OR employee_id+9 < employee_id
>   AND employee_id+10 < employee_id
>   OR employee_id+11 < employee_id
>   AND employee_id+12 < employee_id
>   OR employee_id+13 < employee_id
>   AND employee_id+14 < employee_id
>   OR employee_id+15 < employee_id
>   AND employee_id+16 < employee_id
>   OR employee_id+17 < employee_id
>   AND employee_id+18 < employee_id
>   OR employee_id+19 < employee_id
>   AND employee_id+20 < employee_id
>   OR employee_id+21 < employee_id
>   AND employee_id+22 < employee_id
>   OR employee_id+23 < employee_id
>   AND employee_id+24 < employee_id
>   OR employee_id+25 < employee_id
>   AND employee_id+26 < employee_id
>   OR employee_id+27 < employee_id
>   AND employee_id+28 < employee_id
>   OR employee_id+29 < employee_id
>   AND employee_id+30 < employee_id
>   OR employee_id+31 < employee_id
>   AND employee_id+32 < employee_id
>   OR employee_id+33 < employee_id
>   AND employee_id+34 < employee_id
>   OR employee_id+35 < employee_id
>   AND employee_id+36 < employee_id
>   OR employee_id+37 < employee_id
>   AND employee_id+38 < employee_id
>   OR employee_id+39 < employee_id
>   AND employee_id+40 < employee_id
>   OR employee_id+41 < employee_id
>   AND employee_id+42 < employee_id
>   OR employee_id+43 < employee_id
>   AND employee_id+44 < employee_id
>   OR employee_id+45 < employee_id
>   AND employee_id+46 < employee_id
>   OR employee_id+47 < employee_id
>   AND employee_id+48 < employee_id
>   OR employee_id+49 < employee_id
>   AND TRUE;
> {code}
> Drill consumes more than 6 GB memory.
> One of the issues to fix is to replace {{Deque<Set<Integer>> localVariablesSet;}} with {{Deque<BitSet>}}, it will reduce memory usage significantly.
> Additionally should be investigated why these objects cannot be collected by GC.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)