[jira] [Commented] (DRILL-7372) MethodAnalyzer consumes too much memory

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966314#comment-16966314 ] 

ASF GitHub Bot commented on DRILL-7372:
---------------------------------------

paul-rogers commented on issue #1887: DRILL-7372: MethodAnalyzer consumes too much memory
URL: https://github.com/apache/drill/pull/1887#issuecomment-549193078
 
 
   Just a thought: we proved a couple of years back that the modern JVM does a perfectly fine job of scalar replacement. We also found performance benefits from using "plain Java" without class merging. As a result, our byte code analysis is no longer needed. Just flip one option and we can completely remove this code instead of continuing to maintain it.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> MethodAnalyzer consumes too much memory
> ---------------------------------------
>
>                 Key: DRILL-7372
>                 URL: https://issues.apache.org/jira/browse/DRILL-7372
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.16.0
>            Reporter: Vova Vysotskyi
>            Assignee: Vova Vysotskyi
>            Priority: Major
>              Labels: ready-to-commit
>             Fix For: 1.17.0
>
>
> In the scope of DRILL-6524 was added logic for determining whether a variable is assigned in conditional block to prevent incorrect scalar replacement for such cases.
> But for some queries, this logic consumes too many memory, for example, for the following query:
> {code:sql}
> SELECT *
> FROM cp.`employee.json`
> WHERE employee_id+0 < employee_id
>   OR employee_id+1 < employee_id
>   AND employee_id+2 < employee_id
>   OR employee_id+3 < employee_id
>   AND employee_id+4 < employee_id
>   OR employee_id+5 < employee_id
>   AND employee_id+6 < employee_id
>   OR employee_id+7 < employee_id
>   AND employee_id+8 < employee_id
>   OR employee_id+9 < employee_id
>   AND employee_id+10 < employee_id
>   OR employee_id+11 < employee_id
>   AND employee_id+12 < employee_id
>   OR employee_id+13 < employee_id
>   AND employee_id+14 < employee_id
>   OR employee_id+15 < employee_id
>   AND employee_id+16 < employee_id
>   OR employee_id+17 < employee_id
>   AND employee_id+18 < employee_id
>   OR employee_id+19 < employee_id
>   AND employee_id+20 < employee_id
>   OR employee_id+21 < employee_id
>   AND employee_id+22 < employee_id
>   OR employee_id+23 < employee_id
>   AND employee_id+24 < employee_id
>   OR employee_id+25 < employee_id
>   AND employee_id+26 < employee_id
>   OR employee_id+27 < employee_id
>   AND employee_id+28 < employee_id
>   OR employee_id+29 < employee_id
>   AND employee_id+30 < employee_id
>   OR employee_id+31 < employee_id
>   AND employee_id+32 < employee_id
>   OR employee_id+33 < employee_id
>   AND employee_id+34 < employee_id
>   OR employee_id+35 < employee_id
>   AND employee_id+36 < employee_id
>   OR employee_id+37 < employee_id
>   AND employee_id+38 < employee_id
>   OR employee_id+39 < employee_id
>   AND employee_id+40 < employee_id
>   OR employee_id+41 < employee_id
>   AND employee_id+42 < employee_id
>   OR employee_id+43 < employee_id
>   AND employee_id+44 < employee_id
>   OR employee_id+45 < employee_id
>   AND employee_id+46 < employee_id
>   OR employee_id+47 < employee_id
>   AND employee_id+48 < employee_id
>   OR employee_id+49 < employee_id
>   AND TRUE;
> {code}
> Drill consumes more than 6 GB memory.
> One of the issues to fix is to replace {{Deque<Set<Integer>> localVariablesSet;}} with {{Deque<BitSet>}}, it will reduce memory usage significantly.
> Additionally should be investigated why these objects cannot be collected by GC.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)