[jira] [Commented] (DRILL-7372) MethodAnalyzer consumes too many memory

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963888#comment-16963888 ] 

ASF GitHub Bot commented on DRILL-7372:
---------------------------------------

vvysotskyi commented on pull request #1887: DRILL-7372: MethodAnalyzer consumes too many memory
URL: https://github.com/apache/drill/pull/1887
 
 
   Jira: [DRILL-7372](https://issues.apache.org/jira/browse/DRILL-7372).
   
   - Replaced `Set<Integer>` with `BitSet` - it allowed to reduce heap usage significantly.
   - Avoided copying `localVariablesSet` and `labelsStack` from the source frame when a new frame is created since the branch-sensitive analysis was already done for a source frame.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> MethodAnalyzer consumes too many memory
> ---------------------------------------
>
>                 Key: DRILL-7372
>                 URL: https://issues.apache.org/jira/browse/DRILL-7372
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.16.0
>            Reporter: Vova Vysotskyi
>            Assignee: Vova Vysotskyi
>            Priority: Major
>             Fix For: 1.17.0
>
>
> In the scope of DRILL-6524 was added logic for determining whether a variable is assigned in conditional block to prevent incorrect scalar replacement for such cases.
> But for some queries, this logic consumes too many memory, for example, for the following query:
> {code:sql}
> SELECT *
> FROM cp.`employee.json`
> WHERE employee_id+0 < employee_id
>   OR employee_id+1 < employee_id
>   AND employee_id+2 < employee_id
>   OR employee_id+3 < employee_id
>   AND employee_id+4 < employee_id
>   OR employee_id+5 < employee_id
>   AND employee_id+6 < employee_id
>   OR employee_id+7 < employee_id
>   AND employee_id+8 < employee_id
>   OR employee_id+9 < employee_id
>   AND employee_id+10 < employee_id
>   OR employee_id+11 < employee_id
>   AND employee_id+12 < employee_id
>   OR employee_id+13 < employee_id
>   AND employee_id+14 < employee_id
>   OR employee_id+15 < employee_id
>   AND employee_id+16 < employee_id
>   OR employee_id+17 < employee_id
>   AND employee_id+18 < employee_id
>   OR employee_id+19 < employee_id
>   AND employee_id+20 < employee_id
>   OR employee_id+21 < employee_id
>   AND employee_id+22 < employee_id
>   OR employee_id+23 < employee_id
>   AND employee_id+24 < employee_id
>   OR employee_id+25 < employee_id
>   AND employee_id+26 < employee_id
>   OR employee_id+27 < employee_id
>   AND employee_id+28 < employee_id
>   OR employee_id+29 < employee_id
>   AND employee_id+30 < employee_id
>   OR employee_id+31 < employee_id
>   AND employee_id+32 < employee_id
>   OR employee_id+33 < employee_id
>   AND employee_id+34 < employee_id
>   OR employee_id+35 < employee_id
>   AND employee_id+36 < employee_id
>   OR employee_id+37 < employee_id
>   AND employee_id+38 < employee_id
>   OR employee_id+39 < employee_id
>   AND employee_id+40 < employee_id
>   OR employee_id+41 < employee_id
>   AND employee_id+42 < employee_id
>   OR employee_id+43 < employee_id
>   AND employee_id+44 < employee_id
>   OR employee_id+45 < employee_id
>   AND employee_id+46 < employee_id
>   OR employee_id+47 < employee_id
>   AND employee_id+48 < employee_id
>   OR employee_id+49 < employee_id
>   AND TRUE;
> {code}
> Drill consumes more than 6 GB memory.
> One of the issues to fix is to replace {{Deque<Set<Integer>> localVariablesSet;}} with {{Deque<BitSet>}}, it will reduce memory usage significantly.
> Additionally should be investigated why these objects cannot be collected by GC.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)