[jira] [Updated] (DRILL-7372) MethodAnalyzer consumes too many memory

["Arina Ielchiieva (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arina Ielchiieva updated DRILL-7372:
------------------------------------
    Reviewer: Arina Ielchiieva

> MethodAnalyzer consumes too many memory
> ---------------------------------------
>
>                 Key: DRILL-7372
>                 URL: https://issues.apache.org/jira/browse/DRILL-7372
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.16.0
>            Reporter: Vova Vysotskyi
>            Assignee: Vova Vysotskyi
>            Priority: Major
>             Fix For: 1.17.0
>
>
> In the scope of DRILL-6524 was added logic for determining whether a variable is assigned in conditional block to prevent incorrect scalar replacement for such cases.
> But for some queries, this logic consumes too many memory, for example, for the following query:
> {code:sql}
> SELECT *
> FROM cp.`employee.json`
> WHERE employee_id+0 < employee_id
>   OR employee_id+1 < employee_id
>   AND employee_id+2 < employee_id
>   OR employee_id+3 < employee_id
>   AND employee_id+4 < employee_id
>   OR employee_id+5 < employee_id
>   AND employee_id+6 < employee_id
>   OR employee_id+7 < employee_id
>   AND employee_id+8 < employee_id
>   OR employee_id+9 < employee_id
>   AND employee_id+10 < employee_id
>   OR employee_id+11 < employee_id
>   AND employee_id+12 < employee_id
>   OR employee_id+13 < employee_id
>   AND employee_id+14 < employee_id
>   OR employee_id+15 < employee_id
>   AND employee_id+16 < employee_id
>   OR employee_id+17 < employee_id
>   AND employee_id+18 < employee_id
>   OR employee_id+19 < employee_id
>   AND employee_id+20 < employee_id
>   OR employee_id+21 < employee_id
>   AND employee_id+22 < employee_id
>   OR employee_id+23 < employee_id
>   AND employee_id+24 < employee_id
>   OR employee_id+25 < employee_id
>   AND employee_id+26 < employee_id
>   OR employee_id+27 < employee_id
>   AND employee_id+28 < employee_id
>   OR employee_id+29 < employee_id
>   AND employee_id+30 < employee_id
>   OR employee_id+31 < employee_id
>   AND employee_id+32 < employee_id
>   OR employee_id+33 < employee_id
>   AND employee_id+34 < employee_id
>   OR employee_id+35 < employee_id
>   AND employee_id+36 < employee_id
>   OR employee_id+37 < employee_id
>   AND employee_id+38 < employee_id
>   OR employee_id+39 < employee_id
>   AND employee_id+40 < employee_id
>   OR employee_id+41 < employee_id
>   AND employee_id+42 < employee_id
>   OR employee_id+43 < employee_id
>   AND employee_id+44 < employee_id
>   OR employee_id+45 < employee_id
>   AND employee_id+46 < employee_id
>   OR employee_id+47 < employee_id
>   AND employee_id+48 < employee_id
>   OR employee_id+49 < employee_id
>   AND TRUE;
> {code}
> Drill consumes more than 6 GB memory.
> One of the issues to fix is to replace {{Deque<Set<Integer>> localVariablesSet;}} with {{Deque<BitSet>}}, it will reduce memory usage significantly.
> Additionally should be investigated why these objects cannot be collected by GC.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)