[jira] [Created] (DRILL-7372) MethodAnalyzer consumes too many memory

["Volodymyr Vysotskyi (Jira)" <ji...@apache.org>]

Volodymyr Vysotskyi created DRILL-7372:
------------------------------------------

             Summary: MethodAnalyzer consumes too many memory
                 Key: DRILL-7372
                 URL: https://issues.apache.org/jira/browse/DRILL-7372
             Project: Apache Drill
          Issue Type: Bug
    Affects Versions: 1.16.0
            Reporter: Volodymyr Vysotskyi
            Assignee: Volodymyr Vysotskyi
             Fix For: 1.17.0


In the scope of DRILL-6524 was added logic for determining whether a variable is assigned in conditional block to prevent incorrect scalar replacement for such cases.

But for some queries, this logic consumes too many memory, for example, for the following query:
{code:sql}
SELECT *
FROM cp.`employee.json`
WHERE employee_id+0 < employee_id
  OR employee_id+1 < employee_id
  AND employee_id+2 < employee_id
  OR employee_id+3 < employee_id
  AND employee_id+4 < employee_id
  OR employee_id+5 < employee_id
  AND employee_id+6 < employee_id
  OR employee_id+7 < employee_id
  AND employee_id+8 < employee_id
  OR employee_id+9 < employee_id
  AND employee_id+10 < employee_id
  OR employee_id+11 < employee_id
  AND employee_id+12 < employee_id
  OR employee_id+13 < employee_id
  AND employee_id+14 < employee_id
  OR employee_id+15 < employee_id
  AND employee_id+16 < employee_id
  OR employee_id+17 < employee_id
  AND employee_id+18 < employee_id
  OR employee_id+19 < employee_id
  AND employee_id+20 < employee_id
  OR employee_id+21 < employee_id
  AND employee_id+22 < employee_id
  OR employee_id+23 < employee_id
  AND employee_id+24 < employee_id
  OR employee_id+25 < employee_id
  AND employee_id+26 < employee_id
  OR employee_id+27 < employee_id
  AND employee_id+28 < employee_id
  OR employee_id+29 < employee_id
  AND employee_id+30 < employee_id
  OR employee_id+31 < employee_id
  AND employee_id+32 < employee_id
  OR employee_id+33 < employee_id
  AND employee_id+34 < employee_id
  OR employee_id+35 < employee_id
  AND employee_id+36 < employee_id
  OR employee_id+37 < employee_id
  AND employee_id+38 < employee_id
  OR employee_id+39 < employee_id
  AND employee_id+40 < employee_id
  OR employee_id+41 < employee_id
  AND employee_id+42 < employee_id
  OR employee_id+43 < employee_id
  AND employee_id+44 < employee_id
  OR employee_id+45 < employee_id
  AND employee_id+46 < employee_id
  OR employee_id+47 < employee_id
  AND employee_id+48 < employee_id
  OR employee_id+49 < employee_id
  AND TRUE;
{code}

Drill consumes more than 6 GB memory.

One of the issues to fix is to replace {{Deque<Set<Integer>> localVariablesSet;}} with {{Deque<BitSet>}}, it will reduce memory usage significantly.
Additionally should be investigated why these objects cannot be collected by GC.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)