[jira] [Reopened] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

["Aman Sinha (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aman Sinha reopened DRILL-3765:
-------------------------------

Re-opening since this is becoming a performance issue.  Discussed briefly with [~jni].  The Volcano planner's invocations of the PruneScanRule multiple times during the search exploration is too heavy-weight since the rule itself is complex.  We should ideally move the pruning rules to the Hep planner as long as we ensure that associated rules such as filter push-down, projection push-down are also applied before and after pruning.  

> Partition prune rule is unnecessary fired multiple times. 
> ----------------------------------------------------------
>
>                 Key: DRILL-3765
>                 URL: https://issues.apache.org/jira/browse/DRILL-3765
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>            Reporter: Jinfeng Ni
>            Assignee: Jinfeng Ni
>
> It seems that the partition prune rule may be fired multiple times, even after the first rule execution has pushed the filter into the scan operator. Since partition prune has to build the vectors to contain the partition /file / directory information, to invoke the partition prune rule unnecessary may lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to reduce the chance of hitting OOM exception, while the partition prune rule is executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)