[jira] [Commented] (DRILL-3759) Make partition pruning multi-phased to reduce the working set kept in memory

["John Omernik (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138971#comment-15138971 ] 

John Omernik commented on DRILL-3759:
-------------------------------------

I am tossing my +1 on this JIRA any sort of larger tables in Parquet, where you are running accross multiple subdirectories (consider a years worth of data) with multiple files (even say 25 files per day) would result is excessive query planning do to the parser reviewing all files. Even when a prune operation is requested by the user, causing a very poor user experience on these tables. 

> Make partition pruning multi-phased to reduce the working set kept in memory
> ----------------------------------------------------------------------------
>
>                 Key: DRILL-3759
>                 URL: https://issues.apache.org/jira/browse/DRILL-3759
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Query Planning & Optimization
>    Affects Versions: 1.1.0
>            Reporter: Aman Sinha
>            Assignee: Mehant Baid
>             Fix For: Future
>
>
> Currently, partition pruning gets all file names in the table and applies the pruning.  Suppose the files are spread out over several directories and there is a filter  on dirN,  this is not efficient - both in terms of elapsed time and memory usage.  This has been seen in a few use cases recently. 
> Wherever possible, we should ideally perform the pruning in N steps (where N is the number of directory levels referenced in the filter conditions):   
>   1. Get the directory and  filenames at level i
>   2. Materialize into the in-memory table 
>   3. Apply interpreter-based evaluation of filter condition
>   4. Determine qualifying directories, increment i and repeat from step 1
>  
> This multi phase approach may not be possible for certain types of filters - e,g for disjunctions. This analysis needs to be done. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)