You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Aman Sinha (JIRA)" <ji...@apache.org> on 2015/09/10 21:01:45 UTC
[jira] [Created] (DRILL-3759) Make partition pruning multi-phased
to reduce the working set kept in memory
Aman Sinha created DRILL-3759:
---------------------------------
Summary: Make partition pruning multi-phased to reduce the working set kept in memory
Key: DRILL-3759
URL: https://issues.apache.org/jira/browse/DRILL-3759
Project: Apache Drill
Issue Type: Improvement
Components: Query Planning & Optimization
Affects Versions: 1.1.0
Reporter: Aman Sinha
Assignee: Jinfeng Ni
Currently, partition pruning gets all file names in the table and applies the pruning. Suppose the files are spread out over several directories and there is a filter on dirN, this is not efficient - both in terms of elapsed time and memory usage. This has been seen in a few use cases recently.
Wherever possible, we should ideally perform the pruning in N steps (where N is the number of directory levels referenced in the filter conditions):
1. Get the directory and filenames at level i
2. Materialize into the in-memory table
3. Apply interpreter-based evaluation of filter condition
4. Determine qualifying directories, increment i and repeat from step 1
This multi phase approach may not be possible for certain types of filters - e,g for disjunctions. This analysis needs to be done.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)