[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition

["Aman Sinha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207648#comment-15207648 ] 

Aman Sinha commented on DRILL-4530:
-----------------------------------

On a closer look, we can optimize this type of query which has a single partition by changing the selection root after partition pruning rule is applied.  While populating the partition vectors in ParquetPartitionDescriptor, we can keep track of the dir0, dir1 etc. values that have been encountered and if these are unique, then we can change the selection root to  'A/B/C'.  This will subsequently get used by ParquetGroupScan to read the metadata cache file from the subdirectory instead of top level.  

Note that this type of optimization would apply only when we know that a single partition is being selected.  For a general case (e.g IN filter or range predicate on the directory column) other solutions need to be designed.   However, for the general case it is unlikely that a user will compare its performance with an equivalent query written without the partition filter (it is not easy to express arbitrary WHERE condition into a FROM clause). 

> Improve metadata cache performance for queries with single partition 
> ---------------------------------------------------------------------
>
>                 Key: DRILL-4530
>                 URL: https://issues.apache.org/jira/browse/DRILL-4530
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Query Planning & Optimization
>    Affects Versions: 1.6.0
>            Reporter: Aman Sinha
>            Assignee: Aman Sinha
>
> Consider two types of queries which are run with Parquet metadata caching: 
> {noformat}
> query 1:
> SELECT col FROM  `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query1 elapsed time is 9 sec even though both are accessing the same amount of data.  The user expectation is that they should perform roughly the same.  The main difference comes from reading the bigger metadata cache file at the root level 'A' for query2 and then applying the partitioning filter.  query1 reads a much smaller metadata cache file at the subdirectory level. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)