[jira] [Commented] (DRILL-5796) Filter pruning for multi rowgroup parquet file

["Robert Hou (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736835#comment-16736835 ] 

Robert Hou commented on DRILL-5796:
-----------------------------------

Is there any limits for this feature?  I am testing it with roughly 250 files organized in roughly 20 directories.  There should only be one file that matches the query.  But the Scan operator shows that all 250 files in 20 directories need to be scanned.  Perhaps the optimizer decides not to scan row group stats after some threshold?

> Filter pruning for multi rowgroup parquet file
> ----------------------------------------------
>
>                 Key: DRILL-5796
>                 URL: https://issues.apache.org/jira/browse/DRILL-5796
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Parquet
>            Reporter: Damien Profeta
>            Assignee: Jean-Blas IMBERT
>            Priority: Major
>              Labels: ready-to-commit
>             Fix For: 1.15.0
>
>
> Today, filter pruning use the file name as the partitioning key. This means you can remove a partition only if the whole file is for the same partition. With parquet, you can prune the filter if the rowgroup make a partition of your dataset as the unit of work if the rowgroup not the file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)