[jira] [Commented] (HAWQ-886) Support PXF filter push down for ORC

["Shivram Mani (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HAWQ-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362999#comment-15362999 ] 

Shivram Mani commented on HAWQ-886:
-----------------------------------

Based on an initial performance evaluation on Reading ORC files with features turned on/off
Data ~ 500000 rows 6 columns with primitive data types

Read using naive row based reader ~ 1500ms
Read using Vectorizedbatch reader (default 1024 batch size) ~ 1000ms
Read with filter (7500 rows) ~ 750ms
Read without filter with column projection ~ 850ms
Read with filter with column projection ~ 600ms

Over all we can achieve roughly a 60% speedup over a rather small dataset.

> Support PXF filter push down for ORC
> ------------------------------------
>
>                 Key: HAWQ-886
>                 URL: https://issues.apache.org/jira/browse/HAWQ-886
>             Project: Apache HAWQ
>          Issue Type: New Feature
>          Components: PXF
>            Reporter: Shivram Mani
>            Assignee: Shivram Mani
>             Fix For: 2.1.0
>
>
> Currently PXF via the Hive profile doesn’t pass any of the filter information down while accessing ORC files. The only filter that is possible right now is at the level of partition and is generically done for all Hive tables.
> ORC internally contains file level, stripe level and row level statistics including information such as min,max values etc. For more information refer to https://orc.apache.org/docs/indexes.html
> The proposal here is to possibly introduce a new profile optimized for ORC files and to leverage these stats to improve the performance of HAWQ queries with predicates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)