Predicate pushdown/indexing on ORC file

[Avrilia Floratou <av...@gmail.com>]

Hi all,

I'm using hive-12. I have a file that contains 10 integer columns stored in
ORC format. The ORC file is zlib compressed and indexing is enabled.
I'm running a simple select count(*) with a predicate of the form (Col1 =0
OR col2 = 0 etc). The predicate touches all 10 columns but its selectivity
is 0 (none of the records in the file
satisfy the predicate). I have set the hive.optimize.ppd property to true.
For this query basically none of the stripes should be read. However after
the MR job finishes, I can see that the whole file has been read. Why is
that? Is there any other parameter that I should enable for predicate
pushdown/indexing to work?

Thanks,
Avrilia

Re: Predicate pushdown/indexing on ORC file

[Prasanth Jayachandran <pj...@hortonworks.com>]

AFAIK, ORC uses “hive.optimize.index.filter” hive config to enable predicate pushdown. Can you please try by setting hive.optimize.index.filter to true?

Thanks
Prasanth Jayachandran

On Nov 7, 2013, at 4:04 PM, Avrilia Floratou <av...@gmail.com> wrote:

> Hi all,
> 
> I'm using hive-12. I have a file that contains 10 integer columns stored in ORC format. The ORC file is zlib compressed and indexing is enabled.
> I'm running a simple select count(*) with a predicate of the form (Col1 =0 OR col2 = 0 etc). The predicate touches all 10 columns but its selectivity is 0 (none of the records in the file 
> satisfy the predicate). I have set the hive.optimize.ppd property to true. For this query basically none of the stripes should be read. However after the MR job finishes, I can see that the whole file has been read. Why is that? Is there any other parameter that I should enable for predicate pushdown/indexing to work?
> 
> Thanks,
> Avrilia


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.