You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@orc.apache.org by Kavinder Dhaliwal <ka...@gmail.com> on 2016/10/03 20:58:28 UTC

Supported data types for predicate pushdown

Hi,

I'm working on pushing down filters to ORC but am unclear about the
expected behavior. My understanding is that a search argument filter will
work for data types that have a min/max value and strings can be filtered
via a bloom filter. But what about other data types (date/time, binary
data, etc)?  A comprehensive guide would be helpful to better understand
this

Thanks,
Kavinder

Re: Supported data types for predicate pushdown

Posted by Kavinder Dhaliwal <ka...@gmail.com>.

Thanks Prasanth for the quick response

On Mon, Oct 3, 2016 at 2:02 PM, Prasanth J <j....@gmail.com> wrote:

> No support for binary and complex types yet. Date and timestamp should
> work.
> The way it works is for all supported data types min/max value is checked
> first.
> If it returns true and if the bloom filter exists for that column type
> then bloom filter is checked.
> If min/max itself returns false we don’t check bloom filter.
>
> Thanks
> Prasanth
>
> > On Oct 3, 2016, at 1:58 PM, Kavinder Dhaliwal <ka...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I'm working on pushing down filters to ORC but am unclear about the
> > expected behavior. My understanding is that a search argument filter will
> > work for data types that have a min/max value and strings can be filtered
> > via a bloom filter. But what about other data types (date/time, binary
> > data, etc)?  A comprehensive guide would be helpful to better understand
> > this
> >
> > Thanks,
> > Kavinder
>
>

Re: Supported data types for predicate pushdown

Posted by Prasanth J <j....@gmail.com>.

No support for binary and complex types yet. Date and timestamp should work.
The way it works is for all supported data types min/max value is checked first. 
If it returns true and if the bloom filter exists for that column type then bloom filter is checked.
If min/max itself returns false we don’t check bloom filter.

Thanks
Prasanth

> On Oct 3, 2016, at 1:58 PM, Kavinder Dhaliwal <ka...@gmail.com> wrote:
> 
> Hi,
> 
> I'm working on pushing down filters to ORC but am unclear about the
> expected behavior. My understanding is that a search argument filter will
> work for data types that have a min/max value and strings can be filtered
> via a bloom filter. But what about other data types (date/time, binary
> data, etc)?  A comprehensive guide would be helpful to better understand
> this
> 
> Thanks,
> Kavinder