[Propose] New PXF profile optimized for ORC (predicate pushdown)

[Shivram Mani <sh...@gmail.com>]

I've created the following jira HAWQ-866
<https://issues.apache.org/jira/browse/HAWQ-886> which is focussed on
improving/enhancing the existing PXF profile to read ORC files. The goal is
to make use of the underlying ORC reader's capability of supporting
predicate push-down among others.

Presto has also contributed an alternative ORC reader which provides both
predicate push down and Lazy reads
https://code.facebook.com/posts/370832626374903/even-faster-data-at-the-speed-of-presto-orc/
.

Will be evaluating both the options as part of this effort.


-- 
shivram mani

Re: [Propose] New PXF profile optimized for ORC (predicate pushdown)

[Hubert Zhang <hz...@pivotal.io>]

+1 for lazy reader, It can save a lot of decompression and
deserialization(CPU bound) time.

On Wed, Jul 6, 2016 at 7:00 AM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> On Tue, Jul 5, 2016 at 12:01 PM, Shivram Mani <sh...@gmail.com>
> wrote:
> > I've created the following jira HAWQ-866
> > <https://issues.apache.org/jira/browse/HAWQ-886> which is focussed on
> > improving/enhancing the existing PXF profile to read ORC files. The goal
> is
> > to make use of the underlying ORC reader's capability of supporting
> > predicate push-down among others.
> >
> > Presto has also contributed an alternative ORC reader which provides both
> > predicate push down and Lazy reads
> >
> https://code.facebook.com/posts/370832626374903/even-faster-data-at-the-speed-of-presto-orc/
> > .
> >
> > Will be evaluating both the options as part of this effort.
>
> Great to see this effort! Do you plan to come up with any kind of
> benchmark to
> be able to compare the native ORC reader vs. PXF ORC reader performance
> and capabilities?
>
> Or does it really all just boil down to TPC?
>
> Thanks,
> Roman.
>



-- 
Thanks

Hubert Zhang

Re: [Propose] New PXF profile optimized for ORC (predicate pushdown)

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Tue, Jul 5, 2016 at 12:01 PM, Shivram Mani <sh...@gmail.com> wrote:
> I've created the following jira HAWQ-866
> <https://issues.apache.org/jira/browse/HAWQ-886> which is focussed on
> improving/enhancing the existing PXF profile to read ORC files. The goal is
> to make use of the underlying ORC reader's capability of supporting
> predicate push-down among others.
>
> Presto has also contributed an alternative ORC reader which provides both
> predicate push down and Lazy reads
> https://code.facebook.com/posts/370832626374903/even-faster-data-at-the-speed-of-presto-orc/
> .
>
> Will be evaluating both the options as part of this effort.

Great to see this effort! Do you plan to come up with any kind of benchmark to
be able to compare the native ORC reader vs. PXF ORC reader performance
and capabilities?

Or does it really all just boil down to TPC?

Thanks,
Roman.