[jira] [Created] (PIG-3760) Partition pruning for ORC and Parquet

["Andrew Musselman (JIRA)" <ji...@apache.org>]

Andrew Musselman created PIG-3760:
-------------------------------------

             Summary: Partition pruning for ORC and Parquet
                 Key: PIG-3760
                 URL: https://issues.apache.org/jira/browse/PIG-3760
             Project: Pig
          Issue Type: New Feature
    Affects Versions: 0.12.0
            Reporter: Andrew Musselman
            Priority: Minor
             Fix For: 0.13.0


>From the conversation on dev@pig:

"Partition pruning for ORC is not addressed in PIG-3558. We will need
to do partition pruning for both ORC and Parquet in a new ticket.
Curently there is no interface to deal with this kind of pushdown
(LoadMetadata.setPartitionFilter push the filter to loader, but remove
the filter statement, for ORC/Parquet, filter is a hint, and we need
to do the filter again in Pig even it is pushed to loader), we will
need to define a new interface for that. You are welcome to initiate
the work. I know Aniket is also interested in doing that, so be sure
the talk with him about this work.

Thanks,
Daniel



On Mon, Feb 10, 2014 at 11:42 AM, Andrew Musselman
<an...@gmail.com> wrote:
> I had a chat with a couple people last week about a feature request for
> Pig:  in a "where" or "filter" clause, when loading an ORC file, to skip
> directly to the right offset instead of scanning the whole file."



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)