[jira] [Created] (HIVE-22980) Support custom path filter for ORC tables

["Oleksiy Sayankin (Jira)" <ji...@apache.org>]

Oleksiy Sayankin created HIVE-22980:
---------------------------------------

             Summary: Support custom path filter for ORC tables
                 Key: HIVE-22980
                 URL: https://issues.apache.org/jira/browse/HIVE-22980
             Project: Hive
          Issue Type: New Feature
          Components: ORC
            Reporter: Oleksiy Sayankin
            Assignee: Oleksiy Sayankin


The customer is looking for an option to specify custom path filter for ORC tables. Please find the details below from customer requirement.

Problem Statement/Approach in customer words :

{quote} 
Currently, Orc file input format does not take in path filters set in the property "mapreduce.input.pathfilter.class" OR " mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc files. 

AcidUtils class has a static filter called "hiddenFilters" which is used by ORC to filter input paths. If we can pass the custom filter classes(set in the property mentioned above) to AcidUtils and replace hiddenFilter with a filter that does an "and" operation over hiddenFilter+customFilters, the filters would work well.

On local testing, mapreduce.input.pathfilter.class seems to be working for Text tables but not for ORC tables.
{quote}

Our analysis:

{{OrcInputFormat}} and {{FileInputFormat}} are different implementations for {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is only respected by {{FileInputFormat}}, but not by any other implementations of {{InputFormat}}. The customer wants to have the ability to filter out rows based on path/filenames, current ORC features like bloomfilters and indexes are not good enough for them to minimize number of disk read operations.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)