[jira] [Created] (HIVE-26013) Parquet predicate filters are not properly propogated to task configs at runtime

["Rajesh Balamohan (Jira)" <ji...@apache.org>]

Rajesh Balamohan created HIVE-26013:
---------------------------------------

             Summary: Parquet predicate filters are not properly propogated to task configs at runtime
                 Key: HIVE-26013
                 URL: https://issues.apache.org/jira/browse/HIVE-26013
             Project: Hive
          Issue Type: Bug
            Reporter: Rajesh Balamohan


Hive ParquetRecordReader sets the predicate filter in the config for parquet libs to read.

Ref: [https://github.com/apache/hive/blob/master/ql%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Fio%2Fparquet%2FParquetRecordReaderBase.java#L188]
{code:java}
 ParquetInputFormat.setFilterPredicate(conf, p);
{code}
This internally sets {color:#FF0000}"parquet.private.read.filter.predicate" {color}variable in config.

Ref: [https://github.com/apache/parquet-mr/blob/master/parquet-hadoop%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fparquet%2Fhadoop%2FParquetInputFormat.java#L231]

Config set in compilation phase isn't visible at runtime for the tasks. This causes filters to be lost and tasks run with excessive IO.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)