[jira] [Commented] (HIVE-26013) Parquet predicate filters are not properly propogated to task configs at runtime

["Rajesh Balamohan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-26013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503414#comment-17503414 ] 

Rajesh Balamohan commented on HIVE-26013:
-----------------------------------------

This could be a false alarm as blocks are already getting filtered out for the split.

Ref: [https://github.com/apache/hive/blob/master/ql%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Fio%2Fparquet%2FParquetRecordReaderBase.java#L123]

 

> Parquet predicate filters are not properly propogated to task configs at runtime
> --------------------------------------------------------------------------------
>
>                 Key: HIVE-26013
>                 URL: https://issues.apache.org/jira/browse/HIVE-26013
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>            Priority: Major
>              Labels: performance
>
> Hive ParquetRecordReader sets the predicate filter in the config for parquet libs to read.
> Ref: [https://github.com/apache/hive/blob/master/ql%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Fio%2Fparquet%2FParquetRecordReaderBase.java#L188]
> {code:java}
>  ParquetInputFormat.setFilterPredicate(conf, p);
> {code}
> This internally sets {color:#FF0000}"parquet.private.read.filter.predicate" {color}variable in config.
> Ref: [https://github.com/apache/parquet-mr/blob/master/parquet-hadoop%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fparquet%2Fhadoop%2FParquetInputFormat.java#L231]
> Config set in compilation phase isn't visible at runtime for the tasks. This causes filters to be lost and tasks run with excessive IO.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)