[jira] [Updated] (HIVE-30182) Parquet: a leaf without its runtime value drops the whole row-group filter

["Denys Kuzmenko (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-30182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Denys Kuzmenko updated HIVE-30182:
----------------------------------
    Summary: Parquet: a leaf without its runtime value drops the whole row-group filter  (was: Parquet: a leaf without its runtime value drops the whole)

> Parquet: a leaf without its runtime value drops the whole row-group filter
> --------------------------------------------------------------------------
>
>                 Key: HIVE-30182
>                 URL: https://issues.apache.org/jira/browse/HIVE-30182
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Parquet, Query Processor
>            Reporter: Denys Kuzmenko
>            Priority: Major
>              Labels: pull-request-available
>
> `ParquetFilterPredicateConverter.toFilterPredicate` returns null (no row-group filter at all) when any leaf of the search argument cannot be built. A leaf whose literal is a runtime value (DynamicValue, e.g. a semijoin's min/max) that is not available when the reader builds its filter is such a leaf: BETWEEN/IN get an empty literal list from SearchArgumentImpl.getLiteralList and FilterPredicateLeafBuilder throws "Not able to build 'between' operation filter with [] which needs two literals"; a single-literal leaf throws NoDynamicValuesException. The static leaves of the same conjunction are lost with it, and every failure logs an ERROR with a stack trace: 45,927 of them in one TPC-DS 10 TB run, all from LLAP IO-Elevator threads reading semijoin-filtered Parquet scans.
> Fix: such a leaf filters nothing. The search argument pushes NOT down to the leaves, so dropping a leaf from a conjunction only weakens the filter; a disjunction containing it still filters nothing.
> Not covered here: the runtime values are not reachable from the LLAP IO threads at all, so a semijoin's min/max never prunes Parquet row groups on that path (separate issue).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)