[jira] [Updated] (PARQUET-1217) Incorrect handling of missing values in Statistics

["Gabor Szadovszky (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PARQUET-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabor Szadovszky updated PARQUET-1217:
--------------------------------------
    Fix Version/s: 1.8.3

> Incorrect handling of missing values in Statistics
> --------------------------------------------------
>
>                 Key: PARQUET-1217
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1217
>             Project: Parquet
>          Issue Type: Bug
>    Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0
>            Reporter: Gabor Szadovszky
>            Assignee: Gabor Szadovszky
>            Priority: Major
>             Fix For: 1.10.0, 1.8.3
>
>
> As per the parquet-format specs the min/max values in statistics are optional. Therefore, it is possible to have {{numNulls}} in {{Statistics}} while we don't have min/max values. In {{StatisticsFilter}} we rely on the method [StatisticsFilter.isAllNulls(ColumnChunkMetaData)|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java#L90] to handle the case of {{null}} min/max values which is not correct due to the described scenario. 
>  We shall check {{Statistics.hasNonNullValue()}} any time before using the actual min/max values.
> In addition we don't check if the {{null_count}} is set or not when reading from the parquet file. We simply use the value which is {{0}} in case of unset. In the parquet-mr side the {{Statistics}} object uses the value {{0}} to sign that the {{num_nulls}} is unset. It is incorrect if we are searching for null values and we falsely drop a column chunk thinking there are no null values but the field in the statistics was simply unset.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)