[jira] [Updated] (PARQUET-1217) Incorrect handling of missing values in Statistics

["Gabor Szadovszky (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PARQUET-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabor Szadovszky updated PARQUET-1217:
--------------------------------------
    Summary: Incorrect handling of missing values in Statistics  (was: Missing check for null min/max values in StatisticsFilter)

> Incorrect handling of missing values in Statistics
> --------------------------------------------------
>
>                 Key: PARQUET-1217
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1217
>             Project: Parquet
>          Issue Type: Bug
>    Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0
>            Reporter: Gabor Szadovszky
>            Assignee: Gabor Szadovszky
>            Priority: Major
>
> As per the parquet-format specs the min/max values in statistics are optional. Therefore, it is possible to have {{numNulls}} in {{Statistics}} while we don't have min/max values. In {{StatisticsFilter}} we rely on the method [StatisticsFilter.isAllNulls(ColumnChunkMetaData)|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java#L90] to handle the case of {{null}} min/max values which is not correct due to the described scenario. 
> We shall check {{Statistics.hasNonNullValue()}} any time before using the actual min/max values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)