You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Gang Wu (Jira)" <ji...@apache.org> on 2023/10/18 09:25:00 UTC

[jira] [Resolved] (PARQUET-2352) Update parquet format spec to allow truncation of row group min/max stats

     [ https://issues.apache.org/jira/browse/PARQUET-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gang Wu resolved PARQUET-2352.
------------------------------
    Fix Version/s: format-2.10.0
         Assignee: Raunaq Morarka
       Resolution: Fixed

> Update parquet format spec to allow truncation of row group min/max stats
> -------------------------------------------------------------------------
>
>                 Key: PARQUET-2352
>                 URL: https://issues.apache.org/jira/browse/PARQUET-2352
>             Project: Parquet
>          Issue Type: Improvement
>            Reporter: Raunaq Morarka
>            Assignee: Raunaq Morarka
>            Priority: Major
>             Fix For: format-2.10.0
>
>
> Column index stats are explicitly allowed to be truncated [https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L958]
> However, it seems row group min/max stats are not allowed to be truncated [https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L219] although it is not explicitly clarified like in the column index case. This forces implementations to either drop min/max row group stats for columns with long strings and miss opportunities for filtering row groups or seemingly deviate from spec by truncating min/max row group stats.
> https://issues.apache.org/jira/browse/PARQUET-1685 introduced a feature to parquet-mr which allows users to deviate from spec and configure truncation of min/max row group stats. Unfortunately, there is no way for readers to detect whether truncation took place.
> Since the possibility of truncation exists and is not possible to explicitly detect, attempts to pushdown min/max aggregation to parquet have avoided implementing it for string columns (e.g. https://issues.apache.org/jira/browse/SPARK-36645)
> Given the above situation, the spec should be updated to allow truncation of min/max row group stats. This would align the spec with current reality that string column min/max row group stats could be truncated.
> Additionally, a flag could be added to the stats to specify whether min/max stats are truncated. Reader implementations could then safely implement min/max aggregation pushdown to strings for new data going forward by checking the value of this flag. When the flag is not found on existing data then it could be assumed that the data could be truncated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)