You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Raunaq Morarka (Jira)" <ji...@apache.org> on 2023/10/18 09:48:00 UTC
[jira] [Commented] (PARQUET-411) Format: Add a flag when min/max are truncated
[ https://issues.apache.org/jira/browse/PARQUET-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776580#comment-17776580 ]
Raunaq Morarka commented on PARQUET-411:
----------------------------------------
I believe this issue is addressed by the changes done for https://issues.apache.org/jira/browse/PARQUET-2352
cc: [~wgtmac]
> Format: Add a flag when min/max are truncated
> ---------------------------------------------
>
> Key: PARQUET-411
> URL: https://issues.apache.org/jira/browse/PARQUET-411
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
> Affects Versions: format-2.3.1
> Reporter: Ryan Blue
> Priority: Major
>
> PARQUET-372 drops page and column chunk stats when values are larger than 4k to avoid storing very large values in page headers and the file footer. An alternative approach is to truncate the values, which would still allow filtering on page stats. The problem with truncating values is that the value in stats may not be the true min or max so engines that use these values as the result of aggregations like {{min(col)}} would return incorrect data. We should consider adding metadata to allow truncating values for filtering that captures the fact that the values have been modified.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)