[jira] [Resolved] (IMPALA-15413) Truncate Parquet page-index min/max values on UTF-8 character boundaries

["Dániel Gábor Vankó (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dániel Gábor Vankó resolved IMPALA-15413.
-----------------------------------------
    Resolution: Fixed

> Truncate Parquet page-index min/max values on UTF-8 character boundaries
> ------------------------------------------------------------------------
>
>                 Key: IMPALA-15413
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15413
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>            Reporter: Dániel Gábor Vankó
>            Assignee: Dániel Gábor Vankó
>            Priority: Major
>              Labels: parquet, utf8
>             Fix For: Impala 5.0.0
>
>
> When Impala writes the Parquet page index, it truncates per-page min/max string statistics to PAGE_INDEX_MAX_STRING_LENGTH via TruncateDown() / TruncateUp() in be/src/exec/parquet/hdfs-parquet-table-writer.cc. These truncate at an arbitrary byte offset, which can split a multi-byte UTF-8 sequence and produce a min/max value that is not itself valid UTF-8.
> The Parquet spec requires the truncated min/max in the column/page index to remain valid values of the column's logical type:
> https://github.com/apache/parquet-format/blob/219e3f12a62f9476e830c21e26d030d231f7c017/src/main/thrift/parquet.thrift#L300
> When Impala annotates STRING columns as UTF-8 by default (IMPALA-12675), writing an invalid-UTF-8 min/max into a UTF-8-annotated column produces internally inconsistent metadata and can confuse readers that trust the annotation.
> Proposal:
> - Add UTF-8-aware variants of TruncateDown/TruncateUp that truncate on character boundaries (and, for TruncateUp, increment the last whole code point correctly). Reference implementation: parquet-java's BinaryTruncator:
> https://github.com/apache/parquet-java/blob/master/parquet-column/src/main/java/org/apache/parquet/internal/column/columnindex/BinaryTruncator.java
> - Check other engines' behavior for parity.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org