You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Dániel Gábor Vankó (Jira)" <ji...@apache.org> on 2026/09/23 15:52:00 UTC
[jira] [Updated] (IMPALA-15414) Validate CHAR/VARCHAR values as UTF-8 when writing Parquet
[ https://issues.apache.org/jira/browse/IMPALA-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dániel Gábor Vankó updated IMPALA-15414:
----------------------------------------
Labels: parquet utf8 (was: UTF-8 parquet)
> Validate CHAR/VARCHAR values as UTF-8 when writing Parquet
> ----------------------------------------------------------
>
> Key: IMPALA-15414
> URL: https://issues.apache.org/jira/browse/IMPALA-15414
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Dániel Gábor Vankó
> Priority: Major
> Labels: parquet, utf8
>
> IMPALA-12675 makes Impala annotate STRING Parquet columns as UTF-8 by default and validate their values on the write path, rejecting non-UTF-8 bytes with PARQUET_INVALID_UTF8_STRING. CHAR and VARCHAR columns are always annotated as UTF-8 (unconditionally, in ParquetMetadataUtils::FillSchemaElement()), but unlike STRING they are not validated — so it is currently possible to write non-UTF-8 bytes into a CHAR/VARCHAR Parquet column while still emitting a UTF-8 annotation, i.e. a file whose metadata does not match its contents.
> This asymmetry is deliberate for now: removing the escape hatch for CHAR/VARCHAR would be a larger behavior change and deserves its own patch. This ticket tracks doing that consistently.
> Proposal:
> - Validate CHAR/VARCHAR values as UTF-8 on the Parquet write path, consistent with STRING (BaseColumnWriter::ShouldValidateUtf8() in be/src/exec/parquet/hdfs-parquet-table-writer.cc currently restricts validation to TYPE_STRING).
> - Decide the opt-out story: unlike STRING there is no per-column annotation toggle for CHAR/VARCHAR, so validation must not leave users without a way to store legacy data. Consider gating behind an option or documenting the migration path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org