[jira] [Created] (IMPALA-15414) Validate CHAR/VARCHAR values as UTF-8 when writing Parquet

["Dániel Gábor Vankó (Jira)" <ji...@apache.org>]

Dániel Gábor Vankó created IMPALA-15414:
-------------------------------------------

             Summary: Validate CHAR/VARCHAR values as UTF-8 when writing Parquet
                 Key: IMPALA-15414
                 URL: https://issues.apache.org/jira/browse/IMPALA-15414
             Project: IMPALA
          Issue Type: Improvement
          Components: Backend
            Reporter: Dániel Gábor Vankó


IMPALA-12675 makes Impala annotate STRING Parquet columns as UTF-8 by default and validate their values on the write path, rejecting non-UTF-8 bytes with PARQUET_INVALID_UTF8_STRING. CHAR and VARCHAR columns are always annotated as UTF-8 (unconditionally, in ParquetMetadataUtils::FillSchemaElement()), but unlike STRING they are not validated — so it is currently possible to write non-UTF-8 bytes into a CHAR/VARCHAR Parquet column while still emitting a UTF-8 annotation, i.e. a file whose metadata does not match its contents.

This asymmetry is deliberate for now: removing the escape hatch for CHAR/VARCHAR would be a larger behavior change and deserves its own patch. This ticket tracks doing that consistently.

Proposal:
- Validate CHAR/VARCHAR values as UTF-8 on the Parquet write path, consistent with STRING (BaseColumnWriter::ShouldValidateUtf8() in be/src/exec/parquet/hdfs-parquet-table-writer.cc currently restricts validation to TYPE_STRING).
- Decide the opt-out story: unlike STRING there is no per-column annotation toggle for CHAR/VARCHAR, so validation must not leave users without a way to store legacy data. Consider gating behind an option or documenting the migration path.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org