[jira] [Created] (IMPALA-15411) Add an is_valid_utf8(STRING) builtin to detect non-UTF-8 values

["Dániel Gábor Vankó (Jira)" <ji...@apache.org>]

Dániel Gábor Vankó created IMPALA-15411:
-------------------------------------------

             Summary:  Add an is_valid_utf8(STRING) builtin to detect non-UTF-8 values
                 Key: IMPALA-15411
                 URL: https://issues.apache.org/jira/browse/IMPALA-15411
             Project: IMPALA
          Issue Type: New Feature
            Reporter: Dániel Gábor Vankó
            Assignee: Dániel Gábor Vankó


With UTF-8 validation on the Parquet write path enabled by default (IMPALA-12675), an INSERT/CTAS into a UTF-8-annotated STRING column now fails with PARQUET_INVALID_UTF8_STRING if any value is not valid UTF-8. Users hitting this have no easy SQL-level way to locate the offending rows in their source data.

Expose a scalar builtin:

is_valid_utf8(STRING) -> BOOLEAN

returning true for valid UTF-8 (empty string / NULL handling to be defined), so users can run e.g.:
{code:java}
SELECT * FROM t WHERE NOT is_valid_utf8(s);{code}
Proposal:
- Back it with the existing impala::IsValidUtf8() in be/src/util/utf8-util.\{h,cc} (already SIMD-accelerated via simdutf, added in IMPALA-12675).
- Register the scalar function in the builtins (frontend builtins-db/function registry + backend function implementation).
- Reference it from the PARQUET_INVALID_UTF8_STRING error message and the impala_incompatible_changes.xml migration note so users are pointed at the tool.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)