[jira] [Created] (FLINK-39600) FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

["Gustavo de Morais (Jira)" <ji...@apache.org>]

Gustavo de Morais created FLINK-39600:
-----------------------------------------

             Summary: FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities
                 Key: FLINK-39600
                 URL: https://issues.apache.org/jira/browse/FLINK-39600
             Project: Flink
          Issue Type: New Feature
          Components: Table SQL / API
            Reporter: Gustavo de Morais
            Assignee: Gustavo de Morais


{{CAST(bytes AS STRING)}} today silently replaces invalid UTF-8 with the Unicode replacement character {{{}U+FFFD{}}}. The substitution is irreversible and produces no warning - pipelines keep running while data is permanently corrupted downstream. This also blocks engine optimizations that need injective guarantees (e.g. upsert key propagation through {{{}BINARY -> STRING{}}}).

[FLIP-568|https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities] addresses this by:
 # Making {{CAST(bytes AS STRING)}} throw on invalid UTF-8. {{TRY_CAST}} returns {{{}NULL{}}}. A migration flag restores the legacy behavior.
 # Adding two SQL functions:
 ** {{IS_VALID_UTF8(bytes) -> BOOLEAN}} for routing invalid records to a dead-letter sink
 ** {{MAKE_VALID_UTF8(bytes) -> STRING}} as the explicit, opt-in substitution recipe
 # Adding {{StringData.fromUtf8Bytes(byte[])}} connector API that validates at ingestion and throws on invalid input.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)