You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gustavo de Morais <gu...@gmail.com> on 2026/03/19 14:23:03 UTC

[DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Hi everyone,

Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8 byte
with U+FFFD (?). The substitution is irreversible and produces no warning -
the pipeline keeps running while data is permanently corrupted
downstream. This also means that a CAST from BYTES → STRING → BYTES is not
idempotent, which prevents the engine from applying certain optimizations.
For example, for preserving upsert keys after such CASTs.

I'd like to start a discussion around defining and improving the default
behavior. I've written a short FLIP [1] proposing new utility functions to
handle this explicitly - similar to what other engines like Spark already
do - and changing the default behavior to throw an error instead of
silently corrupting data, while giving users clear options to deal with
invalid bytes.

Looking forward to your feedback and thoughts.

Kind regards,
Gustavo

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities

Re: Re: Re: [DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Posted by Gustavo de Morais <gu...@gmail.com>.
Hi Xuyang, Timo and Sergey,

Thanks for the feedback! I'm kicking off the vote now.

Kind regards,
Gustavo

On Tue, 24 Mar 2026 at 10:59, Xuyang <xy...@163.com> wrote:

> Hi, Gustavo.
> Looks good to update the Internal API independently. +1.
>
>
>
>
>
> --
>
>     Best!
>     Xuyang
>
>
>
> At 2026-03-23 20:50:02, "Gustavo de Morais" <gu...@gmail.com>
> wrote:
> >Hi Sergey,
> >
> >Thanks for the message! I took a look at our support for UTF_16 and I
> don't
> >think any additional UTF_16 support is necessary for the scope of this
> >FLIP. For UTF-16 users should use Decode instead of Cast. Flink stores all
> >strings as UTF-8 internally, so DECODE(bytes, 'UTF-16') already handles
> >that case - it converts to a Java String and re-encodes as UTF-8 before
> >storage. Since the internal representation is always UTF-8, the validation
> >problem is fundamentally a UTF-8 concern and the functions are correctly
> >named. Adding validation functions support for all the character sets we
> >support for DECODE (‘US-ASCII’, ‘ISO-8859-1’, ‘UTF-8’, ‘UTF-16BE’,
> >‘UTF-16LE’, ‘UTF-16) might be something useful for some less frequent use
> >cases but I'd say this is out-of-scope for this FLIP.
> >
> >Kind regards,
> >Gustavo
> >
> >
> >On Fri, 20 Mar 2026 at 20:24, Sergey Nuyanzin <sn...@gmail.com>
> wrote:
> >
> >> Hi Gustavo
> >> thanks for the proposal
> >>
> >> I noticed that you are proposing usage of UTF8 in names (default cast
> >> to string is also using utf8)
> >> however I wonder if it makes sense to introduce similar utf16 similar
> >> functions as Flink supports this as well?
> >>
> >> On Fri, Mar 20, 2026 at 8:09 PM Gustavo de Morais
> >> <gu...@gmail.com> wrote:
> >> >
> >> > Hi Xuyang and Timo,
> >> >
> >> > Thanks for the positive feedback! Regarding your suggestions, Xuyang:
> >> >
> >> > 1. Yes, good point - we should add the fromUtf8Bytes(byte[], int, int)
> >> > overload as well.
> >> > 2. This is also relevant. If we want to do validation during ingestion
> >> > time, this might have performance implications. Since these are
> @Internal
> >> > APIs, they can be changed independently from the FLIP afterwards if it
> >> > makes sense. What's your opinion?
> >> >
> >> > Kind regards,
> >> > Gustavo
> >> >
> >> > On Fri, 20 Mar 2026 at 08:44, Xuyang <xy...@163.com> wrote:
> >> >
> >> > > Hi, Gustavo.
> >> > > Great catch! Thanks for driving this FLIP. Overall LGTM. I just have
> >> two
> >> > > minor points I'd like to confirm with you.
> >> > > 1. Should we also add the overload function `fromUtf8Bytes(byte[],
> int,
> >> > > int)` in StringData?
> >> > > 2. Callers like `ColumnarRowData#getString` and
> >> > > `ColumnarArrayData#getString`  call `StringData.fromBytes` directly.
> >> Should
> >> > > these call sites be migrated in a follow-up, or intentionally left
> >> as-is?
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > >     Best!
> >> > >     Xuyang
> >> > >
> >> > >
> >> > >
> >> > > At 2026-03-19 22:37:28, "Timo Walther" <tw...@apache.org> wrote:
> >> > > >Hi Gustavo,
> >> > > >
> >> > > >thank you for this excellent design document. And thanks for
> >> discovering
> >> > > >this data loss and driving the investigation. We should definitely
> fix
> >> > > >this shortcoming. Also looking at other vendors, it is definitly a
> >> cause
> >> > > >for false assumptions that lead to hard-to-debug inconsistencies.
> >> > > >
> >> > > >+1 for this proposal.
> >> > > >
> >> > > >Cheers,
> >> > > >Timo
> >> > > >
> >> > > >
> >> > > >On 19.03.26 15:23, Gustavo de Morais wrote:
> >> > > >> Hi everyone,
> >> > > >>
> >> > > >> Currently, CAST(bytes AS STRING) silently replaces any invalid
> UTF-8
> >> > > byte
> >> > > >> with U+FFFD (?). The substitution is irreversible and produces no
> >> > > warning -
> >> > > >> the pipeline keeps running while data is permanently corrupted
> >> > > >> downstream. This also means that a CAST from BYTES → STRING →
> BYTES
> >> is
> >> > > not
> >> > > >> idempotent, which prevents the engine from applying certain
> >> > > optimizations.
> >> > > >> For example, for preserving upsert keys after such CASTs.
> >> > > >>
> >> > > >> I'd like to start a discussion around defining and improving the
> >> default
> >> > > >> behavior. I've written a short FLIP [1] proposing new utility
> >> functions
> >> > > to
> >> > > >> handle this explicitly - similar to what other engines like Spark
> >> > > already
> >> > > >> do - and changing the default behavior to throw an error instead
> of
> >> > > >> silently corrupting data, while giving users clear options to
> deal
> >> with
> >> > > >> invalid bytes.
> >> > > >>
> >> > > >> Looking forward to your feedback and thoughts.
> >> > > >>
> >> > > >> Kind regards,
> >> > > >> Gustavo
> >> > > >>
> >> > > >> [1]
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities
> >> > > >>
> >> > >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Sergey
> >>
>

Re:Re: Re: [DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Posted by Xuyang <xy...@163.com>.
Hi, Gustavo. 
Looks good to update the Internal API independently. +1.





--

    Best!
    Xuyang



At 2026-03-23 20:50:02, "Gustavo de Morais" <gu...@gmail.com> wrote:
>Hi Sergey,
>
>Thanks for the message! I took a look at our support for UTF_16 and I don't
>think any additional UTF_16 support is necessary for the scope of this
>FLIP. For UTF-16 users should use Decode instead of Cast. Flink stores all
>strings as UTF-8 internally, so DECODE(bytes, 'UTF-16') already handles
>that case - it converts to a Java String and re-encodes as UTF-8 before
>storage. Since the internal representation is always UTF-8, the validation
>problem is fundamentally a UTF-8 concern and the functions are correctly
>named. Adding validation functions support for all the character sets we
>support for DECODE (‘US-ASCII’, ‘ISO-8859-1’, ‘UTF-8’, ‘UTF-16BE’,
>‘UTF-16LE’, ‘UTF-16) might be something useful for some less frequent use
>cases but I'd say this is out-of-scope for this FLIP.
>
>Kind regards,
>Gustavo
>
>
>On Fri, 20 Mar 2026 at 20:24, Sergey Nuyanzin <sn...@gmail.com> wrote:
>
>> Hi Gustavo
>> thanks for the proposal
>>
>> I noticed that you are proposing usage of UTF8 in names (default cast
>> to string is also using utf8)
>> however I wonder if it makes sense to introduce similar utf16 similar
>> functions as Flink supports this as well?
>>
>> On Fri, Mar 20, 2026 at 8:09 PM Gustavo de Morais
>> <gu...@gmail.com> wrote:
>> >
>> > Hi Xuyang and Timo,
>> >
>> > Thanks for the positive feedback! Regarding your suggestions, Xuyang:
>> >
>> > 1. Yes, good point - we should add the fromUtf8Bytes(byte[], int, int)
>> > overload as well.
>> > 2. This is also relevant. If we want to do validation during ingestion
>> > time, this might have performance implications. Since these are @Internal
>> > APIs, they can be changed independently from the FLIP afterwards if it
>> > makes sense. What's your opinion?
>> >
>> > Kind regards,
>> > Gustavo
>> >
>> > On Fri, 20 Mar 2026 at 08:44, Xuyang <xy...@163.com> wrote:
>> >
>> > > Hi, Gustavo.
>> > > Great catch! Thanks for driving this FLIP. Overall LGTM. I just have
>> two
>> > > minor points I'd like to confirm with you.
>> > > 1. Should we also add the overload function `fromUtf8Bytes(byte[], int,
>> > > int)` in StringData?
>> > > 2. Callers like `ColumnarRowData#getString` and
>> > > `ColumnarArrayData#getString`  call `StringData.fromBytes` directly.
>> Should
>> > > these call sites be migrated in a follow-up, or intentionally left
>> as-is?
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > >     Best!
>> > >     Xuyang
>> > >
>> > >
>> > >
>> > > At 2026-03-19 22:37:28, "Timo Walther" <tw...@apache.org> wrote:
>> > > >Hi Gustavo,
>> > > >
>> > > >thank you for this excellent design document. And thanks for
>> discovering
>> > > >this data loss and driving the investigation. We should definitely fix
>> > > >this shortcoming. Also looking at other vendors, it is definitly a
>> cause
>> > > >for false assumptions that lead to hard-to-debug inconsistencies.
>> > > >
>> > > >+1 for this proposal.
>> > > >
>> > > >Cheers,
>> > > >Timo
>> > > >
>> > > >
>> > > >On 19.03.26 15:23, Gustavo de Morais wrote:
>> > > >> Hi everyone,
>> > > >>
>> > > >> Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8
>> > > byte
>> > > >> with U+FFFD (?). The substitution is irreversible and produces no
>> > > warning -
>> > > >> the pipeline keeps running while data is permanently corrupted
>> > > >> downstream. This also means that a CAST from BYTES → STRING → BYTES
>> is
>> > > not
>> > > >> idempotent, which prevents the engine from applying certain
>> > > optimizations.
>> > > >> For example, for preserving upsert keys after such CASTs.
>> > > >>
>> > > >> I'd like to start a discussion around defining and improving the
>> default
>> > > >> behavior. I've written a short FLIP [1] proposing new utility
>> functions
>> > > to
>> > > >> handle this explicitly - similar to what other engines like Spark
>> > > already
>> > > >> do - and changing the default behavior to throw an error instead of
>> > > >> silently corrupting data, while giving users clear options to deal
>> with
>> > > >> invalid bytes.
>> > > >>
>> > > >> Looking forward to your feedback and thoughts.
>> > > >>
>> > > >> Kind regards,
>> > > >> Gustavo
>> > > >>
>> > > >> [1]
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities
>> > > >>
>> > >
>>
>>
>>
>> --
>> Best regards,
>> Sergey
>>

Re: Re: [DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Posted by Gustavo de Morais <gu...@gmail.com>.
Hi Sergey,

Thanks for the message! I took a look at our support for UTF_16 and I don't
think any additional UTF_16 support is necessary for the scope of this
FLIP. For UTF-16 users should use Decode instead of Cast. Flink stores all
strings as UTF-8 internally, so DECODE(bytes, 'UTF-16') already handles
that case - it converts to a Java String and re-encodes as UTF-8 before
storage. Since the internal representation is always UTF-8, the validation
problem is fundamentally a UTF-8 concern and the functions are correctly
named. Adding validation functions support for all the character sets we
support for DECODE (‘US-ASCII’, ‘ISO-8859-1’, ‘UTF-8’, ‘UTF-16BE’,
‘UTF-16LE’, ‘UTF-16) might be something useful for some less frequent use
cases but I'd say this is out-of-scope for this FLIP.

Kind regards,
Gustavo


On Fri, 20 Mar 2026 at 20:24, Sergey Nuyanzin <sn...@gmail.com> wrote:

> Hi Gustavo
> thanks for the proposal
>
> I noticed that you are proposing usage of UTF8 in names (default cast
> to string is also using utf8)
> however I wonder if it makes sense to introduce similar utf16 similar
> functions as Flink supports this as well?
>
> On Fri, Mar 20, 2026 at 8:09 PM Gustavo de Morais
> <gu...@gmail.com> wrote:
> >
> > Hi Xuyang and Timo,
> >
> > Thanks for the positive feedback! Regarding your suggestions, Xuyang:
> >
> > 1. Yes, good point - we should add the fromUtf8Bytes(byte[], int, int)
> > overload as well.
> > 2. This is also relevant. If we want to do validation during ingestion
> > time, this might have performance implications. Since these are @Internal
> > APIs, they can be changed independently from the FLIP afterwards if it
> > makes sense. What's your opinion?
> >
> > Kind regards,
> > Gustavo
> >
> > On Fri, 20 Mar 2026 at 08:44, Xuyang <xy...@163.com> wrote:
> >
> > > Hi, Gustavo.
> > > Great catch! Thanks for driving this FLIP. Overall LGTM. I just have
> two
> > > minor points I'd like to confirm with you.
> > > 1. Should we also add the overload function `fromUtf8Bytes(byte[], int,
> > > int)` in StringData?
> > > 2. Callers like `ColumnarRowData#getString` and
> > > `ColumnarArrayData#getString`  call `StringData.fromBytes` directly.
> Should
> > > these call sites be migrated in a follow-up, or intentionally left
> as-is?
> > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > >     Best!
> > >     Xuyang
> > >
> > >
> > >
> > > At 2026-03-19 22:37:28, "Timo Walther" <tw...@apache.org> wrote:
> > > >Hi Gustavo,
> > > >
> > > >thank you for this excellent design document. And thanks for
> discovering
> > > >this data loss and driving the investigation. We should definitely fix
> > > >this shortcoming. Also looking at other vendors, it is definitly a
> cause
> > > >for false assumptions that lead to hard-to-debug inconsistencies.
> > > >
> > > >+1 for this proposal.
> > > >
> > > >Cheers,
> > > >Timo
> > > >
> > > >
> > > >On 19.03.26 15:23, Gustavo de Morais wrote:
> > > >> Hi everyone,
> > > >>
> > > >> Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8
> > > byte
> > > >> with U+FFFD (?). The substitution is irreversible and produces no
> > > warning -
> > > >> the pipeline keeps running while data is permanently corrupted
> > > >> downstream. This also means that a CAST from BYTES → STRING → BYTES
> is
> > > not
> > > >> idempotent, which prevents the engine from applying certain
> > > optimizations.
> > > >> For example, for preserving upsert keys after such CASTs.
> > > >>
> > > >> I'd like to start a discussion around defining and improving the
> default
> > > >> behavior. I've written a short FLIP [1] proposing new utility
> functions
> > > to
> > > >> handle this explicitly - similar to what other engines like Spark
> > > already
> > > >> do - and changing the default behavior to throw an error instead of
> > > >> silently corrupting data, while giving users clear options to deal
> with
> > > >> invalid bytes.
> > > >>
> > > >> Looking forward to your feedback and thoughts.
> > > >>
> > > >> Kind regards,
> > > >> Gustavo
> > > >>
> > > >> [1]
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities
> > > >>
> > >
>
>
>
> --
> Best regards,
> Sergey
>

Re: Re: [DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Posted by Sergey Nuyanzin <sn...@gmail.com>.
Hi Gustavo
thanks for the proposal

I noticed that you are proposing usage of UTF8 in names (default cast
to string is also using utf8)
however I wonder if it makes sense to introduce similar utf16 similar
functions as Flink supports this as well?

On Fri, Mar 20, 2026 at 8:09 PM Gustavo de Morais
<gu...@gmail.com> wrote:
>
> Hi Xuyang and Timo,
>
> Thanks for the positive feedback! Regarding your suggestions, Xuyang:
>
> 1. Yes, good point - we should add the fromUtf8Bytes(byte[], int, int)
> overload as well.
> 2. This is also relevant. If we want to do validation during ingestion
> time, this might have performance implications. Since these are @Internal
> APIs, they can be changed independently from the FLIP afterwards if it
> makes sense. What's your opinion?
>
> Kind regards,
> Gustavo
>
> On Fri, 20 Mar 2026 at 08:44, Xuyang <xy...@163.com> wrote:
>
> > Hi, Gustavo.
> > Great catch! Thanks for driving this FLIP. Overall LGTM. I just have two
> > minor points I'd like to confirm with you.
> > 1. Should we also add the overload function `fromUtf8Bytes(byte[], int,
> > int)` in StringData?
> > 2. Callers like `ColumnarRowData#getString` and
> > `ColumnarArrayData#getString`  call `StringData.fromBytes` directly. Should
> > these call sites be migrated in a follow-up, or intentionally left as-is?
> >
> >
> >
> >
> >
> > --
> >
> >     Best!
> >     Xuyang
> >
> >
> >
> > At 2026-03-19 22:37:28, "Timo Walther" <tw...@apache.org> wrote:
> > >Hi Gustavo,
> > >
> > >thank you for this excellent design document. And thanks for discovering
> > >this data loss and driving the investigation. We should definitely fix
> > >this shortcoming. Also looking at other vendors, it is definitly a cause
> > >for false assumptions that lead to hard-to-debug inconsistencies.
> > >
> > >+1 for this proposal.
> > >
> > >Cheers,
> > >Timo
> > >
> > >
> > >On 19.03.26 15:23, Gustavo de Morais wrote:
> > >> Hi everyone,
> > >>
> > >> Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8
> > byte
> > >> with U+FFFD (?). The substitution is irreversible and produces no
> > warning -
> > >> the pipeline keeps running while data is permanently corrupted
> > >> downstream. This also means that a CAST from BYTES → STRING → BYTES is
> > not
> > >> idempotent, which prevents the engine from applying certain
> > optimizations.
> > >> For example, for preserving upsert keys after such CASTs.
> > >>
> > >> I'd like to start a discussion around defining and improving the default
> > >> behavior. I've written a short FLIP [1] proposing new utility functions
> > to
> > >> handle this explicitly - similar to what other engines like Spark
> > already
> > >> do - and changing the default behavior to throw an error instead of
> > >> silently corrupting data, while giving users clear options to deal with
> > >> invalid bytes.
> > >>
> > >> Looking forward to your feedback and thoughts.
> > >>
> > >> Kind regards,
> > >> Gustavo
> > >>
> > >> [1]
> > >>
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities
> > >>
> >



-- 
Best regards,
Sergey

Re: Re: [DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Posted by Gustavo de Morais <gu...@gmail.com>.
Hi Xuyang and Timo,

Thanks for the positive feedback! Regarding your suggestions, Xuyang:

1. Yes, good point - we should add the fromUtf8Bytes(byte[], int, int)
overload as well.
2. This is also relevant. If we want to do validation during ingestion
time, this might have performance implications. Since these are @Internal
APIs, they can be changed independently from the FLIP afterwards if it
makes sense. What's your opinion?

Kind regards,
Gustavo

On Fri, 20 Mar 2026 at 08:44, Xuyang <xy...@163.com> wrote:

> Hi, Gustavo.
> Great catch! Thanks for driving this FLIP. Overall LGTM. I just have two
> minor points I'd like to confirm with you.
> 1. Should we also add the overload function `fromUtf8Bytes(byte[], int,
> int)` in StringData?
> 2. Callers like `ColumnarRowData#getString` and
> `ColumnarArrayData#getString`  call `StringData.fromBytes` directly. Should
> these call sites be migrated in a follow-up, or intentionally left as-is?
>
>
>
>
>
> --
>
>     Best!
>     Xuyang
>
>
>
> At 2026-03-19 22:37:28, "Timo Walther" <tw...@apache.org> wrote:
> >Hi Gustavo,
> >
> >thank you for this excellent design document. And thanks for discovering
> >this data loss and driving the investigation. We should definitely fix
> >this shortcoming. Also looking at other vendors, it is definitly a cause
> >for false assumptions that lead to hard-to-debug inconsistencies.
> >
> >+1 for this proposal.
> >
> >Cheers,
> >Timo
> >
> >
> >On 19.03.26 15:23, Gustavo de Morais wrote:
> >> Hi everyone,
> >>
> >> Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8
> byte
> >> with U+FFFD (?). The substitution is irreversible and produces no
> warning -
> >> the pipeline keeps running while data is permanently corrupted
> >> downstream. This also means that a CAST from BYTES → STRING → BYTES is
> not
> >> idempotent, which prevents the engine from applying certain
> optimizations.
> >> For example, for preserving upsert keys after such CASTs.
> >>
> >> I'd like to start a discussion around defining and improving the default
> >> behavior. I've written a short FLIP [1] proposing new utility functions
> to
> >> handle this explicitly - similar to what other engines like Spark
> already
> >> do - and changing the default behavior to throw an error instead of
> >> silently corrupting data, while giving users clear options to deal with
> >> invalid bytes.
> >>
> >> Looking forward to your feedback and thoughts.
> >>
> >> Kind regards,
> >> Gustavo
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities
> >>
>

Re:Re: [DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Posted by Xuyang <xy...@163.com>.
Hi, Gustavo.
Great catch! Thanks for driving this FLIP. Overall LGTM. I just have two minor points I'd like to confirm with you.
1. Should we also add the overload function `fromUtf8Bytes(byte[], int, int)` in StringData?
2. Callers like `ColumnarRowData#getString` and `ColumnarArrayData#getString`  call `StringData.fromBytes` directly. Should these call sites be migrated in a follow-up, or intentionally left as-is?





--

    Best!
    Xuyang



At 2026-03-19 22:37:28, "Timo Walther" <tw...@apache.org> wrote:
>Hi Gustavo,
>
>thank you for this excellent design document. And thanks for discovering 
>this data loss and driving the investigation. We should definitely fix 
>this shortcoming. Also looking at other vendors, it is definitly a cause 
>for false assumptions that lead to hard-to-debug inconsistencies.
>
>+1 for this proposal.
>
>Cheers,
>Timo
>
>
>On 19.03.26 15:23, Gustavo de Morais wrote:
>> Hi everyone,
>> 
>> Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8 byte
>> with U+FFFD (?). The substitution is irreversible and produces no warning -
>> the pipeline keeps running while data is permanently corrupted
>> downstream. This also means that a CAST from BYTES → STRING → BYTES is not
>> idempotent, which prevents the engine from applying certain optimizations.
>> For example, for preserving upsert keys after such CASTs.
>> 
>> I'd like to start a discussion around defining and improving the default
>> behavior. I've written a short FLIP [1] proposing new utility functions to
>> handle this explicitly - similar to what other engines like Spark already
>> do - and changing the default behavior to throw an error instead of
>> silently corrupting data, while giving users clear options to deal with
>> invalid bytes.
>> 
>> Looking forward to your feedback and thoughts.
>> 
>> Kind regards,
>> Gustavo
>> 
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities
>> 

Re: [DISCUSS] FLIP-568: Strict BYTES-to-STRING CAST with UTF-8 Validation Utilities

Posted by Timo Walther <tw...@apache.org>.
Hi Gustavo,

thank you for this excellent design document. And thanks for discovering 
this data loss and driving the investigation. We should definitely fix 
this shortcoming. Also looking at other vendors, it is definitly a cause 
for false assumptions that lead to hard-to-debug inconsistencies.

+1 for this proposal.

Cheers,
Timo


On 19.03.26 15:23, Gustavo de Morais wrote:
> Hi everyone,
> 
> Currently, CAST(bytes AS STRING) silently replaces any invalid UTF-8 byte
> with U+FFFD (?). The substitution is irreversible and produces no warning -
> the pipeline keeps running while data is permanently corrupted
> downstream. This also means that a CAST from BYTES → STRING → BYTES is not
> idempotent, which prevents the engine from applying certain optimizations.
> For example, for preserving upsert keys after such CASTs.
> 
> I'd like to start a discussion around defining and improving the default
> behavior. I've written a short FLIP [1] proposing new utility functions to
> handle this explicitly - similar to what other engines like Spark already
> do - and changing the default behavior to throw an error instead of
> silently corrupting data, while giving users clear options to deal with
> invalid bytes.
> 
> Looking forward to your feedback and thoughts.
> 
> Kind regards,
> Gustavo
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-568%3A+Strict+BYTES-to-STRING+CAST+with+UTF-8+Validation+Utilities
>