[jira] [Created] (FLINK-40305) VARIANT string and object key decoding ignores the UTF-8 charset

["Ramin Gharib (Jira)" <ji...@apache.org>]

Ramin Gharib created FLINK-40305:
------------------------------------

             Summary: VARIANT string and object key decoding ignores the UTF-8 charset
                 Key: FLINK-40305
                 URL: https://issues.apache.org/jira/browse/FLINK-40305
             Project: Flink
          Issue Type: Bug
          Components: API / Core
            Reporter: Ramin Gharib
            Assignee: Ramin Gharib


{\{BinaryVariantUtil}} decodes variant strings and object field names without passing a charset, so both fall back to the JVM default:
 
{code:java}
// getString
return new String(value, start, length);
// getMetadataKey
return new String(metadata, stringStart + offset, nextOffset - offset);
{code}
 
Both are written as UTF-8 by \{{BinaryVariantInternalBuilder}}, which uses \{{str.getBytes(StandardCharsets.UTF_8)}} for values and \{{key.getBytes(StandardCharsets.UTF_8)}} for keys, so the write and read paths disagree.
 
The default charset is only guaranteed to be UTF-8 from Java 18 onwards (JEP 400). On Java 11 and 17 it is platform dependent, so a variant holding non-ASCII text can decode incorrectly. \{{new String(byte[], ...)}} also substitutes \{{U+FFFD}} for malformed input rather than failing.
 
{\{getMetadataKey}} is the more damaging of the two, since it decodes field names. A mangled name makes \{{getField(name)}} silently return null and corrupts \{{getFieldNames()}} and \{{toJson()}}.
 
This is a regression from the original port. Spark's \{{VariantUtil}}, which this class is derived from, passes the charset in both places:
 
{code:java}
return new String(value, start, length, StandardCharsets.UTF_8);
return new String(metadata, stringStart + offset, nextOffset - offset, StandardCharsets.UTF_8);
{code}
 
h3. Fix
Pass \{{StandardCharsets.UTF_8}} in both calls. Add a round-trip test with a non-ASCII string value and a non-ASCII field name.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)