[jira] [Created] (FLINK-40350) LPAD/RPAD split supplementary-plane characters into unpaired surrogates

["sepuri sai krishna (Jira)" <ji...@apache.org>]

sepuri sai krishna created FLINK-40350:
------------------------------------------

             Summary: LPAD/RPAD split supplementary-plane characters into unpaired surrogates
                 Key: FLINK-40350
                 URL: https://issues.apache.org/jira/browse/FLINK-40350
             Project: Flink
          Issue Type: Bug
          Components: Table SQL / Planner, Table SQL / Runtime
    Affects Versions: 2.3.0, 2.2.0, 2.1.0, 2.0.0
            Reporter: sepuri sai krishna


h3. Problem

{{LPAD}} and {{RPAD}} measure length in UTF-16 code units instead of characters. When the requested length falls in the middle of a supplementary-plane character, the function splits the surrogate pair and returns a string containing an unpaired surrogate, which is not valid Unicode.

In the examples below, {{E}} stands for the single-character string U+1F600 GRINNING FACE, which UTF-16 encodes as the surrogate pair U+D83D U+DE00. It is written as {{E}} rather than inline so that this description stays within the Basic Multilingual Plane.

|| Expression || Returned, as UTF-16 code units || Note ||
| {{LPAD(E, 1, 'x')}} | {{U+D83D}} | unpaired high surrogate: the first half of {{E}} |
| {{RPAD(E, 1, 'x')}} | {{U+D83D}} | unpaired high surrogate: the first half of {{E}} |
| {{RPAD('a', 4, E)}} | {{U+0061 U+D83D U+DE00 U+D83D}} | {{'a'}}, then an intact {{E}}, then a trailing unpaired high surrogate |

The third case is the least obvious: the padding is copied one code unit at a time, so the final repetition of the pad string can stop between the two halves of its character.

The documentation states that the length is measured in characters:

{quote}Returns a new string from string1 left-padded with string2 to a length of integer *characters*.{quote}

{{E}} is one character, so {{LPAD(E, 1, 'x')}} should return {{E}} unchanged rather than half of it.

h3. Impact

Silent corruption of string data. The result is invalid UTF-16 and does not round-trip: encoding it yields a replacement character or malformed bytes, and comparisons and downstream string functions then operate on a value the query never produced. No exception is thrown and nothing is logged.

Affects any query where {{LPAD}} or {{RPAD}} truncates, or where the pad string contains supplementary-plane characters - emoji, historic scripts, and less common CJK ideographs.

h3. Affects

Present in every branch checked from release-1.19 through release-2.3, and on master.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)