[jira] [Commented] (SPARK-57932) regexp_instr and regexp_replace treat string positions as UTF-16 code units instead of code points

["Kousuke Saruta (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-57932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096030#comment-18096030 ] 

Kousuke Saruta commented on SPARK-57932:
----------------------------------------

Note: Fix Version/s need to be changed once any of the following vote doesn't pass:

* 4.0.4 (RC1): 4.0.5 -> 4.0.4
* 4.1.3 (RC1): 4.1.4 -> 4.1.3
* 4.2.0 (RC6): 4.2.1 -> 4.2.0

> regexp_instr and regexp_replace treat string positions as UTF-16 code units instead of code points
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-57932
>                 URL: https://issues.apache.org/jira/browse/SPARK-57932
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.8, 4.2.0, 4.1.2, 4.0.3, 4.3.0, 5.0.0
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0, 5.0.0, 4.2.1, 4.0.5, 4.1.4
>
>
> *The issue:*
> Spark string positions are 1-based code points (instr, locate, position, substring, length). For strings containing supplementary characters (code points > U+FFFF, e.g. emojis), these two functions gave wrong results:
> - regexp_instr('😀ab', 'ab') returned 3 instead of 2.
> - regexp_replace('😀aXa', 'a', 'Z', 3) returned 😀ZXZ instead of 😀aXZ (replaced a match before position); regexp_replace('😀😀', '😀', 'Z', 3) returned 😀Z instead of the input unchanged. This is the sibling of the LIKE fix in SPARK-55453.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org