[jira] [Created] (SPARK-47863) endsWith and startsWith don't work correctly for some collations

["Vladimir Golubev (Jira)" <ji...@apache.org>]

Vladimir Golubev created SPARK-47863:
----------------------------------------

             Summary: endsWith and startsWith don't work correctly for some collations
                 Key: SPARK-47863
                 URL: https://issues.apache.org/jira/browse/SPARK-47863
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.0.0
            Reporter: Vladimir Golubev


*CollationSupport.EndsWIth* and *CollationSupport.StartsWith* use {*}CollationAwareUTF8String.matchAt{*}, which operates byte offsets to compare prefixes/suffixes. This is not correct, since sometimes string parts (suffix/prefix) of different lengths are actually equal in context of case-insensitive and lower-case collations.



Example test cases that highlight the problem:

{{{}- *assertContains("The İo", "i̇o", "UNICODE_CI", true);* for *CollationSupportSuite.*{}}}{{{}{*}testContains{*}.{}}} 
{{- *assertEndsWith("The İo", "i̇o", "UNICODE_CI", true);* for }}{*}{{CollationSupportSuite.}}{*}{{{}{*}testEndsWith{*}.{}}}

{{{}The first passes, since it uses *StringSearch* directly, the second one does not.{}}}{{{}{}}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org