You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "DenineLu (via GitHub)" <gi...@apache.org> on 2024/10/15 05:13:06 UTC

[PR] [][] Fix the split function with limit being cut off incorrectly [spark]

DenineLu opened a new pull request, #48470:
URL: https://github.com/apache/spark/pull/48470

   ### What changes were proposed in this pull request?
   
   The current behavior of the split function is as follows:
   ```
   select split('hello', 'h', 1) // result is ["hello"]
   select split('hello', '-', 1) // result is ["hello"]
   select split('hello', '', 1)  // result is ["h"]
   
   select split('1A2A3A4', 'A', 3) // result is ["1","2","3A4"]
   select split('1A2A3A4', '', 3)  // result is ["1","A","2"]
   ```
   
   However, according to the function's description, when the limit is greater than zero, the last element of the split result should contain the remaining part of the input string.
   ```
   Arguments:
         * str - a string expression to split.
         * regex - a string representing a regular expression. The regex string should be a Java regular expression.
         * limit - an integer expression which controls the number of times the regex is applied.
             * limit > 0: The resulting array's length will not be more than `limit`, and the resulting array's last entry will contain all input beyond the last matched regex.
             * limit <= 0: `regex` will be applied as many times as possible, and the resulting array can be of any size. 
   ```
   
   So, the split function produces incorrect results with an empty regex and a limit. The correct result should be: 
   ```
   select split('hello', '', 1)    // result is ["hello"]
   
   select split('1A2A3A4', '', 3)  // result is ["1","A","2A3A4"]
   ```
   
   ### Why are the changes needed?
   Fix correctness issue.
   
   ### Does this PR introduce _any_ user-facing change?
   Yes.
   When the empty regex parameter is provided along with a limit parameter greater than 0, the output of the split function changes.
   Before this patch
   ```
   select split('hello', '', 1)          // result is ["h"]
   select split('1A2A3A4', '', 3)  // result is ["1","A","2"]
   ```
   After this patch 
   ```
   select split('hello', '', 1)          // result is ["hello"]
   select split('1A2A3A4', '', 3)  // result is ["1","A","2A3A4"]
   ```
   
   ### How was this patch tested?
   Unit tests.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2641848389

   > Please investigate more databases, then we make the decision which is the more suitable behavior.
   
   For the split(str, regex, limit) function, the limit parameter controls the number of times regex is applied. I checked the related functions of mainstream databases. Except for presto and trino, there are basically no similar functions.
   | Database/SQLEngine | Behavior of this function |
   |--------|--------|
   | PostgreSQL | There is no similar function |
   | MySQL | There is no similar function |
   | Oracle | There is no similar function |
   | MariaDB | There is no similar function |
   | Microsoft SQL Server  | There is no similar function |
   | Hive  | There is no similar function |
   | PrestoDB  | The behavior is consistent with this PR, and the functionality is implemented earlier than Spark |
   | Trino  | The behavior is consistent with this PR, and the functionality is implemented earlier than Spark |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804732967


##########
sql/core/src/test/resources/sql-tests/inputs/string-functions.sql:
##########
@@ -27,6 +27,8 @@ select right("abcd", -2), right("abcd", 0), right("abcd", 'a');
 SELECT split('aa1cc2ee3', '[1-9]+');
 SELECT split('aa1cc2ee3', '[1-9]+', 2);
 SELECT split('hello', '');
+SELECT split('hello', '', 1);
+SELECT split('hello', '', 3);
 SELECT split('', '');

Review Comment:
   Thanks, already added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1808400617


##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1484,17 +1485,17 @@ public static UTF8String concatWs(UTF8String separator, UTF8String... inputs) {
   public UTF8String[] split(UTF8String pattern, int limit) {
     // For the empty `pattern` a `split` function ignores trailing empty strings unless original
     // string is empty.
-    if (numBytes() != 0 && pattern.numBytes() == 0) {
+    if (numBytes() != 0 && (pattern.numBytes() == 0 || lowercaseRegexPrefix.equals(pattern))) {

Review Comment:
   how about: instead of checking whether the pattern equals to the (?ui) prefix, we modify the collation implementation (prefixing logic) to avoid appending the prefix at all in the case that pattern is an empty string



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r2227274943


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -6158,6 +6158,15 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
+  val LEGACY_TRUNCATE_FOR_EMPTY_REGEX_SPLIT =
+    buildConf("spark.sql.legacy.truncateForEmptyRegexSplit")
+      .internal()
+      .doc("When set to true, using empty regex to split will discard the trailing string, which " +

Review Comment:
   how to define "trailing string"? everything except for the first char?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-3106614760

   I added a configuration to fall back to the current behavior, can you review it again? @wangyum @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2629282709

   What's the behavior of other mainstream databases?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2638676457

   > What's the behavior of other mainstream databases?
   
   As far as I know, the split function of HIVE and StarRocks does not support the limit parameter, and the behavior of presto is consistent with the revised behavior of this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2639099089

   Please investigate more databases, then we make the decision which is the more suitable behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-3117125832

   thanks, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1808033801


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationRegexpExpressionsSuite.scala:
##########
@@ -57,25 +57,31 @@ class CollationRegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalH
   }
 
   test("StringSplit expression with collated strings") {
-    case class StringSplitTestCase[R](s: String, r: String, collation: String, expected: R)
+    case class StringSplitTestCase[R](s: String, r: String, collation: String,
+      expected: R, limit: Int)
     val testCases = Seq(
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C")),
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C")),
-      StringSplitTestCase("", "", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C")),
-      StringSplitTestCase("", "[1-9]+", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase(null, "[1-9]+", "UTF8_BINARY", null),
-      StringSplitTestCase("1A2B3C", null, "UTF8_BINARY", null),
-      StringSplitTestCase(null, null, "UTF8_BINARY", null)
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C"), -1),
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C"), -1),
+      StringSplitTestCase("", "", "UTF8_BINARY", Seq(""), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), 0),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1A2B3C"), 1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 3),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3C"), 5),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), 100),

Review Comment:
   In the current situation, if `UTF8_LCASE` is applied to an empty string, the condition here will not be met because the value of pattern after being collated by collationAwareRegex is`(?ui)`, meaning that [#37631](https://github.com/apache/spark/pull/37631) does not support truncating the trailing empty string when the pattern is `(?ui)`. 
   ```
   public UTF8String[] split(UTF8String pattern, int limit) {
       // For the empty `pattern` a `split` function ignores trailing empty strings unless original
       // string is empty.
       if (numBytes() != 0 && pattern.numBytes() == 0) {
   ```
   Therefore, it seems that the result is not what we want when limit <= 0. 
   ```
   select split('1A2B3C', '(?ui)', -1);    // result is ["1", "A", "2", "B", "3", "C", ""]
   select split('1A2B3C', '(?ui)', 0);    // result is ["1", "A", "2", "B", "3", "C", ""]
   
   select split('1A2B3C', '(?ui)', 1);    // result is ["1A2B3C"]
   select split('1A2B3C', '(?ui)', 3);    // result is ["1", "A", "2B3C"]
   select split('1A2B3C', '(?ui)', 6);     // result is ["1", "A", "2", "B", "3", "C"]
   select split('1A2B3C', '(?ui)', 100);    // result is ["1", "A", "2", "B", "3", "C"]
   ```
   When the pattern is "(?ui)", a simple and direct approach can be taken to correct the result.
   ```
   public UTF8String[] split(UTF8String pattern, int limit) {
       // For the empty `pattern` a `split` function ignores trailing empty strings unless original
       // string is empty.
       if (numBytes() != 0 && (pattern.numBytes() == 0 || lowercaseRegexPrefix.equals(pattern))) {
   ```
   However, when the pattern is "(?ui)(?ui)" or "(?ui)(?ui)(?ui)", the result still contains a trailing empty string, and I haven't thought of an efficient way to match and resolve it. Should we consider this a reasonable situation?
   Additionally, do you think it is necessary to check in the `CollationSupport.lowercaseRegex` method whether the regex already has a `(?ui)` prefix?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804613614


##########
sql/core/src/test/resources/sql-tests/inputs/string-functions.sql:
##########
@@ -27,6 +27,8 @@ select right("abcd", -2), right("abcd", 0), right("abcd", 'a');
 SELECT split('aa1cc2ee3', '[1-9]+');
 SELECT split('aa1cc2ee3', '[1-9]+', 2);
 SELECT split('hello', '');
+SELECT split('hello', '', 1);
+SELECT split('hello', '', 3);

Review Comment:
   ```suggestion
   SELECT split('hello', '', 0);
   SELECT split('hello', '', 1);
   SELECT split('hello', '', 3);
   SELECT split('hello', '', 5);
   SELECT split('hello', '', 100);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #48470: [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit
URL: https://github.com/apache/spark/pull/48470


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r2227442139


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -6158,6 +6158,15 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
+  val LEGACY_TRUNCATE_FOR_EMPTY_REGEX_SPLIT =
+    buildConf("spark.sql.legacy.truncateForEmptyRegexSplit")
+      .internal()
+      .doc("When set to true, using empty regex to split will discard the trailing string, which " +

Review Comment:
   The previous description was unclear. I've updated it—please take another look!



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -6158,6 +6158,15 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
+  val LEGACY_TRUNCATE_FOR_EMPTY_REGEX_SPLIT =
+    buildConf("spark.sql.legacy.truncateForEmptyRegexSplit")
+      .internal()
+      .doc("When set to true, using empty regex to split will discard the trailing string, which " +

Review Comment:
   > how to define "trailing string"? everything except for the first char?
   
   The previous description was unclear. I've updated it—please take another look!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1808430241


##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1484,17 +1485,17 @@ public static UTF8String concatWs(UTF8String separator, UTF8String... inputs) {
   public UTF8String[] split(UTF8String pattern, int limit) {
     // For the empty `pattern` a `split` function ignores trailing empty strings unless original
     // string is empty.
-    if (numBytes() != 0 && pattern.numBytes() == 0) {
+    if (numBytes() != 0 && (pattern.numBytes() == 0 || lowercaseRegexPrefix.equals(pattern))) {

Review Comment:
   that is an interesting observation, although in that case I don't see why the user's pattern can't be any other flag modifier combination, such as: (?m), (?s), (?x), (?a)
   
   taking this into consideration, there is really nothing special about `lowercaseRegexPrefix`. instead, you should look for a library method that can discern whether a pattern is "functionally" empty, instead of doing a manual check against `lowercaseRegexPrefix`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804730851


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationRegexpExpressionsSuite.scala:
##########
@@ -57,25 +57,28 @@ class CollationRegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalH
   }
 
   test("StringSplit expression with collated strings") {
-    case class StringSplitTestCase[R](s: String, r: String, collation: String, expected: R)
+    case class StringSplitTestCase[R](s: String, r: String, collation: String,
+      expected: R, limit: Int)
     val testCases = Seq(
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C")),
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C")),
-      StringSplitTestCase("", "", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C")),
-      StringSplitTestCase("", "[1-9]+", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase(null, "[1-9]+", "UTF8_BINARY", null),
-      StringSplitTestCase("1A2B3C", null, "UTF8_BINARY", null),
-      StringSplitTestCase(null, null, "UTF8_BINARY", null)
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C"), -1),
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C"), -1),
+      StringSplitTestCase("", "", "UTF8_BINARY", Seq(""), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1A2B3C"), 1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 3),

Review Comment:
   > let's add all of these
   
   Supplemented!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2638675870

   > What's the behavior of other mainstream databases?
   As far as I know, the split function of HIVE and StarRocks does not support the limit parameter, and the behavior of presto is consistent with the revised behavior of this PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r2227442139


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -6158,6 +6158,15 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
+  val LEGACY_TRUNCATE_FOR_EMPTY_REGEX_SPLIT =
+    buildConf("spark.sql.legacy.truncateForEmptyRegexSplit")
+      .internal()
+      .doc("When set to true, using empty regex to split will discard the trailing string, which " +

Review Comment:
   The previous description was unclear. I've updated it—please take another look!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804614836


##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1488,13 +1488,13 @@ public UTF8String[] split(UTF8String pattern, int limit) {
       int newLimit = limit > numChars() || limit <= 0 ? numChars() : limit;
       byte[] input = getBytes();
       int byteIndex = 0;
-      int charIndex = 0;
       UTF8String[] result = new UTF8String[newLimit];
-      while (charIndex < newLimit) {
+      for (int i = 0; i < newLimit - 1; i++) {

Review Comment:
   ```suggestion
         for (int charIndex = 0; charIndex < newLimit - 1; charIndex ++) {
   ```



##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1488,13 +1488,13 @@ public UTF8String[] split(UTF8String pattern, int limit) {
       int newLimit = limit > numChars() || limit <= 0 ? numChars() : limit;
       byte[] input = getBytes();
       int byteIndex = 0;
-      int charIndex = 0;
       UTF8String[] result = new UTF8String[newLimit];
-      while (charIndex < newLimit) {
+      for (int i = 0; i < newLimit - 1; i++) {
         int currCharNumBytes = numBytesForFirstByte(input[byteIndex]);
-        result[charIndex++] = UTF8String.fromBytes(input, byteIndex, currCharNumBytes);
+        result[i] = UTF8String.fromBytes(input, byteIndex, currCharNumBytes);

Review Comment:
   ```suggestion
           result[charIndex] = UTF8String.fromBytes(input, byteIndex, currCharNumBytes);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804624747


##########
sql/core/src/test/resources/sql-tests/inputs/string-functions.sql:
##########
@@ -27,6 +27,8 @@ select right("abcd", -2), right("abcd", 0), right("abcd", 'a');
 SELECT split('aa1cc2ee3', '[1-9]+');
 SELECT split('aa1cc2ee3', '[1-9]+', 2);
 SELECT split('hello', '');
+SELECT split('hello', '', 1);
+SELECT split('hello', '', 3);
 SELECT split('', '');

Review Comment:
   I would also prefer to see:
   ```
   SELECT split('', '', -1);
   SELECT split('', '', 0);
   SELECT split('', '', 1);
   ```
   here, for more complete testing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #48470: [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit
URL: https://github.com/apache/spark/pull/48470


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-3102949996

   Yea I think we should fix this, but given the behavior change is already shipped, we need a legacy config to protect this fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r2227275197


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -6158,6 +6158,15 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
+  val LEGACY_TRUNCATE_FOR_EMPTY_REGEX_SPLIT =
+    buildConf("spark.sql.legacy.truncateForEmptyRegexSplit")
+      .internal()
+      .doc("When set to true, using empty regex to split will discard the trailing string, which " +

Review Comment:
   can we also mention the correct (when flag is false) behavior here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1808417962


##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1484,17 +1485,17 @@ public static UTF8String concatWs(UTF8String separator, UTF8String... inputs) {
   public UTF8String[] split(UTF8String pattern, int limit) {
     // For the empty `pattern` a `split` function ignores trailing empty strings unless original
     // string is empty.
-    if (numBytes() != 0 && pattern.numBytes() == 0) {
+    if (numBytes() != 0 && (pattern.numBytes() == 0 || lowercaseRegexPrefix.equals(pattern))) {

Review Comment:
   I agree with what you're saying, but should we consider that the user's pattern itself might be `(?ui)` and is unrelated to prefixing logic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "TongWei1105 (via GitHub)" <gi...@apache.org>.
TongWei1105 commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2412983967

   cc @wangyum @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804626072


##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1488,13 +1488,13 @@ public UTF8String[] split(UTF8String pattern, int limit) {
       int newLimit = limit > numChars() || limit <= 0 ? numChars() : limit;
       byte[] input = getBytes();
       int byteIndex = 0;
-      int charIndex = 0;
       UTF8String[] result = new UTF8String[newLimit];
-      while (charIndex < newLimit) {
+      for (int i = 0; i < newLimit - 1; i++) {
         int currCharNumBytes = numBytesForFirstByte(input[byteIndex]);
-        result[charIndex++] = UTF8String.fromBytes(input, byteIndex, currCharNumBytes);
+        result[i] = UTF8String.fromBytes(input, byteIndex, currCharNumBytes);
         byteIndex += currCharNumBytes;
       }
+      result[newLimit - 1] = UTF8String.fromBytes(input, byteIndex, numBytes() - byteIndex);

Review Comment:
   > is `ArrayIndexOutOfBoundsException` possible here? what if newLimit=0 (i.e. numChars()=0, limit=-1)
   
   no, this code block will only be entered when the following conditions are met. 
   ```
   if (numBytes() != 0 && pattern.numBytes() == 0)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804612385


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationRegexpExpressionsSuite.scala:
##########
@@ -57,25 +57,28 @@ class CollationRegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalH
   }
 
   test("StringSplit expression with collated strings") {
-    case class StringSplitTestCase[R](s: String, r: String, collation: String, expected: R)
+    case class StringSplitTestCase[R](s: String, r: String, collation: String,
+      expected: R, limit: Int)
     val testCases = Seq(
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C")),
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C")),
-      StringSplitTestCase("", "", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C")),
-      StringSplitTestCase("", "[1-9]+", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase(null, "[1-9]+", "UTF8_BINARY", null),
-      StringSplitTestCase("1A2B3C", null, "UTF8_BINARY", null),
-      StringSplitTestCase(null, null, "UTF8_BINARY", null)
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C"), -1),
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C"), -1),
+      StringSplitTestCase("", "", "UTF8_BINARY", Seq(""), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1A2B3C"), 1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 3),

Review Comment:
   ```suggestion
         StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 0),
         StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 3),
         StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 5),
         StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 100),
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1808703775


##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1484,17 +1485,17 @@ public static UTF8String concatWs(UTF8String separator, UTF8String... inputs) {
   public UTF8String[] split(UTF8String pattern, int limit) {
     // For the empty `pattern` a `split` function ignores trailing empty strings unless original
     // string is empty.
-    if (numBytes() != 0 && pattern.numBytes() == 0) {
+    if (numBytes() != 0 && (pattern.numBytes() == 0 || lowercaseRegexPrefix.equals(pattern))) {

Review Comment:
   Thank you for your explanation. It looks like there’s no way to validate this "weird" situation without losing performance. I made changes according to your advice. Thanks again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2418855511

   > thanks for making this change - however, please add collation-related tests as well, see:
   > 
   > ```
   > test("StringSplit expression with collated strings")
   > ```
   > 
   > in `CollationRegexpExpressionsSuite.scala`
   
   Thank you for your guidance. The relevant tests have been added.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804613614


##########
sql/core/src/test/resources/sql-tests/inputs/string-functions.sql:
##########
@@ -27,6 +27,8 @@ select right("abcd", -2), right("abcd", 0), right("abcd", 'a');
 SELECT split('aa1cc2ee3', '[1-9]+');
 SELECT split('aa1cc2ee3', '[1-9]+', 2);
 SELECT split('hello', '');
+SELECT split('hello', '', 1);
+SELECT split('hello', '', 3);

Review Comment:
   ```suggestion
   SELECT split('hello', '', -1);
   SELECT split('hello', '', 0);
   SELECT split('hello', '', 1);
   SELECT split('hello', '', 3);
   SELECT split('hello', '', 5);
   SELECT split('hello', '', 100);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1806176091


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationRegexpExpressionsSuite.scala:
##########
@@ -57,25 +57,31 @@ class CollationRegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalH
   }
 
   test("StringSplit expression with collated strings") {
-    case class StringSplitTestCase[R](s: String, r: String, collation: String, expected: R)
+    case class StringSplitTestCase[R](s: String, r: String, collation: String,
+      expected: R, limit: Int)
     val testCases = Seq(
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C")),
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C")),
-      StringSplitTestCase("", "", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C")),
-      StringSplitTestCase("", "[1-9]+", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase(null, "[1-9]+", "UTF8_BINARY", null),
-      StringSplitTestCase("1A2B3C", null, "UTF8_BINARY", null),
-      StringSplitTestCase(null, null, "UTF8_BINARY", null)
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C"), -1),
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C"), -1),
+      StringSplitTestCase("", "", "UTF8_BINARY", Seq(""), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), 0),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1A2B3C"), 1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 3),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3C"), 5),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), 100),

Review Comment:
   ```suggestion
         StringSplitTestCase("1A2B3C", "", "UTF8_LCASE", Seq("1", "A", "2", "B", "3", "C"), -1),
         StringSplitTestCase("1A2B3C", "", "UTF8_LCASE", Seq("1", "A", "2", "B", "3", "C"), 0),
         StringSplitTestCase("1A2B3C", "", "UTF8_LCASE", Seq("1A2B3C"), 1),
         StringSplitTestCase("1A2B3C", "", "UTF8_LCASE", Seq("1", "A", "2B3C"), 3),
         StringSplitTestCase("1A2B3C", "", "UTF8_LCASE", Seq("1", "A", "2", "B", "3C"), 5),
         StringSplitTestCase("1A2B3C", "", "UTF8_LCASE", Seq("1", "A", "2", "B", "3", "C"), 100),
   ```
   sorry, I meant to request using collation (other than UTF8_BINARY) here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-3089849108

   @cloud-fan This is a behavior change since Spark 3.4.
     | SELECT str_to_map('p9999.c1.m9999.l5352',   '\\.', '')['p']; | SELECT split('p9999', '', 2);
   -- | -- | --
   Spark <= 3.3 | 9999 | ["p","9999"]
   Spark > 3.4 | 9 | ["p","9"]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "DenineLu (via GitHub)" <gi...@apache.org>.
DenineLu commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-3105879188

   I added a configuration to fall back to the current behavior, can you review it again? @wangyum @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2413145035

   cc @vitaliili-db


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48470:
URL: https://github.com/apache/spark/pull/48470#issuecomment-2628614365

   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804612691


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollationRegexpExpressionsSuite.scala:
##########
@@ -57,25 +57,28 @@ class CollationRegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalH
   }
 
   test("StringSplit expression with collated strings") {
-    case class StringSplitTestCase[R](s: String, r: String, collation: String, expected: R)
+    case class StringSplitTestCase[R](s: String, r: String, collation: String,
+      expected: R, limit: Int)
     val testCases = Seq(
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C")),
-      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", "")),
-      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C")),
-      StringSplitTestCase("", "", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C")),
-      StringSplitTestCase("", "[1-9]+", "UTF8_BINARY", Seq("")),
-      StringSplitTestCase(null, "[1-9]+", "UTF8_BINARY", null),
-      StringSplitTestCase("1A2B3C", null, "UTF8_BINARY", null),
-      StringSplitTestCase(null, null, "UTF8_BINARY", null)
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_BINARY", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_BINARY", Seq("1A2B3C"), -1),
+      StringSplitTestCase("1A2B3C", "[ABC]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[abc]", "UTF8_LCASE", Seq("1", "2", "3", ""), -1),
+      StringSplitTestCase("1A2B3C", "[1-9]+", "UTF8_BINARY", Seq("", "A", "B", "C"), -1),
+      StringSplitTestCase("", "", "UTF8_BINARY", Seq(""), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2", "B", "3", "C"), -1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1A2B3C"), 1),
+      StringSplitTestCase("1A2B3C", "", "UTF8_BINARY", Seq("1", "A", "2B3C"), 3),

Review Comment:
   let's add all of these



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit [spark]

Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #48470:
URL: https://github.com/apache/spark/pull/48470#discussion_r1804617069


##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -1488,13 +1488,13 @@ public UTF8String[] split(UTF8String pattern, int limit) {
       int newLimit = limit > numChars() || limit <= 0 ? numChars() : limit;
       byte[] input = getBytes();
       int byteIndex = 0;
-      int charIndex = 0;
       UTF8String[] result = new UTF8String[newLimit];
-      while (charIndex < newLimit) {
+      for (int i = 0; i < newLimit - 1; i++) {
         int currCharNumBytes = numBytesForFirstByte(input[byteIndex]);
-        result[charIndex++] = UTF8String.fromBytes(input, byteIndex, currCharNumBytes);
+        result[i] = UTF8String.fromBytes(input, byteIndex, currCharNumBytes);
         byteIndex += currCharNumBytes;
       }
+      result[newLimit - 1] = UTF8String.fromBytes(input, byteIndex, numBytes() - byteIndex);

Review Comment:
   is `ArrayIndexOutOfBoundsException` possible here?
   what if newLimit=0 (i.e. numChars()=0, limit=-1)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org