You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jovanpavl-db (via GitHub)" <gi...@apache.org> on 2024/10/18 13:24:32 UTC

[PR] init commit. [spark]

jovanpavl-db opened a new pull request, #48545:
URL: https://github.com/apache/spark/pull/48545

   ### What changes were proposed in this pull request?
   Add RTRIM collation suggestion for collation expressions in case of the wrong name.
   
   
   ### Why are the changes needed?
   Better customer experience. 
   
   
   ### Does this PR introduce _any_ user-facing change?
   Yes.
   
   ### How was this patch tested?
   Added tests in CollationFactorySuite.
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-49663][SQL] Enable RTRIM suggestions in collation expressions [spark]

Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #48545: [SPARK-49663][SQL] Enable RTRIM suggestions in collation expressions
URL: https://github.com/apache/spark/pull/48545


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-49663][SQL] Enable RTRIM suggestions in collation expressions [spark]

Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48545:
URL: https://github.com/apache/spark/pull/48545#discussion_r1820838752


##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -106,21 +106,27 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
       ("UTF8_BINARY_AI","UTF8_BINARY"),
       ("UTF8_BINARY_CI_AI","UTF8_BINARY"),
       ("UTF8_BINARY_AI_CI","UTF8_BINARY"),
+      ("UTF8_BINARY_AI_RTRIM", "UTF8_BINARY_RTRIM"),
+      ("UTF8_BINARY_CI_RTRIM","UTF8_BINARY_RTRIM"),
+      ("UTF8_BINARY_AI_CI_RTRIM","UTF8_BINARY_RTRIM"),

Review Comment:
   let's be consistent and use spaces:
   ```suggestion
         ("UTF8_BINARY_CI_RTRIM", "UTF8_BINARY_RTRIM"),
         ("UTF8_BINARY_AI_CI_RTRIM", "UTF8_BINARY_RTRIM"),
   ```



##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -106,21 +106,27 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
       ("UTF8_BINARY_AI","UTF8_BINARY"),
       ("UTF8_BINARY_CI_AI","UTF8_BINARY"),
       ("UTF8_BINARY_AI_CI","UTF8_BINARY"),
+      ("UTF8_BINARY_AI_RTRIM", "UTF8_BINARY_RTRIM"),
+      ("UTF8_BINARY_CI_RTRIM","UTF8_BINARY_RTRIM"),
+      ("UTF8_BINARY_AI_CI_RTRIM","UTF8_BINARY_RTRIM"),
       ("UTF8_BS","UTF8_LCASE"),
       ("BINARY_UTF8","ar_SAU"),
       ("UTF8_BINARY_A","UTF8_BINARY"),
       ("UNICODE_X","UNICODE"),
       ("UNICODE_CI_X","UNICODE"),
       ("UNICODE_LCASE_X","UNICODE"),
+      ("UNICODE_RTRIM_LCASE_X","UNICODE"),

Review Comment:
   ```suggestion
         ("UNICODE_RTRIM_LCASE_X", "UNICODE"),
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-49663][SQL] Enable RTRIM suggestions in collation expressions [spark]

Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #48545:
URL: https://github.com/apache/spark/pull/48545#issuecomment-2446547146

   +1, LGTM. Merging to master.
   Thank you, @jovanpavl-db and @uros-db for your review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-49663][SQL] Enable RTRIM suggestions in collation expressions [spark]

Posted by "jovanpavl-db (via GitHub)" <gi...@apache.org>.
jovanpavl-db commented on PR #48545:
URL: https://github.com/apache/spark/pull/48545#issuecomment-2443694564

   > Does it make sense to use the Levenshtein distance for suggestions like we do in `UNRESOLVED_FIELD.WITH_SUGGESTION`, see
   > 
   > https://github.com/apache/spark/blob/bb15eb7b91ab775bdb84b6b17353a706794b122d/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala#L75-L97
   
   I went through this implementation and it seems useful, however it can be quite problematic in our case to do suggestion on specifiers (we currently want 100% match on them because they are usually 2 chars long), for example if someone puts _C you can suggest _CI, _CS and even some UNICODE collations like ci. For every wrong specifier we can have multiple suggestions so it can really explode. That's why I think we (currently) stayed out of business of doing suggestions on specifiers but rather whole collation. We can definitely rethink this approach in the future at least for the some of specifiers like trim. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org