You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/17 04:44:20 UTC

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32566: [SPARK-35418][SQL] Add sentences function to functions.{scala,py}

HyukjinKwon commented on a change in pull request #32566:
URL: https://github.com/apache/spark/pull/32566#discussion_r633216739



##########
File path: python/pyspark/sql/functions.py
##########
@@ -2681,6 +2681,30 @@ def overlay(src, replace, pos, len=-1):
     ))
 
 
+def sentences(str, lang="", country=""):

Review comment:
       lang -> language to be consistent

##########
File path: python/pyspark/sql/functions.py
##########
@@ -2681,6 +2681,30 @@ def overlay(src, replace, pos, len=-1):
     ))
 
 
+def sentences(str, lang="", country=""):
+    """
+    Splits a string into arrays of sentences, where each sentence is an array of words.
+    The 'lang' and 'country' arguments are optional, and if omitted, the default locale is used.
+
+    .. versionadded:: 3.2.0
+

Review comment:
       Can we also add:
   
   ```
       Parameters
       ----------
       str: ...
   ```
   
   (see also https://numpydoc.readthedocs.io/en/latest/format.html)

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2867,6 +2867,25 @@ object functions {
     new Overlay(src.expr, replace.expr, pos.expr)
   }
 
+  /**
+   * Splits a string into arrays of sentences, where each sentence is an array of words.
+   * @group string_funcs
+   * @since 3.2.0
+   */
+  def sentences(str: Column, language: String, country: String): Column = withExpr {

Review comment:
       I think we should make it `Column` for both `language` and `country` in case they we make it accept non-literal values (which is also documented on the top of this file).

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
##########
@@ -589,6 +596,6 @@ class StringFunctionsSuite extends QueryTest with SharedSparkSession {
       df2.selectExpr("str_to_map(a)"),
       Seq(Row(Map("a" -> "1", "b" -> "2", "c" -> "3")))
     )
-
   }
+

Review comment:
       no biggie but I would remove these newline changes

##########
File path: python/pyspark/sql/functions.py
##########
@@ -2681,6 +2681,30 @@ def overlay(src, replace, pos, len=-1):
     ))
 
 
+def sentences(str, lang="", country=""):

Review comment:
       no biggie but I would avoid `str` as it shadows the built-in function `str` in Python. Maybe just simply `string`.

##########
File path: python/pyspark/sql/functions.py
##########
@@ -2681,6 +2681,30 @@ def overlay(src, replace, pos, len=-1):
     ))
 
 
+def sentences(str, lang="", country=""):
+    """
+    Splits a string into arrays of sentences, where each sentence is an array of words.
+    The 'lang' and 'country' arguments are optional, and if omitted, the default locale is used.
+
+    .. versionadded:: 3.2.0
+
+    Examples

Review comment:
       ```suggestion
       Examples
       --------
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org