You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by mgaido91 <gi...@git.apache.org> on 2018/04/09 16:22:05 UTC

[GitHub] spark pull request #21011: [SPARK-23916][SQL] Add array_join function

GitHub user mgaido91 opened a pull request:

    https://github.com/apache/spark/pull/21011

    [SPARK-23916][SQL] Add array_join function

    ## What changes were proposed in this pull request?
    
    The PR adds the SQL function `array_join`. The behavior of the function is based on Presto's one.
    
    The function accepts an `array` of `string` which is to be joined, a `string` which is the delimiter to use between the items of the first argument and optionally a `string` which is used to replace `null` values.
    
    ## How was this patch tested?
    
    added UTs


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mgaido91/spark SPARK-23916

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21011
    
----
commit 1408cfcd8f0ad2a571d29b57d71128584ea4b4f0
Author: Marco Gaido <ma...@...>
Date:   2018-04-09T16:16:43Z

    [SPARK-23916][SQL] Add array_join function

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21011: [SPARK-23916][SQL] Add array_join function

Posted by ueshin <gi...@git.apache.org>.

Github user ueshin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21011#discussion_r181380233
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
    @@ -287,3 +288,173 @@ case class ArrayContains(left: Expression, right: Expression)
     
       override def prettyName: String = "array_contains"
     }
    +
    +/**
    + * Creates a String containing all the elements of the input array separated by the delimiter.
    + */
    +@ExpressionDescription(
    +  usage = """
    +    _FUNC_(array, delimiter[, nullReplacement]) - Concatenates the elements of the given array
    +      using the delimiter and an optional string to replace nulls. If no value is set for
    +      nullReplacement, any null value is filtered.""",
    +  examples = """
    +    Examples:
    +      > SELECT _FUNC_(array('hello', 'world'), ' ');
    +       hello world
    +      > SELECT _FUNC_(array('hello', null ,'world'), ' ');
    +       hello world
    +      > SELECT _FUNC_(array('hello', null ,'world'), ' ', ',');
    +       hello , world
    +  """, since = "2.4.0")
    +case class ArrayJoin(
    +    array: Expression,
    +    delimiter: Expression,
    +    nullReplacement: Option[Expression]) extends Expression with ExpectsInputTypes {
    +
    +  def this(array: Expression, delimiter: Expression) = this(array, delimiter, None)
    +
    +  def this(array: Expression, delimiter: Expression, nullReplacement: Expression) =
    +    this(array, delimiter, Some(nullReplacement))
    +
    +  override def inputTypes: Seq[AbstractDataType] = if (nullReplacement.isDefined) {
    +      Seq(ArrayType(StringType), StringType, StringType)
    +    } else {
    +      Seq(ArrayType(StringType), StringType)
    +    }
    --- End diff --
    
    Hmm, I think the indent is 2 spaces in this case. For example, [namedExpressions.scala#L170-L174](https://github.com/mgaido91/spark/blob/e52ff856d42adc5af2e2b2593c2e63d5c3f3a205/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala#L170-L174) or [regexpExpressions.scala#L46-L51](https://github.com/mgaido91/spark/blob/e52ff856d42adc5af2e2b2593c2e63d5c3f3a205/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L46-L51).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21011
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2538/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21011
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function

Posted by mgaido91 <gi...@git.apache.org>.

Github user mgaido91 commented on the issue:

    https://github.com/apache/spark/pull/21011
  
    kindly ping @ueshin 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21011
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21011
  
    **[Test build #89094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89094/testReport)** for PR 21011 at commit [`e52ff85`](https://github.com/apache/spark/commit/e52ff856d42adc5af2e2b2593c2e63d5c3f3a205).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21011
  
    **[Test build #89534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89534/testReport)** for PR 21011 at commit [`b1597d7`](https://github.com/apache/spark/commit/b1597d791b0302a4541cf545956ecf0f3454f056).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21011
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89342/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org