You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by huaxingao <gi...@git.apache.org> on 2018/05/13 16:17:14 UTC
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
GitHub user huaxingao opened a pull request:
https://github.com/apache/spark/pull/21313
[SPARK-24187][R][SQL]Add array_join function to SparkR
## What changes were proposed in this pull request?
This PR adds array_join function to SparkR
## How was this patch tested?
Add unit test in test_sparkSQL.R
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/huaxingao/spark spark-24187
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21313.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21313
----
commit 0c21160751c82c4c453efea72e0553f6802f24a2
Author: Huaxin Gao <hu...@...>
Date: 2018-05-13T16:07:14Z
[SPARK-24187][R][SQL]Adding array_join function to SparkR
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192564574
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,28 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
--- End diff --
Null -> NA?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192578750
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
--- End diff --
@felixcheung scala doesn't have a doc for param delimiter. I added this myself. What I am trying to say is "one or more characters". I will change to "a character string" so it will be
```
@param delimiter a character string to use to concatenate the elements of column.
```
Does this look ok to you?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r187814160
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,28 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with null_replacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
+#' @param null_replacement character(s) to use to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column"),
+ function(x, delimiter, null_replacement = NA) {
--- End diff --
@felixcheung Python uses null_replacement but Scala uses nullReplacement. I will change to nullReplacement.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:
https://github.com/apache/spark/pull/21313
Thank you very much for your help! @HyukjinKwon @felixcheung
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192814246
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter a character string that is used to concatenate the elements of column.
+#' @param nullReplacement a character string that is used to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column", delimiter = "character"),
+ function(x, delimiter, nullReplacement = NULL) {
+ jc <- if (is.null(nullReplacement)) {
+ callJStatic("org.apache.spark.sql.functions", "array_join", x@jc, delimiter)
+ } else {
+ callJStatic("org.apache.spark.sql.functions", "array_join", x@jc, delimiter,
+ nullReplacement)
--- End diff --
It's ```Hello#FooBeautiful#FooWorld!```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91403/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91422/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90552/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3207/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21313
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192564730
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1518,6 +1518,16 @@ test_that("column functions", {
result <- collect(select(df, arrays_overlap(df[[1]], df[[2]])))[[1]]
expect_equal(result, c(TRUE, FALSE, NA))
+ # Test array_join()
+ df <- createDataFrame(list(list(list("Hello", "World!"))))
+ result <- collect(select(df, array_join(df[[1]], "#")))[[1]]
+ expect_equal(result, "Hello#World!")
+ df2 <- createDataFrame(list(list(list("Hello", NA, "World!"))))
--- End diff --
How does it work with NULL?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192578774
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
+#' @param nullReplacement character(s) to use to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column"),
+ function(x, delimiter, nullReplacement = NA) {
--- End diff --
@felixcheung
I will change the default to NULL.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3767/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192577048
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
--- End diff --
I didn't check scala - what's "(s)" here in "character(s)" mean? I ask because "character" refers to the type in R
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3176/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21313
sure, let's try 4-5
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21313
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192925340
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter a character string that is used to concatenate the elements of column.
+#' @param nullReplacement a character string that is used to replace the Null values.
--- End diff --
could you change this to
`nullReplacement an optional character string `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90601/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3181/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #90601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90601/testReport)** for PR 21313 at commit [`4f429e4`](https://github.com/apache/spark/commit/4f429e4c0e66051d77832bc6822212bc40c09dc1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91422/testReport)** for PR 21313 at commit [`92c41c5`](https://github.com/apache/spark/commit/92c41c5c98fe8e8802201cb1799b0cf5b56762ac).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192578796
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
+#' @param nullReplacement character(s) to use to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column"),
--- End diff --
@felixcheung
I will add the type so it will be
```
signature(x = "Column", delimiter = "character"),
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192627641
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter a character string that is used to concatenate the elements of column.
+#' @param nullReplacement a character string that is used to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column", delimiter = "character"),
+ function(x, delimiter, nullReplacement = NULL) {
+ jc <- if (is.null(nullReplacement)) {
+ callJStatic("org.apache.spark.sql.functions", "array_join", x@jc, delimiter)
+ } else {
+ callJStatic("org.apache.spark.sql.functions", "array_join", x@jc, delimiter,
+ nullReplacement)
--- End diff --
`as.character(nullReplacement)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91430/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21313
Yea, it's not necessarily to add all but adding them in a batch sounds making sense at least once you got used to it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91451/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #90552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90552/testReport)** for PR 21313 at commit [`0c21160`](https://github.com/apache/spark/commit/0c21160751c82c4c453efea72e0553f6802f24a2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #90560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90560/testReport)** for PR 21313 at commit [`25977f0`](https://github.com/apache/spark/commit/25977f04b86094d8919a67aaca4985d07b0f303b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #90560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90560/testReport)** for PR 21313 at commit [`25977f0`](https://github.com/apache/spark/commit/25977f04b86094d8919a67aaca4985d07b0f303b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3781/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91422/testReport)** for PR 21313 at commit [`92c41c5`](https://github.com/apache/spark/commit/92c41c5c98fe8e8802201cb1799b0cf5b56762ac).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91403/testReport)** for PR 21313 at commit [`e05e701`](https://github.com/apache/spark/commit/e05e701f3027607fc6942a81e1a9f8d0a5cc6e5f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192627948
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter a character string that is used to concatenate the elements of column.
+#' @param nullReplacement a character string that is used to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column", delimiter = "character"),
+ function(x, delimiter, nullReplacement = NULL) {
+ jc <- if (is.null(nullReplacement)) {
+ callJStatic("org.apache.spark.sql.functions", "array_join", x@jc, delimiter)
+ } else {
+ callJStatic("org.apache.spark.sql.functions", "array_join", x@jc, delimiter,
+ nullReplacement)
--- End diff --
re https://github.com/apache/spark/pull/21313#discussion_r192578750
so what's the behavior if delimiter is more than one character?
like `array_join(df$a, "#Foo", "Beautiful")`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90560/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:
https://github.com/apache/spark/pull/21313
There are still quite a lot of the SQL functions to be added in R. We can bundle several of the functions together in one PR, but I guess it's too much work to add all of them in one PR. @felixcheung @HyukjinKwon
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91475 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91475/testReport)** for PR 21313 at commit [`55b4518`](https://github.com/apache/spark/commit/55b45180f223356887d88c41b4ae538500fe31ae).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91451/testReport)** for PR 21313 at commit [`b0b1415`](https://github.com/apache/spark/commit/b0b14151e42ef638644a6072e0edbec240283c1f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192576992
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
+#' @param nullReplacement character(s) to use to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column"),
+ function(x, delimiter, nullReplacement = NA) {
--- End diff --
wait.. why is `nullReplacement` default to NA? that's a bit unusual
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192564618
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,28 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
+#' @param nullReplacement character(s) to use to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column"),
+ function(x, delimiter, nullReplacement = NA) {
+ jc <- if (is.na(nullReplacement)) {
+ callJStatic("org.apache.spark.sql.functions", "array_join", x@jc, delimiter)
+ }
+ else {
--- End diff --
nit: `} else {`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91475 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91475/testReport)** for PR 21313 at commit [`55b4518`](https://github.com/apache/spark/commit/55b45180f223356887d88c41b4ae538500fe31ae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #90601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90601/testReport)** for PR 21313 at commit [`4f429e4`](https://github.com/apache/spark/commit/4f429e4c0e66051d77832bc6822212bc40c09dc1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #90552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90552/testReport)** for PR 21313 at commit [`0c21160`](https://github.com/apache/spark/commit/0c21160751c82c4c453efea72e0553f6802f24a2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r187810848
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,28 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with null_replacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
+#' @param null_replacement character(s) to use to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column"),
+ function(x, delimiter, null_replacement = NA) {
--- End diff --
generally we don't use `_` in variable name in R. what's the variable name in Scala?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192577008
--- Diff: R/pkg/R/functions.R ---
@@ -3006,6 +3008,27 @@ setMethod("array_contains",
column(jc)
})
+#' @details
+#' \code{array_join}: Concatenates the elements of column using the delimiter.
+#' Null values are replaced with nullReplacement if set, otherwise they are ignored.
+#'
+#' @param delimiter character(s) to use to concatenate the elements of column.
+#' @param nullReplacement character(s) to use to replace the Null values.
+#' @rdname column_collection_functions
+#' @aliases array_join array_join,Column-method
+#' @note array_join since 2.4.0
+setMethod("array_join",
+ signature(x = "Column"),
--- End diff --
is `delimiter` supposed to be a string? it's better to have its type in the signature
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3786/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91430/testReport)** for PR 21313 at commit [`901ff32`](https://github.com/apache/spark/commit/901ff32a03c6ec0c16a0ff7c625781ccf2355a54).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91430/testReport)** for PR 21313 at commit [`901ff32`](https://github.com/apache/spark/commit/901ff32a03c6ec0c16a0ff7c625781ccf2355a54).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91451/testReport)** for PR 21313 at commit [`b0b1415`](https://github.com/apache/spark/commit/b0b14151e42ef638644a6072e0edbec240283c1f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3810/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on the issue:
https://github.com/apache/spark/pull/21313
@felixcheung @HyukjinKwon Any more comments?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91475/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21313
**[Test build #91403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91403/testReport)** for PR 21313 at commit [`e05e701`](https://github.com/apache/spark/commit/e05e701f3027607fc6942a81e1a9f8d0a5cc6e5f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3797/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21313: [SPARK-24187][R][SQL]Add array_join function to S...
Posted by huaxingao <gi...@git.apache.org>.
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21313#discussion_r192574515
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1518,6 +1518,16 @@ test_that("column functions", {
result <- collect(select(df, arrays_overlap(df[[1]], df[[2]])))[[1]]
expect_equal(result, c(TRUE, FALSE, NA))
+ # Test array_join()
+ df <- createDataFrame(list(list(list("Hello", "World!"))))
+ result <- collect(select(df, array_join(df[[1]], "#")))[[1]]
+ expect_equal(result, "Hello#World!")
+ df2 <- createDataFrame(list(list(list("Hello", NA, "World!"))))
--- End diff --
@HyukjinKwon Thank you very much for your review. I will add a test for NULL and also change the }.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21313: [SPARK-24187][R][SQL]Add array_join function to SparkR
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21313
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org