You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by chongguang <gi...@git.apache.org> on 2018/06/17 20:26:46 UTC
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains function deals ...
GitHub user chongguang opened a pull request:
https://github.com/apache/spark/pull/21581
[SPARK-24574][SQL] array_contains function deals with Column type
## What changes were proposed in this pull request?
For the function ```def array_contains(column: Column, value: Any): Column ``` , if we pass the `value` parameter as a Column type, it will yield a runtime exception.
This PR proposes a pattern matching to detect if `value` is of type Column. If yes, it will use the .expr of the column, otherwise it will work as it used to.
## How was this patch tested?
Unit test modified to cover this code change.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chongguang/spark SPARK-24574
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21581.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21581
----
commit 27733f9ad56657925c176ae394114e0429aa9a0b
Author: Chongguang LIU <ch...@...>
Date: 2018-06-17T18:17:15Z
array_contains function deals with Column type for the second parameter.
commit 28aa51554f4c730fae3c8090ac3c268e1ddfa4f8
Author: Chongguang LIU <ch...@...>
Date: 2018-06-17T19:58:57Z
add unit test for Column type
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92017/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains function deals ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196016969
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3077,12 +3077,16 @@ object functions {
//////////////////////////////////////////////////////////////////////////////////////////////
/**
- * Returns null if the array is null, true if the array contains `value`, and false otherwise.
+ * Returns null if the array is null, true if the array contains `value` or the content of
+ * `value` if it is of type Column, and false otherwise.
* @group collection_funcs
* @since 1.5.0
*/
def array_contains(column: Column, value: Any): Column = withExpr {
- ArrayContains(column.expr, Literal(value))
--- End diff --
Oh, I just found @maropu has pointed it out in https://github.com/apache/spark/pull/21581#issuecomment-397928010.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21581
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21581
oh haha FYI that works after it's merged if @chongguang link the email into his Github profile. I asked the same thing in databricks/spark-xml before :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains, array_position...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21581
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92029/testReport)** for PR 21581 at commit [`b6e150c`](https://github.com/apache/spark/commit/b6e150c5e9101c6135ed1020ce97f7e754613035).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains, array_position...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196633968
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -571,6 +571,10 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSQLContext {
df.selectExpr("array_contains(a, 1)"),
Seq(Row(true), Row(false))
)
+ checkAnswer(
+ df.select(array_contains(df("a"), df("c"))),
+ Seq(Row(true), Row(false))
+ )
--- End diff --
Can you add another test to use `selectExpr`, e.g., `df.selectExpr("array_contains(a, c)")`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92028/testReport)** for PR 21581 at commit [`e65611e`](https://github.com/apache/spark/commit/e65611e451bf49b38850af84a89510ac8a749cbd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/21581
LGTM pending tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/21581
But the emails of commits in this pr seem not valid, just for the local computer.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains, array_position...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196633908
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3082,7 +3082,10 @@ object functions {
* @since 1.5.0
*/
def array_contains(column: Column, value: Any): Column = withExpr {
- ArrayContains(column.expr, Literal(value))
+ value match {
--- End diff --
On second thoughts, we should use `lit()`, e.g., `ArrayContains(column.expr, lit(value).expr)`? WDYT? @viirya @maropu @HyukjinKwon
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92017/testReport)** for PR 21581 at commit [`28aa515`](https://github.com/apache/spark/commit/28aa51554f4c730fae3c8090ac3c268e1ddfa4f8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21581
you could just push the newer changes. that will retrigger the tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by chongguang <gi...@git.apache.org>.
Github user chongguang commented on the issue:
https://github.com/apache/spark/pull/21581
@viirya @maropu @HyukjinKwon All the 4 functions have been modified.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains function deals ...
Posted by chongguang <gi...@git.apache.org>.
Github user chongguang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196002327
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3077,12 +3077,16 @@ object functions {
//////////////////////////////////////////////////////////////////////////////////////////////
/**
- * Returns null if the array is null, true if the array contains `value`, and false otherwise.
+ * Returns null if the array is null, true if the array contains `value` or the content of
--- End diff --
Hey, thanks for the message. Do you want me to change the comment back? I see that you have started the test, is it too late?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92015/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92015 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92015/testReport)** for PR 21581 at commit [`28aa515`](https://github.com/apache/spark/commit/28aa51554f4c730fae3c8090ac3c268e1ddfa4f8).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92030/testReport)** for PR 21581 at commit [`8440d5b`](https://github.com/apache/spark/commit/8440d5b08bd33321533635c4b6329ce5c7b843d2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92017/testReport)** for PR 21581 at commit [`28aa515`](https://github.com/apache/spark/commit/28aa51554f4c730fae3c8090ac3c268e1ddfa4f8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21581
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92030/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains function deals ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r195954291
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3077,12 +3077,16 @@ object functions {
//////////////////////////////////////////////////////////////////////////////////////////////
/**
- * Returns null if the array is null, true if the array contains `value`, and false otherwise.
+ * Returns null if the array is null, true if the array contains `value` or the content of
--- End diff --
We need to update this comment? I think `content of value` is a little ambiguous.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains, array_position...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196707308
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3082,7 +3082,10 @@ object functions {
* @since 1.5.0
*/
def array_contains(column: Column, value: Any): Column = withExpr {
- ArrayContains(column.expr, Literal(value))
+ value match {
--- End diff --
@chongguang I mean, use just `ArrayContains(column.expr, lit(value).expr)` instead of `value match { ...`. The `lit()` should handle literals and columns well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92136/testReport)** for PR 21581 at commit [`ddd94f7`](https://github.com/apache/spark/commit/ddd94f7e27f3622db89abd8c4f85975fa0034fff).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92030/testReport)** for PR 21581 at commit [`8440d5b`](https://github.com/apache/spark/commit/8440d5b08bd33321533635c4b6329ce5c7b843d2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains, array_position...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196636915
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3082,7 +3082,10 @@ object functions {
* @since 1.5.0
*/
def array_contains(column: Column, value: Any): Column = withExpr {
- ArrayContains(column.expr, Literal(value))
+ value match {
--- End diff --
Yes, I think so.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92141/testReport)** for PR 21581 at commit [`ddd94f7`](https://github.com/apache/spark/commit/ddd94f7e27f3622db89abd8c4f85975fa0034fff).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains, array_position...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196649798
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3082,7 +3082,10 @@ object functions {
* @since 1.5.0
*/
def array_contains(column: Column, value: Any): Column = withExpr {
- ArrayContains(column.expr, Literal(value))
+ value match {
--- End diff --
+1
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92028/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92126/testReport)** for PR 21581 at commit [`f8c5b43`](https://github.com/apache/spark/commit/f8c5b43ddc3b9209d9b7972e28d237d205190136).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21581
Oh, I see.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains function deals ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196016648
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3077,12 +3077,16 @@ object functions {
//////////////////////////////////////////////////////////////////////////////////////////////
/**
- * Returns null if the array is null, true if the array contains `value`, and false otherwise.
+ * Returns null if the array is null, true if the array contains `value` or the content of
+ * `value` if it is of type Column, and false otherwise.
* @group collection_funcs
* @since 1.5.0
*/
def array_contains(column: Column, value: Any): Column = withExpr {
- ArrayContains(column.expr, Literal(value))
--- End diff --
Other similar expressions are `ArrayPosition`, `ElementAt`, `ArrayRemove`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21581
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains function deals ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196012357
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3077,12 +3077,16 @@ object functions {
//////////////////////////////////////////////////////////////////////////////////////////////
/**
- * Returns null if the array is null, true if the array contains `value`, and false otherwise.
+ * Returns null if the array is null, true if the array contains `value` or the content of
--- End diff --
you can fix now
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21581: [SPARK-24574][SQL] array_contains, array_position...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21581#discussion_r196636085
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3082,7 +3082,10 @@ object functions {
* @since 1.5.0
*/
def array_contains(column: Column, value: Any): Column = withExpr {
- ArrayContains(column.expr, Literal(value))
+ value match {
--- End diff --
Yup, that's what I was thinking too from my glance.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92141/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/21581
Jenkins, retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/21581
@chongguang I think this pr is ready to merge, so I tried, but seems like the commits in this pr aren't connected with your GitHub account. If you want to connect the merge commit to your account, could you let me know the email address connected to your account? Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92126/testReport)** for PR 21581 at commit [`f8c5b43`](https://github.com/apache/spark/commit/f8c5b43ddc3b9209d9b7972e28d237d205190136).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21581
BTW, I found the other 3 similar issues there;
```
scala> Seq((Seq(1, 2, 3), 2)).toDF("a", "b").selectExpr("array_position(a, b)").show
+--------------------+
|array_position(a, b)|
+--------------------+
| 2|
+--------------------+
scala> Seq((Seq(1, 2, 3), 2)).toDF("a", "b").selectExpr("element_at(a, b)").show
+----------------+
|element_at(a, b)|
+----------------+
| 2|
+----------------+
scala> Seq((Seq(1, 2, 3), 2)).toDF("a", "b").selectExpr("array_remove(a, b)").show
+------------------+
|array_remove(a, b)|
+------------------+
| [1, 3]|
+------------------+
```
I think this is a tiny fix, so IMHO this pr might need to address all the issues here. cc: @ueshin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92029/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92126/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92136/testReport)** for PR 21581 at commit [`ddd94f7`](https://github.com/apache/spark/commit/ddd94f7e27f3622db89abd8c4f85975fa0034fff).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by chongguang <gi...@git.apache.org>.
Github user chongguang commented on the issue:
https://github.com/apache/spark/pull/21581
@ueshin Done! :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by chongguang <gi...@git.apache.org>.
Github user chongguang commented on the issue:
https://github.com/apache/spark/pull/21581
Hey @ueshin I just updated the email address linked to my github account, it is now lcg31439@gmail.com
Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92015/testReport)** for PR 21581 at commit [`28aa515`](https://github.com/apache/spark/commit/28aa51554f4c730fae3c8090ac3c268e1ddfa4f8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92141/testReport)** for PR 21581 at commit [`ddd94f7`](https://github.com/apache/spark/commit/ddd94f7e27f3622db89abd8c4f85975fa0034fff).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92028/testReport)** for PR 21581 at commit [`e65611e`](https://github.com/apache/spark/commit/e65611e451bf49b38850af84a89510ac8a749cbd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains function deals with Co...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21581
**[Test build #92029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92029/testReport)** for PR 21581 at commit [`b6e150c`](https://github.com/apache/spark/commit/b6e150c5e9101c6135ed1020ce97f7e754613035).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21581
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92136/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org