You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/03/25 07:40:21 UTC
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/17429
[MINOR][DOCS] Match several documentation changes in Scala to R/Python
## What changes were proposed in this pull request?
This PR proposes to match minor documentations changes in https://github.com/apache/spark/pull/17399 and https://github.com/apache/spark/pull/17380 to R/Python.
## How was this patch tested?
Manual tests in Python , Python tests via `./python/run-tests.py --module=pyspark-sql` and lint-checks for Python/R.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark minor-match-doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17429.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17429
----
commit 5f0ebddc36bb4f63aca162d3f0c23d56860a55b6
Author: hyukjinkwon <gu...@gmail.com>
Date: 2017-03-25T07:15:59Z
Match Scala/Python/R changes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108052897
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
:param col: name of column containing array
:param value: value to check for in array
>>> df = spark.createDataFrame([(["a", "b", "c"],), ([],)], ['data'])
>>> df.select(array_contains(df.data, "a")).collect()
[Row(array_contains(data, a)=True), Row(array_contains(data, a)=False)]
+ >>> df = spark.createDataFrame([(["1", "2", "3"],), ([],)], ['data'])
+ >>> df.select(array_contains(df.data, 1)).collect()
--- End diff --
Yea, the data is string array and the given value is a number. This case is about implicit casting case. I added this example and removed `The collection elements and value must be of the same type`. This one is the bit I described in https://github.com/apache/spark/pull/17429#discussion_r108030625
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17429
**[Test build #75248 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75248/testReport)** for PR 17429 at commit [`d05aba5`](https://github.com/apache/spark/commit/d05aba5b70ad22fd0e5661168dc6deceff51a13e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17429
Thank you @felixcheung, IMO, I think it is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17429
**[Test build #75211 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75211/testReport)** for PR 17429 at commit [`5f0ebdd`](https://github.com/apache/spark/commit/5f0ebddc36bb4f63aca162d3f0c23d56860a55b6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17429
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108053036
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
:param col: name of column containing array
:param value: value to check for in array
>>> df = spark.createDataFrame([(["a", "b", "c"],), ([],)], ['data'])
>>> df.select(array_contains(df.data, "a")).collect()
[Row(array_contains(data, a)=True), Row(array_contains(data, a)=False)]
+ >>> df = spark.createDataFrame([(["1", "2", "3"],), ([],)], ['data'])
+ >>> df.select(array_contains(df.data, 1)).collect()
--- End diff --
Maybe I could just remove this.. though another advantage of this is to test this function with a implicit cast case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17429
I'm fine as-is and then another iteration to handle R/python difference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108046296
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
--- End diff --
like my other comment, probably should say `True` when in Python, @holdenk?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17429
**[Test build #75248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75248/testReport)** for PR 17429 at commit [`d05aba5`](https://github.com/apache/spark/commit/d05aba5b70ad22fd0e5661168dc6deceff51a13e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108046277
--- Diff: R/pkg/R/functions.R ---
@@ -3548,7 +3548,7 @@ setMethod("row_number",
#' array_contains
#'
-#' Returns true if the array contain the value.
+#' Returns null if the array is null, true if the array contains the value, and false otherwise.
--- End diff --
for null, we need to be more careful - `null` in JVM should show up as `NA` in R.
also, should `true` be `TRUE` and `false` be `FALSE` to match R type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17429
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108052977
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
:param col: name of column containing array
:param value: value to check for in array
>>> df = spark.createDataFrame([(["a", "b", "c"],), ([],)], ['data'])
>>> df.select(array_contains(df.data, "a")).collect()
[Row(array_contains(data, a)=True), Row(array_contains(data, a)=False)]
+ >>> df = spark.createDataFrame([(["1", "2", "3"],), ([],)], ['data'])
+ >>> df.select(array_contains(df.data, 1)).collect()
--- End diff --
ah got it. it's quite subtle though, I wonder if it might confuse than help more...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17429
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108030625
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
--- End diff --
Other documentation use `true` rather than `True`. So, I matach this to `true`. I am willing to sweep if anyone feels this should be fixed.
The reason I removed `The collection elements and value must be of the same type` is it seems we can provide other types that are implicitly castable.
This is not documented in Scala/R too. So, I instead provided a doctest as an excpetion in the Python documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108068955
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
:param col: name of column containing array
:param value: value to check for in array
>>> df = spark.createDataFrame([(["a", "b", "c"],), ([],)], ['data'])
>>> df.select(array_contains(df.data, "a")).collect()
[Row(array_contains(data, a)=True), Row(array_contains(data, a)=False)]
+ >>> df = spark.createDataFrame([(["1", "2", "3"],), ([],)], ['data'])
+ >>> df.select(array_contains(df.data, 1)).collect()
--- End diff --
unless Holden thought otherwise, let's remove this from docstring and ok to add to test explicitly :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75211/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17429
**[Test build #75247 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75247/testReport)** for PR 17429 at commit [`33f1328`](https://github.com/apache/spark/commit/33f132808562aaa78446c05f1fb3462603346935).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17429
thanks, merged to master, since part of the fix in scala was in master only. if you think it should also be in branch-2.1, let me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17429
**[Test build #75247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75247/testReport)** for PR 17429 at commit [`33f1328`](https://github.com/apache/spark/commit/33f132808562aaa78446c05f1fb3462603346935).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17429
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17429
**[Test build #75211 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75211/testReport)** for PR 17429 at commit [`5f0ebdd`](https://github.com/apache/spark/commit/5f0ebddc36bb4f63aca162d3f0c23d56860a55b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108047783
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
--- End diff --
I am willing to grep and replace too. Please let me know @holdenk.
FWIW, If deciding this drags this PR, I would like to say we should merge this first if you and @holdenk do not strongly feel about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108052818
--- Diff: python/pyspark/sql/functions.py ---
@@ -1675,15 +1675,18 @@ def array(*cols):
@since(1.5)
def array_contains(col, value):
"""
- Collection function: returns True if the array contains the given value. The collection
- elements and value must be of the same type.
+ Collection function: returns null if the array is null, true if the array contains the
+ given value, and false otherwise.
:param col: name of column containing array
:param value: value to check for in array
>>> df = spark.createDataFrame([(["a", "b", "c"],), ([],)], ['data'])
>>> df.select(array_contains(df.data, "a")).collect()
[Row(array_contains(data, a)=True), Row(array_contains(data, a)=False)]
+ >>> df = spark.createDataFrame([(["1", "2", "3"],), ([],)], ['data'])
+ >>> df.select(array_contains(df.data, 1)).collect()
--- End diff --
generally docstring we keep as simple as possible - any particular reason to add "1" vs the existing "a"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75248/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17429: [MINOR][DOCS] Match several documentation changes...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17429#discussion_r108047754
--- Diff: R/pkg/R/functions.R ---
@@ -3548,7 +3548,7 @@ setMethod("row_number",
#' array_contains
#'
-#' Returns true if the array contain the value.
+#' Returns null if the array is null, true if the array contains the value, and false otherwise.
--- End diff --
Yea, I agree with being careful. For this PR, I followed the others. I skimmed again and it seems we have not used the notation for `None`, `True` and `False` in `functions.py`, and `NA, `TRUE` and `FALSE` in both `functions.R`.
I can grep and replace.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17429: [MINOR][DOCS] Match several documentation changes in Sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17429
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75247/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org