You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/04 09:36:58 UTC
[GitHub] [spark] Ngone51 opened a new pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Ngone51 opened a new pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add examples and parameter description for these Scala functions:
* transform
* exists
* forall
* aggregate
* zip_with
* transform_keys
* transform_values
* map_filter
* map_zip_with
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Better documentation for UX.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Pass Jenkins.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582709862
**[Test build #117962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117962/testReport)** for PR 27449 at commit [`a64017a`](https://github.com/apache/spark/commit/a64017aea41cc856a528e0e2fe0690f220ae16ca).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582874119
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117979/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581912646
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582405919
**[Test build #117920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117920/testReport)** for PR 27449 at commit [`48f7411`](https://github.com/apache/spark/commit/48f74116aa21ddf55d252228fefa612e29441290).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375636476
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3530,6 +3577,12 @@ object functions {
/**
* Applies a function to every key-value pair in a map and returns
* a map with the results of those applications as the new keys for the pairs.
+ * {{{
+ * df.select(transform_keys(col("i"), (k, v) => k + v))
+ * }}}
+ *
+ * @param expr the input map column
+ * @param f (key, value) => new_key, the lambda function to transform the key of input column
Review comment:
`input map column`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581821683
cc @cloud-fan @gatorsmile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374616292
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3410,6 +3410,13 @@ object functions {
/**
* Returns an array of elements after applying a transformation to each element
* in the input array.
+ * {{{
+ * df.select(transform(col("i"), x => x + 1))
+ * df.selectExpr("transform(i, x -> x + 1")
Review comment:
Seems trailing `)` missing in the expression string.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375408066
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
Review comment:
Thanks. Got it. Since this is a new API added at 3.0.0, `MiMa` will be okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582257294
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22659/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582406729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117920/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582392618
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22696/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375385458
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
Review comment:
Hm, does that change the API? I forget when param names matter in Scala.
In any event, 'zero' isn't a terrible name as a reduce-like function needs a 'zero' value that is the result of no aggregation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375636532
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3541,6 +3594,12 @@ object functions {
/**
* Applies a function to every key-value pair in a map and returns
* a map with the results of those applications as the new values for the pairs.
+ * {{{
+ * df.select(transform_values(col("i"), (k, v) => k + v))
+ * }}}
+ *
+ * @param expr the input map column
+ * @param f (key, value) => new_value, the lambda function to transform the value of input column
Review comment:
`input map column`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375393062
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
Review comment:
Sometimes, it's `1` instead of zero. For example, `product` instead of `summation`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375400212
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
Review comment:
Good point. The scala API calls it `z` in `fold` for example. Well if it isn't flagged as an API change by MiMa, OK by me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375393062
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
Review comment:
Sometimes, it's `1` instead of zero. For example, `product` instead of `summation`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582520927
**[Test build #117933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117933/testReport)** for PR 27449 at commit [`083d38b`](https://github.com/apache/spark/commit/083d38ba3f7e245db094b70a5bda84b76b47a50b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582392612
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582874114
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581840984
thanks for doing it! also cc @ueshin @HyukjinKwon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375092137
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3541,6 +3593,12 @@ object functions {
/**
* Applies a function to every key-value pair in a map and returns
* a map with the results of those applications as the new values for the pairs.
+ * {{{
+ * df.select(transform_values(col("i"), (k, v) => k + v))
+ * }}}
+ *
+ * @param expr the input map column
+ * @param f col => v_transformed_col, the lambda function to transform the value of input column
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375076438
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3431,6 +3444,12 @@ object functions {
/**
* Returns whether a predicate holds for one or more elements in the array.
+ * {{{
+ * df.select(exists(col("i"), _ % 2 === 0))
+ * }}}
+ *
+ * @param column the input array column
+ * @param f col => predicate, the Boolean predicate to check the input column
Review comment:
nit: I would just call it `the boolean predicate` instead of `the Boolean predicate` in this file but no big deal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582287846
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375353743
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
Review comment:
Since this are polishing this part, can we rename this column from `zero` to a more meaningful name (e.g. `initialValue`)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582294297
**[Test build #117920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117920/testReport)** for PR 27449 at commit [`48f7411`](https://github.com/apache/spark/commit/48f74116aa21ddf55d252228fefa612e29441290).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582746814
**[Test build #117971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117971/testReport)** for PR 27449 at commit [`41482ae`](https://github.com/apache/spark/commit/41482ae9f947eac1f03fdffc789b141eaa4585f7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582287639
**[Test build #117897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117897/testReport)** for PR 27449 at commit [`aaee074`](https://github.com/apache/spark/commit/aaee074c5b800b3131e92d956e8355b11db4e51b).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582778280
**[Test build #117962 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117962/testReport)** for PR 27449 at commit [`a64017a`](https://github.com/apache/spark/commit/a64017aea41cc856a528e0e2fe0690f220ae16ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582406720
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582256996
**[Test build #117897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117897/testReport)** for PR 27449 at commit [`aaee074`](https://github.com/apache/spark/commit/aaee074c5b800b3131e92d956e8355b11db4e51b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582780853
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117977/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582783395
**[Test build #117979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117979/testReport)** for PR 27449 at commit [`c86900f`](https://github.com/apache/spark/commit/c86900fc55180649ad0f1b195307fdd623cf5923).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747093
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374586457
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3473,8 +3502,8 @@ object functions {
* df.selectExpr("filter(col, (x, i) -> i % 2 == 0)")
* }}}
*
- * @param column: the input array column
- * @param f: (col, index) => predicate, the Boolean predicate to filter the input column
+ * @param column the input array column
+ * @param f: (col, index) => predicate the Boolean predicate to filter the input column
Review comment:
How about `@param f a boolean predicate to filter the input column. Its signature is (col, index) => predicate`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582708562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22726/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582873144
**[Test build #117979 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117979/testReport)** for PR 27449 at commit [`c86900f`](https://github.com/apache/spark/commit/c86900fc55180649ad0f1b195307fdd623cf5923).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582392118
**[Test build #117933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117933/testReport)** for PR 27449 at commit [`083d38b`](https://github.com/apache/spark/commit/083d38ba3f7e245db094b70a5bda84b76b47a50b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582708554
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747403
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117971/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582294297
**[Test build #117920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117920/testReport)** for PR 27449 at commit [`48f7411`](https://github.com/apache/spark/commit/48f74116aa21ddf55d252228fefa612e29441290).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582783395
**[Test build #117979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117979/testReport)** for PR 27449 at commit [`c86900f`](https://github.com/apache/spark/commit/c86900fc55180649ad0f1b195307fdd623cf5923).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374586457
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3473,8 +3502,8 @@ object functions {
* df.selectExpr("filter(col, (x, i) -> i % 2 == 0)")
* }}}
*
- * @param column: the input array column
- * @param f: (col, index) => predicate, the Boolean predicate to filter the input column
+ * @param column the input array column
+ * @param f: (col, index) => predicate the Boolean predicate to filter the input column
Review comment:
How about `@param f a boolean function to filter the input column. Its signature is (col, index) => predicate`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375092341
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3551,6 +3609,12 @@ object functions {
/**
* Returns a map whose key-value pairs satisfy a predicate.
+ * {{{
+ * df.select(map_filter(col("m"), (k, v) => k * 10 === v))
+ * }}}
+ *
+ * @param expr the input map column
+ * @param f (k, v) => predicate, the Boolean predicate to filter the input map column
Review comment:
`(key, value) => predicate`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582521815
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582287851
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117897/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747400
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375091273
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,15 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
+ * @param merge (c, v) => c, the merge function to reduce all elements into a single state
Review comment:
`c`, `v` is too vague. How about `(combined_value, input_value) => combined_value, the merge function to merge an input value to the combined_value`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582778968
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22735/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747093
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582874114
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582287851
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117897/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582769879
**[Test build #117977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117977/testReport)** for PR 27449 at commit [`c86900f`](https://github.com/apache/spark/commit/c86900fc55180649ad0f1b195307fdd623cf5923).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375636219
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,18 +3511,28 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param initialValue the initial value
+ * @param merge (combined_value, input_value) => combined_value, the merge function to merge
+ * an input value to the combined_value
+ * @param finish col => final_col, the lambda function to convert the single state into
Review comment:
nit: `combined_value => final_value, the lambda function to convert the combined value of all inputs to final result` seems better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374618495
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3473,8 +3502,8 @@ object functions {
* df.selectExpr("filter(col, (x, i) -> i % 2 == 0)")
* }}}
*
- * @param column: the input array column
- * @param f: (col, index) => predicate, the Boolean predicate to filter the input column
+ * @param column the input array column
+ * @param f: (col, index) => predicate the Boolean predicate to filter the input column
Review comment:
Yea, I was about to point out. Just removing `:` should be good enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582392618
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22696/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374586965
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3421,6 +3428,14 @@ object functions {
/**
* Returns an array of elements after applying a transformation to each element
* in the input array.
+ * {{{
+ * df.select(transform(col("i"), (x, i) => x + i))
+ * df.selectExpr("transform(i, (x, i) -> x + i))")
+ * }}}
+ *
+ * @param column the input array column
+ * @param f: (col, index) => transformed_col the lambda function to filter the input column
+ * given the index. Indices start at 0.
Review comment:
Examples are copied from tests and it starts from 0 if you look at the test result:
https://github.com/apache/spark/blob/8aebc80e0e67bcb1aa300b8c8b1a209159237632/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala#L2026
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582770249
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582292257
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582292257
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582780840
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582778976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117962/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582708554
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582783863
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582783863
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581834770
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22588/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375636679
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3561,6 +3626,14 @@ object functions {
/**
* Merge two given maps, key-wise into a single map using a function.
+ * {{{
+ * df.select(map_zip_with(df("m1"), df("m2"), (k, v1, v2) => k === v1 + v2))
+ * }}}
+ *
+ * @param left the left input map column
+ * @param right the right input map column
+ * @param f (key, value1, value2) => new_value, the lambda function to merge two values of
Review comment:
`to merge the map values.` should be good enough
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582778976
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117962/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582780840
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581912659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117825/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374586011
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3473,8 +3502,8 @@ object functions {
* df.selectExpr("filter(col, (x, i) -> i % 2 == 0)")
* }}}
*
- * @param column: the input array column
- * @param f: (col, index) => predicate, the Boolean predicate to filter the input column
+ * @param column the input array column
+ * @param f: (col, index) => predicate the Boolean predicate to filter the input column
Review comment:
we should also remove `:` here too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375083536
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3431,6 +3444,12 @@ object functions {
/**
* Returns whether a predicate holds for one or more elements in the array.
+ * {{{
+ * df.select(exists(col("i"), _ % 2 === 0))
+ * }}}
+ *
+ * @param column the input array column
+ * @param f col => predicate, the Boolean predicate to check the input column
Review comment:
It's intentionally fixed by #27382.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747400
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581912646
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582709862
**[Test build #117962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117962/testReport)** for PR 27449 at commit [`a64017a`](https://github.com/apache/spark/commit/a64017aea41cc856a528e0e2fe0690f220ae16ca).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582292265
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22682/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582885866
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581834760
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374616518
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3410,6 +3410,13 @@ object functions {
/**
* Returns an array of elements after applying a transformation to each element
* in the input array.
+ * {{{
+ * df.select(transform(col("i"), x => x + 1))
+ * df.selectExpr("transform(i, x -> x + 1")
Review comment:
Actually, do we need this? `"transform(i, x -> x + 1)"` is SQL specific examples that have to be located with `ExpressionInfo`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747403
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117971/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375180600
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
+ * @param merge (combined_value, input_value) => combined_value, the merge function to to merge
+ * an input value to the combined_value
+ * @param finish col => final_col, the lambda function to convert the single state into
+ * final result
Review comment:
nit: need one more space at the beginning
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582874119
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117979/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582708562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22726/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582780801
**[Test build #117977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117977/testReport)** for PR 27449 at commit [`c86900f`](https://github.com/apache/spark/commit/c86900fc55180649ad0f1b195307fdd623cf5923).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375180415
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
+ * @param merge (combined_value, input_value) => combined_value, the merge function to to merge
Review comment:
typo: `to to` -> `to`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582778968
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582762313
there are code style issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375088061
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3431,6 +3444,12 @@ object functions {
/**
* Returns whether a predicate holds for one or more elements in the array.
+ * {{{
+ * df.select(exists(col("i"), _ % 2 === 0))
+ * }}}
+ *
+ * @param column the input array column
+ * @param f col => predicate, the Boolean predicate to check the input column
Review comment:
okie. that's fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582392118
**[Test build #117933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117933/testReport)** for PR 27449 at commit [`083d38b`](https://github.com/apache/spark/commit/083d38ba3f7e245db094b70a5bda84b76b47a50b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582257294
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22659/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375092082
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3530,6 +3576,12 @@ object functions {
/**
* Applies a function to every key-value pair in a map and returns
* a map with the results of those applications as the new keys for the pairs.
+ * {{{
+ * df.select(transform_keys(col("i"), (k, v) => k + v))
+ * }}}
+ *
+ * @param expr the input map column
+ * @param f col => k_transformed_col, the lambda function to transform the key of input column
Review comment:
`(key, value) => new_key`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582521804
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581834760
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375180719
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3508,7 +3541,14 @@ object functions {
/**
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x))
+ * }}}
*
+ * @param expr the input array column
+ * @param zero the initial value
+ * @param merge (combined_value, input_value) => combined_value, the merge function to to merge
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582780853
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117977/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374585048
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3473,8 +3502,8 @@ object functions {
* df.selectExpr("filter(col, (x, i) -> i % 2 == 0)")
* }}}
*
- * @param column: the input array column
- * @param f: (col, index) => predicate, the Boolean predicate to filter the input column
Review comment:
shall we keep the comma?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582769879
**[Test build #117977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117977/testReport)** for PR 27449 at commit [`c86900f`](https://github.com/apache/spark/commit/c86900fc55180649ad0f1b195307fdd623cf5923).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582732933
A few last comments, thanks for polishing it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375614088
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,16 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
Review comment:
Thanks all! I test Mima locally and it's good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375091273
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3488,6 +3511,15 @@ object functions {
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state. The final state is converted into the final result
* by applying a finish function.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x, _ * 10))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
+ * @param merge (c, v) => c, the merge function to reduce all elements into a single state
Review comment:
`c`, `v` is too vague. How about `(combined_value, input_value) => combined_value`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374584299
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3421,6 +3428,14 @@ object functions {
/**
* Returns an array of elements after applying a transformation to each element
* in the input array.
+ * {{{
+ * df.select(transform(col("i"), (x, i) => x + i))
+ * df.selectExpr("transform(i, (x, i) -> x + i))")
+ * }}}
+ *
+ * @param column the input array column
+ * @param f: (col, index) => transformed_col the lambda function to filter the input column
+ * given the index. Indices start at 0.
Review comment:
have you run the example and verify that the indices start at 0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582770256
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22741/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581834770
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22588/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582782591
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375091349
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3508,6 +3540,13 @@ object functions {
/**
* Applies a binary operator to an initial state and all elements in the array,
* and reduces this to a single state.
+ * {{{
+ * df.select(aggregate(col("i"), lit(0), (acc, x) => acc + x))
+ * }}}
+ *
+ * @param expr the input array column
+ * @param zero the initial value
+ * @param merge (c, v) => c, the merge function to reduce all elements into a single state
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581912659
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117825/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582406729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117920/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582287846
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374588036
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3473,8 +3502,8 @@ object functions {
* df.selectExpr("filter(col, (x, i) -> i % 2 == 0)")
* }}}
*
- * @param column: the input array column
- * @param f: (col, index) => predicate, the Boolean predicate to filter the input column
+ * @param column the input array column
+ * @param f: (col, index) => predicate the Boolean predicate to filter the input column
Review comment:
It's a little wordy from my view..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582406720
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747102
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22735/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582256996
**[Test build #117897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117897/testReport)** for PR 27449 at commit [`aaee074`](https://github.com/apache/spark/commit/aaee074c5b800b3131e92d956e8355b11db4e51b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582257284
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582521804
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582521815
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581823065
**[Test build #117825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117825/testReport)** for PR 27449 at commit [`17d5f78`](https://github.com/apache/spark/commit/17d5f787277d900291816267ae7d7f8f252818d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582783874
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22744/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582392612
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582783874
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22744/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581823065
**[Test build #117825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117825/testReport)** for PR 27449 at commit [`17d5f78`](https://github.com/apache/spark/commit/17d5f787277d900291816267ae7d7f8f252818d6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r375092574
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3561,6 +3625,14 @@ object functions {
/**
* Merge two given maps, key-wise into a single map using a function.
+ * {{{
+ * df.select(map_zip_with(df("m1"), df("m2"), (k, v1, v2) => k === v1 + v2))
+ * }}}
+ *
+ * @param left the left input map column
+ * @param right the right input map column
+ * @param f (k, v1, v2) => v, the lambda function to merge two values of input map columns into
Review comment:
`(key, value1, value2) => new_value`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582746814
**[Test build #117971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117971/testReport)** for PR 27449 at commit [`41482ae`](https://github.com/apache/spark/commit/41482ae9f947eac1f03fdffc789b141eaa4585f7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#discussion_r374615259
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -3473,8 +3502,8 @@ object functions {
* df.selectExpr("filter(col, (x, i) -> i % 2 == 0)")
* }}}
*
- * @param column: the input array column
- * @param f: (col, index) => predicate, the Boolean predicate to filter the input column
+ * @param column the input array column
+ * @param f: (col, index) => predicate the Boolean predicate to filter the input column
Review comment:
then let's just remove `:`. Javadoc doesn't work well with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582747390
**[Test build #117971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117971/testReport)** for PR 27449 at commit [`41482ae`](https://github.com/apache/spark/commit/41482ae9f947eac1f03fdffc789b141eaa4585f7).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582292265
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22682/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582770249
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582770256
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22741/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-582257284
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27449:
[SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala
functions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27449: [SPARK-27297][DOC][FOLLOW-UP] Improve documentation for various Scala functions
URL: https://github.com/apache/spark/pull/27449#issuecomment-581911915
**[Test build #117825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117825/testReport)** for PR 27449 at commit [`17d5f78`](https://github.com/apache/spark/commit/17d5f787277d900291816267ae7d7f8f252818d6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org