You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by junyangq <gi...@git.apache.org> on 2016/08/24 02:33:59 UTC
[GitHub] spark pull request #14779: [SparkR][Minor] Add more examples to window funct...
GitHub user junyangq opened a pull request:
https://github.com/apache/spark/pull/14779
[SparkR][Minor] Add more examples to window function docs
## What changes were proposed in this pull request?
This PR adds more examples to window function docs to make them more accessible to the users.
It also fixes default value issues for `lag` and `lead`.
## How was this patch tested?
Manual test, R unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/junyangq/spark SPARKR-FixWindowFunctionDocs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14779.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14779
----
commit fe76c69f78721e8825dbf4b27af728a147102c72
Author: Junyang Qian <ju...@databricks.com>
Date: 2016-08-24T02:25:43Z
Add more examples to window function docs.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14779: [SparkR][Minor] Add more examples to window funct...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14779#discussion_r76100314
--- Diff: R/pkg/R/functions.R ---
@@ -3194,25 +3206,34 @@ setMethod("lag",
#' lead
#'
#' Window function: returns the value that is \code{offset} rows after the current row, and
-#' NULL if there is less than \code{offset} rows after the current row. For example,
-#' an \code{offset} of one will return the next row at any given point in the window partition.
+#' \code{defaultValue} if there is less than \code{offset} rows after the current row.
+#' For example, an \code{offset} of one will return the next row at any given point
+#' in the window partition.
#'
#' This is equivalent to the \code{LEAD} function in SQL.
#'
#' @param x Column to compute on
-#' @param offset Number of rows to offset
+#' @param offset Number of rows to offset. If not specified, the default is 1.
#' @param defaultValue (Optional) default value to use
#'
#' @rdname lead
#' @name lead
#' @family window_funcs
#' @aliases lead,characterOrColumn,numeric-method
#' @export
-#' @examples \dontrun{lead(df$c)}
+#' @examples \dontrun{
+#' df <- createDataFrame(mtcars)
+#'
+#' # Partition by am (transmission) and order by hp (horsepower)
+#' ws <- orderBy(windowPartitionBy("am"), "hp")
+#'
+#' # Lag mpg values by 1 row on the partition-and-ordered table
--- End diff --
lag -> lead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64365/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64365/consoleFull)** for PR 14779 at commit [`7ac95d5`](https://github.com/apache/spark/commit/7ac95d5a65936d6a4a9d58db1babd4dd021ed996).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64363/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64363 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64363/consoleFull)** for PR 14779 at commit [`32cf76c`](https://github.com/apache/spark/commit/32cf76cbe5fbd2bf8201f6377ad89eae7e93824e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14779: [SparkR][Minor] Add more examples to window funct...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14779#discussion_r76100591
--- Diff: R/pkg/R/functions.R ---
@@ -3200,19 +3212,27 @@ setMethod("lag",
#' This is equivalent to the \code{LEAD} function in SQL.
#'
#' @param x Column to compute on
-#' @param offset Number of rows to offset
+#' @param offset Number of rows to offset. If not specified, the default is 1.
--- End diff --
in @rdname lag, there's a bit more information on these @param. Perhaps here with @rdname lead we could do similar?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64366/consoleFull)** for PR 14779 at commit [`7324a89`](https://github.com/apache/spark/commit/7324a89a664092b886bd60eae49254d71dc28b45).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64327/consoleFull)** for PR 14779 at commit [`fe76c69`](https://github.com/apache/spark/commit/fe76c69f78721e8825dbf4b27af728a147102c72).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64365/consoleFull)** for PR 14779 at commit [`7ac95d5`](https://github.com/apache/spark/commit/7ac95d5a65936d6a4a9d58db1babd4dd021ed996).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14779: [SparkR][Minor] Add more examples to window funct...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14779#discussion_r76000716
--- Diff: R/pkg/R/functions.R ---
@@ -3200,19 +3212,27 @@ setMethod("lag",
#' This is equivalent to the \code{LEAD} function in SQL.
#'
#' @param x Column to compute on
-#' @param offset Number of rows to offset
+#' @param offset Number of rows to offset. If not specified, the default is 1.
--- End diff --
perhaps explain more what offset/defaultValue will do like in Python?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14779: [SparkR][Minor] Add more examples to window funct...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14779
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64366/consoleFull)** for PR 14779 at commit [`7324a89`](https://github.com/apache/spark/commit/7324a89a664092b886bd60eae49254d71dc28b45).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64327 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64327/consoleFull)** for PR 14779 at commit [`fe76c69`](https://github.com/apache/spark/commit/fe76c69f78721e8825dbf4b27af728a147102c72).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64327/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14779: [SparkR][Minor] Add more examples to window funct...
Posted by junyangq <gi...@git.apache.org>.
Github user junyangq commented on a diff in the pull request:
https://github.com/apache/spark/pull/14779#discussion_r76098894
--- Diff: R/pkg/R/functions.R ---
@@ -3200,19 +3212,27 @@ setMethod("lag",
#' This is equivalent to the \code{LEAD} function in SQL.
#'
#' @param x Column to compute on
-#' @param offset Number of rows to offset
+#' @param offset Number of rows to offset. If not specified, the default is 1.
--- End diff --
This is actually in the description?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64366/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14779
**[Test build #64363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64363/consoleFull)** for PR 14779 at commit [`32cf76c`](https://github.com/apache/spark/commit/32cf76cbe5fbd2bf8201f6377ad89eae7e93824e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14779: [SparkR][Minor] Add more examples to window funct...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14779#discussion_r76000011
--- Diff: R/pkg/R/functions.R ---
@@ -3239,7 +3259,15 @@ setMethod("lead",
#' @aliases ntile,numeric-method
#' @family window_funcs
#' @export
-#' @examples \dontrun{ntile(1)}
+#' @examples \dontrun{
+#' df <- createDataFrame(iris)
+#'
+#' # Partition by Species and order by Sepal_Length
+#' ws <- orderBy(windowPartitionBy("Species"), "Sepal_Length")
+#'
+#' # Get ntil group id (1-4) for Sepal_Length
--- End diff --
ntile?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/14779
LGTM. waiting for Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14779
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14779: [SparkR][Minor] Add more examples to window function doc...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/14779
btw, you might want to avoid using `iris` for example. There has been several bugs where people would like to address the limitation with column names with `.` - if that is fixed then the example would be wrong to reference column as `Sepal_Length` instead of its original name `Sepal.Length`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org