You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ptkool <gi...@git.apache.org> on 2017/11/06 13:15:15 UTC
[GitHub] spark pull request #19672: [SPARK-22456] Add support for dayofweek function
GitHub user ptkool opened a pull request:
https://github.com/apache/spark/pull/19672
[SPARK-22456] Add support for dayofweek function
## What changes were proposed in this pull request?
This PR adds support for a new function called `dayofweek` that returns the day of the week of the given argument as an integer value in the range 1-7, where 1 represents Sunday.
## How was this patch tested?
Unit tests and manual tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ptkool/spark day_of_week_function
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19672.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19672
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83611 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83611/testReport)** for PR 19672 at commit [`a7fde0f`](https://github.com/apache/spark/commit/a7fde0f541fd4d8cc2cc824b7544bccaaefe9a61).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83518/testReport)** for PR 19672 at commit [`636e3e8`](https://github.com/apache/spark/commit/636e3e82e7f5a47e7598aa2309bc53d2235d63ce).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83520/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19672: [SPARK-22456][SQL] Add support for dayofweek func...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19672
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83518/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83611/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83520/testReport)** for PR 19672 at commit [`1ef1f70`](https://github.com/apache/spark/commit/1ef1f708cdc526484fd2edae3e463a233e750d05).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83564/testReport)** for PR 19672 at commit [`72b4dfc`](https://github.com/apache/spark/commit/72b4dfc0f8f29f5c5a5b9556ab4692c9563c437d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19672
Merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19672: [SPARK-22456] Add support for dayofweek function
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19672#discussion_r149124210
--- Diff: python/pyspark/sql/functions.py ---
@@ -869,6 +869,19 @@ def month(col):
return Column(sc._jvm.functions.month(_to_java_column(col)))
+@since(2.2)
--- End diff --
Maybe, `@since(2.3)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19672: [SPARK-22456] Add support for dayofweek function
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19672#discussion_r149302867
--- Diff: python/pyspark/sql/functions.py ---
@@ -869,6 +869,19 @@ def month(col):
return Column(sc._jvm.functions.month(_to_java_column(col)))
+@since(2.3)
+def dayofweek(col):
+ """
+ Extract the day of the week of a given date as integer.
--- End diff --
Let's match the doc with Scala one BTW.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83549/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83564/testReport)** for PR 19672 at commit [`72b4dfc`](https://github.com/apache/spark/commit/72b4dfc0f8f29f5c5a5b9556ab4692c9563c437d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83549/testReport)** for PR 19672 at commit [`edbe583`](https://github.com/apache/spark/commit/edbe583583a066a03df5b97a3067412f1fa51887).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83518/testReport)** for PR 19672 at commit [`636e3e8`](https://github.com/apache/spark/commit/636e3e82e7f5a47e7598aa2309bc53d2235d63ce).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19672: [SPARK-22456][SQL] Add support for dayofweek func...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19672#discussion_r149493317
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
@@ -376,6 +376,7 @@ object FunctionRegistry {
expression[DayOfMonth]("day"),
expression[DayOfYear]("dayofyear"),
expression[DayOfMonth]("dayofmonth"),
+ expression[DayOfWeek]("dayofweek"),
--- End diff --
I think we don't need to add it again (L397)?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19672
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83520/testReport)** for PR 19672 at commit [`1ef1f70`](https://github.com/apache/spark/commit/1ef1f708cdc526484fd2edae3e463a233e750d05).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19672: [SPARK-22456] Add support for dayofweek function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19672#discussion_r149275610
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2550,6 +2550,13 @@ object functions {
* @group datetime_funcs
* @since 1.5.0
*/
+ def dayofweek(e: Column): Column = withExpr { DayOfWeek(e.expr) }
+
+ /**
+ * Extracts the day of the month as an integer from a given date/timestamp/string.
+ * @group datetime_funcs
+ * @since 2.3.0
--- End diff --
Revert back to `1.5.0` here and modify `@since` to `2.3.0` in the doc for `dayofweek`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19672: [SPARK-22456] Add support for dayofweek function
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19672#discussion_r149122409
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2550,6 +2550,13 @@ object functions {
* @group datetime_funcs
* @since 1.5.0
--- End diff --
I think it should be `2.3.0`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83611 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83611/testReport)** for PR 19672 at commit [`a7fde0f`](https://github.com/apache/spark/commit/a7fde0f541fd4d8cc2cc824b7544bccaaefe9a61).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19672
**[Test build #83549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83549/testReport)** for PR 19672 at commit [`edbe583`](https://github.com/apache/spark/commit/edbe583583a066a03df5b97a3067412f1fa51887).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19672: [SPARK-22456] Add support for dayofweek function
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/19672#discussion_r149256287
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2550,6 +2550,13 @@ object functions {
* @group datetime_funcs
* @since 1.5.0
*/
+ def dayofweek(e: Column): Column = withExpr { DayOfWeek(e.expr) }
--- End diff --
In the doc, change `month` to `week`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19672: [SPARK-22456][SQL] Add support for dayofweek function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19672
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83564/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org