You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/02 08:08:10 UTC
[GitHub] [spark] gengliangwang opened a new pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
gengliangwang opened a new pull request #30566:
URL: https://github.com/apache/spark/pull/30566
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
As https://github.com/apache/spark/pull/28534 adds functions from [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions) for converting numbers to timestamp, this PR is to add functions UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS for converting timestamp to numbers.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. Symmetry of the conversion functions
2. Casting timestamp type to numeric types is disallowed in ANSI mode, we should provide functions for users to complete the conversion.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
3 new functions UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS for converting timestamp to long type.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737296850
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737223640
**[Test build #132051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132051/testReport)** for PR 30566 at commit [`45495c7`](https://github.com/apache/spark/commit/45495c773285b6693b8ac399f8b0ec3877baaa74).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737380136
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534358281
##########
File path: sql/core/src/test/resources/sql-tests/inputs/datetime.sql
##########
@@ -14,6 +14,10 @@ select TIMESTAMP_MILLIS(-92233720368547758);
select TIMESTAMP_SECONDS(0.1234567);
-- truncation is OK for float/double
select TIMESTAMP_SECONDS(0.1234567d), TIMESTAMP_SECONDS(FLOAT(0.1234567));
+-- UNIX_SECONDS, UNIX_MILLISECONDS and UNIX_MICROSECONDS
+select UNIX_SECONDS(TIMESTAMP('2020-12-01 14:30:08')), UNIX_SECONDS(TIMESTAMP('2020-12-01 14:30:08.999999')), UNIX_SECONDS(null);
+select UNIX_MILLIS(TIMESTAMP('2020-12-01 14:30:08')), UNIX_MILLIS(TIMESTAMP('2020-12-01 14:30:08.999999')), UNIX_MILLIS(null);
+select UNIX_MICROS(TIMESTAMP('2020-12-01 14:30:08')), UNIX_MICROS(TIMESTAMP('2020-12-01 14:30:08.999999')), UNIX_MICROS(null);
Review comment:
Could you add timezone into timestamp literals?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737304975
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737405566
**[Test build #132067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132067/testReport)** for PR 30566 at commit [`025fdd2`](https://github.com/apache/spark/commit/025fdd2898338c206920811787b109128acf9184).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737257308
**[Test build #132053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132053/testReport)** for PR 30566 at commit [`82ba97b`](https://github.com/apache/spark/commit/82ba97b8418e373faaab34dc6ee21014f86258f8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737257308
**[Test build #132053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132053/testReport)** for PR 30566 at commit [`82ba97b`](https://github.com/apache/spark/commit/82ba97b8418e373faaab34dc6ee21014f86258f8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534139368
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -1197,6 +1197,38 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkResult(Int.MinValue.toLong - 100)
}
+ test("UNIX_SECONDS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixSeconds(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixSeconds(timestamp), 1L)
+ // Truncates higher levels of precision
+ timestamp = Literal(new Timestamp(1999L))
+ checkEvaluation(UnixSeconds(timestamp), 1L)
+ }
+
+ test("UNIX_MILLIS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixMillis(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixMillis(timestamp), 1000L)
+ // Truncates higher levels of precision
+ val timestampWithNanos = new Timestamp(1000L)
+ timestampWithNanos.setNanos(999999)
+ checkEvaluation(UnixMillis(Literal(timestampWithNanos)), 1000L)
+ }
+
+ test("UNIX_MICROS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixMicros(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixMicros(timestamp), 1000000L)
+ // Truncates higher levels of precision
+ val timestampWithNanos = new Timestamp(1000L)
+ timestampWithNanos.setNanos(999999)
Review comment:
I see. Let's just use `timestampWithNanos.setNanos(1000)` since there won't be truncation here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737405566
**[Test build #132067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132067/testReport)** for PR 30566 at commit [`025fdd2`](https://github.com/apache/spark/commit/025fdd2898338c206920811787b109128acf9184).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737223640
**[Test build #132051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132051/testReport)** for PR 30566 at commit [`45495c7`](https://github.com/apache/spark/commit/45495c773285b6693b8ac399f8b0ec3877baaa74).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737408805
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737198454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737339995
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737469530
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737198454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737469530
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737360822
**[Test build #132061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132061/testReport)** for PR 30566 at commit [`270aa95`](https://github.com/apache/spark/commit/270aa9566658d82a17d1f2c0d91a1863a8d40fb2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737296731
**[Test build #132056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132056/testReport)** for PR 30566 at commit [`f5e4b53`](https://github.com/apache/spark/commit/f5e4b53e19975131a46e17620031631c5222e451).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737373539
**[Test build #132066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132066/testReport)** for PR 30566 at commit [`46c8651`](https://github.com/apache/spark/commit/46c8651ff59272d45f0b6bc24825abb1a8c3f7c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534100028
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -1197,6 +1197,38 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkResult(Int.MinValue.toLong - 100)
}
+ test("UNIX_SECONDS") {
Review comment:
let's also test null and negative input (that truncates)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737340236
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737106328
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737411012
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737166050
**[Test build #132042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132042/testReport)** for PR 30566 at commit [`35ef5d2`](https://github.com/apache/spark/commit/35ef5d2bca23c059826931dd37e4f93823c0aa37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737068122
**[Test build #132031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132031/testReport)** for PR 30566 at commit [`0b31348`](https://github.com/apache/spark/commit/0b31348794415313a128c3bfe173ac9f72d4cf17).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737097898
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737340030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534099449
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -524,6 +524,79 @@ case class MicrosToTimestamp(child: Expression)
override def prettyName: String = "timestamp_micros"
}
+abstract class TimestampToLongBase extends UnaryExpression
+ with ExpectsInputTypes with NullIntolerant {
+
+ protected def scaleFactor: Long
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(TimestampType)
+
+ override def dataType: DataType = LongType
+
+ override def nullSafeEval(input: Any): Any = {
+ Math.floorDiv(input.asInstanceOf[Number].longValue(), scaleFactor)
+ }
+
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ if (scaleFactor == 1) {
+ defineCodeGen(ctx, ev, c => c)
+ } else {
+ defineCodeGen(ctx, ev, c => s"java.lang.Math.floorDiv($c, ${scaleFactor}L)")
+ }
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(timestamp) - Returns the number of seconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_(TIMESTAMP('1970-01-01 00:00:01'));
+ 1
+ """,
+ group = "datetime_funcs",
+ since = "3.1.0")
+// scalastyle:on line.size.limit
+case class UnixSeconds(child: Expression) extends TimestampToLongBase {
+ override def scaleFactor: Long = MICROS_PER_SECOND
+
+ override def prettyName: String = "unix_seconds"
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(timestamp) - Returns the number of milliseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_(TIMESTAMP('1970-01-01 00:00:01'));
+ 1000
+ """,
+ group = "datetime_funcs",
+ since = "3.1.0")
+// scalastyle:on line.size.limit
+case class UnixMillis(child: Expression) extends TimestampToLongBase {
+ override def scaleFactor: Long = MICROS_PER_MILLIS
+
+ override def prettyName: String = "unix_millis"
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(timestamp) - Returns the number of microseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision.",
Review comment:
This function doesn't truncate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737373539
**[Test build #132066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132066/testReport)** for PR 30566 at commit [`46c8651`](https://github.com/apache/spark/commit/46c8651ff59272d45f0b6bc24825abb1a8c3f7c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737339997
**[Test build #132062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132062/testReport)** for PR 30566 at commit [`46c8651`](https://github.com/apache/spark/commit/46c8651ff59272d45f0b6bc24825abb1a8c3f7c2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737262579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534100819
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -1197,6 +1197,38 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkResult(Int.MinValue.toLong - 100)
}
+ test("UNIX_SECONDS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixSeconds(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixSeconds(timestamp), 1L)
+ // Truncates higher levels of precision
+ timestamp = Literal(new Timestamp(1999L))
+ checkEvaluation(UnixSeconds(timestamp), 1L)
+ }
+
+ test("UNIX_MILLIS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixMillis(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixMillis(timestamp), 1000L)
+ // Truncates higher levels of precision
+ val timestampWithNanos = new Timestamp(1000L)
+ timestampWithNanos.setNanos(999999)
+ checkEvaluation(UnixMillis(Literal(timestampWithNanos)), 1000L)
+ }
+
+ test("UNIX_MICROS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixMicros(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixMicros(timestamp), 1000000L)
+ // Truncates higher levels of precision
+ val timestampWithNanos = new Timestamp(1000L)
+ timestampWithNanos.setNanos(999999)
Review comment:
maybe we can use `Instant` to test these functions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737337639
**[Test build #132061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132061/testReport)** for PR 30566 at commit [`270aa95`](https://github.com/apache/spark/commit/270aa9566658d82a17d1f2c0d91a1863a8d40fb2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737408805
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534373050
##########
File path: sql/core/src/test/resources/sql-tests/inputs/datetime.sql
##########
@@ -14,6 +14,10 @@ select TIMESTAMP_MILLIS(-92233720368547758);
select TIMESTAMP_SECONDS(0.1234567);
-- truncation is OK for float/double
select TIMESTAMP_SECONDS(0.1234567d), TIMESTAMP_SECONDS(FLOAT(0.1234567));
+-- UNIX_SECONDS, UNIX_MILLISECONDS and UNIX_MICROSECONDS
+select UNIX_SECONDS(TIMESTAMP('2020-12-01 14:30:08')), UNIX_SECONDS(TIMESTAMP('2020-12-01 14:30:08.999999')), UNIX_SECONDS(null);
+select UNIX_MILLIS(TIMESTAMP('2020-12-01 14:30:08')), UNIX_MILLIS(TIMESTAMP('2020-12-01 14:30:08.999999')), UNIX_MILLIS(null);
+select UNIX_MICROS(TIMESTAMP('2020-12-01 14:30:08')), UNIX_MICROS(TIMESTAMP('2020-12-01 14:30:08.999999')), UNIX_MICROS(null);
Review comment:
Thanks, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534354487
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
##########
@@ -151,6 +151,9 @@ class ExpressionInfoSuite extends SparkFunSuite with SharedSparkSession {
"org.apache.spark.sql.catalyst.expressions.CurrentTimestamp",
"org.apache.spark.sql.catalyst.expressions.CurrentTimeZone",
"org.apache.spark.sql.catalyst.expressions.Now",
+ "org.apache.spark.sql.catalyst.expressions.UnixSeconds",
+ "org.apache.spark.sql.catalyst.expressions.UnixMillis",
+ "org.apache.spark.sql.catalyst.expressions.UnixMicros",
Review comment:
Sounds better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534373157
##########
File path: sql/core/src/test/resources/sql-functions/sql-expression-schema.md
##########
@@ -289,6 +289,9 @@
| org.apache.spark.sql.catalyst.expressions.UnaryMinus | negative | SELECT negative(1) | struct<negative(1):int> |
| org.apache.spark.sql.catalyst.expressions.UnaryPositive | positive | SELECT positive(1) | struct<(+ 1):int> |
| org.apache.spark.sql.catalyst.expressions.Unhex | unhex | SELECT decode(unhex('537061726B2053514C'), 'UTF-8') | struct<decode(unhex(537061726B2053514C), UTF-8):string> |
+| org.apache.spark.sql.catalyst.expressions.UnixMicros | unix_micros | SELECT unix_micros(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_micros(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.UnixMillis | unix_millis | SELECT unix_millis(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_millis(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.UnixSeconds | unix_seconds | SELECT unix_seconds(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_seconds(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
Review comment:
Thanks, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737410493
**[Test build #132056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132056/testReport)** for PR 30566 at commit [`f5e4b53`](https://github.com/apache/spark/commit/f5e4b53e19975131a46e17620031631c5222e451).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737368117
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737222180
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737476235
**[Test build #132067 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132067/testReport)** for PR 30566 at commit [`025fdd2`](https://github.com/apache/spark/commit/025fdd2898338c206920811787b109128acf9184).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737097898
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737337639
**[Test build #132061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132061/testReport)** for PR 30566 at commit [`270aa95`](https://github.com/apache/spark/commit/270aa9566658d82a17d1f2c0d91a1863a8d40fb2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737262579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737080039
Probably, you need to add these new functions in the ignore set:
https://github.com/apache/spark/blob/a4788ee8c61e1373e6eded41bb57d84c68149968/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala#L148-L153
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737296731
**[Test build #132056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132056/testReport)** for PR 30566 at commit [`f5e4b53`](https://github.com/apache/spark/commit/f5e4b53e19975131a46e17620031631c5222e451).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #30566:
URL: https://github.com/apache/spark/pull/30566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737405108
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737159277
> Probably, you need to add these new functions in the ignore set:
@maropu Sorry but can you explain the reason? I am not familiar with this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737106328
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737407288
Thank you for update, @gengliangwang !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737380136
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534100617
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -1197,6 +1197,38 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkResult(Int.MinValue.toLong - 100)
}
+ test("UNIX_SECONDS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixSeconds(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixSeconds(timestamp), 1L)
+ // Truncates higher levels of precision
+ timestamp = Literal(new Timestamp(1999L))
+ checkEvaluation(UnixSeconds(timestamp), 1L)
+ }
+
+ test("UNIX_MILLIS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixMillis(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixMillis(timestamp), 1000L)
+ // Truncates higher levels of precision
+ val timestampWithNanos = new Timestamp(1000L)
+ timestampWithNanos.setNanos(999999)
+ checkEvaluation(UnixMillis(Literal(timestampWithNanos)), 1000L)
+ }
+
+ test("UNIX_MICROS") {
+ var timestamp = Literal(new Timestamp(0L))
+ checkEvaluation(UnixMicros(timestamp), 0L)
+ timestamp = Literal(new Timestamp(1000L))
+ checkEvaluation(UnixMicros(timestamp), 1000000L)
+ // Truncates higher levels of precision
+ val timestampWithNanos = new Timestamp(1000L)
+ timestampWithNanos.setNanos(999999)
Review comment:
The truncation happens in `Literal.apply`, not `UnixMicros`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737399917
**[Test build #132062 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132062/testReport)** for PR 30566 at commit [`46c8651`](https://github.com/apache/spark/commit/46c8651ff59272d45f0b6bc24825abb1a8c3f7c2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737340030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737370997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737405108
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737476863
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737218383
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737296634
**[Test build #132051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132051/testReport)** for PR 30566 at commit [`45495c7`](https://github.com/apache/spark/commit/45495c773285b6693b8ac399f8b0ec3877baaa74).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737207436
**[Test build #132042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132042/testReport)** for PR 30566 at commit [`35ef5d2`](https://github.com/apache/spark/commit/35ef5d2bca23c059826931dd37e4f93823c0aa37).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534100028
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -1197,6 +1197,38 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkResult(Int.MinValue.toLong - 100)
}
+ test("UNIX_SECONDS") {
Review comment:
let's also test null and negative input
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737394335
**[Test build #132066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132066/testReport)** for PR 30566 at commit [`46c8651`](https://github.com/apache/spark/commit/46c8651ff59272d45f0b6bc24825abb1a8c3f7c2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737304975
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534298963
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
##########
@@ -151,6 +151,9 @@ class ExpressionInfoSuite extends SparkFunSuite with SharedSparkSession {
"org.apache.spark.sql.catalyst.expressions.CurrentTimestamp",
"org.apache.spark.sql.catalyst.expressions.CurrentTimeZone",
"org.apache.spark.sql.catalyst.expressions.Now",
+ "org.apache.spark.sql.catalyst.expressions.UnixSeconds",
+ "org.apache.spark.sql.catalyst.expressions.UnixMillis",
+ "org.apache.spark.sql.catalyst.expressions.UnixMicros",
Review comment:
Originally, the timestamp creation in the examples will be affected by the local timezone
```
_FUNC_(TIMESTAMP('1970-01-01 00:00:01'))
```
Now I update the example and add the timezone in the timestamp literal. We can test examples.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534290625
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
##########
@@ -151,6 +151,9 @@ class ExpressionInfoSuite extends SparkFunSuite with SharedSparkSession {
"org.apache.spark.sql.catalyst.expressions.CurrentTimestamp",
"org.apache.spark.sql.catalyst.expressions.CurrentTimeZone",
"org.apache.spark.sql.catalyst.expressions.Now",
+ "org.apache.spark.sql.catalyst.expressions.UnixSeconds",
+ "org.apache.spark.sql.catalyst.expressions.UnixMillis",
+ "org.apache.spark.sql.catalyst.expressions.UnixMicros",
Review comment:
why can't we run the example SQL for them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737065946
I will have follow-up PR for adding DATE_FROM_UNIX_DATE/UNIX_DATE and update the error message of ANSI casting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737476863
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737068122
**[Test build #132031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132031/testReport)** for PR 30566 at commit [`0b31348`](https://github.com/apache/spark/commit/0b31348794415313a128c3bfe173ac9f72d4cf17).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737411012
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737166050
**[Test build #132042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132042/testReport)** for PR 30566 at commit [`35ef5d2`](https://github.com/apache/spark/commit/35ef5d2bca23c059826931dd37e4f93823c0aa37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737222180
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737296850
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30566:
URL: https://github.com/apache/spark/pull/30566#issuecomment-737085447
**[Test build #132031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132031/testReport)** for PR 30566 at commit [`0b31348`](https://github.com/apache/spark/commit/0b31348794415313a128c3bfe173ac9f72d4cf17).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30566: [SPARK-33627][SQL] Add new function UNIX_SECONDS, UNIX_MILLIS and UNIX_MICROS
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30566:
URL: https://github.com/apache/spark/pull/30566#discussion_r534364686
##########
File path: sql/core/src/test/resources/sql-functions/sql-expression-schema.md
##########
@@ -289,6 +289,9 @@
| org.apache.spark.sql.catalyst.expressions.UnaryMinus | negative | SELECT negative(1) | struct<negative(1):int> |
| org.apache.spark.sql.catalyst.expressions.UnaryPositive | positive | SELECT positive(1) | struct<(+ 1):int> |
| org.apache.spark.sql.catalyst.expressions.Unhex | unhex | SELECT decode(unhex('537061726B2053514C'), 'UTF-8') | struct<decode(unhex(537061726B2053514C), UTF-8):string> |
+| org.apache.spark.sql.catalyst.expressions.UnixMicros | unix_micros | SELECT unix_micros(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_micros(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.UnixMillis | unix_millis | SELECT unix_millis(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_millis(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.UnixSeconds | unix_seconds | SELECT unix_seconds(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_seconds(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
Review comment:
Please regenerate this, @gengliangwang .
```
-| org.apache.spark.sql.catalyst.expressions.UnixMicros | unix_micros | SELECT unix_micros(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_micros(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
-| org.apache.spark.sql.catalyst.expressions.UnixMillis | unix_millis | SELECT unix_millis(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_millis(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
-| org.apache.spark.sql.catalyst.expressions.UnixSeconds | unix_seconds | SELECT unix_seconds(TIMESTAMP('1970-01-01 00:00:01')) | struct<unix_seconds(CAST(1970-01-01 00:00:01 AS TIMESTAMP)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.UnixMicros | unix_micros | SELECT unix_micros(TIMESTAMP('1970-01-01 00:00:01Z')) | struct<unix_micros(CAST(1970-01-01 00:00:01Z AS TIMESTAMP)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.UnixMillis | unix_millis | SELECT unix_millis(TIMESTAMP('1970-01-01 00:00:01Z')) | struct<unix_millis(CAST(1970-01-01 00:00:01Z AS TIMESTAMP)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.UnixSeconds | unix_seconds | SELECT unix_seconds(TIMESTAMP('1970-01-01 00:00:01Z')) | struct<unix_seconds(CAST(1970-01-01 00:00:01Z AS TIMESTAMP)):bigint> |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org