You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/08 04:59:14 UTC
[GitHub] [spark] gengliangwang opened a new pull request #32814: [SPARK-35664][SQL] Support java.time. LocalDateTime as an external type of TimestampWithoutTZ type
gengliangwang opened a new pull request #32814:
URL: https://github.com/apache/spark/pull/32814
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error message, please read the guideline first:
https://spark.apache.org/error-message-guidelines.html
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
In the PR, I propose to extend Spark SQL API to accept java.time.LocalDateTime as an external type of recently added new Catalyst type - TimestampWithoutTZ. The Java class java.time.LocalDateTime has similar semantic to ANSI SQL timestamp without timezone type, and it is the most suitable to be an external type for TimestampWithoutTZType. In more details:
* Added TimestampWithoutTZConverter which converts java.time.LocalDateTime instances to/from internal representation of the Catalyst type TimestampWithoutTZType (to Long type). The TimestampWithoutTZConverter object uses new methods of DateTimeUtils:
* localDateTimeToMicros() converts the input date time to the total length in microseconds.
* microsToLocalDateTime() obtains a java.time.LocalDateTime
* Support new type TimestampWithoutTZType in RowEncoder via the methods createDeserializerForLocalDateTime() and createSerializerForLocalDateTime().
* Extended the Literal API to construct literals from java.time.LocalDateTime instances.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
To allow users parallelization of java.time.LocalDateTime collections, and construct timestamp without time zone columns. Also to collect such columns back to the driver side.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
The PR extends existing functionality. So, users can parallelize instances of the java.time.LocalDateTime class and collect them back.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
New unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856826204
@gengliangwang I wonder why did you removed the checks for `Long.MaxValue/MinValue`:
<img width="864" alt="Screenshot 2021-06-08 at 17 37 07" src="https://user-images.githubusercontent.com/1580697/121205356-455ca280-c880-11eb-8ddf-08773a3c375a.png">
Does `instantToMicros` overflow on the values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856613524
**[Test build #139469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139469/testReport)** for PR 32814 at commit [`ba44cb1`](https://github.com/apache/spark/commit/ba44cb182a3b62f4eaae6ac9462ea525a3a1e3de).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857322106
**[Test build #139520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139520/testReport)** for PR 32814 at commit [`35b78f1`](https://github.com/apache/spark/commit/35b78f1868ed59091e3d3fba93ac8a9b5ac95db3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856738860
@gengliangwang Have you done similar changes as in https://github.com/apache/spark/pull/31729, correct? Just want to avoid comparing of this PR to similar PRs for intervals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856876797
> Yes. The Long.MinValue case contains negative seconds and positive nanoseconds.
@gengliangwang Could you open an JIRA for the issue, so, we will fix this separately from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857442919
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44062/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856638618
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43990/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856564512
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43970/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856608278
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139447/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32814:
URL: https://github.com/apache/spark/pull/32814#discussion_r647450310
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -71,6 +71,15 @@ object DateTimeUtils {
instantToMicros(instant)
}
+ def microsToLocalDateTime(micros: Long): LocalDateTime = {
+ getLocalDateTime(micros, ZoneId.of("UTC"))
+ }
+
+ def localDateTimeToMicros(localDateTime: LocalDateTime): Long = {
+ localDateTime.toEpochSecond(ZoneOffset.UTC) * MICROS_PER_SECOND +
+ localDateTime.getNano / NANOS_PER_MICROS
Review comment:
Also, please, add tests to check long overflow.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -71,6 +71,15 @@ object DateTimeUtils {
instantToMicros(instant)
}
+ def microsToLocalDateTime(micros: Long): LocalDateTime = {
+ getLocalDateTime(micros, ZoneId.of("UTC"))
+ }
+
+ def localDateTimeToMicros(localDateTime: LocalDateTime): Long = {
+ localDateTime.toEpochSecond(ZoneOffset.UTC) * MICROS_PER_SECOND +
+ localDateTime.getNano / NANOS_PER_MICROS
Review comment:
Potentially, `*` and `+` can overflow. Could you use `Math.multiplyExact`
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -71,6 +71,15 @@ object DateTimeUtils {
instantToMicros(instant)
}
+ def microsToLocalDateTime(micros: Long): LocalDateTime = {
+ getLocalDateTime(micros, ZoneId.of("UTC"))
Review comment:
```suggestion
getLocalDateTime(micros, ZoneOffset.UTC)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856535551
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43970/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856820200
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139487/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856820145
**[Test build #139487 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139487/testReport)** for PR 32814 at commit [`0418133`](https://github.com/apache/spark/commit/0418133b1b68be1c1c675b5f81c0f6c8602f5018).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856688405
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856739900
@MaxGekk Yes I follow #31729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857361737
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44045/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856451528
**[Test build #139447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139447/testReport)** for PR 32814 at commit [`1101f55`](https://github.com/apache/spark/commit/1101f5550f7dd032fec39a211cb7e4bc345e565b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32814:
URL: https://github.com/apache/spark/pull/32814#discussion_r647216313
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
##########
@@ -634,6 +634,27 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
}
}
+ test("SPARK-35099: microseconds to LocalDateTime") {
+ assert(microsToLocalDateTime(0) == LocalDateTime.parse("1970-01-01T00:00:00"))
+ assert(microsToLocalDateTime(100) == LocalDateTime.parse("1970-01-01T00:00:00.0001"))
+ assert(microsToLocalDateTime(100000000) == LocalDateTime.parse("1970-01-01T00:01:40"))
+ assert(microsToLocalDateTime(100000000000L) == LocalDateTime.parse("1970-01-02T03:46:40"))
Review comment:
let's test negative value as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856646708
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43990/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856826204
@gengliangwang I wonder why did you remove the checks for `Long.MaxValue/MinValue`:
<img width="864" alt="Screenshot 2021-06-08 at 17 37 07" src="https://user-images.githubusercontent.com/1580697/121205356-455ca280-c880-11eb-8ddf-08773a3c375a.png">
Does `instantToMicros` overflow on the values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32814:
URL: https://github.com/apache/spark/pull/32814#discussion_r647972577
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
##########
@@ -634,6 +634,39 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
}
}
+ test("SPARK-35099: microseconds to LocalDateTime") {
+ assert(microsToLocalDateTime(0) == LocalDateTime.parse("1970-01-01T00:00:00"))
+ assert(microsToLocalDateTime(100) == LocalDateTime.parse("1970-01-01T00:00:00.0001"))
+ assert(microsToLocalDateTime(100000000) == LocalDateTime.parse("1970-01-01T00:01:40"))
+ assert(microsToLocalDateTime(100000000000L) == LocalDateTime.parse("1970-01-02T03:46:40"))
+ assert(microsToLocalDateTime(253402300799999999L) ==
+ LocalDateTime.parse("9999-12-31T23:59:59.999999"))
+ assert(microsToLocalDateTime(Long.MinValue) ==
+ LocalDateTime.parse("-290308-12-21T19:59:05.224192"))
+ assert(microsToLocalDateTime(Long.MaxValue) ==
+ LocalDateTime.parse("+294247-01-10T04:00:54.775807"))
+ }
+
+ test("SPARK-35099: LocalDateTime to microseconds") {
Review comment:
Please, fix the JIRA id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857389904
**[Test build #139538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139538/testReport)** for PR 32814 at commit [`a2326e0`](https://github.com/apache/spark/commit/a2326e0abc8cca80032213476d53ee5b0b9228ea).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857455555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857342525
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44045/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857387110
**[Test build #139538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139538/testReport)** for PR 32814 at commit [`a2326e0`](https://github.com/apache/spark/commit/a2326e0abc8cca80032213476d53ee5b0b9228ea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32814:
URL: https://github.com/apache/spark/pull/32814#discussion_r647973376
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
##########
@@ -634,6 +634,39 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
}
}
+ test("SPARK-35099: microseconds to LocalDateTime") {
Review comment:
Done, thanks.
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
##########
@@ -634,6 +634,39 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
}
}
+ test("SPARK-35099: microseconds to LocalDateTime") {
+ assert(microsToLocalDateTime(0) == LocalDateTime.parse("1970-01-01T00:00:00"))
+ assert(microsToLocalDateTime(100) == LocalDateTime.parse("1970-01-01T00:00:00.0001"))
+ assert(microsToLocalDateTime(100000000) == LocalDateTime.parse("1970-01-01T00:01:40"))
+ assert(microsToLocalDateTime(100000000000L) == LocalDateTime.parse("1970-01-02T03:46:40"))
+ assert(microsToLocalDateTime(253402300799999999L) ==
+ LocalDateTime.parse("9999-12-31T23:59:59.999999"))
+ assert(microsToLocalDateTime(Long.MinValue) ==
+ LocalDateTime.parse("-290308-12-21T19:59:05.224192"))
+ assert(microsToLocalDateTime(Long.MaxValue) ==
+ LocalDateTime.parse("+294247-01-10T04:00:54.775807"))
+ }
+
+ test("SPARK-35099: LocalDateTime to microseconds") {
Review comment:
Done, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856598207
**[Test build #139447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139447/testReport)** for PR 32814 at commit [`1101f55`](https://github.com/apache/spark/commit/1101f5550f7dd032fec39a211cb7e4bc345e565b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856665398
**[Test build #139469 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139469/testReport)** for PR 32814 at commit [`ba44cb1`](https://github.com/apache/spark/commit/ba44cb182a3b62f4eaae6ac9462ea525a3a1e3de).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856889287
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44010/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856646708
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43990/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856474722
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43970/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857387110
**[Test build #139538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139538/testReport)** for PR 32814 at commit [`a2326e0`](https://github.com/apache/spark/commit/a2326e0abc8cca80032213476d53ee5b0b9228ea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856667236
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856912090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856857466
> Does instantToMicros overflow on the values?
@MaxGekk Yes.
```
scala> import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
scala> instantToMicros(microsToInstant(Long.MinValue))
java.lang.ArithmeticException: long overflow
at java.lang.Math.multiplyExact(Math.java:892)
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:389)
... 49 elided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856820200
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139487/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856688380
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856860749
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44010/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32814:
URL: https://github.com/apache/spark/pull/32814#discussion_r647972335
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
##########
@@ -634,6 +634,39 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
}
}
+ test("SPARK-35099: microseconds to LocalDateTime") {
Review comment:
JIRA ID is incorrect, please, change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856451528
**[Test build #139447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139447/testReport)** for PR 32814 at commit [`1101f55`](https://github.com/apache/spark/commit/1101f5550f7dd032fec39a211cb7e4bc345e565b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang edited a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856857466
> Does instantToMicros overflow on the values?
@MaxGekk Yes. The Long.MinValue case contains negative seconds and positive nanoseconds. Thus it overflows when we try to convert it back.
```
scala> import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
scala> instantToMicros(microsToInstant(Long.MinValue))
java.lang.ArithmeticException: long overflow
at java.lang.Math.multiplyExact(Math.java:892)
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:389)
... 49 elided
scala> microsToInstant(Long.MinValue).getNano
res10: Int = 224192000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856885851
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856688405
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857439167
@cloud-fan @MaxGekk Thanks for the review.
Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857389942
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139538/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857455555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856912090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang edited a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856857466
> Does instantToMicros overflow on the values?
@MaxGekk Yes. The Long.MinValue case contains negative seconds and positive nanoseconds. Thus it overflows when we try to convert it back. We can fix it but it is too corner. The fix makes performance worse, too.
```
scala> import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
scala> instantToMicros(microsToInstant(Long.MinValue))
java.lang.ArithmeticException: long overflow
at java.lang.Math.multiplyExact(Math.java:892)
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:389)
... 49 elided
scala> microsToInstant(Long.MinValue).getNano
res10: Int = 224192000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856608278
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139447/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856815838
**[Test build #139487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139487/testReport)** for PR 32814 at commit [`0418133`](https://github.com/apache/spark/commit/0418133b1b68be1c1c675b5f81c0f6c8602f5018).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857438239
**[Test build #139520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139520/testReport)** for PR 32814 at commit [`35b78f1`](https://github.com/apache/spark/commit/35b78f1868ed59091e3d3fba93ac8a9b5ac95db3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856678657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857353329
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44045/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856815838
**[Test build #139487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139487/testReport)** for PR 32814 at commit [`0418133`](https://github.com/apache/spark/commit/0418133b1b68be1c1c675b5f81c0f6c8602f5018).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856858561
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856613524
**[Test build #139469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139469/testReport)** for PR 32814 at commit [`ba44cb1`](https://github.com/apache/spark/commit/ba44cb182a3b62f4eaae6ac9462ea525a3a1e3de).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857389942
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139538/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #32814:
URL: https://github.com/apache/spark/pull/32814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856564512
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43970/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857420871
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44062/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856678657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857322106
**[Test build #139520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139520/testReport)** for PR 32814 at commit [`35b78f1`](https://github.com/apache/spark/commit/35b78f1868ed59091e3d3fba93ac8a9b5ac95db3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-857361737
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44045/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32814:
URL: https://github.com/apache/spark/pull/32814#discussion_r647453714
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -71,6 +71,15 @@ object DateTimeUtils {
instantToMicros(instant)
}
+ def microsToLocalDateTime(micros: Long): LocalDateTime = {
+ getLocalDateTime(micros, ZoneId.of("UTC"))
+ }
+
+ def localDateTimeToMicros(localDateTime: LocalDateTime): Long = {
+ localDateTime.toEpochSecond(ZoneOffset.UTC) * MICROS_PER_SECOND +
+ localDateTime.getNano / NANOS_PER_MICROS
Review comment:
BTW, I replaced `*` by `multiplyExact`, and got:
```
java.lang.ArithmeticException: long overflow
at java.lang.Math.multiplyExact(Math.java:892)
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.localDateTimeToMicros(DateTimeUtils.scala:79)
at org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite.$anonfun$new$51(DateTimeUtilsSuite.scala:660)
```
on your new test:
```scala
assert(DateTimeUtils.localDateTimeToMicros(LocalDateTime.parse("-290308-12-21T19:59:05.224192"))
== Long.MinValue)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32814: [SPARK-35664][SQL] Support java.time.LocalDateTime as an external type of TimestampWithoutTZ type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32814:
URL: https://github.com/apache/spark/pull/32814#issuecomment-856885192
@MaxGekk https://issues.apache.org/jira/browse/SPARK-35679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org