You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/10 17:45:40 UTC
[GitHub] [spark] MaxGekk opened a new pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
MaxGekk opened a new pull request #31799:
URL: https://github.com/apache/spark/pull/31799
### What changes were proposed in this pull request?
In the PR, I propose to especially handle the amount of seconds `-9223372036855` in `IntervalUtils. durationToMicros()`. Starting from the amount (any durations with the second field < `-9223372036855`), input durations cannot fit to `Long` in the conversion to microseconds. For example, the amount of microseconds = `Long.MinValue = -9223372036854775808` can be represented in two forms:
1. seconds = -9223372036854, nanoAdjustment = -775808, or
2. seconds = -9223372036855, nanoAdjustment = +224192
And the method `Duration.ofSeconds()` produces the last form but such form causes overflow while converting `-9223372036855` seconds to microseconds.
In the PR, I propose to convert the second form to the first one if the second field of input duration is equal to `-9223372036855`.
### Why are the changes needed?
The changes fix the issue demonstrated by the code:
```scala
scala> durationToMicros(microsToDuration(Long.MinValue))
java.lang.ArithmeticException: long overflow
at java.lang.Math.multiplyExact(Math.java:892)
at org.apache.spark.sql.catalyst.util.IntervalUtils$.durationToMicros(IntervalUtils.scala:782)
... 49 elided
```
The `durationToMicros()` method cannot handle valid output of `microsToDuration()`.
### Does this PR introduce _any_ user-facing change?
Should not since new interval types has not been released yet.
### How was this patch tested?
By running new UT from `IntervalUtilsSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592140205
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
+ Math.addExact(us, (duration.getNano - NANOS_PER_SECOND) / NANOS_PER_MICROS)
Review comment:
ah， right
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796034345
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-795959674
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-795944077
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135941/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592105151
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
+ Math.addExact(us, (duration.getNano - NANOS_PER_SECOND) / NANOS_PER_MICROS)
Review comment:
is it possible that `duration.getNano - NANOS_PER_SECOND` overflows? Or `duration.getNano` is guaranteed to be positive?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796598156
**[Test build #135969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135969/testReport)** for PR 31799 at commit [`b107722`](https://github.com/apache/spark/commit/b107722103f55f4305accb676c7c46540e39a0dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31799:
URL: https://github.com/apache/spark/pull/31799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796598156
**[Test build #135969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135969/testReport)** for PR 31799 at commit [`b107722`](https://github.com/apache/spark/commit/b107722103f55f4305accb676c7c46540e39a0dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796500474
@cloud-fan @HyukjinKwon @yaooqinn Could you review this fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592138777
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
+ Math.addExact(us, (duration.getNano - NANOS_PER_SECOND) / NANOS_PER_MICROS)
Review comment:
then we are safe. Can we add a comment or assert to explain that this won't overflow?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592131789
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
+ Math.addExact(us, (duration.getNano - NANOS_PER_SECOND) / NANOS_PER_MICROS)
Review comment:
It seems a mess at java-side already
```scala
scala> java.time.Duration.ofNanos(-1)
res6: java.time.Duration = PT-0.000000001S
scala> java.time.Duration.ofNanos(-1).get
get getClass getNano getSeconds getUnits
scala> java.time.Duration.ofNanos(-1).getNano
res9: Int = 999999999
scala> java.time.Duration.ofNanos(-1000).getNano
res10: Int = 999999000
scala> java.time.Duration.ofNanos(Int.MinValue).getNano
res12: Int = 852516352
scala> java.time.Duration.ofNanos(Int.M).getNano
MaxValue MinValue
scala> java.time.Duration.ofNanos(Int.MaxValue).getNano
res15: Int = 147483647
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796012578
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796665658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40553/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796772432
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135969/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592129047
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
Review comment:
nit: microseconds
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796772432
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135969/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796814236
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592134384
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
+ Math.addExact(us, (duration.getNano - NANOS_PER_SECOND) / NANOS_PER_MICROS)
Review comment:
Why? `getNano` returns an adjustment to seconds, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592131113
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
Review comment:
`us` is an alias for microseconds ;-) , see https://en.wikipedia.org/wiki/Microsecond
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-795944077
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135941/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796653983
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40553/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796743057
**[Test build #135969 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135969/testReport)** for PR 31799 at commit [`b107722`](https://github.com/apache/spark/commit/b107722103f55f4305accb676c7c46540e39a0dc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31799:
URL: https://github.com/apache/spark/pull/31799#discussion_r592107485
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -779,9 +782,14 @@ object IntervalUtils {
* @throws ArithmeticException If numeric overflow occurs
*/
def durationToMicros(duration: Duration): Long = {
- val us = Math.multiplyExact(duration.getSeconds, MICROS_PER_SECOND)
- val result = Math.addExact(us, duration.getNano / NANOS_PER_MICROS)
- result
+ val seconds = duration.getSeconds
+ if (seconds == minDurationSeconds) {
+ val us = (minDurationSeconds + 1) * MICROS_PER_SECOND
+ Math.addExact(us, (duration.getNano - NANOS_PER_SECOND) / NANOS_PER_MICROS)
Review comment:
According to Java doc for `Duration.getNano`:
```java
* @return the nanoseconds within the second part of the length of the duration, from 0 to 999,999,999
*/
public int getNano() {
return nanos;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796034345
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796639219
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40553/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31799: [SPARK-34695][SQL] Fix long overflow in conversion of minimum duration to microseconds
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31799:
URL: https://github.com/apache/spark/pull/31799#issuecomment-796665658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40553/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org