You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/03/14 15:53:35 UTC
[spark] branch branch-3.4 updated: [SPARK-42770][CONNECT] Add `truncatedTo(ChronoUnit.MICROS)` to make `SQLImplicitsTestSuite` in Java 17 daily test GA task pass
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new f36325d76cf [SPARK-42770][CONNECT] Add `truncatedTo(ChronoUnit.MICROS)` to make `SQLImplicitsTestSuite` in Java 17 daily test GA task pass
f36325d76cf is described below
commit f36325d76cf00dd7baef513a20a57686f41b87dd
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Tue Mar 14 08:53:15 2023 -0700
[SPARK-42770][CONNECT] Add `truncatedTo(ChronoUnit.MICROS)` to make `SQLImplicitsTestSuite` in Java 17 daily test GA task pass
### What changes were proposed in this pull request?
Run `LocalDateTime.now()` and `Instant.now()` with Java 8 & 11 always get microseconds on both Linux and MacOS, but there are some differences when using Java 17, it will get accurate nanoseconds on Linux, but still get the microseconds on MacOS.
On Linux(CentOs)
```
jshell> java.time.LocalDateTime.now()
$1 ==> 2023-03-13T18:09:12.498162194
jshell> java.time.Instant.now()
$2 ==> 2023-03-13T10:09:16.013993186Z
```
On MacOS
```
jshell> java.time.LocalDateTime.now()
$1 ==> 2023-03-13T17:13:47.485897
jshell> java.time.Instant.now()
$2 ==> 2023-03-13T09:15:12.031850Z
```
At present, Spark always converts them to microseconds, this will cause `test implicit encoder resolution` in SQLImplicitsTestSuite test fail when using Java 17 on Linux, so this pr add `truncatedTo(ChronoUnit.MICROS)` when
testing on Linux using Java 17 to ensure the accuracy of test input data is also microseconds.
### Why are the changes needed?
Make Java 17 daily test GA task run successfully. The Java 17 daily test GA task failed as follows:
```
[info] - test implicit encoder resolution *** FAILED *** (1 second, 329 milliseconds)
4429[info] 2023-03-02T23:00:20.404434 did not equal 2023-03-02T23:00:20.404434875 (SQLImplicitsTestSuite.scala:63)
4430[info] org.scalatest.exceptions.TestFailedException:
4431[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
4432[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
4433[info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
4434[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
4435[info] at org.apache.spark.sql.SQLImplicitsTestSuite.testImplicit$1(SQLImplicitsTestSuite.scala:63)
4436[info] at org.apache.spark.sql.SQLImplicitsTestSuite.$anonfun$new$2(SQLImplicitsTestSuite.scala:133)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual checked with Java 17
Closes #40395 from LuciferYang/SPARK-42770.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 3a2571c3978e7271388b5e56267e3bd60c5a4712)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../org/apache/spark/sql/SQLImplicitsTestSuite.scala | 20 +++++++++++++++++---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SQLImplicitsTestSuite.scala b/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SQLImplicitsTestSuite.scala
index f3261ac4850..470736fbebe 100644
--- a/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SQLImplicitsTestSuite.scala
+++ b/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SQLImplicitsTestSuite.scala
@@ -18,9 +18,11 @@ package org.apache.spark.sql
import java.sql.{Date, Timestamp}
import java.time.{Duration, Instant, LocalDate, LocalDateTime, Period}
+import java.time.temporal.ChronoUnit
import java.util.concurrent.atomic.AtomicLong
import io.grpc.inprocess.InProcessChannelBuilder
+import org.apache.commons.lang3.{JavaVersion, SystemUtils}
import org.scalatest.BeforeAndAfterAll
import org.apache.spark.connect.proto
@@ -130,9 +132,21 @@ class SQLImplicitsTestSuite extends ConnectFunSuite with BeforeAndAfterAll {
testImplicit(BigDecimal(decimal))
testImplicit(Date.valueOf(LocalDate.now()))
testImplicit(LocalDate.now())
- testImplicit(LocalDateTime.now())
- testImplicit(Instant.now())
- testImplicit(Timestamp.from(Instant.now()))
+ // SPARK-42770: Run `LocalDateTime.now()` and `Instant.now()` with Java 8 & 11 always
+ // get microseconds on both Linux and MacOS, but there are some differences when
+ // using Java 17, it will get accurate nanoseconds on Linux, but still get the microseconds
+ // on MacOS. At present, Spark always converts them to microseconds, this will cause the
+ // test fail when using Java 17 on Linux, so add `truncatedTo(ChronoUnit.MICROS)` when
+ // testing on Linux using Java 17 to ensure the accuracy of input data is microseconds.
+ if (SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17) && SystemUtils.IS_OS_LINUX) {
+ testImplicit(LocalDateTime.now().truncatedTo(ChronoUnit.MICROS))
+ testImplicit(Instant.now().truncatedTo(ChronoUnit.MICROS))
+ testImplicit(Timestamp.from(Instant.now().truncatedTo(ChronoUnit.MICROS)))
+ } else {
+ testImplicit(LocalDateTime.now())
+ testImplicit(Instant.now())
+ testImplicit(Timestamp.from(Instant.now()))
+ }
testImplicit(Period.ofYears(2))
testImplicit(Duration.ofMinutes(77))
testImplicit(SaveMode.Append)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org