[GitHub] [spark] MaxGekk commented on a change in pull request #24311: [SPARK-27401][SQL] Refactoring conversion of Timestamp to/from java.sql.Timestamp

[GitBox <gi...@apache.org>]

MaxGekk commented on a change in pull request #24311: [SPARK-27401][SQL] Refactoring conversion of Timestamp to/from java.sql.Timestamp
URL: https://github.com/apache/spark/pull/24311#discussion_r272820174
 
 

 ##########
 File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala
 ##########
 @@ -102,8 +102,14 @@ object LiteralGenerator {
   lazy val dateLiteralGen: Gen[Literal] =
     for { d <- Arbitrary.arbInt.arbitrary } yield Literal.create(new Date(d), DateType)
 
-  lazy val timestampLiteralGen: Gen[Literal] =
-    for { t <- Arbitrary.arbLong.arbitrary } yield Literal.create(new Timestamp(t), TimestampType)
+  private def yearToMillis(year: Long): Long = {
+    year * 365 * 24 * 3600 * 1000L
+  }
+
+  lazy val timestampLiteralGen: Gen[Literal] = {
+    for { millis <- Gen.choose(yearToMillis(-9999), yearToMillis(10000)) }
+      yield Literal.create(new Timestamp(millis), TimestampType)
+  }
 
 Review comment:
   I changed the random generator for timestamps, and limit max/min values for milliseconds since epoch because the long random generator can produce milliseconds that caused Long overflow at conversion of milliseconds to microseconds. Internally as you know we store microseconds since epoch in `TimestampType`. For example, the old (current) generator can create an instance of `java.sql.Timestamp(-3948373668011580000, 570000000)`. New function `fromJavaTimestamp` calls `instantToMicros`, and inside of it we use `multiplyExact` which can detect `Long` overflow on multiplication:
   ```scala
   def instantToMicros(instant: Instant): Long = {
     val us = Math.multiplyExact(-3948373668011580, 1000000)
       ...
   }
   ```
   <img width="873" alt="Screen Shot 2019-04-07 at 08 41 30" src="https://user-images.githubusercontent.com/1580697/55679740-fa961780-5910-11e9-914c-11a1336894e8.png">
   
   Previous (current) implementation doesn't detect the overflow at all. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org