You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/14 18:49:06 UTC
[GitHub] [spark] MaxGekk commented on a diff in pull request #36546: [SPARK-37544][SQL] Correct date arithmetic in sequences
MaxGekk commented on code in PR #36546:
URL: https://github.com/apache/spark/pull/36546#discussion_r873062601
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala:
##########
@@ -964,6 +964,50 @@ class CollectionExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper
}
}
+ test("SPARK-37544: Time zone should not affect date sequence with month interval") {
+ outstandingZoneIds.foreach { zid =>
+ DateTimeTestUtils.withDefaultTimeZone(zid) {
+ checkEvaluation(new Sequence(
+ Literal(Date.valueOf("2021-01-01")),
+ Literal(Date.valueOf("2022-01-01")),
+ Literal(stringToInterval("interval 3 month"))),
+ Seq(
+ Date.valueOf("2021-01-01"),
+ Date.valueOf("2021-04-01"),
+ Date.valueOf("2021-07-01"),
+ Date.valueOf("2021-10-01"),
+ Date.valueOf("2022-01-01")))
+ }
+ }
+
+ // however, time zone should still affect sequences generated using hours interval, especially
+ // if the sequence's start-stop includes a "spring forward".
+ // take, for example, the following Spark date arithmetic:
Review Comment:
A sentence should begin from an upper case letter?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala:
##########
@@ -3062,6 +3080,9 @@ object Sequence {
"org.apache.spark.sql.catalyst.util.DateTimeUtils.timestampNTZAddInterval"
}
+ private val daysToMicrosCode = "org.apache.spark.sql.catalyst.util.DateTimeUtils.daysToMicros"
+ private val microsToDaysCode = "org.apache.spark.sql.catalyst.util.DateTimeUtils.microsToDays"
+
Review Comment:
Could you introduce a val for DateTimeUtils:
```scala
private val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
```
and re-use it here (and above):
```scala
private val addIntervalCode = outerDataType match {
case TimestampType | DateType => s"$dtu.timestampAddInterval"
case TimestampNTZType => s"$dtu.timestampNTZAddInterval"
}
private val daysToMicrosCode = s"$dtu.daysToMicros"
private val microsToDaysCode = s"$dtu.microsToDays"
```
Look at datetimeExpressions.scala for examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org