[GitHub] [spark] bersprockets commented on a diff in pull request #37513: [SPARK-39184][SQL] Handle undersized result array in date and timestamp sequences

[GitBox <gi...@apache.org>]

bersprockets commented on code in PR #37513:
URL: https://github.com/apache/spark/pull/37513#discussion_r945329087


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala:
##########
@@ -3055,17 +3055,25 @@ object Sequence {
         val startMicros: Long = toMicros(num.toLong(start), scale)
         val stopMicros: Long = toMicros(num.toLong(stop), scale)
 
-        val maxEstimatedArrayLength =
+        val estimatedArrayLength =
           getSequenceLength(startMicros, stopMicros, input3, intervalStepInMicros)

Review Comment:
   Another approach I tested was "fudging" the stopMicros by adding 1 hour. That also worked without adding too many extra cases of overestimation (which requires slicing the array on completion). However, that approach works only until some time zone decides to have two "spring forwards" with no intervening "fall backward", or a "spring forward" of more than one hour. Both of those situations are unlikely to occur, but always a possibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org