[jira] [Updated] (SPARK-33404) "date_trunc" expression returns incorrect results

["Sean R. Owen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-33404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean R. Owen updated SPARK-33404:
---------------------------------
    Fix Version/s:     (was: 3.1.0)

> "date_trunc" expression returns incorrect results
> -------------------------------------------------
>
>                 Key: SPARK-33404
>                 URL: https://issues.apache.org/jira/browse/SPARK-33404
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.0.0, 3.0.1
>            Reporter: Utkarsh Agarwal
>            Priority: Major
>              Labels: correctness
>
> `date_trunc` SQL expression returns incorrect results for {{minute}} formatting string.
> Context: The {{minute}} formatting string should truncate the timestamps such that the seconds is set to ZERO.
> Repro (run the following commands in spark-shell):
> {quote}
> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
> spark.sql("SELECT date_trunc('minute', '1769-10-17 17:10:02')").show()
> {quote}
> Spark currently incorrectly returns 
> {quote}
> 1769-10-17 17:10:02
> {quote}
> against the expected return value of 
> {quote}
> 1769-10-17 17:10:00
> {quote}
> This happens as {{truncTimestamp}} in package {{org.apache.spark.sql.catalyst.util.DateTimeUtils}} incorrectly assumes that time zone offsets can never have the granularity of a second and thus does not account for time zone adjustment when truncating the timestamp to {{minute}}. 
> This assumption is currently used when truncating the timestamps to {{microsecond, millisecond, second, or minute}}. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org