You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sumeet (Jira)" <ji...@apache.org> on 2021/07/13 17:13:00 UTC
[jira] [Commented] (SPARK-36065) date_trunc returns incorrect
output
[ https://issues.apache.org/jira/browse/SPARK-36065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380050#comment-17380050 ]
Sumeet commented on SPARK-36065:
--------------------------------
cc [~maxgekk]
> date_trunc returns incorrect output
> -----------------------------------
>
> Key: SPARK-36065
> URL: https://issues.apache.org/jira/browse/SPARK-36065
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.0, 3.3.0
> Reporter: Sumeet
> Priority: Major
> Labels: date_trunc, sql, timestamp
>
> Hi,
> Running date_trunc on any hour of "1891-10-01" returns incorrect output for "Europe/Bratislava" timezone.
> Use the following steps in order to reproduce the issue:
> * Run spark-shell using:
> {code:java}
> TZ="Europe/Bratislava" ./bin/spark-shell --conf spark.driver.extraJavaOptions='-Duser.timezone=Europe/Bratislava' --conf spark.executor.extraJavaOptions='-Duser.timezone=Europe/Bratislava' --conf spark.sql.session.timeZone="Europe/Bratislava"{code}
> * Generate test data:
> {code:java}
> ((0 until 9).map(hour => s"1891-10-01 00:0$hour:00") ++ (10 until 24).map(hour => s"1891-10-01 00:$hour:00")).toDF("ts_string").createOrReplaceTempView("temp_ts")
> {code}
> * Run query:
> {code:java}
> sql("select ts_string, cast(ts_string as TIMESTAMP) as ts, date_trunc('day', ts_string) from temp_ts").show(false)
> {code}
> * Output:
> {code:java}
> +-------------------+-------------------+--------------------------+
> |ts_string |ts |date_trunc(day, ts_string)|
> +-------------------+-------------------+--------------------------+
> |1891-10-01 00:00:00|1891-10-01 00:02:16|1891-10-01 00:02:16 |
> |1891-10-01 00:01:00|1891-10-01 00:03:16|1891-10-01 00:02:16 |
> |1891-10-01 00:02:00|1891-10-01 00:04:16|1891-10-01 00:02:16 |
> |1891-10-01 00:03:00|1891-10-01 00:03:00|1891-10-01 00:02:16 |
> |1891-10-01 00:04:00|1891-10-01 00:04:00|1891-10-01 00:02:16 |
> |1891-10-01 00:05:00|1891-10-01 00:05:00|1891-10-01 00:02:16 |
> |1891-10-01 00:06:00|1891-10-01 00:06:00|1891-10-01 00:02:16 |
> |1891-10-01 00:07:00|1891-10-01 00:07:00|1891-10-01 00:02:16 |
> |1891-10-01 00:08:00|1891-10-01 00:08:00|1891-10-01 00:02:16 |
> |1891-10-01 00:10:00|1891-10-01 00:10:00|1891-10-01 00:02:16 |
> |1891-10-01 00:11:00|1891-10-01 00:11:00|1891-10-01 00:02:16 |
> |1891-10-01 00:12:00|1891-10-01 00:12:00|1891-10-01 00:02:16 |
> |1891-10-01 00:13:00|1891-10-01 00:13:00|1891-10-01 00:02:16 |
> |1891-10-01 00:14:00|1891-10-01 00:14:00|1891-10-01 00:02:16 |
> |1891-10-01 00:15:00|1891-10-01 00:15:00|1891-10-01 00:02:16 |
> |1891-10-01 00:16:00|1891-10-01 00:16:00|1891-10-01 00:02:16 |
> |1891-10-01 00:17:00|1891-10-01 00:17:00|1891-10-01 00:02:16 |
> |1891-10-01 00:18:00|1891-10-01 00:18:00|1891-10-01 00:02:16 |
> |1891-10-01 00:19:00|1891-10-01 00:19:00|1891-10-01 00:02:16 |
> |1891-10-01 00:20:00|1891-10-01 00:20:00|1891-10-01 00:02:16 |
> +-------------------+-------------------+--------------------------+
> only showing top 20 rows
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org