You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "pavithra ramachandran (Jira)" <ji...@apache.org> on 2020/06/26 07:12:00 UTC
[jira] [Issue Comment Deleted] (SPARK-31887) Date casting to string
is giving wrong value
[ https://issues.apache.org/jira/browse/SPARK-31887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
pavithra ramachandran updated SPARK-31887:
------------------------------------------
Comment: was deleted
(was: i would like to check this.)
> Date casting to string is giving wrong value
> --------------------------------------------
>
> Key: SPARK-31887
> URL: https://issues.apache.org/jira/browse/SPARK-31887
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.5
> Environment: The spark is running on cluster mode with Mesos.
>
> Mesos agents are dockerised running on Ubuntu 18.
>
> Timezone setting of docker instance: UTC
> Timezone of server hosting docker: America/New_York
> Timezone of driver machine: America/New_York
> Reporter: Amit Gupta
> Priority: Major
>
> The code converts the string to date and then write it in csv.
> {code:java}
> val x = Seq(("2020-02-19", "2020-02-19 05:11:00")).toDF("a", "b").select('a.cast("date"), 'b.cast("timestamp"))
> x.show()
> +----------+-------------------+
> | a| b|
> +----------+-------------------+
> |2020-02-19|2020-02-19 05:11:00|
> +----------+-------------------+
> x.write.mode("overwrite").option("header", true).csv("/tmp/test1.csv")
> {code}
>
> The date written in CSV file is different:
> {code:java}
> > snakebite cat "/tmp/test1.csv/*.csv"
> a,b
> 2020-02-18,2020-02-19T05:11:00.000Z{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org