You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Amit Gupta (Jira)" <ji...@apache.org> on 2020/06/01 12:10:00 UTC
[jira] [Created] (SPARK-31887) Date casting to string is giving
wrong value
Amit Gupta created SPARK-31887:
----------------------------------
Summary: Date casting to string is giving wrong value
Key: SPARK-31887
URL: https://issues.apache.org/jira/browse/SPARK-31887
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.5
Environment: The spark is running on cluster mode with Mesos.
Mesos agents are dockerised running on Ubuntu 18.
Timezone setting of docker instance: UTC
Timezone of server hosting docker: America/New_York
Timezone of driver machine: America/New_York
Reporter: Amit Gupta
The code converts the string to date and then write it in csv.
{code:java}
val x = Seq(("2020-02-19", "2020-02-19 05:11:00")).toDF("a", "b").select('a.cast("date"), 'b.cast("timestamp"))
x.show()
+----------+-------------------+
| a| b|
+----------+-------------------+
|2020-02-19|2020-02-19 05:11:00|
+----------+-------------------+
x.write.mode("overwrite").option("header", true).csv("/tmp/test1.csv")
{code}
The date written in CSV file is different:
{code:java}
> snakebite cat "/tmp/test1.csv/*.csv"
a,b
2020-02-18,2020-02-19T05:11:00.000Z{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org