You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by sk skk <sp...@gmail.com> on 2018/01/12 00:48:47 UTC

Timestamp changing while writing

Hello,

I am using createDataframe and passing java row rdd and schema . But it is
changing the time value when I write that data frame to a parquet file.

Can any one help .

Thank you,
Sudhir

Re: Timestamp changing while writing

Posted by Bryan Cutler <cu...@gmail.com>.

Spark internally stores timestamps as UTC values, so cearteDataFrame will
covert from local time zone to UTC. I think there was a Jira to correct
parquet output. Are the values you are seeing offset from your local time
zone?

On Jan 11, 2018 4:49 PM, "sk skk" <sp...@gmail.com> wrote:

> Hello,
>
> I am using createDataframe and passing java row rdd and schema . But it is
> changing the time value when I write that data frame to a parquet file.
>
> Can any one help .
>
> Thank you,
> Sudhir
>