[jira] [Commented] (SPARK-14057) sql time stamps do not respect time zones

["Vijay Parmar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275987#comment-15275987 ] 

Vijay Parmar commented on SPARK-14057:
--------------------------------------

I have few suggestions to make here after looking into the issue along with referring google and other sources:-

1. We can make use of the built-in java.time.package which is available in Java 8 and higher versions (http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/util/Date.java).
    In short, here a new instance would be created to adjust the Time-Zone.
   
   In java.util.date package the class in most of the cases ignores the Time-Zone.

  We could try implementing this package.


2.  This code snippet can be handy :-

ZoneId zoneLondon = ZoneId.of("London"); 
ZonedDateTime nowLondon = ZonedDateTime.now ( zoneLondon );

ZoneId zoneSingapore = ZoneId.of("Singapore"); 
ZonedDateTime nowSingapore = nowLondon.withZoneSameInstant( zoneSingapore );
ZonedDateTime nowUTC = nowLondon.withZoneSameInstant( ZoneOffset.UTC );

3. We need to look into the SQL side code also To have an understanding how the Time is getting captured and stored once it is received from this end.

I will still keep on looking iinto the issue and will update you. Meanwhile, I also wait for your comment(s) on my suggestions.  




> sql time stamps do not respect time zones
> -----------------------------------------
>
>                 Key: SPARK-14057
>                 URL: https://issues.apache.org/jira/browse/SPARK-14057
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0
>            Reporter: Andrew Davidson
>            Priority: Minor
>
> we have time stamp data. The time stamp data is UTC how ever when we load the data into spark data frames, the system assume the time stamps are in the local time zone. This causes problems for our data scientists. Often they pull data from our data center into their local macs. The data centers run UTC. There computers are typically in PST or EST.
> It is possible to hack around this problem
> This cause a lot of errors in their analysis
> A complete description of this issue can be found in the following mail msg
> https://www.mail-archive.com/user@spark.apache.org/msg48121.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org