You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2017/08/28 21:24:00 UTC

[jira] [Created] (ARROW-1425) [Python] Document semantic differences between Spark timestamps and Arrow timestamps

Wes McKinney created ARROW-1425:
-----------------------------------

             Summary: [Python] Document semantic differences between Spark timestamps and Arrow timestamps
                 Key: ARROW-1425
                 URL: https://issues.apache.org/jira/browse/ARROW-1425
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Python
            Reporter: Wes McKinney
             Fix For: 0.7.0


The way that Spark treats non-timezone-aware timestamps as session local can be problematic when using pyarrow which may view the data coming from toPandas() as time zone naive (but with fields as though it were UTC, not session local). We should document carefully how to properly handle the data coming from Spark to avoid problems.

cc [~bryanc] [~holdenkarau]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)