You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Weston Pace (Jira)" <ji...@apache.org> on 2021/09/23 18:02:00 UTC
[jira] [Commented] (ARROW-14104) Reading Lists of Timestamps from
parquet files in Arrow 5.0.0 fails to preserve the TimeZone - unlike in
Arrow 4.0.0
[ https://issues.apache.org/jira/browse/ARROW-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419375#comment-17419375 ]
Weston Pace commented on ARROW-14104:
-------------------------------------
I wasn't able to reproduce this using the example script. Could you possibly provide a few more details around how you are creating and then reading the files? Could you upload a sample file that should have the America/Denver timezone but does not seem to read that way? That way we can isolate if the issue is reading or writing the file.
I tested both writing with 5.0 and reading with 5.0 and also writing with 4.0 and reading with 5.0.
I did have to modify your script slightly to get it to run which may have affected things. Here is what I used:
{code:python}
import datetime
import pyarrow as pa
import pyarrow.parquet as pq
column = pa.array([[datetime.datetime(2023, 9, 23, 11)]], pa.list_(pa.timestamp('us', 'America/New_York')))
t = pa.Table.from_arrays([column], names=['TimestampColumn'])
pq.write_table(t, "example.parq")
t2 = pq.read_table("example.parq")
print(t2)
{code}
> Reading Lists of Timestamps from parquet files in Arrow 5.0.0 fails to preserve the TimeZone - unlike in Arrow 4.0.0
> --------------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-14104
> URL: https://issues.apache.org/jira/browse/ARROW-14104
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Parquet, Python
> Affects Versions: 5.0.0
> Reporter: Sarah Gilmore
> Priority: Minor
>
> In Arrow 4.0.0 it is possible to round-trip the TimeZone property of List<Timestamp> columns to and from parquet files:
> {code:java}
> >>> import pyarrow as pa
> >>> import pyarrow.parquet as pq
> >>> column = pa.array([[datetime.datetime(2023, 9, 23, 11)]], pa.list_(pa.timestamp('us', 'America/New_York'));
> >>> t = pa.Table.from_arrays([column], name=['TimestampColumn']);
> >>> pq.write_table(t, "example.parq");
> >>> t2 = pq.read_table("example.parq");
> >>> t2
> pyarrow.Table
> Dates: list<item: timestamp[us, tz=America/Denver]>
> child 0, item: timestamp[us, tz=America/Denver]
> {code}
> However, if you read the same parquet file in pyarrow 5.0.0, the TimeZone is set to UTC:
> {code:java}
> >>> t3 = pq.read_table("example.parq");
> >>> t3
> pyarrow.Table
> Dates: list<item: timestamp[us, tz=UTC]>
> child 0, item: timestamp[us, tz=UTC]
> {code}
>
> I noticed that the TimeZone is preserved in Arrow 5.0 when reading non-nested timestamp columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)