[jira] [Comment Edited] (ARROW-14104) Reading Lists of Timestamps from parquet files in Arrow 5.0.0 fails to preserve the TimeZone - unlike in Arrow 4.0.0

["Sarah Gilmore (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17430530#comment-17430530 ] 

Sarah Gilmore edited comment on ARROW-14104 at 10/19/21, 12:57 PM:
-------------------------------------------------------------------

So sorry about the delay.  [~jorisvandenbossche] and [~westonpace].

I've attached two files ([^exampleArrow4.parq] and [^exampleArrow5.parq]) that were both created with the following code:
{code:java}
// code placeholder
import pyarrow as pa
import pyarrow.parquet as pq

import datetime
>>> column = pa.array([[datetime.datetime(2023, 9, 23, 11)]], pa.list_(pa.timestamp('us', 'America/New_York')));
>>> t = pa.Table.from_arrays([column], names=['TimestampColumn']);
>>> pq.write_table(t, "example.parq"); 

{code}
  

I ran the code snippet above in pyarrow 4.0.0 and pyarrow 5.0.0 to create exampleArrow4.parq and exampleArrow5.parq, respectively. 

 

Here's the output of reading both files in pyarrow 4.0.0: 
{code:java}
// code placeholder
import pyarrow as pa
import pyarrow.parquet as pq

>>> t1 = pq.read_table("exampleArrow4.parq")
>>> t1
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=America/New_York]>
  child 0, item: timestamp[us, tz=America/New_York]

>>> t2 = pq.read_table("exampleArrow5.parq")
>>> t2
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=America/New_York]>
  child 0, item: timestamp[us, tz=America/New_York]{code}
 The TimeZone is read in properly for both files.

 

Here's the output of reading both files in pyarrow 5.0.0:
{code:java}
// code placeholder
import pyarrow as pa import pyarrow.parquet as pq

>>> t1 = pq.read_table("exampleArrow4.parq")
>>> t1
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=UTC]>
  child 0, item: timestamp[us, tz=UTC]

>>> t2 = pq.read_table("exampleArrow5.parq")
>>> t2
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=UTC]>
  child 0, item: timestamp[us, tz=UTC]
{code}
 

It looks like pyarrow 5.0.0 writes out the TimeZone information, but doesn't read it in properly.

 

 

 


was (Author: sgilmore):
So sorry about the delay.  [~jorisvandenbossche] and [~westonpace].

I've attached two files ([^exampleArrow4.parq] and [^exampleArrow5.parq]) that were both created with the following code:

 
{code:java}
// code placeholder
import pyarrow as pa
import pyarrow.parquet as pq

import datetime
>>> column = pa.array([[datetime.datetime(2023, 9, 23, 11)]], pa.list_(pa.timestamp('us', 'America/New_York')));
>>> t = pa.Table.from_arrays([column], names=['TimestampColumn']);
>>> pq.write_table(t, "example.parq"); 

{code}
 

 

I ran the code snippet above in pyarrow 4.0.0 and pyarrow 5.0.0 to create exampleArrow4.parq and exampleArrow5.parq, respectively. 

 

Here's the output of reading both files in pyarrow 4.0.0:

 
{code:java}
// code placeholder
import pyarrow as pa
import pyarrow.parquet as pq

>>> t1 = pq.read_table("exampleArrow4.parq")
>>> t1
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=America/New_York]>
  child 0, item: timestamp[us, tz=America/New_York]

>>> t2 = pq.read_table("exampleArrow5.parq")
>>> t2
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=America/New_York]>
  child 0, item: timestamp[us, tz=America/New_York]{code}
 

The TimeZone is read in properly for both files.

 

Here's the output of reading both files in pyarrow 5.0.0:

 
{code:java}
// code placeholder
import pyarrow as pa import pyarrow.parquet as pq

>>> t1 = pq.read_table("exampleArrow4.parq")
>>> t1
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=UTC]>
  child 0, item: timestamp[us, tz=UTC]

>>> t2 = pq.read_table("exampleArrow5.parq")
>>> t2
pyarrow.Table
TimestampColumn: list<item: timestamp[us, tz=UTC]>
  child 0, item: timestamp[us, tz=UTC]
{code}
 

 

It looks like pyarrow 5.0.0 writes out the TimeZone information, but doesn't read it in properly.

 

 

 

> Reading Lists of Timestamps from parquet files in Arrow 5.0.0 fails to preserve the TimeZone - unlike in Arrow 4.0.0
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-14104
>                 URL: https://issues.apache.org/jira/browse/ARROW-14104
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, Parquet, Python
>    Affects Versions: 5.0.0
>            Reporter: Sarah Gilmore
>            Priority: Minor
>         Attachments: exampleArrow4.parq, exampleArrow5.parq
>
>
> In Arrow 4.0.0 it is possible to round-trip the TimeZone property of List<Timestamp> columns to and from parquet files: 
> {code:java}
> >>> import pyarrow as pa
> >>> import pyarrow.parquet as pq
> >>> import datetime 
> >>> column = pa.array([[datetime.datetime(2023, 9, 23, 11)]], pa.list_(pa.timestamp('us', 'America/New_York')));
> >>> t = pa.Table.from_arrays([column], names=['TimestampColumn']);
> >>> pq.write_table(t, "example.parq");
> >>> t2 = pq.read_table("example.parq");
> >>> t2
> pyarrow.Table
> Dates: list<item: timestamp[us, tz=America/New_York]>
>   child 0, item: timestamp[us, tz=America/New_York]
> {code}
> However, if you read the same parquet file in pyarrow 5.0.0, the TimeZone is set to UTC:
> {code:java}
> >>> t3 = pq.read_table("example.parq");
> >>> t3
> pyarrow.Table
> Dates: list<item: timestamp[us, tz=UTC]>
>   child 0, item: timestamp[us, tz=UTC]
>  {code}
>  
> I noticed that the TimeZone is preserved in Arrow 5.0 when reading non-nested timestamp columns. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)