You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Robert Nishihara (JIRA)" <ji...@apache.org> on 2018/02/09 07:55:00 UTC
[jira] [Updated] (ARROW-2122) Pyarrow fails to serialize dataframe
with timestamp.
[ https://issues.apache.org/jira/browse/ARROW-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Nishihara updated ARROW-2122:
------------------------------------
Description:
The bug can be reproduced as follows.
{code:java}
import pyarrow as pa
import pandas as pd
df = pd.DataFrame({'A': [pd.Timestamp('2012-11-11 00:00:00+01:00'), pd.NaT]})
s = pa.serialize(df).to_buffer()
new_df = pa.deserialize(s) # this fails{code}
The last line fails with
{code:java}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "serialization.pxi", line 441, in pyarrow.lib.deserialize
File "serialization.pxi", line 404, in pyarrow.lib.deserialize_from
File "serialization.pxi", line 257, in pyarrow.lib.SerializedPyObject.deserialize
File "serialization.pxi", line 174, in pyarrow.lib.SerializationContext._deserialize_callback
File "/home/ubuntu/arrow/python/pyarrow/serialization.py", line 77, in _deserialize_pandas_dataframe
return pdcompat.serialized_dict_to_dataframe(data)
File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 446, in serialized_dict_to_dataframe
for block in data['blocks']]
File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 446, in <listcomp>
for block in data['blocks']]
File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 466, in _reconstruct_block
dtype = _make_datetimetz(item['timezone'])
File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 481, in _make_datetimetz
return DatetimeTZDtype('ns', tz=tz)
File "/home/ubuntu/anaconda3/lib/python3.5/site-packages/pandas/core/dtypes/dtypes.py", line 409, in __new__
raise ValueError("DatetimeTZDtype constructor must have a tz "
ValueError: DatetimeTZDtype constructor must have a tz supplied{code}
was:
The bug can be reproduced as follows.
{code:java}
import pyarrow as pa
import pandas as pd
s = pa.serialize({code}
> Pyarrow fails to serialize dataframe with timestamp.
> ----------------------------------------------------
>
> Key: ARROW-2122
> URL: https://issues.apache.org/jira/browse/ARROW-2122
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Robert Nishihara
> Priority: Major
>
> The bug can be reproduced as follows.
> {code:java}
> import pyarrow as pa
> import pandas as pd
> df = pd.DataFrame({'A': [pd.Timestamp('2012-11-11 00:00:00+01:00'), pd.NaT]})
> s = pa.serialize(df).to_buffer()
> new_df = pa.deserialize(s) # this fails{code}
> The last line fails with
> {code:java}
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "serialization.pxi", line 441, in pyarrow.lib.deserialize
> File "serialization.pxi", line 404, in pyarrow.lib.deserialize_from
> File "serialization.pxi", line 257, in pyarrow.lib.SerializedPyObject.deserialize
> File "serialization.pxi", line 174, in pyarrow.lib.SerializationContext._deserialize_callback
> File "/home/ubuntu/arrow/python/pyarrow/serialization.py", line 77, in _deserialize_pandas_dataframe
> return pdcompat.serialized_dict_to_dataframe(data)
> File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 446, in serialized_dict_to_dataframe
> for block in data['blocks']]
> File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 446, in <listcomp>
> for block in data['blocks']]
> File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 466, in _reconstruct_block
> dtype = _make_datetimetz(item['timezone'])
> File "/home/ubuntu/arrow/python/pyarrow/pandas_compat.py", line 481, in _make_datetimetz
> return DatetimeTZDtype('ns', tz=tz)
> File "/home/ubuntu/anaconda3/lib/python3.5/site-packages/pandas/core/dtypes/dtypes.py", line 409, in __new__
> raise ValueError("DatetimeTZDtype constructor must have a tz "
> ValueError: DatetimeTZDtype constructor must have a tz supplied{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)