You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Daniel Figus (Jira)" <ji...@apache.org> on 2020/05/26 14:44:00 UTC
[jira] [Created] (ARROW-8944) [Python] Pandas - Parquet - Pandas
roundtrip causes out of bounds timestamp
Daniel Figus created ARROW-8944:
-----------------------------------
Summary: [Python] Pandas - Parquet - Pandas roundtrip causes out of bounds timestamp
Key: ARROW-8944
URL: https://issues.apache.org/jira/browse/ARROW-8944
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.17.1, 0.17.0
Environment: pandas==1.0.3
pyarrow==0.17.1
Python==3,7.6 @ Windows 10 64Bit
Reporter: Daniel Figus
The following pandas -> parquet -> pandas roudtrip raises an out of bounds timestamp error with pyarrow 0.17.0 and 0.17.1:
{code:python}
import pandas
target = 'ts_roundtrip.parquet'
dataframe = pandas.DataFrame({'id':[1,2,3],'timestamp':['', '', '']})
dataframe['timestamp'] = pandas.to_datetime(dataframe['timestamp'],errors='raise')
dataframe2 = pandas.DataFrame({'id':[4,5,6,7],'timestamp':['', '2020-03-02T03:03:17.791062Z','','']})
dataframe2['timestamp'] = pandas.to_datetime(dataframe2['timestamp'],errors='raise')
dataframe = dataframe.append(dataframe2)
print(dataframe.head(10))
dataframe.to_parquet(target, coerce_timestamps=None, index=False, version='2.0')
dataframe_new = pandas.read_parquet(target)
print(dataframe_new.head())
{code}
Output:
{noformat}
id timestamp
0 1 NaT
1 2 NaT
2 3 NaT
0 4 NaT
1 5 2020-03-02 03:03:17.791062+00:00
2 6 NaT
3 7 NaT
Traceback (most recent call last):
File "c:\some\path\pyarrow_ts_test.py", line 16, in <module>
dataframe_new = pandas.read_parquet(target)
File "c:\some\path\venv\lib\site-packages\pandas\io\parquet.py", line 310, in read_parquet
return impl.read(path, columns=columns, **kwargs)
File "c:\some\path\venv\lib\site-packages\pandas\io\parquet.py", line 125, in read
path, columns=columns, **kwargs
File "pyarrow\array.pxi", line 587, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow\table.pxi", line 1640, in pyarrow.lib.Table._to_pandas
File "c:\some\path\venv\lib\site-packages\pyarrow\pandas_compat.py", line 766, in table_to_blockmanager
blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
File "c:\some\path\venv\lib\site-packages\pyarrow\pandas_compat.py", line 1102, in _table_to_blocks
list(extension_columns.keys()))
File "pyarrow\table.pxi", line 1107, in pyarrow.lib.table_to_blocks
File "pyarrow\error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Casting from timestamp[us] to timestamp[ns] would result in out of bounds timestamp: -62135596800000000
{noformat}
Background:
We have a dataset with a timestamp column that is sparsely populated and originates from many json files. So it is very likely that in some of those json files there is no timestamp (as string in ISO format) and instead just an empty string. Each JSON file was read into a pandas dataframe, the timestamp column casted to datetime and all dataframes appended. That was done with pyarrow<0.17.0 and those parquet files cannot be read any longer and result in the above mentioned error message as well.
A closer look at our old parquets show that the NaTs are converted to "1754-08-30 22:43:41.128654848" when reading back to a pandas dataframe :(. You get the same result when you run the above code and pyarrow==0.16.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)