You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2020/03/17 09:23:00 UTC
[jira] [Commented] (ARROW-8115) [Python] NaT not handled correctly
when writing
[ https://issues.apache.org/jira/browse/ARROW-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060762#comment-17060762 ]
Joris Van den Bossche commented on ARROW-8115:
----------------------------------------------
[~IlyaOrson] thanks for the report!
This is an issue with converting the pandas DataFrame to an Arrow table (so not related with the actual writing to parquet)
The issue is that pandas creates an object dtype column with datetime.date objects using NaT as missing value sentinel. That combination doesn't seem to be supported by arrow:
{code}
In [15]: pa.array([pd.NaT, datetime.date(2019, 5, 1)])
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-15-dd2b1a934774> in <module>
----> 1 pa.array([pd.NaT, datetime.date(2019, 5, 1)])
~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()
~/scipy/repos/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()
TypeError: an integer is required (got type datetime.date)
{code}
> [Python] NaT not handled correctly when writing
> -----------------------------------------------
>
> Key: ARROW-8115
> URL: https://issues.apache.org/jira/browse/ARROW-8115
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Environment: <details>
> [paste the output of ``pd.show_versions()`` here below this line]
> INSTALLED VERSIONS
> ------------------
> commit : None
> python : 3.7.4.final.0
> python-bits : 64
> OS : Windows
> OS-release : 10
> machine : AMD64
> processor : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
> byteorder : little
> LC_ALL : None
> LANG : None
> LOCALE : None.None
> pandas : 1.0.1
> numpy : 1.18.1
> pytz : 2019.3
> dateutil : 2.8.0
> pip : 20.0.2
> setuptools : 45.2.0.post20200210
> Cython : None
> pytest : 5.3.5
> hypothesis : None
> sphinx : None
> blosc : None
> feather : None
> xlsxwriter : None
> lxml.etree : 4.4.2
> html5lib : 1.0.1
> pymysql : None
> psycopg2 : None
> jinja2 : 2.11.1
> IPython : 7.12.0
> pandas_datareader: None
> bs4 : 4.8.1
> bottleneck : None
> fastparquet : None
> gcsfs : None
> lxml.etree : 4.4.2
> matplotlib : 3.1.2
> numexpr : None
> odfpy : None
> openpyxl : 3.0.3
> pandas_gbq : None
> pyarrow : 0.16.0
> pytables : None
> pytest : 5.3.5
> pyxlsb : None
> s3fs : 0.4.0
> scipy : None
> sqlalchemy : None
> tables : None
> tabulate : None
> xarray : None
> xlrd : 1.2.0
> xlwt : None
> xlsxwriter : None
> numba : None
> </details>
> Reporter: Ilya Orson Sandoval
> Priority: Major
>
> h3. Code Sample
> {code:python}
> import pandas as pd
> df = pd.DataFrame({"date": ["", "2019-05-01"]})
> df.date = pd.to_datetime(df.date).dt.date
> df.to_parquet("issue_NaT_parquet")
> {code}
> h3. Problem description
> The above gives me the following error:
> {code}
> ---------------------------------------------------------------------------
> TypeError Traceback (most recent call last)
> <ipython-input-13-432405bef6ac> in <module>
> ----> 1 df.to_parquet("test.parquet")
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\util\_decorators.py in wrapper(*args, **kwargs)
> 212 else:
> 213 kwargs[new_arg_name] = new_arg_value
> --> 214 return func(*args, **kwargs)
> 215
> 216 return cast(F, wrapper)
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\core\frame.py in to_parquet(self, path, engine, compression, index, partition_cols, **kwargs)
> 2114 index=index,
> 2115 partition_cols=partition_cols,
> -> 2116 **kwargs,
> 2117 )
> 2118
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\io\parquet.py in to_parquet(df, path, engine, compression, index, partition_cols, **kwargs)
> 262 index=index,
> 263 partition_cols=partition_cols,
> --> 264 **kwargs,
> 265 )
> 266
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\io\parquet.py in write(self, df, path, compression, coerce_timestamps, index, partition_cols, **kwargs)
> 99 from_pandas_kwargs["preserve_index"] = index
> 100
> --> 101 table = self.api.Table.from_pandas(df, **from_pandas_kwargs)
> 102 if partition_cols is not None:
> 103 self.api.parquet.write_to_dataset(
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\table.pxi in pyarrow.lib.Table.from_pandas()
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in dataframe_to_arrays(df, schema, preserve_index, nthreads, columns, safe)
> 573 if nthreads == 1:
> 574 arrays = [convert_column(c, f)
> --> 575 for c, f in zip(columns_to_convert, convert_fields)]
> 576 else:
> 577 from concurrent import futures
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in <listcomp>(.0)
> 573 if nthreads == 1:
> 574 arrays = [convert_column(c, f)
> --> 575 for c, f in zip(columns_to_convert, convert_fields)]
> 576 else:
> 577 from concurrent import futures
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in convert_column(col, field)
> 558
> 559 try:
> --> 560 result = pa.array(col, type=type_, from_pandas=True, safe=safe)
> 561 except (pa.ArrowInvalid,
> 562 pa.ArrowNotImplementedError,
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\array.pxi in pyarrow.lib.array()
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\array.pxi in pyarrow.lib._ndarray_to_array()
> TypeError: an integer is required (got type datetime.date){quote}
> {code}
> h3. Expected Output
> Parquet with null values mixed with date values.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)