[jira] [Created] (ARROW-8115) [Python] NaT not handled correctly when writing

["Ilya Orson Sandoval (Jira)" <ji...@apache.org>]

Ilya Orson Sandoval created ARROW-8115:
------------------------------------------

             Summary: [Python] NaT not handled correctly when writing
                 Key: ARROW-8115
                 URL: https://issues.apache.org/jira/browse/ARROW-8115
             Project: Apache Arrow
          Issue Type: Bug
         Environment: <details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 1.0.1
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.0
pip              : 20.0.2
setuptools       : 45.2.0.post20200210
Cython           : None
pytest           : 5.3.5
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.4.2
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.12.0
pandas_datareader: None
bs4              : 4.8.1
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.2
matplotlib       : 3.1.2
numexpr          : None
odfpy            : None
openpyxl         : 3.0.3
pandas_gbq       : None
pyarrow          : 0.16.0
pytables         : None
pytest           : 5.3.5
pyxlsb           : None
s3fs             : 0.4.0
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : None
numba            : None

</details>
            Reporter: Ilya Orson Sandoval


#### Code Sample

```python
import pandas as pd
df = pd.DataFrame({"date": ["", "2019-05-01"]})
df.date = pd.to_datetime(df.date).dt.date
df.to_parquet("issue_NaT_parquet")
```

#### Problem description

The above gives me the following error:
<details>

> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
> <ipython-input-11-432405bef6ac> in <module>
> ----> 1 df.to_parquet("test.parquet")
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\util\_decorators.py in wrapper(*args, **kwargs)
>     212                 else:
>     213                     kwargs[new_arg_name] = new_arg_value
> --> 214             return func(*args, **kwargs)
>     215
>     216         return cast(F, wrapper)
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\core\frame.py in to_parquet(self, path, engine, compression, index, partition_cols, **kwargs)
> 
>    2114             index=index,
>    2115             partition_cols=partition_cols,
> -> 2116             **kwargs,
>    2117         )
>    2118
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\io\parquet.py in to_parquet(df, path, engine, compression, index, partition_cols, **kwargs)
>     262         index=index,
>     263         partition_cols=partition_cols,
> --> 264         **kwargs,
>     265     )
>     266
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pandas\io\parquet.py in write(self, df, path, compression, coerce_timestamps, index, partition_cols,
>  **kwargs)
>      99             from_pandas_kwargs["preserve_index"] = index
>     100
> --> 101         table = self.api.Table.from_pandas(df, **from_pandas_kwargs)
>     102         if partition_cols is not None:
>     103             self.api.parquet.write_to_dataset(
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\table.pxi in pyarrow.lib.Table.from_pandas()
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in dataframe_to_arrays(df, schema, preserve_index, nthreads, columns, safe)
> 
>     573     if nthreads == 1:
>     574         arrays = [convert_column(c, f)
> --> 575                   for c, f in zip(columns_to_convert, convert_fields)]
>     576     else:
>     577         from concurrent import futures
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in <listcomp>(.0)
>     573     if nthreads == 1:
>     574         arrays = [convert_column(c, f)
> --> 575                   for c, f in zip(columns_to_convert, convert_fields)]
>     576     else:
>     577         from concurrent import futures
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\pandas_compat.py in convert_column(col, field)
>     558
>     559         try:
> --> 560             result = pa.array(col, type=type_, from_pandas=True, safe=safe)
>     561         except (pa.ArrowInvalid,
>     562                 pa.ArrowNotImplementedError,
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\array.pxi in pyarrow.lib.array()
> 
> ~\AppData\Local\Continuum\miniconda3\lib\site-packages\pyarrow\array.pxi in pyarrow.lib._ndarray_to_array()
> 
> TypeError: an integer is required (got type datetime.date)

</details>

#### Expected Output

Parquet with null values mixed with date values.

#### Output of ``pd.show_versions()``



--
This message was sent by Atlassian Jira
(v8.3.4#803005)