[jira] [Created] (ARROW-7345) [Python] Writing partitions with NaNs silently drops data

["Karl Dunkle Werner (Jira)" <ji...@apache.org>]

Karl Dunkle Werner created ARROW-7345:
-----------------------------------------

             Summary: [Python] Writing partitions with NaNs silently drops data
                 Key: ARROW-7345
                 URL: https://issues.apache.org/jira/browse/ARROW-7345
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 0.15.1
            Reporter: Karl Dunkle Werner


When writing a partitioned table, if the partitioning column has NA values, they're silently dropped. I think it would be helpful if there was a warning. Even better, from my perspective, would be writing out those partitions with a directory name like {{partition_col=NaN}}. 

Here's a small example where only the {{b = 2}} group is written out and the {{b = NaN}} group is dropped.

{code:python}
import os
import tempfile
import pyarrow.json
import pyarrow.parquet
from pathlib import Path

# Create a dataset with NaN:
json_str = """
{"a": 1, "b": 2}
{"a": 2, "b": null}
"""
with tempfile.NamedTemporaryFile() as tf:
    tf = Path(tf.name)
    tf.write_text(json_str)
    table = pyarrow.json.read_json(tf)

# Write out a partitioned dataset, using the NaN-containing column
with tempfile.TemporaryDirectory() as out_dir:
    pyarrow.parquet.write_to_dataset(table, out_dir, partition_cols=["b"])
    print(os.listdir(out_dir))
    read_table = pyarrow.parquet.read_table(out_dir)
print(f"Wrote out {table.shape[0]} rows, read back {read_table.shape[0]} row")

# Output:
#> ['b=2.0']
#> Wrote out 2 rows, read back 1 row
{code}
 
It looks like this caused by pandas dropping NaNs when doing [the {{groupby}} here|https://github.com/apache/arrow/blob/b16a3b53092ccfbc67e5a4e5c90be5913a67c8a5/python/pyarrow/parquet.py#L1434].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)