[I] Strange behavior when null is partition value in parquet datasets with filename and directory partitioning [arrow]

["rohanjain101 (via GitHub)" <gi...@apache.org>]

rohanjain101 opened a new issue, #39938:
URL: https://github.com/apache/arrow/issues/39938

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   ```
   null_table = pa.Table.from_pydict({"A": ["", None], "B": [None, ""], "C": ["E", "F"]})
   partitioning = pa.dataset.FilenamePartitioning(pa.schema({"A": pa.string()}))
   pa.parquet.write_to_dataset(null_table, r"dir\filename\", partitioning=partitioning)
   >>> pa.parquet.read_table(  r"dir\filename\", partitioning=partitioning)
   pyarrow.Table
   A: string
   ----
   A: []
   >>>
   ```
   
   Null does not roundtrip when it is one of the partition values, and filename or directory partitioning are used.
   
   When mixing null and empty string with directory partitioning, it also doesn't roundtrip:
   
   ```
   >>> partitioning = pa.dataset.DirectoryPartitioning(pa.schema({"A": pa.string()}))
   >>> pa.parquet.write_to_dataset(null_table, r"dir\directory\", partitioning=partitioning)
   >>> pa.parquet.read_table( r"dir\directory\", partitioning=partitioning)
   pyarrow.Table
   B: string
   C: string
   A: string
   ----
   B: [[null],[""]]
   C: [["E"],["F"]]
   A: [[null],[null]]
   >>>
   ```
   
   I would expect A to be:
   
   `A: [[""],[null]]`
   
   Which is what the original table had. Also, if there are 2 partition columns, and the first one has a null value, an error is raised, but if there's only 1 partition column, then no error is raised, which seems inconsistent:
   
   ```
   >>> partitioning = pa.dataset.DirectoryPartitioning(pa.schema({"A": pa.string(), "B": pa.string()}))
   >>> pa.parquet.write_to_dataset(null_table, r"dir\", partitioning=partitioning)
   pyarrow.lib.ArrowInvalid: No partition key for A but a key was provided subsequently for B.
   >>>
   ```
   
   What is the expected behavior when null is in a partition column? Is it expected to work, or should an error always be raised?
   
   
   
   ### Component(s)
   
   C++, Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org