Re: [I] [C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs. [arrow]

["fenfeng9 (via GitHub)" <gi...@apache.org>]

fenfeng9 commented on issue #50470:
URL: https://github.com/apache/arrow/issues/50470#issuecomment-4952158701

   A null parent struct masks its child slots. Therefore, an underlying Arrow child-array slot may be null when the parent struct is null, even if the child field is non-nullable. This is valid because the child is logically absent. The Parquet writer incorrectly rejects this case.
   
   
   #### Reproduce
   
   ```python
   import pyarrow as pa
   import pyarrow.parquet as pq
   
   
   value_field = pa.field("value", pa.int32(), nullable=False)
   
   # The physical child slot is null.
   value_column = pa.array([None], type=pa.int32())
   
   # True means that the corresponding parent struct is null.
   parent_is_null = pa.array([True])
   
   array = pa.StructArray.from_arrays(
       arrays=[value_column],
       fields=[value_field],
       mask=parent_is_null,
   )
   
   print("logical array:", array.to_pylist())
   print("physical child:", array.field("value").to_pylist())
   
   pq.write_table(pa.table({"data": array}), "output.parquet")
   ```
   #### Result
   
   ```python
   logical array: [None]
   physical child: [None]
   Traceback (most recent call last):
     File "/path/to/test.py", line 22, in <module>
       pq.write_table(pa.table({"data": array}), "output.parquet")
       ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/path/to/site-packages/pyarrow/parquet/core.py", line 2015, in write_table
       writer.write_table(table, row_group_size=row_group_size)
       ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/path/to/site-packages/pyarrow/parquet/core.py", line 1180, in write_table
       self.writer.write_table(table, row_group_size=row_group_size)
       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "pyarrow/_parquet.pyx", line 2397, in pyarrow._parquet.ParquetWriter.write_table
     File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
   pyarrow.lib.ArrowInvalid: Column 'value' is declared non-nullable but contains nulls
   ```
   
   #### Analysis
   This appears to be a regression introduced by #44921. 
   
   The check uses `leaf_array.null_count()` without accounting for null parent structs, so it incorrectly rejects child nulls masked by a null parent.
   
   https://github.com/apache/arrow/blob/5815dc0b4e3d9239828b47d427c9759fbc506e09/cpp/src/parquet/column_writer.cc#L1432-L1442
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org