You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jqin61 (via GitHub)" <gi...@apache.org> on 2024/02/01 16:23:26 UTC
[I] Support Null Column Write in Unpartitioned Table [iceberg-python]
jqin61 opened a new issue, #347:
URL: https://github.com/apache/iceberg-python/issues/347
### Feature Request / Improvement
Hi I noticed the existing append and overwrite functions would break if any column of the input arrow table consists only of nulls. For example:
```
year_data = pa.array([None], type=pa.int64())
n_legs_data = pa.array([4], type=pa.int64()) # Assuming legs are always non-null and integer
animals_data = pa.array(["Flamingo"], type=pa.string())
# Create an arrow table with one column full of nulls
arrow_table_nulls = pa.Table.from_arrays([year_data, n_legs_data, animals_data], names=['year', 'n_legs', 'animals'])
catalog = load_catalog("lacus", **properties)
iceberg_table_nulls = catalog.load_table('test.append_arrow_with_nulls')
iceberg_table_nulls.append(table)
```
would break in the step of metadata collection of the written parquet file.
The error looks like this:
```
...
/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py in fill_parquet_file_metadata(data_file, parquet_metadata, stats_columns, parquet_column_mapping)
1689
1690 for k, agg in col_aggs.items():
-> 1691 _min = agg.min_as_bytes()
1692 if _min is not None:
1693 lower_bounds[k] = _min
/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py in min_as_bytes(self)
1341
1342 def min_as_bytes(self) -> bytes:
-> 1343 return self.serialize(
1344 self.current_min
1345 if self.trunc_length is None
/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py in serialize(self, value)
1332
1333 def serialize(self, value: Any) -> bytes:
-> 1334 return to_bytes(self.primitive_type, value)
1335
1336 def update_min(self, val: Any) -> None:
/lib/python3.10/functools.py in wrapper(*args, **kw)
887 '1 positional argument')
888
--> 889 return dispatch(args[0].__class__)(*args, **kw)
890
891 funcname = getattr(func, '__name__', 'singledispatch function')
/lib/python3.10/site-packages/pyiceberg/conversions.py in _(_, value)
183 @to_bytes.register(LongType)
184 def _(_: PrimitiveType, value: int) -> bytes:
--> 185 return _LONG_STRUCT.pack(value)
186
187
error: required argument is not an integer
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Null Column Write in Unpartitioned Table [iceberg-python]
Posted by "jqin61 (via GitHub)" <gi...@apache.org>.
jqin61 commented on issue #347:
URL: https://github.com/apache/iceberg-python/issues/347#issuecomment-1921708521
move to a bug fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Null Column Write in Unpartitioned Table [iceberg-python]
Posted by "jqin61 (via GitHub)" <gi...@apache.org>.
jqin61 closed issue #347: Support Null Column Write in Unpartitioned Table
URL: https://github.com/apache/iceberg-python/issues/347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org