You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kris-gaudel (via GitHub)" <gi...@apache.org> on 2025/08/20 18:24:00 UTC
[PR] Document null field handling for PyArrow [iceberg-python]
kris-gaudel opened a new pull request, #2365:
URL: https://github.com/apache/iceberg-python/pull/2365
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Documentation for handling of null field for PyArrow
# Are these changes tested?
NA
# Are there any user-facing changes?
NA
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Document null field handling for PyArrow [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #2365:
URL: https://github.com/apache/iceberg-python/pull/2365#discussion_r2289198957
##########
pyiceberg/schema.py:
##########
@@ -1760,8 +1769,16 @@ def _is_field_compatible(self, lhs: NestedField) -> bool:
promote(rhs.field_type, lhs.field_type)
self.rich_table.add_row("✅", str(lhs), str(rhs))
return True
- except ResolveError:
- self.rich_table.add_row("❌", str(lhs), str(rhs))
+ except ResolveError as e:
+ # UnknownType can only be promoted to Primitive types
+ if isinstance(rhs.field_type, UnknownType):
+ if isinstance(lhs.field_type, (ListType, MapType, StructType)):
Review Comment:
```suggestion
if not isinstance(lhs.field_type, PrimitiveType):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Document null field handling for PyArrow [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #2365:
URL: https://github.com/apache/iceberg-python/pull/2365#issuecomment-3207966403
Thanks for working on this @kris-gaudel 🙌 This looks great! Would it be possible to throw in a test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Document null field handling for PyArrow [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #2365:
URL: https://github.com/apache/iceberg-python/pull/2365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Document null field handling for PyArrow [iceberg-python]
Posted by "kris-gaudel (via GitHub)" <gi...@apache.org>.
kris-gaudel commented on PR #2365:
URL: https://github.com/apache/iceberg-python/pull/2365#issuecomment-3208137218
Added 2 unit tests to check proper handling behaviour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Document null field handling for PyArrow [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #2365:
URL: https://github.com/apache/iceberg-python/pull/2365#discussion_r2289204315
##########
pyiceberg/schema.py:
##########
@@ -1760,8 +1769,16 @@ def _is_field_compatible(self, lhs: NestedField) -> bool:
promote(rhs.field_type, lhs.field_type)
self.rich_table.add_row("✅", str(lhs), str(rhs))
return True
- except ResolveError:
- self.rich_table.add_row("❌", str(lhs), str(rhs))
+ except ResolveError as e:
+ # UnknownType can only be promoted to Primitive types
+ if isinstance(rhs.field_type, UnknownType):
+ if isinstance(lhs.field_type, (ListType, MapType, StructType)):
+ error_msg = f"PyArrow null type (UnknownType) cannot be promoted to non-primitive type {lhs.field_type}. UnknownType can only be promoted to primitive types (string, int, boolean, etc.) in V3+ tables."
+ else:
+ error_msg = f"PyArrow null type (UnknownType) cannot be promoted to {lhs.field_type}. This may be due to table format version limitations (V1/V2 tables don't support UnknownType promotion)."
Review Comment:
I would leave out the `PyArrow` null type since this is in `schema.py`, rather than `pyarrow.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org