You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "010Soham (via GitHub)" <gi...@apache.org> on 2025/12/19 04:34:46 UTC
[PR] Core: Improve error for null/unknown schema types in table creation [iceberg-python]
010Soham opened a new pull request, #2843:
URL: https://github.com/apache/iceberg-python/pull/2843
What changed
- Add a client-side validation error when a PyArrow `null` type is used with format version < 3, including the field path and a hint to use a concrete type or format-version 3.
- Add a unit test to verify the clear error message when converting a PyArrow schema with a null field.
Why
- Prevents misleading REST errors like “Cannot parse type string… unknown” and points directly to the offending field.
Testing
- make lint
- uv run pytest tests/catalog/test_base.py -k rejects_null_type -v
- make test (timed out at ~42%)
Closes #2539
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Improve error for null/unknown schema types in table creation [iceberg-python]
Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on code in PR #2843:
URL: https://github.com/apache/iceberg-python/pull/2843#discussion_r2671196788
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1435,6 +1435,13 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType:
elif pa.types.is_null(primitive):
# PyArrow null type (pa.null()) is converted to Iceberg UnknownType
# UnknownType can be promoted to any primitive type in V3+ tables per the Iceberg spec
+ if self._format_version < 3:
+ field_path = ".".join(self._field_names) if self._field_names else "<root>"
+ raise ValueError(
+ "Null type (pa.null()) is not supported in Iceberg format version "
+ f"{self._format_version}. Field: {field_path}. "
+ "Use a concrete type (string, int, boolean, etc.) or set format-version=3."
Review Comment:
thanks @geruh
Updated the error message to mention format-version 3+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Improve error for null/unknown schema types in table creation [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2843:
URL: https://github.com/apache/iceberg-python/pull/2843#issuecomment-3724755802
Thanks for the PR @010Soham and @geruh @Fokko for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Improve error for null/unknown schema types in table creation [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2843:
URL: https://github.com/apache/iceberg-python/pull/2843#discussion_r2646404096
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1435,6 +1435,13 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType:
elif pa.types.is_null(primitive):
# PyArrow null type (pa.null()) is converted to Iceberg UnknownType
# UnknownType can be promoted to any primitive type in V3+ tables per the Iceberg spec
+ if self._format_version < 3:
+ field_path = ".".join(self._field_names) if self._field_names else "<root>"
+ raise ValueError(
+ "Null type (pa.null()) is not supported in Iceberg format version "
+ f"{self._format_version}. Field: {field_path}. "
+ "Use a concrete type (string, int, boolean, etc.) or set format-version=3."
Review Comment:
nit: wdyt about future proofing with a message like: `Requires v3+ or use a concrete type.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Improve error for null/unknown schema types in table creation [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2843:
URL: https://github.com/apache/iceberg-python/pull/2843#discussion_r2646154990
##########
tests/catalog/test_base.py:
##########
@@ -207,6 +207,16 @@ def test_convert_schema_if_needed(
assert expected == catalog._convert_schema_if_needed(schema)
+def test_convert_schema_if_needed_rejects_null_type(catalog: InMemoryCatalog) -> None:
+ schema = pa.schema([pa.field("n1", pa.null())])
+ with pytest.raises(ValueError) as exc_info:
+ catalog._convert_schema_if_needed(schema)
+ message = str(exc_info.value)
+ assert "Null type" in message
+ assert "n1" in message
+ assert "format-version=3" in message
Review Comment:
this is the full error message
```
'Null type (pa.null()) is not supported in Iceberg format version 2. Field: n1. Use a concrete type (string, int, boolean, etc.) or set format-version=3.'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Improve error for null/unknown schema types in table creation [iceberg-python]
Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on PR #2843:
URL: https://github.com/apache/iceberg-python/pull/2843#issuecomment-3722513298
@Fokko
Have added the changes prescribed @geruh
let me know if that sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Improve error for null/unknown schema types in table creation [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2843:
URL: https://github.com/apache/iceberg-python/pull/2843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org