You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sunxiaojian (via GitHub)" <gi...@apache.org> on 2025/03/28 07:37:03 UTC
[PR] fixed [iceberg-python]
sunxiaojian opened a new pull request, #1860:
URL: https://github.com/apache/iceberg-python/pull/1860
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Closes #1851
# Are these changes tested?
test_types.py
# Are there any user-facing changes?
N/A
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Pass in the type as a string [iceberg-python]
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #1860:
URL: https://github.com/apache/iceberg-python/pull/1860#issuecomment-2763764233
@kevinjqliu Thanks for the review! I've fixed all the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Pass in the type as a string [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1860:
URL: https://github.com/apache/iceberg-python/pull/1860#discussion_r2019120254
##########
tests/test_types.py:
##########
@@ -231,6 +231,22 @@ def test_nested_field() -> None:
assert "validation errors for NestedField" in str(exc_info.value)
+def test_nested_field_type_as_str_unsupported() -> None:
+ with pytest.raises(ValueError) as exc_info:
+ _ = (NestedField(1, "field", "list", required=True),)
Review Comment:
nit: lets also test other complex type `list/map/struct`
##########
pyiceberg/types.py:
##########
@@ -324,7 +324,7 @@ def __init__(
self,
field_id: Optional[int] = None,
name: Optional[str] = None,
- field_type: Optional[IcebergType] = None,
+ field_type: Optional[IcebergType | str] = None,
Review Comment:
nit: add to the example to the docstring above with `str` type
##########
tests/test_types.py:
##########
@@ -231,6 +231,22 @@ def test_nested_field() -> None:
assert "validation errors for NestedField" in str(exc_info.value)
+def test_nested_field_type_as_str_unsupported() -> None:
+ with pytest.raises(ValueError) as exc_info:
+ _ = (NestedField(1, "field", "list", required=True),)
+ assert "Unsupported field type: list" in str(exc_info.value)
+
+
+def test_nested_field_type_as_str_struct() -> None:
Review Comment:
nit: include other types that are part of `handle_primitive_type` here as well
##########
pyiceberg/types.py:
##########
@@ -340,6 +340,12 @@ def __init__(
data["doc"] = doc
data["initial-default"] = data["initial-default"] if "initial-default" in data else initial_default
data["write-default"] = data["write-default"] if "write-default" in data else write_default
+ if isinstance(data["type"], str):
+ try:
+ data["type"] = IcebergType.handle_primitive_type(data["type"], None)
+ except ValueError as e:
+ raise ValueError(f"Unsupported field type: {data['type']}.") from e
+
Review Comment:
nit: what do you think about using a pydantic field validator here?
https://docs.pydantic.dev/latest/concepts/validators/#field-before-validator
```
@validator("field_type", pre=True)
def convert_field_type(cls, v):
"""Convert string values into IcebergType instances."""
if isinstance(v, str):
try:
return IcebergType.handle_primitive_type(v, None)
except ValueError as e:
raise ValueError(f"Unsupported field type: {v}") from e
return v # If already an IcebergType, keep as is
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Pass in the type as a string [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1860:
URL: https://github.com/apache/iceberg-python/pull/1860#discussion_r2019955353
##########
tests/test_types.py:
##########
@@ -231,6 +246,27 @@ def test_nested_field() -> None:
assert "validation errors for NestedField" in str(exc_info.value)
+def test_nested_field_type_as_str_unsupported() -> None:
+ unsupported_types = ["list", "map", "struct"]
+ for type_str in unsupported_types:
+ with pytest.raises(ValueError) as exc_info:
+ _ = NestedField(1, "field", type_str, required=True)
+ assert f"Unsupported field type: '{type_str}'" in str(exc_info.value)
+
+
+@pytest.mark.parametrize("type_str, type_class", primitive_types.items())
+def test_nested_field_type_as_str(type_str: str, type_class: type) -> None:
Review Comment:
nit: parametrizing the test actually is slower compared to just running a for loop into the function
```
for type_str, type_class in primitive_types.items():
field_var = NestedField(
1,
"field",
type_str,
required=True,
)
assert isinstance(
field_var.field_type, type_class
), f"Expected {type_class.__name__}, got {field_var.field_type.__class__.__name__}"
```
##########
tests/test_types.py:
##########
@@ -231,6 +246,27 @@ def test_nested_field() -> None:
assert "validation errors for NestedField" in str(exc_info.value)
+def test_nested_field_type_as_str_unsupported() -> None:
+ unsupported_types = ["list", "map", "struct"]
+ for type_str in unsupported_types:
+ with pytest.raises(ValueError) as exc_info:
+ _ = NestedField(1, "field", type_str, required=True)
+ assert f"Unsupported field type: '{type_str}'" in str(exc_info.value)
+
+
+@pytest.mark.parametrize("type_str, type_class", primitive_types.items())
+def test_nested_field_type_as_str(type_str: str, type_class: type) -> None:
Review Comment:
nit:
```suggestion
def test_nested_field_primitive_type_as_str(type_str: str, type_class: type) -> None:
```
##########
tests/test_types.py:
##########
@@ -231,6 +246,27 @@ def test_nested_field() -> None:
assert "validation errors for NestedField" in str(exc_info.value)
+def test_nested_field_type_as_str_unsupported() -> None:
+ unsupported_types = ["list", "map", "struct"]
+ for type_str in unsupported_types:
+ with pytest.raises(ValueError) as exc_info:
+ _ = NestedField(1, "field", type_str, required=True)
+ assert f"Unsupported field type: '{type_str}'" in str(exc_info.value)
+
+
+@pytest.mark.parametrize("type_str, type_class", primitive_types.items())
+def test_nested_field_type_as_str(type_str: str, type_class: type) -> None:
+ field_var = NestedField(
+ 1,
+ "field",
+ type_str,
+ required=True,
+ )
+ assert isinstance(
+ field_var.field_type, type_class
+ ), f"Expected {type_class.__name__}, got {field_var.field_type.__class__.__name__}"
+
Review Comment:
nit can we add a test for this case too?
```
except ValueError as e:
raise ValueError(f"Unsupported field type: '{v}'") from e
```
##########
tests/test_types.py:
##########
@@ -231,6 +246,27 @@ def test_nested_field() -> None:
assert "validation errors for NestedField" in str(exc_info.value)
+def test_nested_field_type_as_str_unsupported() -> None:
Review Comment:
nit:
```suggestion
def test_nested_field_complex_type_as_str_unsupported() -> None:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Pass in the type as a string [iceberg-python]
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #1860:
URL: https://github.com/apache/iceberg-python/pull/1860#issuecomment-2764609478
@kevinjqliu Fixed all the comments. PTAL, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Pass data type as string representation to `NestedField` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1860:
URL: https://github.com/apache/iceberg-python/pull/1860#issuecomment-2764670827
Thanks for the contribution @sunxiaojian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Pass data type as string representation to `NestedField` [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #1860:
URL: https://github.com/apache/iceberg-python/pull/1860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org