You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "010Soham (via GitHub)" <gi...@apache.org> on 2025/12/19 09:59:41 UTC
[PR] Core: Respect partition evolution in inspect.partitions [iceberg-python]
010Soham opened a new pull request, #2845:
URL: https://github.com/apache/iceberg-python/pull/2845
## What does this change do?
`table.inspect.partitions()` now builds its partition schema using only the partition specs
that appear in the selected snapshot’s manifests. This prevents new partition fields
from showing up as `None` for data written with older specs.
## Why is this needed?
When partition specs evolve, inspecting old snapshots should show the old partition
shape only. The previous behavior mixed all specs together, which produced misleading
`None` fields.
## How was this tested?
- make lint
- uv run python -m pytest tests/io/test_pyarrow.py -k "partition_evolution" -v
- make test (timed out at ~42%)
Closes #1120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Respect partition evolution in inspect.partitions [iceberg-python]
Posted by "010Soham (via GitHub)" <gi...@apache.org>.
010Soham commented on PR #2845:
URL: https://github.com/apache/iceberg-python/pull/2845#issuecomment-3723827386
Glad it was useful @kevinjqliu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Respect partition evolution in inspect.partitions [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2845:
URL: https://github.com/apache/iceberg-python/pull/2845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Respect partition evolution in inspect.partitions [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2845:
URL: https://github.com/apache/iceberg-python/pull/2845#discussion_r2646162357
##########
tests/io/test_pyarrow.py:
##########
@@ -2588,6 +2588,34 @@ def test_inspect_partition_for_nested_field(catalog: InMemoryCatalog) -> None:
assert {part["part"] for part in partitions} == {"data-a", "data-b"}
+def test_inspect_partitions_respects_partition_evolution(catalog: InMemoryCatalog) -> None:
+ schema = Schema(
+ NestedField(id=1, name="dt", field_type=DateType(), required=False),
+ NestedField(id=2, name="category", field_type=StringType(), required=False),
+ )
+ spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="dt"))
+ catalog.create_namespace("default")
+ table = catalog.create_table("default.test_partition_evolution_inspect", schema=schema, partition_spec=spec)
Review Comment:
```suggestion
table = catalog.create_table("default.test_inspect_partitions_respects_partition_evolution", schema=schema, partition_spec=spec)
```
in case theres conflict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Respect partition evolution in inspect.partitions [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2845:
URL: https://github.com/apache/iceberg-python/pull/2845#issuecomment-3690429160
Thanks for the PR @010Soham
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org