You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tusharchou (via GitHub)" <gi...@apache.org> on 2026/03/15 12:03:01 UTC
[PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
tusharchou opened a new pull request, #3149:
URL: https://github.com/apache/iceberg-python/pull/3149
## Rationale
While reviewing PR #3011 (manifest pruning optimization), I identified a correctness
gap when tables have undergone partition spec evolution.
When `dynamic_partition_overwrite` is called on a table with mixed `partition_spec_id`s
in its snapshot, the delete predicate was built using only the **current** partition spec.
This caused `inclusive_projection` to fail silently when evaluating older manifests —
the predicate contained field references (e.g. `region`) that have no corresponding
partition field in the old spec, causing the manifest evaluator to skip those manifests
entirely. The result is silent data duplication: stale rows from old spec manifests are
never deleted.
## Changes
- `pyiceberg/table/__init__.py`: `dynamic_partition_overwrite` now iterates over all
`partition_spec_id`s present in the current snapshot and builds a per-spec delete
predicate, projecting the new data files' partition values into each historical spec's
coordinate space before evaluating.
- `tests/integration/test_manifest_pruning_spec_evolution.py`: two regression tests added:
1. Mixed-spec snapshot — overwrite a partition present under both spec-0 and spec-1
2. Overwrite a partition that exists **only** in spec-0 manifests (the silent data
duplication case — no exception raised, wrong rows survive)
## Are these changes tested?
Yes — two new integration tests using the SQLite in-memory catalog, no external
services required.
## Are there any user-facing changes?
Yes — `dynamic_partition_overwrite` now correctly deletes all matching rows across
all historical partition specs, fixing silent data duplication on evolved tables.
## Related
- Fixes #3148
- Related to #3011 (manifest pruning optimization that exposed this gap)
- Related to #1108 (prior spec evolution fix in manifest rewriting by @Fokko)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#discussion_r3000066172
##########
pyiceberg/table/__init__.py:
##########
@@ -542,12 +542,69 @@ def dynamic_partition_overwrite(
)
)
- partitions_to_overwrite = {data_file.partition for data_file in data_files}
- delete_filter = self._build_partition_predicate(
- partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
- )
- self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+ # partitions_to_overwrite = {data_file.partition for data_file in data_files}
+ # delete_filter = self._build_partition_predicate(
+ # partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
+ # )
+ # self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+
+ # Build the partition predicate per-spec to handle tables that have
+ # undergone partition spec evolution. Manifests in the snapshot may be
+ # written under different (older) specs. We need to project the overwrite
+ # partitions into each historical spec's coordinate space so that the
+ # manifest evaluator correctly identifies which old manifests to delete.
+ current_spec = self.table_metadata.spec()
+ current_schema = self.table_metadata.schema()
+ # Collect the source column names (e.g. "category") that are being
+ # overwritten — these are stable across spec evolution (only field IDs matter).
+ overwrite_source_ids = {field.source_id for field in current_spec.fields}
+
+ delete_filter: BooleanExpression = AlwaysFalse()
+
+ # For each historical spec in the snapshot, build a predicate using
+ # only the fields that spec knows about, matched against the
+ # corresponding positions in the new data files' partition records.
+ snapshot = self.table_metadata.snapshot_by_name(branch or MAIN_BRANCH)
+ if snapshot is not None:
+ spec_ids_in_snapshot = {m.partition_spec_id for m in snapshot.manifests(io=self._table.io)}
Review Comment:
Also, iterating over the manifests is pretty expensive. Do we need to do that upfront?
##########
pyiceberg/table/__init__.py:
##########
@@ -542,12 +542,69 @@ def dynamic_partition_overwrite(
)
)
- partitions_to_overwrite = {data_file.partition for data_file in data_files}
- delete_filter = self._build_partition_predicate(
- partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
- )
- self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+ # partitions_to_overwrite = {data_file.partition for data_file in data_files}
+ # delete_filter = self._build_partition_predicate(
+ # partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
+ # )
+ # self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+
+ # Build the partition predicate per-spec to handle tables that have
+ # undergone partition spec evolution. Manifests in the snapshot may be
+ # written under different (older) specs. We need to project the overwrite
+ # partitions into each historical spec's coordinate space so that the
+ # manifest evaluator correctly identifies which old manifests to delete.
+ current_spec = self.table_metadata.spec()
+ current_schema = self.table_metadata.schema()
+ # Collect the source column names (e.g. "category") that are being
+ # overwritten — these are stable across spec evolution (only field IDs matter).
+ overwrite_source_ids = {field.source_id for field in current_spec.fields}
+
+ delete_filter: BooleanExpression = AlwaysFalse()
+
+ # For each historical spec in the snapshot, build a predicate using
+ # only the fields that spec knows about, matched against the
+ # corresponding positions in the new data files' partition records.
+ snapshot = self.table_metadata.snapshot_by_name(branch or MAIN_BRANCH)
+ if snapshot is not None:
+ spec_ids_in_snapshot = {m.partition_spec_id for m in snapshot.manifests(io=self._table.io)}
Review Comment:
I think this logic is a bit wonky, if the branch doesn't exists then we silently fall back to the main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#discussion_r3000068221
##########
pyiceberg/table/__init__.py:
##########
@@ -542,12 +542,69 @@ def dynamic_partition_overwrite(
)
)
- partitions_to_overwrite = {data_file.partition for data_file in data_files}
- delete_filter = self._build_partition_predicate(
- partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
- )
- self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+ # partitions_to_overwrite = {data_file.partition for data_file in data_files}
+ # delete_filter = self._build_partition_predicate(
+ # partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
+ # )
+ # self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+
+ # Build the partition predicate per-spec to handle tables that have
+ # undergone partition spec evolution. Manifests in the snapshot may be
+ # written under different (older) specs. We need to project the overwrite
+ # partitions into each historical spec's coordinate space so that the
+ # manifest evaluator correctly identifies which old manifests to delete.
+ current_spec = self.table_metadata.spec()
+ current_schema = self.table_metadata.schema()
+ # Collect the source column names (e.g. "category") that are being
+ # overwritten — these are stable across spec evolution (only field IDs matter).
+ overwrite_source_ids = {field.source_id for field in current_spec.fields}
+
+ delete_filter: BooleanExpression = AlwaysFalse()
+
+ # For each historical spec in the snapshot, build a predicate using
+ # only the fields that spec knows about, matched against the
+ # corresponding positions in the new data files' partition records.
+ snapshot = self.table_metadata.snapshot_by_name(branch or MAIN_BRANCH)
+ if snapshot is not None:
Review Comment:
I think this logic is a bit wonky, if the branch doesn't exists then we silently fall back to the main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#issuecomment-4323450561
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#discussion_r3000070565
##########
pyiceberg/table/__init__.py:
##########
@@ -542,12 +542,69 @@ def dynamic_partition_overwrite(
)
)
- partitions_to_overwrite = {data_file.partition for data_file in data_files}
- delete_filter = self._build_partition_predicate(
- partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
- )
- self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+ # partitions_to_overwrite = {data_file.partition for data_file in data_files}
+ # delete_filter = self._build_partition_predicate(
+ # partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
+ # )
+ # self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+
+ # Build the partition predicate per-spec to handle tables that have
+ # undergone partition spec evolution. Manifests in the snapshot may be
+ # written under different (older) specs. We need to project the overwrite
+ # partitions into each historical spec's coordinate space so that the
+ # manifest evaluator correctly identifies which old manifests to delete.
+ current_spec = self.table_metadata.spec()
+ current_schema = self.table_metadata.schema()
+ # Collect the source column names (e.g. "category") that are being
+ # overwritten — these are stable across spec evolution (only field IDs matter).
+ overwrite_source_ids = {field.source_id for field in current_spec.fields}
+
+ delete_filter: BooleanExpression = AlwaysFalse()
+
+ # For each historical spec in the snapshot, build a predicate using
+ # only the fields that spec knows about, matched against the
+ # corresponding positions in the new data files' partition records.
+ snapshot = self.table_metadata.snapshot_by_name(branch or MAIN_BRANCH)
+ if snapshot is not None:
+ spec_ids_in_snapshot = {m.partition_spec_id for m in snapshot.manifests(io=self._table.io)}
Review Comment:
Do we need to collect these upfront? This is pretty expensive as it will do IO to pull all the manifests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#discussion_r3000066172
##########
pyiceberg/table/__init__.py:
##########
@@ -542,12 +542,69 @@ def dynamic_partition_overwrite(
)
)
- partitions_to_overwrite = {data_file.partition for data_file in data_files}
- delete_filter = self._build_partition_predicate(
- partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
- )
- self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+ # partitions_to_overwrite = {data_file.partition for data_file in data_files}
+ # delete_filter = self._build_partition_predicate(
+ # partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
+ # )
+ # self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+
+ # Build the partition predicate per-spec to handle tables that have
+ # undergone partition spec evolution. Manifests in the snapshot may be
+ # written under different (older) specs. We need to project the overwrite
+ # partitions into each historical spec's coordinate space so that the
+ # manifest evaluator correctly identifies which old manifests to delete.
+ current_spec = self.table_metadata.spec()
+ current_schema = self.table_metadata.schema()
+ # Collect the source column names (e.g. "category") that are being
+ # overwritten — these are stable across spec evolution (only field IDs matter).
+ overwrite_source_ids = {field.source_id for field in current_spec.fields}
+
+ delete_filter: BooleanExpression = AlwaysFalse()
+
+ # For each historical spec in the snapshot, build a predicate using
+ # only the fields that spec knows about, matched against the
+ # corresponding positions in the new data files' partition records.
+ snapshot = self.table_metadata.snapshot_by_name(branch or MAIN_BRANCH)
+ if snapshot is not None:
+ spec_ids_in_snapshot = {m.partition_spec_id for m in snapshot.manifests(io=self._table.io)}
Review Comment:
Also, iterating over the manifests is pretty expensive. Do we need to do that upfront?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3149: fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution
URL: https://github.com/apache/iceberg-python/pull/3149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "tusharchou (via GitHub)" <gi...@apache.org>.
tusharchou commented on PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#issuecomment-4063360610
AI Disclosure
AI was used to help understand the code base and draft code changes. All code changes have been thoroughly reviewed, ensuring that the code changes are in line with a broader understanding of the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#discussion_r3000061428
##########
pyiceberg/table/__init__.py:
##########
@@ -542,12 +542,69 @@ def dynamic_partition_overwrite(
)
)
- partitions_to_overwrite = {data_file.partition for data_file in data_files}
- delete_filter = self._build_partition_predicate(
- partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
- )
- self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+ # partitions_to_overwrite = {data_file.partition for data_file in data_files}
+ # delete_filter = self._build_partition_predicate(
+ # partition_records=partitions_to_overwrite, spec=self.table_metadata.spec(), schema=self.table_metadata.schema()
+ # )
+ # self.delete(delete_filter=delete_filter, snapshot_properties=snapshot_properties, branch=branch)
+
+ # Build the partition predicate per-spec to handle tables that have
+ # undergone partition spec evolution. Manifests in the snapshot may be
+ # written under different (older) specs. We need to project the overwrite
+ # partitions into each historical spec's coordinate space so that the
+ # manifest evaluator correctly identifies which old manifests to delete.
+ current_spec = self.table_metadata.spec()
+ current_schema = self.table_metadata.schema()
+ # Collect the source column names (e.g. "category") that are being
+ # overwritten — these are stable across spec evolution (only field IDs matter).
+ overwrite_source_ids = {field.source_id for field in current_spec.fields}
+
+ delete_filter: BooleanExpression = AlwaysFalse()
+
+ # For each historical spec in the snapshot, build a predicate using
+ # only the fields that spec knows about, matched against the
+ # corresponding positions in the new data files' partition records.
+ snapshot = self.table_metadata.snapshot_by_name(branch or MAIN_BRANCH)
+ if snapshot is not None:
+ spec_ids_in_snapshot = {m.partition_spec_id for m in snapshot.manifests(io=self._table.io)}
Review Comment:
I think this logic is a bit wonky, if the branch doesn't exists then we silently fall back to the main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#discussion_r3000082905
##########
tests/integration/test_manifest_pruning_spec_evolution.py:
##########
@@ -0,0 +1,283 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Regression / investigation test for manifest pruning correctness under partition spec evolution.
+
+Context
+-------
+PR #3011 (merged Feb 20 2026) added manifest pruning to _OverwriteFiles and _DeleteFiles
+in pyiceberg/table/update/snapshot.py. The pruning builds a partition predicate from the
+*current* partition spec and evaluates it against every manifest in the snapshot via a
+KeyDefaultDict of per-spec evaluators.
+
+The question this test file investigates:
+ When a table has been through partition spec evolution, its snapshot may contain manifests
+ written under *different* partition_spec_ids. Does the manifest evaluator correctly resolve
+ each manifest's own spec before deciding whether to include or skip it?
+
+If the answer is "no", the overwrite will silently skip manifests from the old spec, leaving
+stale data files that should have been deleted -- a silent correctness bug.
+
+How to run
+----------
+ pytest tests/integration/test_manifest_pruning_spec_evolution.py -v
+"""
+
+import tempfile
+from typing import Any
+
+import pyarrow as pa
+
+from pyiceberg.catalog import Catalog, load_catalog
+from pyiceberg.partitioning import PartitionField, PartitionSpec
+from pyiceberg.schema import Schema
+from pyiceberg.transforms import IdentityTransform
+from pyiceberg.types import LongType, NestedField, StringType
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+SCHEMA = Schema(
+ NestedField(field_id=1, name="category", field_type=StringType(), required=False),
+ NestedField(field_id=2, name="region", field_type=StringType(), required=False),
+ NestedField(field_id=3, name="value", field_type=LongType(), required=False),
+)
+
+# Spec 0: partitioned only by category
+SPEC_V0 = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="category"))
+
+
+def make_catalog(warehouse: str) -> Catalog:
+ """Spin up a local SQLite-backed catalog -- no services needed."""
+ return load_catalog(
+ "test",
+ type="sql",
+ uri=f"sqlite:///{warehouse}/catalog.db",
+ warehouse=f"file://{warehouse}",
+ )
+
+
+def arrow_table(rows: list[dict[str, Any]]) -> pa.Table:
+ return pa.Table.from_pylist(
+ rows,
+ schema=pa.schema(
+ [
+ pa.field("category", pa.string()),
+ pa.field("region", pa.string()),
+ pa.field("value", pa.int64()),
+ ]
+ ),
+ )
+
+
+# ---------------------------------------------------------------------------
+# Test 1: Mixed spec snapshot -- overwrite partition present in both specs
+# ---------------------------------------------------------------------------
+
+
+def test_overwrite_after_partition_spec_evolution_correctness() -> None:
+ """
+ Verifies that dynamic_partition_overwrite correctly replaces ALL data files
+ for the target partition, including those written under a previous partition spec.
+
+ Setup:
+ - Spec 0: partition by identity(category)
+ - Write A(1,2,3) and B(10,11) under spec 0
+ - Evolve to spec 1: add identity(region)
+ - Write A(100,101) and B(200) under spec 1
+ - Overwrite category=A with new rows (999, 888)
+
+ Expected after overwrite:
+ - Only new A rows: values [888, 999]
+ - All B rows untouched: values [10, 11, 200]
+ - Total: 5 rows
+
+ Bug (pre-fix): spec-0 A manifests are skipped by the evaluator,
+ leaving stale A rows (1, 2, 3) in the table -> 8 rows total.
Review Comment:
Hmm. I'm not sure if we can make this change. Existing users might be used to the existing behavior. When we change this, it will drop all many more rows than before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: dynamic_partition_overwrite builds per-spec delete predicates after partition spec evolution [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3149:
URL: https://github.com/apache/iceberg-python/pull/3149#issuecomment-4367598549
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org