You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/08/07 22:58:07 UTC
[PR] Add ManageSnapshots.cherry_pick_snapshot [iceberg-python]
1fanwang opened a new pull request, #3769:
URL: https://github.com/apache/iceberg-python/pull/3769
<!-- Relates to #737 -->
# Rationale for this change
pyiceberg can stage a write on a branch and validate it there, but cannot publish it. Write and Audit work; Publish does not, so the last step happens outside Python — typically a Spark session calling `system.cherrypick_snapshot`.
This adds `ManageSnapshots.cherry_pick_snapshot()`, mirroring Java's `CherryPickOperation`:
- An **append** is replayed as a new snapshot on top of current, carrying the data files it added and recording `source-snapshot-id`, plus `published-wap-id` when staged with a `wap.id`.
- Any **other operation** is fast-forwarded to when its parent is already current, and rejected otherwise with Java's message.
- Picking an existing ancestor is a no-op, and a `wap.id` may only be published once.
**Not covered:** dynamic overwrite (`OVERWRITE` with `replace-partitions`), which Java also replays. It raises here rather than silently doing nothing. Worth a follow-up.
# Prior art
[#750](https://github.com/apache/iceberg-python/pull/750) by @chinmay-bhat implemented this in 2024 and was closed by the stale bot with no review on merit; credit for getting there first is theirs. I rewrote rather than rebased, since `ManageSnapshots` has moved to `pyiceberg/table/update/snapshot.py` and the producer API changed underneath it. This one is narrower: a single method, no `publish_changes(wap_id)` and no dedicated producer class, since `fast_append()` already does the work.
@chinmay-bhat — glad to hand this back if you would rather carry it.
Touches the same class as [#3649](https://github.com/apache/iceberg-python/pull/3649). Happy to rebase onto whichever lands first and reuse its helper and exception types.
# Are these changes tested?
**Integration**, against the REST catalog and Hive metastore from `dev/docker-compose-integration.yml`. It stages a branch write with a `wap.id`, advances `main` so the pick is a replay not a fast-forward, publishes, then checks the summary properties and that a second publish is refused.
<details><summary>Red, with <code>pyiceberg/table/update/snapshot.py</code> at upstream/main</summary>
```
$ git checkout upstream/main -- pyiceberg/table/update/snapshot.py
$ pytest tests/integration/test_snapshot_operations.py -m integration -k cherry_pick -q
E AttributeError: 'ManageSnapshots' object has no attribute 'cherry_pick_snapshot'
tests/integration/test_snapshot_operations.py:366: AttributeError
FAILED ...::test_cherry_pick_snapshot_publishes_staged_write[session_catalog_hive]
FAILED ...::test_cherry_pick_snapshot_publishes_staged_write[session_catalog]
2 failed, 23 deselected
```
</details>
<details><summary>Green, with the change restored</summary>
```
$ pytest tests/integration/test_snapshot_operations.py -m integration -k cherry_pick -v
test_cherry_pick_snapshot_publishes_staged_write[session_catalog_hive] PASSED
test_cherry_pick_snapshot_publishes_staged_write[session_catalog] PASSED
```
The table goes from `[1, 9]` to `[1, 2, 3, 9]`, the published snapshot carries
`source-snapshot-id` and `published-wap-id=etl-001`, and re-publishing raises
`Duplicate request to cherry pick wap id that was published already: etl-001`.
</details>
Ran the docs example verbatim too: 1 row before publish, 3 after, wap id recorded.
**Unit**, across the `memory`, `sql`, and `sql_without_rowcount` catalogs: replay onto a moved-on `main`, wap id recorded, duplicate publish rejected, append replayed when its parent is current, non-append fast-forwarded, ancestor no-op, and both rejection paths. Same red on unpatched source.
Whole unit and integration suites pass; `prek run -a` clean.
# Are there any user-facing changes?
`ManageSnapshots.cherry_pick_snapshot(snapshot_id)`, plus a Write-Audit-Publish section in `mkdocs/docs/api.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add ManageSnapshots.cherry_pick_snapshot [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3769: Add ManageSnapshots.cherry_pick_snapshot
URL: https://github.com/apache/iceberg-python/pull/3769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add ManageSnapshots.cherry_pick_snapshot [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3769:
URL: https://github.com/apache/iceberg-python/pull/3769#issuecomment-5672830504
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add ManageSnapshots.cherry_pick_snapshot [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3769:
URL: https://github.com/apache/iceberg-python/pull/3769#issuecomment-5577224983
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add ManageSnapshots.cherry_pick_snapshot [iceberg-python]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #3769:
URL: https://github.com/apache/iceberg-python/pull/3769#discussion_r3739934581
##########
mkdocs/docs/api.md:
##########
@@ -1483,6 +1483,38 @@ Remove an existing branch:
table.manage_snapshots().remove_branch("dev").commit()
```
+### Write-Audit-Publish
+
+Stage a write on a branch, validate it, then publish it to the table with
+`cherry_pick_snapshot`. The staged data is invisible to readers of the table until it is
+published.
+
+```python
+# Write: stage the changes on an audit branch
+table.manage_snapshots().create_branch(
+ snapshot_id=table.metadata.current_snapshot_id,
+ branch_name="audit-2024-01-15",
+).commit()
+
+table = catalog.load_table("db.table")
+table.append(new_rows, branch="audit-2024-01-15", snapshot_properties={"wap.id": "etl-2024-01-15"})
+
+# Audit: validate the staged data without affecting readers of the table
+table = catalog.load_table("db.table")
+staged = table.metadata.refs["audit-2024-01-15"].snapshot_id
+assert len(table.scan(snapshot_id=staged).to_arrow()) > 0
+
+# Publish: replay the staged changes onto the current table state
+table.manage_snapshots().cherry_pick_snapshot(staged).commit()
+```
+
+The published snapshot records `source-snapshot-id`, and `published-wap-id` when the staged
+snapshot carried a `wap.id`. A given `wap.id` can only be published once.
+
+If the table has not changed since the branch was cut, the branch is fast-forwarded rather than
+replayed. Picking a snapshot that is already an ancestor of the current state does nothing. Only
+append snapshots can be replayed; anything else raises.
Review Comment:
Good catch, fixed in 8abb5b5. That paragraph described an earlier draft where a pick was fast-forwarded whenever the table had not moved. The implementation follows Java: appends are always replayed, so the wap trail is recorded either way, and fast-forward is the fallback for a non-append whose parent is current.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add ManageSnapshots.cherry_pick_snapshot [iceberg-python]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3769:
URL: https://github.com/apache/iceberg-python/pull/3769#discussion_r3739269753
##########
mkdocs/docs/api.md:
##########
@@ -1483,6 +1483,38 @@ Remove an existing branch:
table.manage_snapshots().remove_branch("dev").commit()
```
+### Write-Audit-Publish
+
+Stage a write on a branch, validate it, then publish it to the table with
+`cherry_pick_snapshot`. The staged data is invisible to readers of the table until it is
+published.
+
+```python
+# Write: stage the changes on an audit branch
+table.manage_snapshots().create_branch(
+ snapshot_id=table.metadata.current_snapshot_id,
+ branch_name="audit-2024-01-15",
+).commit()
+
+table = catalog.load_table("db.table")
+table.append(new_rows, branch="audit-2024-01-15", snapshot_properties={"wap.id": "etl-2024-01-15"})
+
+# Audit: validate the staged data without affecting readers of the table
+table = catalog.load_table("db.table")
+staged = table.metadata.refs["audit-2024-01-15"].snapshot_id
+assert len(table.scan(snapshot_id=staged).to_arrow()) > 0
+
+# Publish: replay the staged changes onto the current table state
+table.manage_snapshots().cherry_pick_snapshot(staged).commit()
+```
+
+The published snapshot records `source-snapshot-id`, and `published-wap-id` when the staged
+snapshot carried a `wap.id`. A given `wap.id` can only be published once.
+
+If the table has not changed since the branch was cut, the branch is fast-forwarded rather than
+replayed. Picking a snapshot that is already an ancestor of the current state does nothing. Only
+append snapshots can be replayed; anything else raises.
Review Comment:
The Write-Audit-Publish docs say an unchanged table is “fast-forwarded rather than replayed,” and that “anything else raises.” This contradicts the implementation/tests where append snapshots are always replayed (even when the picked snapshot’s parent is current) and non-append snapshots can be fast-forwarded when their parent is current. Please update this paragraph to match the actual semantics of `ManageSnapshots.cherry_pick_snapshot`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org