You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "syun64 (via GitHub)" <gi...@apache.org> on 2024/05/14 16:40:47 UTC
[I] Support Snapshot Management Operations: Cherry Pick Operation [iceberg-python]
syun64 opened a new issue, #737:
URL: https://github.com/apache/iceberg-python/issues/737
### Feature Request / Improvement
Following is a list of operations that are supported in Spark:
- rollback_to_snapshot (set_ref_snapshot)
- rollback_to_timestamp (set_ref_snapshot)
- set_current_snapshot (set_ref_snapshot)
- cherrypick_snapshot
- publish_changes
- fast_forward
[set_ref_snapshot](https://github.com/apache/iceberg-python/pull/728) support will be introduced, but it would also be nice to add snapshot migration operations that generate new snapshots from existing ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-2518821922
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations: Cherry Pick Operation [iceberg-python]
Posted by "chinmay-bhat (via GitHub)" <gi...@apache.org>.
chinmay-bhat commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-2113081895
@syun64 I have a few questions about the operations and I couldn't find more info in the docs. Apologies if these have been answered elsewhere.
- How does rollback_to_timestamp use `set_ref_snapshot()`? In the [rollback_to_timestamp documentation](https://iceberg.apache.org/docs/1.5.1/spark-procedures/#rollback_to_timestamp), inputs are Table and timestamp, and I can't find a `snapshot_by_timestamp()` api to get the snapshot_id.
- for cherrypick_snapshot and publish_changes, wouldn't we need an `add_snapshot()` table api ? I noticed both `add_snapshot()` and `set_ref_snapshot()` were removed in the [same PR](https://github.com/apache/iceberg-python/pull/471/files#diff-23e8153e0fd497a9212215bd2067068f3b56fa071770c7ef326db3d3d03cee9bL328). Do we bring back `add_snapshot()` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-2123759077
Hi @syun64 @chinmay-bhat. Thank you for driving these features. I saw some PRs raised by @chinmay-bhat : #748, #750, #758. Those are great!
I would like to discuss what APIs we want to expose to users. I am actually leaning toward removing `set_ref_snapshot` from public API. The reason is that `set_ref_snapshot` itself requires too many details from users and thus error-prone. It shall sit behind APIs like `rollback_to_snapshot`, `create_tag`, `cherryPick` which have more specific functionality and simpler to use.
We may also want to group these APIs into a class (e.g. `ManageSnapshots`), like the [java implementation](https://github.com/apache/iceberg/blob/70b7aa534b2c79ccd7b6c0e0fd1be980772bb20a/api/src/main/java/org/apache/iceberg/ManageSnapshots.java#L44-L57). So the resulting user experience with these APIs will be
```python
with table.transaction() as txn:
... other updates
with txn.manage_snapshots() as ms:
ms.createTag("Tag_A", 0)
ms.createTag("Tag_B", 1)
```
We could also keep `table.createTag` for convenience. Similar to the usage of [UpdateSchema](https://py.iceberg.apache.org/api/#schema-evolution).
What do you think about this? Appreciate any thoughts and suggestions!
Also kindly looping @Fokko to this thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-2113105812
> * How does rollback_to_timestamp use `set_ref_snapshot()`? In the [rollback_to_timestamp documentation](https://iceberg.apache.org/docs/1.5.1/spark-procedures/#rollback_to_timestamp), inputs are Table and timestamp, and I can't find a `snapshot_by_timestamp()` api to get the snapshot_id.
- Yeah I think it would be helpful to introduce a `snapshot_by_timestamp` utility function to get the snapshot - just like you mentioned, that would help recover feature parity with the existing Java API
> * for cherrypick_snapshot and publish_changes ([docs](https://iceberg.apache.org/docs/1.5.1/spark-procedures/#cherrypick_snapshot)), wouldn't we need an `add_snapshot()` table api ? I noticed both `add_snapshot()` and `set_ref_snapshot()` were removed in the [same PR](https://github.com/apache/iceberg-python/pull/471/files#diff-23e8153e0fd497a9212215bd2067068f3b56fa071770c7ef326db3d3d03cee9bL328). Do we bring back `add_snapshot()` as well?
I don't think `add_snapshot` needs to be an API because the function is incredibly simple, in that it just adds a AddSnapshotUpdate table update and AssertTableUUID requirement. I think instead, functions like cherrypick_snapshot, publish_changes should be separate APIs that builds the new snapshot and then makes a commit with the updated snapshot. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "chinmay-bhat (via GitHub)" <gi...@apache.org>.
chinmay-bhat commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-2113132702
> Yeah I think it would be helpful to introduce a snapshot_by_timestamp utility function to get the snapshot - just like you mentioned, that would help recover feature parity with the existing Java API
Yeah, looking at the [Java api](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SetSnapshotOperation.java#L68), something like `findLatestAncestorOlderThan` makes sense.
> WDYT?
Sounds good to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "chinmay-bhat (via GitHub)" <gi...@apache.org>.
chinmay-bhat commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-2124351316
@Honah thank you for your response.
I agree that we should hide the `set_ref_snapshot` from the public API. I also like the idea of creating a `ManageSnapshots` inner class in `Transaction` to organise the APIs, while exposing them through `Transaction` and `Table`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #737: Support Snapshot Management Operations
URL: https://github.com/apache/iceberg-python/issues/737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-2489807554
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Snapshot Management Operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #737:
URL: https://github.com/apache/iceberg-python/issues/737#issuecomment-4550277140
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org