You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aiborodin (via GitHub)" <gi...@apache.org> on 2025/11/06 07:01:44 UTC
[PR] Implement Snapshot validation API [iceberg]
aiborodin opened a new pull request, #14514:
URL: https://github.com/apache/iceberg/pull/14514
This change implements the ability to add custom `Snapshot` validations to the existing `SnapshotUpdate` API.
**Why?**
Custom `Snapshot` validation is necessary for non-idempotent table update operations, which rely on the existing state for correctness and exactly-once delivery. Applications like Flink and Kafka Connect use snapshot properties to store their idempotence keys and identify their base state during recovery.
Due to the nature of concurrent commits in Iceberg, the commit's parent snapshots may change, even during the ongoing commit operation. For example, a duplicate commit may happen after the currently staged commit request refreshes its snapshot history, see [SnapshotProducer::refresh()](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L254)). This creates a problem for the upstream client code, as it currently can't check information of the new base snapshots to identify idempotence violations - concurrently committed Kafka offsets in Kafka connect or duplicate `max-committed-checkpoint-id` in Flink.
This change addresses this problem and allows clients to implement custom idempotence validations.
This change is the parent of the following PRs:
The Kafka connect issue fix:
The Flink duplication fix:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "aiborodin (via GitHub)" <gi...@apache.org>.
aiborodin commented on PR #14514:
URL: https://github.com/apache/iceberg/pull/14514#issuecomment-3499722627
@rdblue What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #14514:
URL: https://github.com/apache/iceberg/pull/14514#issuecomment-3503713278
I think that the starting snapshot ID is often based on what an operation read. This validation is likely going to be based on the current version when the transaction started. Since those are different, I would not reuse starting snapshot ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14514: Implement Snapshot validation API for commits
URL: https://github.com/apache/iceberg/pull/14514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "aiborodin (via GitHub)" <gi...@apache.org>.
aiborodin commented on PR #14514:
URL: https://github.com/apache/iceberg/pull/14514#issuecomment-3500636027
> I think that difference could lead to needing two separate values when combining history validation with other validations that use startingSnapshotId.
Wouldn't both the history validation and other validations want to check the same set of parent snapshots?
I can't think of any use case where someone would want to validate different ranges of new base snapshots, and it feels like both use cases should have the same entry point for setting the starting snapshot.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #14514:
URL: https://github.com/apache/iceberg/pull/14514#issuecomment-3499972703
I just thought of another issue with this approach. This use of `startingSnapshotId` conflicts with other uses. Here, it determines the starting point of validation, which is where the table state was when the transaction started (because KC determined what to commit and the current offset from the current snapshot). In other places, it is used for the table version that was actually read -- which may not be current when the operation starts. I think that difference could lead to needing two separate values when combining history validation with other validations that use `startingSnapshotId`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14514:
URL: https://github.com/apache/iceberg/pull/14514#issuecomment-3672841623
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "aiborodin (via GitHub)" <gi...@apache.org>.
aiborodin commented on PR #14514:
URL: https://github.com/apache/iceberg/pull/14514#issuecomment-3509429357
> This validation is likely going to be based on the current version when the transaction started.
But would it be reasonable for the operation that has read a different version (not the current one) to run the custom validation from that read version? Why would it want to run the custom validation from the current instead of the read version if it based all its update assumptions on that read version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Implement Snapshot validation API for commits [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14514:
URL: https://github.com/apache/iceberg/pull/14514#issuecomment-3639474025
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org