You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "NoahKusaba (via GitHub)" <gi...@apache.org> on 2026/10/07 20:54:45 UTC

[PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

NoahKusaba opened a new pull request, #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372

   ## Which issue does this PR close?
   
   - Related to #3134 (covers its idempotent-write case without making `TransactionAction` public).
   
   ## What changes are included in this PR?
   
   An engine that reruns a task can commit the same append twice. Checking for an earlier commit before `Transaction::commit` costs an extra table load and still races, because `do_commit` reloads the table and rebuilds the actions.
   
   - `FastAppendAction::with_idempotency_key(key)` records `key` in the snapshot summary (`IDEMPOTENCY_KEY_SUMMARY_PROPERTY`). On every commit attempt it walks `main`'s ancestry in the refreshed table and skips the append if a snapshot already has the key. A racing loser fails its ref requirement, retries, finds the key and skips. A rerun appending the same files is skipped rather than rejected as a duplicate. Flink's committer uses the same pattern with `flink.job-id`.
   - `snapshot_with_idempotency_key(&table, key)` returns the snapshot on `main` holding a key. A caller that records a value unique to its commit call in the snapshot properties can compare it to tell whether its append was committed or skipped.
   - `do_commit` no longer calls `update_table` when no action produced updates or requirements, as in Java. Before, such a commit still wrote a new metadata file (for example, `expire_snapshots` with nothing to expire).
   
   An append with no files and a key commits a snapshot recording the key, so its rerun is skipped too. Keys on expired or rolled-back snapshots are not found.
   
   ## Are these changes tested?
   
   Yes, unit tests in `transaction/append.rs`:
   - reruns from a stale table, with the same files, and with the key below later snapshots
   - an append without files
   - a race, replayed through a mock catalog in sequence (load, rejected update, load)
   - the lookup helper
   
   And in `transaction/mod.rs`, a commit without updates writing no metadata file.
   
   ## AI Disclosure
   
   Drafted with Claude Code; I reviewed it and ran the tests, clippy and fmt.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#issuecomment-6074214574

   @comphead thanks for the review, all addressed in db9d1a5f6:
   - Dropped `test_rerun_finds_its_key_below_later_snapshots`. `test_append_without_files_records_its_key` is replaced (see the zero-file thread), so the full summary-map assertion is gone too.
   - Added `test_skipped_append_keeps_the_other_updates`: a property update in the same transaction still commits when the append is skipped.
   - A skip now logs `tracing::info!` with the key and the snapshot ID that holds it.
   - Renamed `snapshot_properties()` to `summary_properties()`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#discussion_r4212115645


##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -66,6 +93,35 @@ impl FastAppendAction {
         self
     }
 
+    /// Make this append idempotent under `key`.
+    ///
+    /// The key is recorded in the new snapshot's summary under
+    /// [`IDEMPOTENCY_KEY_SUMMARY_PROPERTY`]. On every commit attempt, including
+    /// retries against a refreshed table, the action first walks the ancestry of
+    /// the `main` branch; if a snapshot already carries `key`, the append
+    /// produces no updates and the transaction commits nothing for it. The
+    /// check comes before the added files are validated, so a rerun appending
+    /// the files the first run committed is skipped, not rejected as a
+    /// duplicate.
+    ///
+    /// This lets a caller safely rerun a commit that may already have

Review Comment:
   Please state two consequences for callers here. Attempts under one key must append equivalent data, because a rerun with different files is skipped without any signal (`test_rerun_from_a_stale_table_commits_nothing` relies on this). And the files of a skipped attempt are not referenced by the table, but they can be the very files the winning attempt committed when the paths match, so cleanup after a skip needs care.



##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -66,6 +93,35 @@ impl FastAppendAction {
         self
     }
 
+    /// Make this append idempotent under `key`.
+    ///
+    /// The key is recorded in the new snapshot's summary under
+    /// [`IDEMPOTENCY_KEY_SUMMARY_PROPERTY`]. On every commit attempt, including
+    /// retries against a refreshed table, the action first walks the ancestry of
+    /// the `main` branch; if a snapshot already carries `key`, the append
+    /// produces no updates and the transaction commits nothing for it. The
+    /// check comes before the added files are validated, so a rerun appending
+    /// the files the first run committed is skipped, not rejected as a
+    /// duplicate.
+    ///
+    /// This lets a caller safely rerun a commit that may already have
+    /// succeeded (for example a task retried by a distributed engine). If two
+    /// attempts race, the loser fails its `main` ref requirement, retries
+    /// against the refreshed table, sees the winner's key and skips. To learn
+    /// whether this append was the one committed, see
+    /// [`snapshot_with_idempotency_key`].
+    ///
+    /// An append with no data files still commits a snapshot recording `key`,

Review Comment:
   A zero-file append only works today because `SnapshotProducer::produce_manifests` accepts a properties-only append (`snapshot.rs:345`). That branch is marked as a workaround to clean up (#1548). If the cleanup removes it, an append with a key and no files would likely start failing with `PreconditionFailed`, which contradicts this paragraph.
   
   Is the empty snapshot needed, for example so a caller can use the key as a completion marker? A retried zero-file job adds no data, so there is nothing to protect from duplication. If the marker is wanted, please note the dependency in the TODO in `snapshot.rs`. If not, skipping a zero-file append would avoid an empty snapshot per job and let this paragraph and `test_append_without_files_records_its_key` go.



##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -22,18 +22,44 @@ use async_trait::async_trait;
 use uuid::Uuid;
 
 use crate::error::Result;
-use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation};
+use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation, SnapshotRef};
 use crate::table::Table;
 use crate::transaction::snapshot::{
     DefaultManifestProcess, SnapshotProduceOperation, SnapshotProducer,
 };
 use crate::transaction::{ActionCommit, TransactionAction};
+use crate::util::snapshot::ancestors_of;
+
+/// Snapshot summary property that records the idempotency key of a fast append.
+/// See [`FastAppendAction::with_idempotency_key`].
+pub const IDEMPOTENCY_KEY_SUMMARY_PROPERTY: &str = "iceberg-rust.idempotency-key";
+
+/// Returns the snapshot on the `main` branch of `table` whose summary records
+/// idempotency key `key`, if any.
+///
+/// After committing a [`FastAppendAction`] with
+/// [`FastAppendAction::with_idempotency_key`], this finds the snapshot holding
+/// the key, whether this commit or an earlier one created it. To tell the two
+/// apart, record a value unique to the commit call with
+/// [`FastAppendAction::set_snapshot_properties`] and compare it with the
+/// returned snapshot's summary.
+pub fn snapshot_with_idempotency_key(table: &Table, key: &str) -> Option<SnapshotRef> {

Review Comment:
   Does a downstream user already need `snapshot_with_idempotency_key` and `IDEMPOTENCY_KEY_SUMMARY_PROPERTY` to be public? By its own docs the helper cannot tell a caller whether its append won. The caller has to add a second unique property and compare. If nothing needs them yet, `pub(crate)` keeps the new API to `with_idempotency_key`, and widening later is not a breaking change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba closed pull request #3372: feat(transaction): add idempotency key to fast append
URL: https://github.com/apache/iceberg-rust/pull/3372


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on code in PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#discussion_r4226581503


##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -22,18 +22,44 @@ use async_trait::async_trait;
 use uuid::Uuid;
 
 use crate::error::Result;
-use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation};
+use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation, SnapshotRef};
 use crate::table::Table;
 use crate::transaction::snapshot::{
     DefaultManifestProcess, SnapshotProduceOperation, SnapshotProducer,
 };
 use crate::transaction::{ActionCommit, TransactionAction};
+use crate::util::snapshot::ancestors_of;
+
+/// Snapshot summary property that records the idempotency key of a fast append.
+/// See [`FastAppendAction::with_idempotency_key`].
+pub const IDEMPOTENCY_KEY_SUMMARY_PROPERTY: &str = "iceberg-rust.idempotency-key";
+
+/// Returns the snapshot on the `main` branch of `table` whose summary records
+/// idempotency key `key`, if any.
+///
+/// After committing a [`FastAppendAction`] with
+/// [`FastAppendAction::with_idempotency_key`], this finds the snapshot holding
+/// the key, whether this commit or an earlier one created it. To tell the two
+/// apart, record a value unique to the commit call with
+/// [`FastAppendAction::set_snapshot_properties`] and compare it with the
+/// returned snapshot's summary.
+pub fn snapshot_with_idempotency_key(table: &Table, key: &str) -> Option<SnapshotRef> {

Review Comment:
   Agreed, made `snapshot_with_idempotency_key` private; only `commit` uses it. I kept `IDEMPOTENCY_KEY_SUMMARY_PROPERTY` public because it names a property stored in table metadata, so anyone reading snapshot summaries needs the name.
   



##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -66,6 +93,35 @@ impl FastAppendAction {
         self
     }
 
+    /// Make this append idempotent under `key`.
+    ///
+    /// The key is recorded in the new snapshot's summary under
+    /// [`IDEMPOTENCY_KEY_SUMMARY_PROPERTY`]. On every commit attempt, including
+    /// retries against a refreshed table, the action first walks the ancestry of
+    /// the `main` branch; if a snapshot already carries `key`, the append
+    /// produces no updates and the transaction commits nothing for it. The
+    /// check comes before the added files are validated, so a rerun appending
+    /// the files the first run committed is skipped, not rejected as a
+    /// duplicate.
+    ///
+    /// This lets a caller safely rerun a commit that may already have

Review Comment:
   Done. The doc now says to use one key per logical write, since an append whose key is already committed is dropped whatever its files and the commit still returns `Ok`. It also says that after a skip, only files the table does not reference should be deleted, because a rerun's files can be the committed ones.
   



##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -66,6 +93,35 @@ impl FastAppendAction {
         self
     }
 
+    /// Make this append idempotent under `key`.
+    ///
+    /// The key is recorded in the new snapshot's summary under
+    /// [`IDEMPOTENCY_KEY_SUMMARY_PROPERTY`]. On every commit attempt, including
+    /// retries against a refreshed table, the action first walks the ancestry of
+    /// the `main` branch; if a snapshot already carries `key`, the append
+    /// produces no updates and the transaction commits nothing for it. The
+    /// check comes before the added files are validated, so a rerun appending
+    /// the files the first run committed is skipped, not rejected as a
+    /// duplicate.
+    ///
+    /// This lets a caller safely rerun a commit that may already have
+    /// succeeded (for example a task retried by a distributed engine). If two
+    /// attempts race, the loser fails its `main` ref requirement, retries
+    /// against the refreshed table, sees the winner's key and skips. To learn
+    /// whether this append was the one committed, see
+    /// [`snapshot_with_idempotency_key`].
+    ///
+    /// An append with no data files still commits a snapshot recording `key`,

Review Comment:
   Agreed, the empty snapshot isn't needed. A keyed append with no data files and no snapshot properties now commits nothing and returns `Ok`, so nothing new depends on the #1548 workaround. With caller-set snapshot properties it still commits as before, so the key never drops them. The paragraph and `test_append_without_files_records_its_key` are replaced by `test_append_without_files_commits_nothing` and `test_append_without_files_but_with_properties_commits`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#issuecomment-6099852709

   I meant to cc @alexanderbianchi, as this may be relevant to datafusion-distributed, but I'm not sure how it handles partial executor commits. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "alexanderbianchi (via GitHub)" <gi...@apache.org>.
alexanderbianchi commented on PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#issuecomment-6113527621

   Today datafusion-distributed hasn't put any effort into writes. It is something we'd like to get too, but the current priority is the read path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#issuecomment-6046693711

   cc @mbutrovich  @comphead  @gabotechs 
   
   This is a major blocker for correct inserts into iceberg for distributed engines. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on code in PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#discussion_r4226581812


##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -66,6 +93,35 @@ impl FastAppendAction {
         self
     }
 
+    /// Make this append idempotent under `key`.
+    ///
+    /// The key is recorded in the new snapshot's summary under
+    /// [`IDEMPOTENCY_KEY_SUMMARY_PROPERTY`]. On every commit attempt, including
+    /// retries against a refreshed table, the action first walks the ancestry of
+    /// the `main` branch; if a snapshot already carries `key`, the append
+    /// produces no updates and the transaction commits nothing for it. The
+    /// check comes before the added files are validated, so a rerun appending
+    /// the files the first run committed is skipped, not rejected as a
+    /// duplicate.
+    ///
+    /// This lets a caller safely rerun a commit that may already have
+    /// succeeded (for example a task retried by a distributed engine). If two
+    /// attempts race, the loser fails its `main` ref requirement, retries
+    /// against the refreshed table, sees the winner's key and skips. To learn
+    /// whether this append was the one committed, see
+    /// [`snapshot_with_idempotency_key`].
+    ///
+    /// An append with no data files still commits a snapshot recording `key`,

Review Comment:
   I originally kept it in as AI pointed out consistency with Flink's write design, but I'm overall indifferent and you're right the empty snapshot isn't needed. A keyed append with no data files and no snapshot properties now commits nothing and returns `Ok`, so nothing new depends on the #1548 workaround. With caller-set snapshot properties it still commits as before, so the key never drops them. The paragraph and `test_append_without_files_records_its_key` are replaced by `test_append_without_files_commits_nothing` and `test_append_without_files_but_with_properties_commits`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add idempotency key to fast append [iceberg-rust]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #3372:
URL: https://github.com/apache/iceberg-rust/pull/3372#issuecomment-6255819905

   Actually this isn't the write solution. Closing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org