You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "CTTY (via GitHub)" <gi...@apache.org> on 2025/07/01 18:43:19 UTC

[PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

CTTY opened a new pull request, #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes #1387 
   
   ## What changes are included in this PR?
   
   <!--
   Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
   -->
   - Added retry logic using `backon` 
   
   ## Are these changes tested?
   TODO
   <!--
   Specify what test covers (unit test, integration test, etc.).
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#discussion_r2203855133


##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -152,13 +159,59 @@ impl Transaction {
     }
 
     /// Commit transaction.
-    pub async fn commit(mut self, catalog: &dyn Catalog) -> Result<Table> {
+    pub async fn commit(self, catalog: &dyn Catalog) -> Result<Table> {
         if self.actions.is_empty() {
             // nothing to commit
             return Ok(self.table);
         }
 
-        self.do_commit(catalog).await
+        let backoff = Self::build_backoff(self.table.metadata().properties());
+        let tx = self;
+
+        (|mut tx: Transaction| async {
+            let result = tx.do_commit(catalog).await;
+            (tx, result)
+        })
+        .retry(backoff)
+        .sleep(tokio::time::sleep)
+        .context(tx)
+        .when(|e| e.retryable())
+        .await
+        .1
+    }
+
+    fn build_backoff(props: &HashMap<String, String>) -> ExponentialBackoff {
+        ExponentialBuilder::new()
+            .with_min_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MIN_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MIN_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.min-wait-ms"),
+            ))
+            .with_max_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MAX_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MAX_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.max-wait-ms"),
+            ))
+            .with_total_delay(Some(Duration::from_millis(
+                props
+                    .get(COMMIT_TOTAL_RETRY_TIME_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.total-timeout-ms"),
+            )))
+            .with_max_times(
+                props
+                    .get(COMMIT_NUM_RETRIES)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_NUM_RETRIES_DEFAULT))
+                    .expect("Invalid value for commit.retry.num-retries"),
+            )
+            .with_factor(2.0)
+            .build()

Review Comment:
   I've created the issue to track the task of centralizing properties: https://github.com/apache/iceberg-rust/issues/1505



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#issuecomment-3025236950

   > Hi @dentiny , this is a good question:) I'm thinking of using a mock catalog, but haven't figured out exactly how
   
   I use [`mockall`](https://docs.rs/mockall/latest/mockall/) on traits.
   Just for reference:)
   - https://github.com/Mooncake-Labs/moonlink/blob/a20e83d95a2685e033a9b139884695be33ba7665/src/moonlink/src/storage/iceberg/table_manager.rs#L37
   - https://github.com/Mooncake-Labs/moonlink/blob/a20e83d95a2685e033a9b139884695be33ba7665/src/moonlink/src/table_handler/tests.rs#L1111-L1134


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#discussion_r2192250914


##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -152,13 +163,59 @@ impl Transaction {
     }
 
     /// Commit transaction.
-    pub async fn commit(mut self, catalog: &dyn Catalog) -> Result<Table> {
+    pub async fn commit(self, catalog: &dyn Catalog) -> Result<Table> {
         if self.actions.is_empty() {
             // nothing to commit
             return Ok(self.table);
         }
 
-        self.do_commit(catalog).await
+        let backoff = Self::build_backoff(self.table.metadata().properties());
+        let tx = self;
+
+        (|mut tx: Transaction| async {
+            let result = tx.do_commit(catalog).await;
+            (tx, result)
+        })
+        .retry(backoff)
+        .sleep(tokio::time::sleep)
+        .context(tx)
+        .when(|e| e.retryable())
+        .await
+        .1
+    }
+
+    fn build_backoff(props: &HashMap<String, String>) -> ExponentialBackoff {
+        ExponentialBuilder::new()
+            .with_min_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MIN_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MIN_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.min-wait-ms"),

Review Comment:
   We should not panic here, we should return error.



##########
crates/iceberg/src/spec/table_metadata.rs:
##########
@@ -98,6 +98,26 @@ pub const RESERVED_PROPERTIES: [&str; 9] = [
     PROPERTY_DEFAULT_SORT_ORDER,
 ];
 
+/// Property key for number of commit retries.
+pub const COMMIT_NUM_RETRIES: &str = "commit.retry.num-retries";
+/// Default value for number of commit retries.
+pub const COMMIT_NUM_RETRIES_DEFAULT: usize = 4;
+
+/// Property key for minimum wait time (ms) between retries.
+pub const COMMIT_MIN_RETRY_WAIT_MS: &str = "commit.retry.min-wait-ms";
+/// Default value for minimum wait time (ms) between retries.
+pub const COMMIT_MIN_RETRY_WAIT_MS_DEFAULT: u64 = 100;
+
+/// Property key for maximum wait time (ms) between retries.
+pub const COMMIT_MAX_RETRY_WAIT_MS: &str = "commit.retry.max-wait-ms";
+/// Default value for maximum wait time (ms) between retries.
+pub const COMMIT_MAX_RETRY_WAIT_MS_DEFAULT: u64 = 60 * 1000; // 1 minute
+
+/// Property key for total maximum retry time (ms).
+pub const COMMIT_TOTAL_RETRY_TIME_MS: &str = "commit.retry.total-timeout-ms";
+/// Default value for total maximum retry time (ms).
+pub const COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT: u64 = 30 * 60 * 1000; // 30 minutes

Review Comment:
   We should add `PROPERTY_` prefix following current convention.



##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -152,13 +159,59 @@ impl Transaction {
     }
 
     /// Commit transaction.
-    pub async fn commit(mut self, catalog: &dyn Catalog) -> Result<Table> {
+    pub async fn commit(self, catalog: &dyn Catalog) -> Result<Table> {
         if self.actions.is_empty() {
             // nothing to commit
             return Ok(self.table);
         }
 
-        self.do_commit(catalog).await
+        let backoff = Self::build_backoff(self.table.metadata().properties());
+        let tx = self;
+
+        (|mut tx: Transaction| async {
+            let result = tx.do_commit(catalog).await;
+            (tx, result)
+        })
+        .retry(backoff)
+        .sleep(tokio::time::sleep)
+        .context(tx)
+        .when(|e| e.retryable())
+        .await
+        .1
+    }
+
+    fn build_backoff(props: &HashMap<String, String>) -> ExponentialBackoff {
+        ExponentialBuilder::new()
+            .with_min_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MIN_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MIN_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.min-wait-ms"),
+            ))
+            .with_max_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MAX_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MAX_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.max-wait-ms"),
+            ))
+            .with_total_delay(Some(Duration::from_millis(
+                props
+                    .get(COMMIT_TOTAL_RETRY_TIME_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.total-timeout-ms"),
+            )))
+            .with_max_times(
+                props
+                    .get(COMMIT_NUM_RETRIES)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_NUM_RETRIES_DEFAULT))
+                    .expect("Invalid value for commit.retry.num-retries"),
+            )
+            .with_factor(2.0)
+            .build()

Review Comment:
   I'm in favor of 1. I agree that we could have a module like `table_properties` to contain all the properties, and have sth to parse properties in a type safe way like following:
   ```
   pub struct TableProperties {
        #[key="commit.num.retries", default="1"]
        commitRetries: i32
   }
   ```
   
   We could track this in following up issue.



##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -262,4 +323,174 @@ mod tests {
             .build()
             .unwrap()
     }
+
+    /// Helper function to create a test table with retry properties
+    fn setup_test_table(num_retries: &str) -> Table {
+        let table = make_v2_table();
+
+        // Set retry properties
+        let mut props = HashMap::new();
+        props.insert("commit.retry.min-wait-ms".to_string(), "10".to_string());
+        props.insert("commit.retry.max-wait-ms".to_string(), "100".to_string());
+        props.insert(
+            "commit.retry.total-timeout-ms".to_string(),
+            "1000".to_string(),
+        );
+        props.insert(
+            "commit.retry.num-retries".to_string(),
+            num_retries.to_string(),
+        );
+
+        // Update table properties
+        let metadata = table
+            .metadata()
+            .clone()
+            .into_builder(None)
+            .set_properties(props)
+            .unwrap()
+            .build()
+            .unwrap()
+            .metadata;
+
+        table.with_metadata(Arc::new(metadata))
+    }
+
+    /// Helper function to create a transaction with a simple update action
+    fn create_test_transaction(table: &Table) -> Transaction {
+        let tx = Transaction::new(table);
+        tx.update_table_properties()
+            .set("test.key".to_string(), "test.value".to_string())
+            .apply(tx)
+            .unwrap()
+    }
+
+    /// Helper function to set up a mock catalog with retryable errors
+    fn setup_mock_catalog_with_retryable_errors(
+        success_after_attempts: Option<u32>,
+        expected_calls: usize,
+    ) -> MockCatalog {
+        let mut mock_catalog = MockCatalog::new();
+
+        mock_catalog
+            .expect_load_table()
+            .returning_st(|_| Box::pin(async move { Ok(make_v2_table()) }));
+
+        mock_catalog
+            .expect_update_table()
+            .times(expected_calls)
+            .returning_st(move |_| {
+                if let Some(attempts) = success_after_attempts {
+                    static mut ATTEMPTS: u32 = 0;
+                    unsafe {

Review Comment:
   Why we need `unsafe` here? Modifying static var? We could use `Atomic` variable to avoid using `unsafe`



##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -152,13 +163,59 @@ impl Transaction {
     }
 
     /// Commit transaction.
-    pub async fn commit(mut self, catalog: &dyn Catalog) -> Result<Table> {
+    pub async fn commit(self, catalog: &dyn Catalog) -> Result<Table> {
         if self.actions.is_empty() {
             // nothing to commit
             return Ok(self.table);
         }
 
-        self.do_commit(catalog).await
+        let backoff = Self::build_backoff(self.table.metadata().properties());
+        let tx = self;
+
+        (|mut tx: Transaction| async {
+            let result = tx.do_commit(catalog).await;
+            (tx, result)
+        })
+        .retry(backoff)
+        .sleep(tokio::time::sleep)
+        .context(tx)
+        .when(|e| e.retryable())
+        .await
+        .1
+    }
+
+    fn build_backoff(props: &HashMap<String, String>) -> ExponentialBackoff {
+        ExponentialBuilder::new()
+            .with_min_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MIN_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MIN_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.min-wait-ms"),
+            ))
+            .with_max_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MAX_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MAX_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.max-wait-ms"),

Review Comment:
   Ditto.



##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -152,13 +163,59 @@ impl Transaction {
     }
 
     /// Commit transaction.
-    pub async fn commit(mut self, catalog: &dyn Catalog) -> Result<Table> {
+    pub async fn commit(self, catalog: &dyn Catalog) -> Result<Table> {
         if self.actions.is_empty() {
             // nothing to commit
             return Ok(self.table);
         }
 
-        self.do_commit(catalog).await
+        let backoff = Self::build_backoff(self.table.metadata().properties());
+        let tx = self;
+
+        (|mut tx: Transaction| async {
+            let result = tx.do_commit(catalog).await;
+            (tx, result)
+        })
+        .retry(backoff)
+        .sleep(tokio::time::sleep)
+        .context(tx)
+        .when(|e| e.retryable())
+        .await
+        .1
+    }
+
+    fn build_backoff(props: &HashMap<String, String>) -> ExponentialBackoff {
+        ExponentialBuilder::new()
+            .with_min_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MIN_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MIN_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.min-wait-ms"),
+            ))
+            .with_max_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MAX_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MAX_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.max-wait-ms"),
+            ))
+            .with_total_delay(Some(Duration::from_millis(
+                props
+                    .get(COMMIT_TOTAL_RETRY_TIME_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.total-timeout-ms"),
+            )))
+            .with_max_times(
+                props
+                    .get(COMMIT_NUM_RETRIES)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_NUM_RETRIES_DEFAULT))
+                    .expect("Invalid value for commit.retry.num-retries"),

Review Comment:
   Ditto.



##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -152,13 +163,59 @@ impl Transaction {
     }
 
     /// Commit transaction.
-    pub async fn commit(mut self, catalog: &dyn Catalog) -> Result<Table> {
+    pub async fn commit(self, catalog: &dyn Catalog) -> Result<Table> {
         if self.actions.is_empty() {
             // nothing to commit
             return Ok(self.table);
         }
 
-        self.do_commit(catalog).await
+        let backoff = Self::build_backoff(self.table.metadata().properties());
+        let tx = self;
+
+        (|mut tx: Transaction| async {
+            let result = tx.do_commit(catalog).await;
+            (tx, result)
+        })
+        .retry(backoff)
+        .sleep(tokio::time::sleep)
+        .context(tx)
+        .when(|e| e.retryable())
+        .await
+        .1
+    }
+
+    fn build_backoff(props: &HashMap<String, String>) -> ExponentialBackoff {
+        ExponentialBuilder::new()
+            .with_min_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MIN_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MIN_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.min-wait-ms"),
+            ))
+            .with_max_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MAX_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MAX_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.max-wait-ms"),
+            ))
+            .with_total_delay(Some(Duration::from_millis(
+                props
+                    .get(COMMIT_TOTAL_RETRY_TIME_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.total-timeout-ms"),

Review Comment:
   Ditto.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#discussion_r2205765539


##########
Cargo.toml:
##########
@@ -82,6 +83,7 @@ itertools = "0.13"
 linkedbytes = "0.1.8"
 metainfo = "0.7.14"
 mimalloc = "0.1.46"
+mockall = "0.13.1"

Review Comment:
   Yes, I saw you created a PR already: https://github.com/apache/iceberg-rust/pull/1507
   
   Thanks for catching this!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#issuecomment-3033003159

   cc @liurenjie1024 @Xuanwo this PR is ready for review, PTAL, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#discussion_r2178995973


##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -152,13 +159,59 @@ impl Transaction {
     }
 
     /// Commit transaction.
-    pub async fn commit(mut self, catalog: &dyn Catalog) -> Result<Table> {
+    pub async fn commit(self, catalog: &dyn Catalog) -> Result<Table> {
         if self.actions.is_empty() {
             // nothing to commit
             return Ok(self.table);
         }
 
-        self.do_commit(catalog).await
+        let backoff = Self::build_backoff(self.table.metadata().properties());
+        let tx = self;
+
+        (|mut tx: Transaction| async {
+            let result = tx.do_commit(catalog).await;
+            (tx, result)
+        })
+        .retry(backoff)
+        .sleep(tokio::time::sleep)
+        .context(tx)
+        .when(|e| e.retryable())
+        .await
+        .1
+    }
+
+    fn build_backoff(props: &HashMap<String, String>) -> ExponentialBackoff {
+        ExponentialBuilder::new()
+            .with_min_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MIN_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MIN_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.min-wait-ms"),
+            ))
+            .with_max_delay(Duration::from_millis(
+                props
+                    .get(COMMIT_MAX_RETRY_WAIT_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_MAX_RETRY_WAIT_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.max-wait-ms"),
+            ))
+            .with_total_delay(Some(Duration::from_millis(
+                props
+                    .get(COMMIT_TOTAL_RETRY_TIME_MS)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_TOTAL_RETRY_TIME_MS_DEFAULT))
+                    .expect("Invalid value for commit.retry.total-timeout-ms"),
+            )))
+            .with_max_times(
+                props
+                    .get(COMMIT_NUM_RETRIES)
+                    .map(|s| s.parse())
+                    .unwrap_or_else(|| Ok(COMMIT_NUM_RETRIES_DEFAULT))
+                    .expect("Invalid value for commit.retry.num-retries"),
+            )
+            .with_factor(2.0)
+            .build()

Review Comment:
   Not really related to this change, but something I noticed while writing the PR:
   1. There is not a centralized properties class, which makes properties hard to locate in the code base: Some properties are defined in table_metadata.rs, and some are defined in the classes that use them: [example](https://github.com/apache/iceberg-rust/blob/e93dd04e1ba661c783b05b5dced424c624be8af5/crates/iceberg/src/writer/file_writer/location_generator.rs#L34)
   2. No orthogonal way to fetch and parse property value, which makes error handling inconsistent. For example, this [LOC](https://github.com/apache/iceberg-rust/blob/772ae1ab194aa039b653648d9d8310d074609619/crates/iceberg/src/transaction/snapshot.rs#L300) will silently fallback to the default value if property parsing failed
   
   For 1, we can have a dedicated file like `table_metadata.rs` to hold all hardcoded property keys, or we can have a property struct to define properties we support like [what datafusion does](https://github.com/apache/datafusion/blob/9d06baf45298bc736966d0239fa78ec7a434ca54/datafusion/common/src/config.rs#L197)
   
   For 2, something like iceberg-java's [PropertyUtil](https://github.com/apache/iceberg/blob/fe23584fc3af9f0ea1371989030b0a99affb233f/core/src/main/java/org/apache/iceberg/util/PropertyUtil.java#L35) can be very useful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#issuecomment-3079684007

   This PR introduces retries, but I think there are two main reasons to retry:
   
   - *Due to network interruptions.* E.g., some network issue, or a malfunctioning load-balancer (5xx errors). In this case, we want to retry each of the calls to the REST catalog.
   - *Due to a commit conflict.* E.g., because another client has changed the state of the table, and the requirements no longer hold. The REST catalog will [return a 409](https://github.com/apache/iceberg/blob/8ed1c216503b7193924ca57bd2694025660ac02c/open-api/rest-catalog-open-api.yaml#L1375-L1381), and the retry will fail again.
   
   I assume that this focusses on the second point. But to make this effective we need to check if we still can do the commit, or we have to redo the whole operation (for details, check Russell's talk on [Transactions and Snapshot Isolation](https://www.youtube.com/watch?v=zhxw3Rrslak)). Right now, it will just fail right away because the requirements are not updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#issuecomment-3081245436

   Hi @Fokko ,
   
   On the question of to retry or not retry I think you will find this [thread](https://github.com/apache/iceberg-rust/pull/1383) interesting:) --- tldr: It's controlled by a field `retrable` in `Error`, and we can flip this field to true or false under different scenarios. So the problem of when to retry will be deferred to the `retryable` field instead of the retrying logic itself.
   
   We also added the `TransactionAction` trait. Now every action should:
   - Instead of holding staging states, it should only define how updates and requirements can be generated from the base Table. The base table will be passed in via `commit()` and can be different for every attempt.
   - Only generate new updates and requirements(`ActionCommit`) exactly [when committing](https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/transaction/mod.rs#L254)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#issuecomment-3082522642

   Hi, @Fokko Thanks for the comments.
   
   > I assume that this focusses on the second point.
   
   Yes.
   
   > But to make this effective we need to check if we still can do the commit, or we have to redo the whole operation (for details, check Russell's talk on [Transactions and Snapshot Isolation](https://www.youtube.com/watch?v=zhxw3Rrslak)).
   
   In fact, we will redo it. We recently did a refactoring to the transaction api, and it will reproduce the updates and requirements in each retry. You can check the `do_commit` method.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#discussion_r2205478373


##########
Cargo.toml:
##########
@@ -82,6 +83,7 @@ itertools = "0.13"
 linkedbytes = "0.1.8"
 metainfo = "0.7.14"
 mimalloc = "0.1.46"
+mockall = "0.13.1"

Review Comment:
   Should we add it to dev dependency? Since it's only used in tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): Add retry logic to transaction [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #1484:
URL: https://github.com/apache/iceberg-rust/pull/1484#issuecomment-3025187150

   > curious how do we plan to test the retry? i.e. is it possible to use mock catalog and inject retriable / non-retriable error?
   
   Hi @dentiny , this is a good question:) I'm thinking of using a mock catalog, but haven't figured exactly how


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org