[I] Make Metadata Timestamp Clock Skew Configurable [iceberg]

["chandrasekhar-188k (via GitHub)" <gi...@apache.org>]

chandrasekhar-188k opened a new issue, #17285:
URL: https://github.com/apache/iceberg/issues/17285

   ### Feature Request / Improvement
   
   Background
   
   Iceberg currently validates metadata timestamps using a hardcoded one-minute tolerance (TableMetadata.ONE_MINUTE) when constructing new table metadata during a commit.
   
   This validation helps detect situations where metadata timestamps are significantly ahead of the local system clock, which could indicate incorrect system time or metadata corruption.
   
   However, in deployments where multiple compute engines (for example, Spark, Flink, Trino, or custom applications) operate on the same Iceberg table from different clusters, temporary clock skew greater than one minute can occur despite otherwise healthy deployments. In such cases, valid commits may fail because the metadata timestamp appears to be too far in the future relative to the committing process.
   
   Although proper clock synchronization (for example, via NTP) is recommended, operators may still encounter larger clock skew in distributed or multi-region environments.
   
   Problem
   
   The allowable clock skew is currently hardcoded as one minute and cannot be configured.
   
   As a result:
   
   Valid commits may fail when clocks differ by more than one minute.
   Operators have no mechanism to tune this behavior for their deployment.
   The only current workaround is improving clock synchronization across all participating systems.
   Proposed Solution
   
   Introduce a JVM system property to configure the maximum allowable clock skew used during metadata timestamp validation.
   
   System Property
   iceberg.commit.allowed-clock-skew-ms
   Default Value
   60000
   
   (Existing behavior remains unchanged.)
   
   Example
   -Diceberg.commit.allowed-clock-skew-ms=300000
   
   This would allow metadata timestamps to be up to five minutes ahead of the local system clock before the commit is rejected.
   
   Behavior
   
   The validation logic would determine the allowed clock skew as follows:
   
   Read the JVM system property iceberg.commit.allowed-clock-skew-ms.
   If the property is present and valid, use its value.
   Otherwise, fall back to the current default of 60,000 milliseconds.
   
   This proposal does not change the default behavior for existing deployments.
   
   Compatibility
   
   This proposal is fully backward compatible.
   
   Existing applications continue to use the current one-minute tolerance.
   No table metadata changes are required.
   No catalog changes are required.
   No API changes are required.
   Future Work
   
   A future enhancement could introduce catalog-level configuration, allowing operators to configure the allowable clock skew once per catalog instead of per JVM.
   
   One possible configuration hierarchy would be:
   
   Catalog property
           ↓
   System property
           ↓
   Default (60000 ms)
   
   The current proposal intentionally limits the scope to a JVM system property to minimize implementation complexity while addressing the immediate operational need.
   
   ### Query engine
   
   None
   
   ### Willingness to contribute
   
   - [x] I can contribute this improvement/feature independently
   - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
   - [ ] I cannot contribute this improvement/feature at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org