You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "stevenzwu (via GitHub)" <gi...@apache.org> on 2026/05/12 00:11:33 UTC
[PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
stevenzwu opened a new pull request, #16294:
URL: https://github.com/apache/iceberg/pull/16294
## Summary
Adds a v4 spec requirement that a snapshot's `timestamp-ms` must be strictly greater than the parent snapshot's `timestamp-ms` on the same branch, and documents the Lamport-clock style fast-forward writers must use when the wall clock would not satisfy that constraint.
- `format/spec.md` Snapshots section: notes the v4-only requirement directly under the snapshot field table.
- `format/spec.md` Appendix E: adds a new `Version 4` subsection (above `Version 3`) listing the snapshot-timestamp rule. This is the first v4 entry in Appendix E and is intentionally minimal\u2014other v4 changes (row-level timestamp, etc.) can be appended as they land.
No table or schema fields are added. v1\u2013v3 semantics are unchanged.
## Why
The v4 row-level `_last_updated_timestamp_ms` metadata column is inherited from each manifest entry's commit timestamp, which in turn comes from the snapshot's `timestamp-ms`. Time-travel queries and \"last updated\" semantics on that column rely on `timestamp-ms` being monotonically increasing with sequence number; otherwise a brief backward jump in the writer's wall clock can produce snapshots that appear out-of-order in time but in-order in sequence. Enforcing monotonicity at write time keeps the two orderings consistent without requiring readers to defensively clamp values.
The Lamport-style fast-forward (`max(now, parent.timestamp-ms + 1)`) keeps commits unblocked when the wall clock has drifted: it advances by the minimum amount required, leaving steady-state behavior unchanged when the clock is healthy.
## Companion PR
Reference implementation in `SnapshotProducer` is proposed in apache/iceberg#16293.
## Test plan
- [x] `format/spec.md` renders correctly (note added to the v1\u2013v3 Snapshots tab; new `Version 4` subsection in Appendix E above `Version 3`)
- [ ] Spec review
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3502774316
##########
format/spec.md:
##########
@@ -1908,6 +1910,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
+* The requirement applies only to snapshots committed while the table is at format version 4 or higher; snapshots that existed before a table was upgraded to v4 are not constrained.
Review Comment:
I see but doesn't it makes time travel broken ? as if we are letting the correct timestamp being overriden with v4 ? how about we say first commit post upgrade is ok to not follow assuming that commit is true to the commit time and then after that we can enforce the constraints
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3228610211
##########
format/spec.md:
##########
@@ -751,6 +751,8 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. Root snapshots (those without a parent) are not constrained by this requirement.
Review Comment:
Good catch. Pushed an update that explicitly carves out pre-upgrade snapshots:
> In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. The requirement applies only to snapshots committed while the table is at format version 4 or higher; root snapshots, and snapshots that existed before a table was upgraded to v4, are not constrained.
So a table migrated from v3 with non-monotonic timestamps is conformant: the existing snapshots are grandfathered in, and only new snapshots committed at v4+ must satisfy the rule. A new v4 snapshot whose parent happens to be a pre-upgrade snapshot must still beat that parent's `timestamp-ms`, which will naturally fast-forward past any anomaly inherited from v3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3770281538
##########
format/spec.md:
##########
@@ -960,6 +960,8 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. The requirement applies only to snapshots committed while the table is at format version 4 or higher; root snapshots, and snapshots that existed before a table was upgraded to v4, are not constrained.
Review Comment:
I think we actually have a note about this in the appendix about how you should do time travel because they aren't
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu merged PR #16294:
URL: https://github.com/apache/iceberg/pull/16294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3658540862
##########
format/spec.md:
##########
@@ -1908,6 +1910,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
Review Comment:
Thanks for adding this. BTW this is almost identical to Delta Lake's [In-commit timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps) feature which also adds Lamport clocks to commits.
##########
format/spec.md:
##########
@@ -960,6 +960,8 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. The requirement applies only to snapshots committed while the table is at format version 4 or higher; root snapshots, and snapshots that existed before a table was upgraded to v4, are not constrained.
Review Comment:
The rule says it "applies only to snapshots committed while the table is at format version 4 or higher," but as far as I can tell nothing durably records where that boundary is with v4 snapshots and old grandfathered v3 snapshots.
In Delta, we record the enablement point so readers can identify the boundary.
Maybe we can optionally add a sentence to the reader-facing part of the spec stating that pre-v4 snapshots may violate monotonicity and readers must not assume the full snapshot chain is monotonic. (for time travel purposes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3226695427
##########
format/spec.md:
##########
@@ -751,6 +751,8 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. Root snapshots (those without a parent) are not constrained by this requirement.
Review Comment:
how about tables migrated from v3 to v4 ... with broken timestamps ? do we need to say something about them ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3508894098
##########
format/spec.md:
##########
@@ -1908,6 +1910,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
+* The requirement applies only to snapshots committed while the table is at format version 4 or higher; snapshots that existed before a table was upgraded to v4 are not constrained.
Review Comment:
Today, time travel using timestamp is already sort of broken today if the snapshot timestamp is non monotonic.
> how about we say first commit post upgrade is ok to not follow assuming that commit is true to the commit time and then after that we can enforce the constraints
This assumes v4 writers would have better client clocks than v3 writers, which is not true. We will continue to have the possibility of clock drift after v4 upgrade.
This spec and [commit implementation change](https://github.com/apache/iceberg/pull/16293/changes) will ensure that we will always monotonic commit timestamp for snapshots committed in v4 version.
I am not sure special treatment of the first commit adds significant value or protection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3502695669
##########
format/spec.md:
##########
@@ -1908,6 +1910,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
+* The requirement applies only to snapshots committed while the table is at format version 4 or higher; snapshots that existed before a table was upgraded to v4 are not constrained.
Review Comment:
v4 snapshot will still be able to commit. assume the "bad" timestamp from a v3 snapshot is `t1`. the v4 snapshot can just increment by 1 ms as `t1+1`, which is the behavior in the Java impl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#issuecomment-4686367076
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #16294: Spec: require monotonically increasing snapshot timestamp-ms in v4
URL: https://github.com/apache/iceberg/pull/16294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3532717247
##########
format/spec.md:
##########
@@ -1908,6 +1910,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
+* The requirement applies only to snapshots committed while the table is at format version 4 or higher; snapshots that existed before a table was upgraded to v4 are not constrained.
Review Comment:
Thats fair, i believe there is no way to gaurantee that v4 writes clock will not be broken and once the table is upgraded we can relybly find which snapshot was created in v3, i think then its reasonable to leave this for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3502624862
##########
format/spec.md:
##########
@@ -1908,6 +1910,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
+* The requirement applies only to snapshots committed while the table is at format version 4 or higher; snapshots that existed before a table was upgraded to v4 are not constrained.
Review Comment:
how about a case where table was in v3 format and future timestamp was committed and now v4 snapshot is committing the right timestamp, this would not let v4 timestamp snapshot commit ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r4220351039
##########
format/spec.md:
##########
@@ -966,6 +966,8 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. The requirement applies only to snapshots committed while the table is at format version 4 or higher; snapshots with no parent, and snapshots that existed before a table was upgraded to v4, are not constrained.
Review Comment:
See below comment, I think there's a way to express this requirement in a combined statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3770312590
##########
format/spec.md:
##########
@@ -960,6 +960,8 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. The requirement applies only to snapshots committed while the table is at format version 4 or higher; root snapshots, and snapshots that existed before a table was upgraded to v4, are not constrained.
Review Comment:
> but as far as I can tell nothing durably records where that boundary is with v4 snapshots and old grandfathered v3 snapshots.
We should add a `format-version` field to the snapshot metadata. This is what I did in PR #17523 for the `Snapshot` interface change to support v4 root manifest file pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#issuecomment-4747343108
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r3658540862
##########
format/spec.md:
##########
@@ -1908,6 +1910,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
Review Comment:
Thanks for adding this. BTW this is almost identical to Delta Lake's [In-commit timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps) feature which also adds Lamport clocks to commits. (which is awesome!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: require monotonically increasing snapshot timestamp-ms in v4 [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #16294:
URL: https://github.com/apache/iceberg/pull/16294#discussion_r4220241366
##########
format/spec.md:
##########
@@ -966,6 +966,8 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+In format version 4 and above, a snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch. The requirement applies only to snapshots committed while the table is at format version 4 or higher; snapshots with no parent, and snapshots that existed before a table was upgraded to v4, are not constrained.
Review Comment:
Maybe "Starting in v4, a snapshot's..."
and then for the final sentence "snapshots that were committed before a table was upgraded to v4 or higher do not have this requirement"
##########
format/spec.md:
##########
@@ -1918,6 +1920,12 @@ Reading v4 metadata:
* Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution))
* When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification))
+Snapshot timestamp changes:
+
+* A snapshot's `timestamp-ms` must be strictly greater than the `timestamp-ms` of its parent snapshot on the same branch.
+* The requirement applies only to snapshots committed while the table is at format version 4 or higher.
Review Comment:
Not sure this needs to be said again, but if we do maybe there's a way to combine it with the first sentence:
```
For snapshots committed while the table is at format version 4 or higher, a snapshot’s timestamp-ms must be strictly greater than its parent snapshot’s timestamp-ms on the same branch.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org