You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Steve <ho...@gmail.com> on 2025/11/06 23:24:29 UTC
[Discuss] Seeking input on PR to unlink table metadata's last updated timestamp from snapshot timestamp
Hey all:
I would like to reach out to the Iceberg community to seek input and
thoughts on PR 14504, which proposes changing the behavior of table
metadata's last updated timestamp.
This was discussed yesterday on the Iceberg community sync. The current
Java SDK implementation ties the `last-updated-ms` field in TableMetadata
to the snapshot creation timestamp when the snapshot was added. Ryan shared
his initial motivation but also indicated that he's open to change. We also
confirmed that the last-updated-at timestamp in TableMetadata is purely
informational and provides no guarantee.
Unlinking timestamps allows them to reflect their own state, particularly
given the separation of concerns between the engine's role of authoring
snapshots and the catalog's role of writing metadata.json.
Looking forward to your feedback.
https://github.com/apache/iceberg/pull/14504
Thanks,
Hongyue
Re: [Discuss] Seeking input on PR to unlink table metadata's last updated timestamp from snapshot timestamp
Posted by Steve <ho...@gmail.com>.
Hey Ryan:
Thanks again for the earlier review and guidance on
apache/iceberg#14504. I've addressed your previous comments and hope
you can take another look.
https://github.com/apache/iceberg/pull/14504
Thanks,
Steve
On Fri, Nov 7, 2025 at 3:27 PM Steven Wu <st...@gmail.com> wrote:
>
> > Can we populate the Delta table commit time(9:58) for the first change?
>
> Yufei, it is possible but it would require us to update snapshot producer APIs for all DDL operations. That also includes the transaction APIs. Hence it is a quite disruptive change.
>
> Separation of timestamps also makes sense conceptually. In the context of REST catalog where snapshot file is generated at the client side and the metadata.json file is generated at server side.
>
> As Hongyue said, only snapshot timestamps are used for time travel. Otherwise, timestamp values are mainly for informational purposes.
>
>
> On Fri, Nov 7, 2025 at 6:55 AM Eduard Tudenhöfner <et...@apache.org> wrote:
>>
>> After reading the problem that can happen it makes sense to me to unlink the table metadata's lastUpdatedMillis from the snapshot's lastUpdatedMillis.
>>
>> On Fri, Nov 7, 2025 at 7:09 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>> Can we populate the Delta table commit time(9:58) for the first change?
>>>
>>> Yufei
>>>
>>>
>>> On Thu, Nov 6, 2025 at 9:59 PM Steven Wu <st...@gmail.com> wrote:
>>>>
>>>> Let me add some additional context in addition to what Hongyue said.
>>>>
>>>> Imaging a metadata translation use case from Delta to Iceberg. If the Iceberg table metadata stamp is set to the snapshot timestamp (as today), we can get into the following anomaly state.
>>>>
>>>> At 10:00, a schema change (committed at 9:58 at the source Delta table) was translated to the Iceberg metadata. Since DDL doesn't add a new snapshot to the table, the table metadata timestamp was set to 10:00 (system clock).
>>>> At 10:01, a DML change (committed at 9:59 in the source Delta table) was translated to the Iceberg metadata. The Iceberg snapshot timestamp should be set to 9:59 for consistent time travel behavior. Current Java implementation would set the table metadata timestamp to 9:59 (same as snapshot timestamp).
>>>>
>>>> Now, the table metadata timestamp went backwards in the 2nd commit above. We can avoid such timestamp anomaly behavior with the proposed change. With that, the 2nd commit would set the table timestamp to 10:01 and set the snapshot timestamp to 9:59.
>>>>
>>>>
>>>>
>>>> On Thu, Nov 6, 2025 at 3:24 PM Steve <ho...@gmail.com> wrote:
>>>>>
>>>>> Hey all:
>>>>>
>>>>> I would like to reach out to the Iceberg community to seek input and thoughts on PR 14504, which proposes changing the behavior of table metadata's last updated timestamp.
>>>>>
>>>>> This was discussed yesterday on the Iceberg community sync. The current Java SDK implementation ties the `last-updated-ms` field in TableMetadata to the snapshot creation timestamp when the snapshot was added. Ryan shared his initial motivation but also indicated that he's open to change. We also confirmed that the last-updated-at timestamp in TableMetadata is purely informational and provides no guarantee.
>>>>>
>>>>> Unlinking timestamps allows them to reflect their own state, particularly given the separation of concerns between the engine's role of authoring snapshots and the catalog's role of writing metadata.json.
>>>>>
>>>>> Looking forward to your feedback.
>>>>>
>>>>> https://github.com/apache/iceberg/pull/14504
>>>>>
>>>>> Thanks,
>>>>> Hongyue
Re: [Discuss] Seeking input on PR to unlink table metadata's last updated timestamp from snapshot timestamp
Posted by Steven Wu <st...@gmail.com>.
> Can we populate the Delta table commit time(9:58) for the first change?
Yufei, it is possible but it would require us to update snapshot producer
APIs for all DDL operations. That also includes the transaction APIs. Hence
it is a quite disruptive change.
Separation of timestamps also makes sense conceptually. In the context of
REST catalog where snapshot file is generated at the client side and the
metadata.json file is generated at server side.
As Hongyue said, only snapshot timestamps are used for time travel.
Otherwise, timestamp values are mainly for informational purposes.
On Fri, Nov 7, 2025 at 6:55 AM Eduard Tudenhöfner <et...@apache.org>
wrote:
> After reading the problem that can happen it makes sense to me to unlink
> the table metadata's lastUpdatedMillis from the snapshot's
> lastUpdatedMillis.
>
> On Fri, Nov 7, 2025 at 7:09 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Can we populate the Delta table commit time(9:58) for the first change?
>>
>> Yufei
>>
>>
>> On Thu, Nov 6, 2025 at 9:59 PM Steven Wu <st...@gmail.com> wrote:
>>
>>> Let me add some additional context in addition to what Hongyue said.
>>>
>>> Imaging a metadata translation use case from Delta to Iceberg. If the
>>> Iceberg table metadata stamp is set to the snapshot timestamp (as today),
>>> we can get into the following anomaly state.
>>>
>>> - At 10:00, a schema change (committed at 9:58 at the source Delta
>>> table) was translated to the Iceberg metadata. Since DDL doesn't add a new
>>> snapshot to the table, the table metadata timestamp was set to 10:00
>>> (system clock).
>>> - At 10:01, a DML change (committed at 9:59 in the source Delta
>>> table) was translated to the Iceberg metadata. The Iceberg snapshot
>>> timestamp should be set to 9:59 for consistent time travel behavior.
>>> Current Java implementation would set the table metadata timestamp to 9:59
>>> (same as snapshot timestamp).
>>>
>>> Now, the table metadata timestamp went backwards in the 2nd commit
>>> above. We can avoid such timestamp anomaly behavior with the proposed
>>> change. With that, the 2nd commit would set the table timestamp to 10:01
>>> and set the snapshot timestamp to 9:59.
>>>
>>>
>>>
>>> On Thu, Nov 6, 2025 at 3:24 PM Steve <ho...@gmail.com> wrote:
>>>
>>>> Hey all:
>>>>
>>>> I would like to reach out to the Iceberg community to seek input and
>>>> thoughts on PR 14504, which proposes changing the behavior of table
>>>> metadata's last updated timestamp.
>>>>
>>>> This was discussed yesterday on the Iceberg community sync. The current
>>>> Java SDK implementation ties the `last-updated-ms` field in TableMetadata
>>>> to the snapshot creation timestamp when the snapshot was added. Ryan shared
>>>> his initial motivation but also indicated that he's open to change. We also
>>>> confirmed that the last-updated-at timestamp in TableMetadata is purely
>>>> informational and provides no guarantee.
>>>>
>>>> Unlinking timestamps allows them to reflect their own state,
>>>> particularly given the separation of concerns between the engine's role of
>>>> authoring snapshots and the catalog's role of writing metadata.json.
>>>>
>>>> Looking forward to your feedback.
>>>>
>>>> https://github.com/apache/iceberg/pull/14504
>>>>
>>>> Thanks,
>>>> Hongyue
>>>>
>>>
Re: [Discuss] Seeking input on PR to unlink table metadata's last updated timestamp from snapshot timestamp
Posted by Eduard Tudenhöfner <et...@apache.org>.
After reading the problem that can happen it makes sense to me to unlink
the table metadata's lastUpdatedMillis from the snapshot's
lastUpdatedMillis.
On Fri, Nov 7, 2025 at 7:09 AM Yufei Gu <fl...@gmail.com> wrote:
> Can we populate the Delta table commit time(9:58) for the first change?
>
> Yufei
>
>
> On Thu, Nov 6, 2025 at 9:59 PM Steven Wu <st...@gmail.com> wrote:
>
>> Let me add some additional context in addition to what Hongyue said.
>>
>> Imaging a metadata translation use case from Delta to Iceberg. If the
>> Iceberg table metadata stamp is set to the snapshot timestamp (as today),
>> we can get into the following anomaly state.
>>
>> - At 10:00, a schema change (committed at 9:58 at the source Delta
>> table) was translated to the Iceberg metadata. Since DDL doesn't add a new
>> snapshot to the table, the table metadata timestamp was set to 10:00
>> (system clock).
>> - At 10:01, a DML change (committed at 9:59 in the source Delta
>> table) was translated to the Iceberg metadata. The Iceberg snapshot
>> timestamp should be set to 9:59 for consistent time travel behavior.
>> Current Java implementation would set the table metadata timestamp to 9:59
>> (same as snapshot timestamp).
>>
>> Now, the table metadata timestamp went backwards in the 2nd commit above.
>> We can avoid such timestamp anomaly behavior with the proposed change. With
>> that, the 2nd commit would set the table timestamp to 10:01 and set the
>> snapshot timestamp to 9:59.
>>
>>
>>
>> On Thu, Nov 6, 2025 at 3:24 PM Steve <ho...@gmail.com> wrote:
>>
>>> Hey all:
>>>
>>> I would like to reach out to the Iceberg community to seek input and
>>> thoughts on PR 14504, which proposes changing the behavior of table
>>> metadata's last updated timestamp.
>>>
>>> This was discussed yesterday on the Iceberg community sync. The current
>>> Java SDK implementation ties the `last-updated-ms` field in TableMetadata
>>> to the snapshot creation timestamp when the snapshot was added. Ryan shared
>>> his initial motivation but also indicated that he's open to change. We also
>>> confirmed that the last-updated-at timestamp in TableMetadata is purely
>>> informational and provides no guarantee.
>>>
>>> Unlinking timestamps allows them to reflect their own state,
>>> particularly given the separation of concerns between the engine's role of
>>> authoring snapshots and the catalog's role of writing metadata.json.
>>>
>>> Looking forward to your feedback.
>>>
>>> https://github.com/apache/iceberg/pull/14504
>>>
>>> Thanks,
>>> Hongyue
>>>
>>
Re: [Discuss] Seeking input on PR to unlink table metadata's last updated timestamp from snapshot timestamp
Posted by Yufei Gu <fl...@gmail.com>.
Can we populate the Delta table commit time(9:58) for the first change?
Yufei
On Thu, Nov 6, 2025 at 9:59 PM Steven Wu <st...@gmail.com> wrote:
> Let me add some additional context in addition to what Hongyue said.
>
> Imaging a metadata translation use case from Delta to Iceberg. If the
> Iceberg table metadata stamp is set to the snapshot timestamp (as today),
> we can get into the following anomaly state.
>
> - At 10:00, a schema change (committed at 9:58 at the source Delta
> table) was translated to the Iceberg metadata. Since DDL doesn't add a new
> snapshot to the table, the table metadata timestamp was set to 10:00
> (system clock).
> - At 10:01, a DML change (committed at 9:59 in the source Delta table)
> was translated to the Iceberg metadata. The Iceberg snapshot timestamp
> should be set to 9:59 for consistent time travel behavior. Current Java
> implementation would set the table metadata timestamp to 9:59 (same as
> snapshot timestamp).
>
> Now, the table metadata timestamp went backwards in the 2nd commit above.
> We can avoid such timestamp anomaly behavior with the proposed change. With
> that, the 2nd commit would set the table timestamp to 10:01 and set the
> snapshot timestamp to 9:59.
>
>
>
> On Thu, Nov 6, 2025 at 3:24 PM Steve <ho...@gmail.com> wrote:
>
>> Hey all:
>>
>> I would like to reach out to the Iceberg community to seek input and
>> thoughts on PR 14504, which proposes changing the behavior of table
>> metadata's last updated timestamp.
>>
>> This was discussed yesterday on the Iceberg community sync. The current
>> Java SDK implementation ties the `last-updated-ms` field in TableMetadata
>> to the snapshot creation timestamp when the snapshot was added. Ryan shared
>> his initial motivation but also indicated that he's open to change. We also
>> confirmed that the last-updated-at timestamp in TableMetadata is purely
>> informational and provides no guarantee.
>>
>> Unlinking timestamps allows them to reflect their own state, particularly
>> given the separation of concerns between the engine's role of authoring
>> snapshots and the catalog's role of writing metadata.json.
>>
>> Looking forward to your feedback.
>>
>> https://github.com/apache/iceberg/pull/14504
>>
>> Thanks,
>> Hongyue
>>
>
Re: [Discuss] Seeking input on PR to unlink table metadata's last updated timestamp from snapshot timestamp
Posted by Steven Wu <st...@gmail.com>.
Let me add some additional context in addition to what Hongyue said.
Imaging a metadata translation use case from Delta to Iceberg. If the
Iceberg table metadata stamp is set to the snapshot timestamp (as today),
we can get into the following anomaly state.
- At 10:00, a schema change (committed at 9:58 at the source Delta
table) was translated to the Iceberg metadata. Since DDL doesn't add a new
snapshot to the table, the table metadata timestamp was set to 10:00
(system clock).
- At 10:01, a DML change (committed at 9:59 in the source Delta table)
was translated to the Iceberg metadata. The Iceberg snapshot timestamp
should be set to 9:59 for consistent time travel behavior. Current Java
implementation would set the table metadata timestamp to 9:59 (same as
snapshot timestamp).
Now, the table metadata timestamp went backwards in the 2nd commit above.
We can avoid such timestamp anomaly behavior with the proposed change. With
that, the 2nd commit would set the table timestamp to 10:01 and set the
snapshot timestamp to 9:59.
On Thu, Nov 6, 2025 at 3:24 PM Steve <ho...@gmail.com> wrote:
> Hey all:
>
> I would like to reach out to the Iceberg community to seek input and
> thoughts on PR 14504, which proposes changing the behavior of table
> metadata's last updated timestamp.
>
> This was discussed yesterday on the Iceberg community sync. The current
> Java SDK implementation ties the `last-updated-ms` field in TableMetadata
> to the snapshot creation timestamp when the snapshot was added. Ryan shared
> his initial motivation but also indicated that he's open to change. We also
> confirmed that the last-updated-at timestamp in TableMetadata is purely
> informational and provides no guarantee.
>
> Unlinking timestamps allows them to reflect their own state, particularly
> given the separation of concerns between the engine's role of authoring
> snapshots and the catalog's role of writing metadata.json.
>
> Looking forward to your feedback.
>
> https://github.com/apache/iceberg/pull/14504
>
> Thanks,
> Hongyue
>