You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Steve Zhang <ho...@apple.com.INVALID> on 2025/02/07 18:16:56 UTC
Table metadata swap not work for REST Catalog (#12134)
Hey Iceberg Experts:
I am seeking assistance and insights regarding an issue we’ve encountered with RESTTableOperations and its inability to support on-demand table metadata swaps. We are currently adopting the REST-based catalog from Hive and have noticed a potential gap in the TableOperations.commit() API. Typically, we use the commit API to revert a table to a previously known state, as demonstrated below:
String deisredMetadataPath = "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
ops.commit(ops.current(), TableMetadataParser.read(ops.io(), dedeisredMetadataPath));
However, this approach is no longer working with the REST-based catalog. I suspect that the issue may be related to how the update type is modeled in RESTTableOperations. I have shared a unit test that reproduces the problem on https://github.com/apache/iceberg/issues/12134, where it works on JDBC and in-memory catalogs, but not with RESTCatalog.
Best Regards,
Steve Zhang
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by Steve Zhang <ho...@apple.com.INVALID>.
Hi Iceberg Community,
I wanted to bring up a discussion regarding the current TableOperations.commit logic and its impact on registerTablewith the overwrite option in my https://github.com/apache/iceberg/pull/12228#discussion_r1972591876. Currently, the commit logic always writes a new metadata.json for atomic swaps of table metadata. This design makes it difficult to directly set a user-provided metadata.json as the latest table metadata in the catalog when registering a table with overwrite.
The current workaround is to drop the existing table and re-register it with the provided metadata.json. However, this approach introduces a potential issue: lack of atomicity, which can lead to failures in intermediate states. For example, if concurrent writes or table drops occur between the deletion and re-registration, it may lead to inconsistent or unexpected results.
To address this, we would love to hear the community’s thoughts on:
Potential approaches to allow registerTable with overwrite to perform an atomic swap while respecting the user-provided metadata.json.
Implications on changing table UUID, whether or not to allow table UUID change when user provided metadata.json have a different table UUI as the existing one.
We appreciate any insights or suggestions you may have.
Best,
Steve Zhang
> On Feb 10, 2025, at 4:47 PM, Steve Zhang <ho...@apple.com.INVALID> wrote:
>
> Thank you Russell and Ryan.
>
> Let me start to work on a new API to support force table registration in catalog.
>
> Thanks,
> Steve Zhang
>
>
>
>> On Feb 10, 2025, at 4:29 PM, rdblue@gmail.com wrote:
>>
>> Yeah, it sounds like a "register table force" is the right concept here. I think we want to make sure that table updates remain change-based as the best practice in the REST API. But there are some irregular use cases that justify having some mechanism to completely replace the state (like push-based mirroring). I think it makes sense to revisit mirroring and this use case and come up with a path forward.
>>
>> On Mon, Feb 10, 2025 at 3:12 PM Russell Spitzer <russell.spitzer@gmail.com <ma...@gmail.com>> wrote:
>>> I still would like a "register table" force" option
>>>
>>> On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang <ho...@apple.com.invalid> wrote:
>>>> Thank you Dan for your detailed reply. Based on your explanation, do you think it would be worthwhile to support non-linear or complete metadata replacements in the REST implementation? I am happy to contribute but might need some guidance from the community on the best approach.
>>>>
>>>> For additional context, we explored into the workaround of using a combination of dropping table and re-registering the table with concerns of reading in between. There’s also an attempt to add a force option to the register-table API (https://github.com/apache/iceberg/pull/5327), which would allow for metadata swap on an existing table. However, it was suggested that use TableOperations.commit(base, new) is preferred to achieve atomicity.
>>>>
>>>> Thanks,
>>>> Steve Zhang
>>>>
>>>>
>>>>
>>>>> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dweeks@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> Hey Steve,
>>>>>
>>>>> I think the issue here is that you're using the commit api in table operations to perform a non-incremental/linear change to the metadata. The REST implementation is a little more strict in that it builds a set of updates based on the mutations made to the metadata and the commit process applies those changes. In this scenario, no changes have been made and the call is attempting a complete replacement.
>>>>>
>>>>> The other implementations are just blindly swapping the location, so while that operation does achieve the effect you're looking for, it's not the right semantics for the commit.
>>>>>
>>>>> You might want to consider using the "register table" operation instead, which takes the table identifier and location to perform this type of swap.
>>>>>
>>>>> -Dan
>>>>>
>>>>> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang <ho...@apple.com.invalid> wrote:
>>>>>> Hey Iceberg Experts:
>>>>>>
>>>>>> I am seeking assistance and insights regarding an issue we’ve encountered with RESTTableOperations and its inability to support on-demand table metadata swaps. We are currently adopting the REST-based catalog from Hive and have noticed a potential gap in the TableOperations.commit() API. Typically, we use the commit API to revert a table to a previously known state, as demonstrated below:
>>>>>>
>>>>>> String deisredMetadataPath = "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
>>>>>> ops.commit(ops.current(), TableMetadataParser.read(ops.io <http://ops.io/>(), dedeisredMetadataPath));
>>>>>>
>>>>>> However, this approach is no longer working with the REST-based catalog. I suspect that the issue may be related to how the update type is modeled in RESTTableOperations. I have shared a unit test that reproduces the problem on https://github.com/apache/iceberg/issues/12134, where it works on JDBC and in-memory catalogs, but not with RESTCatalog.
>>>>>>
>>>>>> Best Regards,
>>>>>> Steve Zhang
>>>>>>
>>>>>>
>>>>>>
>>>>
>
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by roryqi <ro...@apache.org>.
Agree. Gravitino‘s Iceberg REST service provides similar mechanism to
notify the changes. You can see
https://gravitino.apache.org/docs/0.8.0-incubating/iceberg-rest-service#event-listener
Yufei Gu <fl...@gmail.com> 于2025年2月11日周二 10:01写道：
> The push-based mirroring highlighted by Ryan is a popular use case.
> Polaris has already implemented notification APIs to address this, and
> there have been several Iceberg community discussions surrounding it. If I
> recall correctly, we generally agreed that the notification API is
> beneficial per last discussion. Polaris supports various notification types(
> https://github.com/apache/polaris/blob/be343f1b484d2d79877978ff55f3ad36031dc4a5/spec/polaris-catalog-apis/notifications-api.yaml#L108).
> The "UPDATE" type provides the same functionality as "register table
> force". I believe using notification APIs, or a similar concept is the
> right approach for achieving push-based mirroring, as "register table
> force" on its own isn't sufficient. Here is a list of notification types
> used in Polaris for reference:
> - CREATE
> - UPDATE
> - DROP
> - VALIDATE
>
> Yufei
>
>
> On Mon, Feb 10, 2025 at 4:49 PM Steve Zhang
> <ho...@apple.com.invalid> wrote:
>
>> Thank you Russell and Ryan.
>>
>> Let me start to work on a new API to support force table registration
>> in catalog.
>>
>> Thanks,
>> Steve Zhang
>>
>>
>>
>> On Feb 10, 2025, at 4:29 PM, rdblue@gmail.com wrote:
>>
>> Yeah, it sounds like a "register table force" is the right concept here.
>> I think we want to make sure that table updates remain change-based as
>> the best practice in the REST API. But there are some irregular use cases
>> that justify having some mechanism to completely replace the state (like
>> push-based mirroring). I think it makes sense to revisit mirroring and this
>> use case and come up with a path forward.
>>
>> On Mon, Feb 10, 2025 at 3:12 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>>> I still would like a "register table" force" option
>>>
>>> On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang
>>> <ho...@apple.com.invalid> wrote:
>>>
>>>> Thank you Dan for your detailed reply. Based on your explanation, do
>>>> you think it would be worthwhile to support non-linear or complete metadata
>>>> replacements in the REST implementation? I am happy to contribute but might
>>>> need some guidance from the community on the best approach.
>>>>
>>>> For additional context, we explored into the workaround of using a
>>>> combination of dropping table and re-registering the table with concerns of
>>>> reading in between. There’s also an attempt to add a force option to the
>>>> register-table API (https://github.com/apache/iceberg/pull/5327),
>>>> which would allow for metadata swap on an existing table. However, it was
>>>> suggested that use TableOperations.commit(base, new) is preferred to
>>>> achieve atomicity.
>>>>
>>>> Thanks,
>>>> Steve Zhang
>>>>
>>>>
>>>>
>>>> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dw...@apache.org> wrote:
>>>>
>>>> Hey Steve,
>>>>
>>>> I think the issue here is that you're using the commit api in table
>>>> operations to perform a non-incremental/linear change to the metadata. The
>>>> REST implementation is a little more strict in that it builds a set of
>>>> updates based on the mutations made to the metadata and the commit process
>>>> applies those changes. In this scenario, no changes have been made and the
>>>> call is attempting a complete replacement.
>>>>
>>>> The other implementations are just blindly swapping the location, so
>>>> while that operation does achieve the effect you're looking for, it's not
>>>> the right semantics for the commit.
>>>>
>>>> You might want to consider using the "register table" operation
>>>> instead, which takes the table identifier and location to perform this type
>>>> of swap.
>>>>
>>>> -Dan
>>>>
>>>> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang
>>>> <ho...@apple.com.invalid> wrote:
>>>>
>>>>> Hey Iceberg Experts:
>>>>>
>>>>> I am seeking assistance and insights regarding an issue we’ve
>>>>> encountered with RESTTableOperations and its inability to support on-demand
>>>>> table metadata swaps. We are currently adopting the REST-based catalog from
>>>>> Hive and have noticed a potential gap in the TableOperations.commit()
>>>>> API. Typically, we use the commit API to revert a table to a previously
>>>>> known state, as demonstrated below:
>>>>>
>>>>> String deisredMetadataPath =
>>>>> "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
>>>>> ops.commit(ops.current(), TableMetadataParser.read(ops.io(),
>>>>> dedeisredMetadataPath));
>>>>>
>>>>> However, this approach is no longer working with the REST-based
>>>>> catalog. I suspect that the issue may be related to how the update type is
>>>>> modeled in RESTTableOperations. I have shared a unit test that reproduces
>>>>> the problem on https://github.com/apache/iceberg/issues/12134, where
>>>>> it works on JDBC and in-memory catalogs, but not with RESTCatalog.
>>>>>
>>>>> Best Regards,
>>>>> Steve Zhang
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by Yufei Gu <fl...@gmail.com>.
The push-based mirroring highlighted by Ryan is a popular use case. Polaris
has already implemented notification APIs to address this, and there have
been several Iceberg community discussions surrounding it. If I recall
correctly, we generally agreed that the notification API is beneficial per
last discussion. Polaris supports various notification types(
https://github.com/apache/polaris/blob/be343f1b484d2d79877978ff55f3ad36031dc4a5/spec/polaris-catalog-apis/notifications-api.yaml#L108).
The "UPDATE" type provides the same functionality as "register table
force". I believe using notification APIs, or a similar concept is the
right approach for achieving push-based mirroring, as "register table
force" on its own isn't sufficient. Here is a list of notification types
used in Polaris for reference:
- CREATE
- UPDATE
- DROP
- VALIDATE
Yufei
On Mon, Feb 10, 2025 at 4:49 PM Steve Zhang <ho...@apple.com.invalid>
wrote:
> Thank you Russell and Ryan.
>
> Let me start to work on a new API to support force table registration in
> catalog.
>
> Thanks,
> Steve Zhang
>
>
>
> On Feb 10, 2025, at 4:29 PM, rdblue@gmail.com wrote:
>
> Yeah, it sounds like a "register table force" is the right concept here. I
> think we want to make sure that table updates remain change-based as
> the best practice in the REST API. But there are some irregular use cases
> that justify having some mechanism to completely replace the state (like
> push-based mirroring). I think it makes sense to revisit mirroring and this
> use case and come up with a path forward.
>
> On Mon, Feb 10, 2025 at 3:12 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> I still would like a "register table" force" option
>>
>> On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang
>> <ho...@apple.com.invalid> wrote:
>>
>>> Thank you Dan for your detailed reply. Based on your explanation, do you
>>> think it would be worthwhile to support non-linear or complete metadata
>>> replacements in the REST implementation? I am happy to contribute but might
>>> need some guidance from the community on the best approach.
>>>
>>> For additional context, we explored into the workaround of using a
>>> combination of dropping table and re-registering the table with concerns of
>>> reading in between. There’s also an attempt to add a force option to the
>>> register-table API (https://github.com/apache/iceberg/pull/5327), which
>>> would allow for metadata swap on an existing table. However, it was
>>> suggested that use TableOperations.commit(base, new) is preferred to
>>> achieve atomicity.
>>>
>>> Thanks,
>>> Steve Zhang
>>>
>>>
>>>
>>> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dw...@apache.org> wrote:
>>>
>>> Hey Steve,
>>>
>>> I think the issue here is that you're using the commit api in table
>>> operations to perform a non-incremental/linear change to the metadata. The
>>> REST implementation is a little more strict in that it builds a set of
>>> updates based on the mutations made to the metadata and the commit process
>>> applies those changes. In this scenario, no changes have been made and the
>>> call is attempting a complete replacement.
>>>
>>> The other implementations are just blindly swapping the location, so
>>> while that operation does achieve the effect you're looking for, it's not
>>> the right semantics for the commit.
>>>
>>> You might want to consider using the "register table" operation instead,
>>> which takes the table identifier and location to perform this type of swap.
>>>
>>> -Dan
>>>
>>> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang
>>> <ho...@apple.com.invalid> wrote:
>>>
>>>> Hey Iceberg Experts:
>>>>
>>>> I am seeking assistance and insights regarding an issue we’ve
>>>> encountered with RESTTableOperations and its inability to support on-demand
>>>> table metadata swaps. We are currently adopting the REST-based catalog from
>>>> Hive and have noticed a potential gap in the TableOperations.commit()
>>>> API. Typically, we use the commit API to revert a table to a previously
>>>> known state, as demonstrated below:
>>>>
>>>> String deisredMetadataPath =
>>>> "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
>>>> ops.commit(ops.current(), TableMetadataParser.read(ops.io(),
>>>> dedeisredMetadataPath));
>>>>
>>>> However, this approach is no longer working with the REST-based
>>>> catalog. I suspect that the issue may be related to how the update type is
>>>> modeled in RESTTableOperations. I have shared a unit test that reproduces
>>>> the problem on https://github.com/apache/iceberg/issues/12134, where
>>>> it works on JDBC and in-memory catalogs, but not with RESTCatalog.
>>>>
>>>> Best Regards,
>>>> Steve Zhang
>>>>
>>>>
>>>>
>>>>
>>>
>
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by Steve Zhang <ho...@apple.com.INVALID>.
Thank you Russell and Ryan.
Let me start to work on a new API to support force table registration in catalog.
Thanks,
Steve Zhang
> On Feb 10, 2025, at 4:29 PM, rdblue@gmail.com wrote:
>
> Yeah, it sounds like a "register table force" is the right concept here. I think we want to make sure that table updates remain change-based as the best practice in the REST API. But there are some irregular use cases that justify having some mechanism to completely replace the state (like push-based mirroring). I think it makes sense to revisit mirroring and this use case and come up with a path forward.
>
> On Mon, Feb 10, 2025 at 3:12 PM Russell Spitzer <russell.spitzer@gmail.com <ma...@gmail.com>> wrote:
>> I still would like a "register table" force" option
>>
>> On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang <ho...@apple.com.invalid> wrote:
>>> Thank you Dan for your detailed reply. Based on your explanation, do you think it would be worthwhile to support non-linear or complete metadata replacements in the REST implementation? I am happy to contribute but might need some guidance from the community on the best approach.
>>>
>>> For additional context, we explored into the workaround of using a combination of dropping table and re-registering the table with concerns of reading in between. There’s also an attempt to add a force option to the register-table API (https://github.com/apache/iceberg/pull/5327), which would allow for metadata swap on an existing table. However, it was suggested that use TableOperations.commit(base, new) is preferred to achieve atomicity.
>>>
>>> Thanks,
>>> Steve Zhang
>>>
>>>
>>>
>>>> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dweeks@apache.org <ma...@apache.org>> wrote:
>>>>
>>>> Hey Steve,
>>>>
>>>> I think the issue here is that you're using the commit api in table operations to perform a non-incremental/linear change to the metadata. The REST implementation is a little more strict in that it builds a set of updates based on the mutations made to the metadata and the commit process applies those changes. In this scenario, no changes have been made and the call is attempting a complete replacement.
>>>>
>>>> The other implementations are just blindly swapping the location, so while that operation does achieve the effect you're looking for, it's not the right semantics for the commit.
>>>>
>>>> You might want to consider using the "register table" operation instead, which takes the table identifier and location to perform this type of swap.
>>>>
>>>> -Dan
>>>>
>>>> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang <ho...@apple.com.invalid> wrote:
>>>>> Hey Iceberg Experts:
>>>>>
>>>>> I am seeking assistance and insights regarding an issue we’ve encountered with RESTTableOperations and its inability to support on-demand table metadata swaps. We are currently adopting the REST-based catalog from Hive and have noticed a potential gap in the TableOperations.commit() API. Typically, we use the commit API to revert a table to a previously known state, as demonstrated below:
>>>>>
>>>>> String deisredMetadataPath = "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
>>>>> ops.commit(ops.current(), TableMetadataParser.read(ops.io <http://ops.io/>(), dedeisredMetadataPath));
>>>>>
>>>>> However, this approach is no longer working with the REST-based catalog. I suspect that the issue may be related to how the update type is modeled in RESTTableOperations. I have shared a unit test that reproduces the problem on https://github.com/apache/iceberg/issues/12134, where it works on JDBC and in-memory catalogs, but not with RESTCatalog.
>>>>>
>>>>> Best Regards,
>>>>> Steve Zhang
>>>>>
>>>>>
>>>>>
>>>
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by "rdblue@gmail.com" <rd...@gmail.com>.
Yeah, it sounds like a "register table force" is the right concept here. I
think we want to make sure that table updates remain change-based as
the best practice in the REST API. But there are some irregular use cases
that justify having some mechanism to completely replace the state (like
push-based mirroring). I think it makes sense to revisit mirroring and this
use case and come up with a path forward.
On Mon, Feb 10, 2025 at 3:12 PM Russell Spitzer <ru...@gmail.com>
wrote:
> I still would like a "register table" force" option
>
> On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang
> <ho...@apple.com.invalid> wrote:
>
>> Thank you Dan for your detailed reply. Based on your explanation, do you
>> think it would be worthwhile to support non-linear or complete metadata
>> replacements in the REST implementation? I am happy to contribute but might
>> need some guidance from the community on the best approach.
>>
>> For additional context, we explored into the workaround of using a
>> combination of dropping table and re-registering the table with concerns of
>> reading in between. There’s also an attempt to add a force option to the
>> register-table API (https://github.com/apache/iceberg/pull/5327), which
>> would allow for metadata swap on an existing table. However, it was
>> suggested that use TableOperations.commit(base, new) is preferred to
>> achieve atomicity.
>>
>> Thanks,
>> Steve Zhang
>>
>>
>>
>> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dw...@apache.org> wrote:
>>
>> Hey Steve,
>>
>> I think the issue here is that you're using the commit api in table
>> operations to perform a non-incremental/linear change to the metadata. The
>> REST implementation is a little more strict in that it builds a set of
>> updates based on the mutations made to the metadata and the commit process
>> applies those changes. In this scenario, no changes have been made and the
>> call is attempting a complete replacement.
>>
>> The other implementations are just blindly swapping the location, so
>> while that operation does achieve the effect you're looking for, it's not
>> the right semantics for the commit.
>>
>> You might want to consider using the "register table" operation instead,
>> which takes the table identifier and location to perform this type of swap.
>>
>> -Dan
>>
>> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang
>> <ho...@apple.com.invalid> wrote:
>>
>>> Hey Iceberg Experts:
>>>
>>> I am seeking assistance and insights regarding an issue we’ve
>>> encountered with RESTTableOperations and its inability to support on-demand
>>> table metadata swaps. We are currently adopting the REST-based catalog from
>>> Hive and have noticed a potential gap in the TableOperations.commit()
>>> API. Typically, we use the commit API to revert a table to a previously
>>> known state, as demonstrated below:
>>>
>>> String deisredMetadataPath =
>>> "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
>>> ops.commit(ops.current(), TableMetadataParser.read(ops.io(),
>>> dedeisredMetadataPath));
>>>
>>> However, this approach is no longer working with the REST-based
>>> catalog. I suspect that the issue may be related to how the update type is
>>> modeled in RESTTableOperations. I have shared a unit test that reproduces
>>> the problem on https://github.com/apache/iceberg/issues/12134, where it
>>> works on JDBC and in-memory catalogs, but not with RESTCatalog.
>>>
>>> Best Regards,
>>> Steve Zhang
>>>
>>>
>>>
>>>
>>
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by Russell Spitzer <ru...@gmail.com>.
I still would like a "register table" force" option
On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang <ho...@apple.com.invalid>
wrote:
> Thank you Dan for your detailed reply. Based on your explanation, do you
> think it would be worthwhile to support non-linear or complete metadata
> replacements in the REST implementation? I am happy to contribute but might
> need some guidance from the community on the best approach.
>
> For additional context, we explored into the workaround of using a
> combination of dropping table and re-registering the table with concerns of
> reading in between. There’s also an attempt to add a force option to the
> register-table API (https://github.com/apache/iceberg/pull/5327), which
> would allow for metadata swap on an existing table. However, it was
> suggested that use TableOperations.commit(base, new) is preferred to
> achieve atomicity.
>
> Thanks,
> Steve Zhang
>
>
>
> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dw...@apache.org> wrote:
>
> Hey Steve,
>
> I think the issue here is that you're using the commit api in table
> operations to perform a non-incremental/linear change to the metadata. The
> REST implementation is a little more strict in that it builds a set of
> updates based on the mutations made to the metadata and the commit process
> applies those changes. In this scenario, no changes have been made and the
> call is attempting a complete replacement.
>
> The other implementations are just blindly swapping the location, so while
> that operation does achieve the effect you're looking for, it's not the
> right semantics for the commit.
>
> You might want to consider using the "register table" operation instead,
> which takes the table identifier and location to perform this type of swap.
>
> -Dan
>
> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang
> <ho...@apple.com.invalid> wrote:
>
>> Hey Iceberg Experts:
>>
>> I am seeking assistance and insights regarding an issue we’ve
>> encountered with RESTTableOperations and its inability to support on-demand
>> table metadata swaps. We are currently adopting the REST-based catalog from
>> Hive and have noticed a potential gap in the TableOperations.commit()
>> API. Typically, we use the commit API to revert a table to a previously
>> known state, as demonstrated below:
>>
>> String deisredMetadataPath =
>> "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
>> ops.commit(ops.current(), TableMetadataParser.read(ops.io(),
>> dedeisredMetadataPath));
>>
>> However, this approach is no longer working with the REST-based
>> catalog. I suspect that the issue may be related to how the update type is
>> modeled in RESTTableOperations. I have shared a unit test that reproduces
>> the problem on https://github.com/apache/iceberg/issues/12134, where it
>> works on JDBC and in-memory catalogs, but not with RESTCatalog.
>>
>> Best Regards,
>> Steve Zhang
>>
>>
>>
>>
>
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by Steve Zhang <ho...@apple.com.INVALID>.
Thank you Dan for your detailed reply. Based on your explanation, do you think it would be worthwhile to support non-linear or complete metadata replacements in the REST implementation? I am happy to contribute but might need some guidance from the community on the best approach.
For additional context, we explored into the workaround of using a combination of dropping table and re-registering the table with concerns of reading in between. There’s also an attempt to add a force option to the register-table API (https://github.com/apache/iceberg/pull/5327), which would allow for metadata swap on an existing table. However, it was suggested that use TableOperations.commit(base, new) is preferred to achieve atomicity.
Thanks,
Steve Zhang
> On Feb 10, 2025, at 1:49 PM, Daniel Weeks <dw...@apache.org> wrote:
>
> Hey Steve,
>
> I think the issue here is that you're using the commit api in table operations to perform a non-incremental/linear change to the metadata. The REST implementation is a little more strict in that it builds a set of updates based on the mutations made to the metadata and the commit process applies those changes. In this scenario, no changes have been made and the call is attempting a complete replacement.
>
> The other implementations are just blindly swapping the location, so while that operation does achieve the effect you're looking for, it's not the right semantics for the commit.
>
> You might want to consider using the "register table" operation instead, which takes the table identifier and location to perform this type of swap.
>
> -Dan
>
> On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang <ho...@apple.com.invalid> wrote:
>> Hey Iceberg Experts:
>>
>> I am seeking assistance and insights regarding an issue we’ve encountered with RESTTableOperations and its inability to support on-demand table metadata swaps. We are currently adopting the REST-based catalog from Hive and have noticed a potential gap in the TableOperations.commit() API. Typically, we use the commit API to revert a table to a previously known state, as demonstrated below:
>>
>> String deisredMetadataPath = "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
>> ops.commit(ops.current(), TableMetadataParser.read(ops.io <http://ops.io/>(), dedeisredMetadataPath));
>>
>> However, this approach is no longer working with the REST-based catalog. I suspect that the issue may be related to how the update type is modeled in RESTTableOperations. I have shared a unit test that reproduces the problem on https://github.com/apache/iceberg/issues/12134, where it works on JDBC and in-memory catalogs, but not with RESTCatalog.
>>
>> Best Regards,
>> Steve Zhang
>>
>>
>>
Re: Table metadata swap not work for REST Catalog (#12134)
Posted by Daniel Weeks <dw...@apache.org>.
Hey Steve,
I think the issue here is that you're using the commit api in table
operations to perform a non-incremental/linear change to the metadata. The
REST implementation is a little more strict in that it builds a set of
updates based on the mutations made to the metadata and the commit process
applies those changes. In this scenario, no changes have been made and the
call is attempting a complete replacement.
The other implementations are just blindly swapping the location, so while
that operation does achieve the effect you're looking for, it's not the
right semantics for the commit.
You might want to consider using the "register table" operation instead,
which takes the table identifier and location to perform this type of swap.
-Dan
On Fri, Feb 7, 2025 at 10:17 AM Steve Zhang <ho...@apple.com.invalid>
wrote:
> Hey Iceberg Experts:
>
> I am seeking assistance and insights regarding an issue we’ve
> encountered with RESTTableOperations and its inability to support on-demand
> table metadata swaps. We are currently adopting the REST-based catalog from
> Hive and have noticed a potential gap in the TableOperations.commit()
> API. Typically, we use the commit API to revert a table to a previously
> known state, as demonstrated below:
>
> String deisredMetadataPath =
> "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
> ops.commit(ops.current(), TableMetadataParser.read(ops.io(),
> dedeisredMetadataPath));
>
> However, this approach is no longer working with the REST-based catalog.
> I suspect that the issue may be related to how the update type is modeled
> in RESTTableOperations. I have shared a unit test that reproduces the
> problem on https://github.com/apache/iceberg/issues/12134, where it works
> on JDBC and in-memory catalogs, but not with RESTCatalog.
>
> Best Regards,
> Steve Zhang
>
>
>
>