You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dramaticlly (via GitHub)" <gi...@apache.org> on 2025/01/30 00:54:05 UTC

[I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

dramaticlly opened a new issue, #12134:
URL: https://github.com/apache/iceberg/issues/12134

   ### Apache Iceberg version
   
   1.7.1 (latest release)
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   Before migrate to REST catalog, we rely on following `TableOperations.commit` API call to swap table metadata atomically. 
   
   ```java
   String deisredMetadataPath = "/var/newdb/table/metadata/00003-579b23d1-4ca5-4acf-85ec-081e1699cb83.metadata.json""
   tblOps.commit(ops.current(), TableMetadataParser.read(originalTable.io(), dedeisredMetadataPath));
   ```
   
   However this is no longer working in REST based catalog, I suspect it might relate to [how update type was modeled here](https://github.com/apache/iceberg/blob/a2b8008da7bc26e03248a35eeee60d1cc7e8499d/core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java#L116) where `metadata.changes()` return empty when read from parser and end up with empty changeset in update table POST call. 
   
   This can be reproduced by adding following test case in  org.apache.iceberg.catalog.CatalogTests.java where all other catalogs are functioning as expected but only failure for TestRESTCatalog
   
   <img width="640" alt="Image" src="https://github.com/user-attachments/assets/bde2d284-18e9-415f-b544-0e4f921ec47a" />
   
   repro: 
   ```java
     @Test
     public void testTableOperationCommit() {
       C catalog = catalog();
   
       if (requiresNamespaceCreate()) {
         catalog.createNamespace(TABLE.namespace());
       }
   
       Map<String, String> properties =
           ImmutableMap.of("user", "someone", "created-at", "2023-01-15T00:00:01");
       Table originalTable =
           catalog
               .buildTable(TABLE, SCHEMA)
               .withPartitionSpec(SPEC)
               .withSortOrder(WRITE_ORDER)
               .withProperties(properties)
               .create();
       TableOperations ops = ((BaseTable) originalTable).operations();
       String original = ops.current().metadataFileLocation();
       FileIO io = ops.io();
   
       originalTable.newFastAppend().appendFile(FILE_A).commit();
       originalTable.newFastAppend().appendFile(FILE_B).commit();
   
       String metadataLocation = ops.refresh().metadataFileLocation();
       System.out.printf("After write, metadata location is:" + metadataLocation);
   
       ops.commit(ops.refresh(), TableMetadataParser.read(originalTable.io(), original));
   
       originalTable.refresh();
       TableMetadata actual = ((BaseTable) originalTable).operations().current();
       TableMetadata expected = new StaticTableOperations(original, io).current();
   
       assertThat(actual.properties())
           .as("Props must match")
           .containsAllEntriesOf(expected.properties());
       assertThat(actual.schema().asStruct())
           .as("Schema must match")
           .isEqualTo(expected.schema().asStruct());
       assertThat(actual.specs()).as("Specs must match").isEqualTo(expected.specs());
       assertThat(actual.sortOrders()).as("Sort orders must match").isEqualTo(expected.sortOrders());
       assertThat(actual.currentSnapshot())
           .as("Current snapshot must match")
           .isEqualTo(expected.currentSnapshot());
       assertThat(actual.snapshots()).as("Snapshots must match").isEqualTo(expected.snapshots());
       assertThat(actual.snapshotLog()).as("History must match").isEqualTo(expected.snapshotLog());
   
       TestHelpers.assertSameSchemaMap(actual.schemasById(), expected.schemasById());
   
       assertThat(actual)
           .isEqualToIgnoringGivenFields(
               expected,
               "metadataFileLocation",
               "schemas",
               "specs",
               "sortOrders",
               "properties",
               "schemasById",
               "specsById",
               "sortOrdersById",
               "snapshots");
     }
   ```
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on issue #12134:
URL: https://github.com/apache/iceberg/issues/12134#issuecomment-2649320278

   I added this on the mailing list but thought I'd drop it here as well:
   
   I think the issue here is that you're using the commit api in table operations to perform a non-incremental/linear change to the metadata.  The REST implementation is a little more strict in that it builds a set of updates based on the mutations made to the metadata and the commit process applies those changes.  In this scenario, no changes have been made and the call is attempting a complete replacement.
   
   The other implementations are just blindly swapping the location, so while that operation does achieve the effect you're looking for, it's not the right semantics for the commit.
   
   You might want to consider using the "register table" operation instead, which takes the table identifier and location to perform this type of swap.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #12134:
URL: https://github.com/apache/iceberg/issues/12134#issuecomment-2623289459

   @rdblue @nastra appreciate your insight on this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #12134:
URL: https://github.com/apache/iceberg/issues/12134#issuecomment-2649328210

   > I added this on the mailing list but thought I'd drop it here as well:
   > 
   > I think the issue here is that you're using the commit api in table operations to perform a non-incremental/linear change to the metadata. The REST implementation is a little more strict in that it builds a set of updates based on the mutations made to the metadata and the commit process applies those changes. In this scenario, no changes have been made and the call is attempting a complete replacement.
   > 
   > The other implementations are just blindly swapping the location, so while that operation does achieve the effect you're looking for, it's not the right semantics for the commit.
   > 
   > You might want to consider using the "register table" operation instead, which takes the table identifier and location to perform this type of swap.
   
   Thank you @danielcweeks for the detailed reply. I actually trying to rely on this as a replacement for "drop + register table" operation as TableOperations.commit can be done atomically suggested by @rdblue way back in https://github.com/apache/iceberg/pull/5327#issuecomment-1207478697. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12134:
URL: https://github.com/apache/iceberg/issues/12134#issuecomment-3941994708

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12134: RESTTableOperations does not support table metadata swap like others TableOperations did
URL: https://github.com/apache/iceberg/issues/12134


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12134:
URL: https://github.com/apache/iceberg/issues/12134#issuecomment-3172256652

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #12134:
URL: https://github.com/apache/iceberg/issues/12134#issuecomment-3176468047

   not stale


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] RESTTableOperations does not support table metadata swap like others TableOperations did [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12134:
URL: https://github.com/apache/iceberg/issues/12134#issuecomment-3865826546

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org