You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gaborkaszab (via GitHub)" <gi...@apache.org> on 2025/11/28 15:29:47 UTC

[PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

gaborkaszab opened a new pull request, #14709:
URL: https://github.com/apache/iceberg/pull/14709

   With RESTTableOperations it's not required to perform a refresh after committing a merge append because the commit already refreshes table metadata. Initially an appendFiles().commit() required the following messages sent to REST catalog:
     1) Load table at the beginning of SnapshotProducer.apply()
     2) Update table to send the updates to the catalog
     3) Load table again in MergingSnapshotProducer.updateEvent()
   The last step isn't needed when ops is RESTTableOperations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2708816617


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java:
##########
@@ -4326,6 +4326,23 @@ public void testTableCacheAgeDoesNotRefreshesAfterAccess() {
         .isEqualTo(HALF_OF_TABLE_EXPIRATION);
   }
 
+  @Test
+  public void testNumLoadTableCallsForMergeAppend() {
+    RESTCatalogAdapter adapter = Mockito.spy(new RESTCatalogAdapter(backendCatalog));
+
+    RESTCatalog catalog = catalog(adapter);
+
+    catalog.createNamespace(TABLE.namespace());
+
+    BaseTable table = (BaseTable) catalog.createTable(TABLE, SCHEMA);
+
+    table.newAppend().appendFile(FILE_A).commit();

Review Comment:
   Before the commit, we may want to extract out the CreateSnapshotEvent and assert it's what we expect, since that's the part we're changing?



##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -3271,6 +3271,11 @@ public void testCatalogWithCustomMetricsReporter() throws IOException {
     assertThat(CustomMetricsReporter.SCAN_COUNTER.get()).isEqualTo(1);
     // reset counter in case subclasses run this test multiple times
     CustomMetricsReporter.SCAN_COUNTER.set(0);
+
+    // append file through MergeAppend and check and reset counter
+    table.newAppend().appendFile(FILE_A).commit();
+    assertThat(CustomMetricsReporter.COMMIT_COUNTER.get()).isEqualTo(1);
+    CustomMetricsReporter.COMMIT_COUNTER.set(0);

Review Comment:
   Sorry not entirely following, what additional behavior related to this change are we trying test here? This just looks like it verifies the commit is performed and the metrics after that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3744552314

   Hey @flyrain 
   Do you think we can wrap this up and merge, unless there are further comments?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2712025933


##########
core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java:
##########
@@ -4326,6 +4326,23 @@ public void testTableCacheAgeDoesNotRefreshesAfterAccess() {
         .isEqualTo(HALF_OF_TABLE_EXPIRATION);
   }
 
+  @Test
+  public void testNumLoadTableCallsForMergeAppend() {
+    RESTCatalogAdapter adapter = Mockito.spy(new RESTCatalogAdapter(backendCatalog));
+
+    RESTCatalog catalog = catalog(adapter);
+
+    catalog.createNamespace(TABLE.namespace());
+
+    BaseTable table = (BaseTable) catalog.createTable(TABLE, SCHEMA);
+
+    table.newAppend().appendFile(FILE_A).commit();

Review Comment:
   I added verification of the outgoing CommitReport on the metrics endpoint



##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -3271,6 +3271,11 @@ public void testCatalogWithCustomMetricsReporter() throws IOException {
     assertThat(CustomMetricsReporter.SCAN_COUNTER.get()).isEqualTo(1);
     // reset counter in case subclasses run this test multiple times
     CustomMetricsReporter.SCAN_COUNTER.set(0);
+
+    // append file through MergeAppend and check and reset counter
+    table.newAppend().appendFile(FILE_A).commit();
+    assertThat(CustomMetricsReporter.COMMIT_COUNTER.get()).isEqualTo(1);
+    CustomMetricsReporter.COMMIT_COUNTER.set(0);

Review Comment:
   I'm not sure either :) Probably I found some test gap here, but I agree this addition is unrelated to the intent of the PR. Removing this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3718792708

   Hey @flyrain ,
   Do you think you can take a look at this and maybe merge if there are no further comments? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3836447811

   I'll go ahead and merge, thanks again for the improvement @gaborkaszab ! thanks @flyrain for reviewing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3835244340

   Hey @amogh-jahagirdar,
   Do you think you can take another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3772449449

   Resolved merge conflict.
   @flyrain do you think you can take a quick look at this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2605941782


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
   @Override
   public Object updateEvent() {
     long snapshotId = snapshotId();
-    Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+    Snapshot justSaved = ops().current().snapshot(snapshotId);

Review Comment:
   That is the other approach I was considering. I didn't want (Merging)SnapshotProducer to know about what is the underlying TableOps implementation and I preferred this more general approach. Basically, instead of asking "is this REST ops?" I ask "do we have this snapshot" that seemed more robust and general.
   WDYT @flyrain ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3836557209

   Thanks for the reviews @amogh-jahagirdar @flyrain !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3631536368

   @flyrain @amogh-jahagirdar @nastra Would you mind taking a look at this improvement?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2619984035


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
   @Override
   public Object updateEvent() {
     long snapshotId = snapshotId();
-    Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+    Snapshot justSaved = ops().current().snapshot(snapshotId);

Review Comment:
   Thanks for pointing that out, @flyrain ! I updated the description to be a bit more generic, and use RESTTableOperations as an example.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3595503224

   Hey @flyrain @amogh-jahagirdar @nastra ,
   You might be interested in this change (I found you active within this area) where I found that when I do `table.newAppend().appendFile(...).commit()` then there is an excess load table request going to the REST catalog. In case I don't miss anything, I think we can skip this step because RESTTableOperations.commit() already refreshed table metadata.
   
   Note, I also see an unconditional table refresh in `CherryPickOperation.updateEvent()` but I didn't have the time to investigate that one further. In case there is support for the current enhancement, I can take a look at the cherry pick one in a separate PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #14709:
URL: https://github.com/apache/iceberg/pull/14709


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2604900299


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
   @Override
   public Object updateEvent() {
     long snapshotId = snapshotId();
-    Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+    Snapshot justSaved = ops().current().snapshot(snapshotId);

Review Comment:
   Thanks a lot for the changes, @gaborkaszab ! Should we check if `ops` is an instance of `RESTTableOperations` to ensure this is REST operation? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2616604010


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
   @Override
   public Object updateEvent() {
     long snapshotId = snapshotId();
-    Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+    Snapshot justSaved = ops().current().snapshot(snapshotId);

Review Comment:
   Thanks for the explanation. To be clear, I'm all for a generic way instead of dedicating to one type of table operation. My confusion and concern came from the PR description as the following:
   
   > With RESTTableOperations it's not required to perform a refresh after committing a merge append because the commit already refreshes table metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org