You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gaborkaszab (via GitHub)" <gi...@apache.org> on 2025/11/28 15:29:47 UTC
[PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
gaborkaszab opened a new pull request, #14709:
URL: https://github.com/apache/iceberg/pull/14709
With RESTTableOperations it's not required to perform a refresh after committing a merge append because the commit already refreshes table metadata. Initially an appendFiles().commit() required the following messages sent to REST catalog:
1) Load table at the beginning of SnapshotProducer.apply()
2) Update table to send the updates to the catalog
3) Load table again in MergingSnapshotProducer.updateEvent()
The last step isn't needed when ops is RESTTableOperations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2708816617
##########
core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java:
##########
@@ -4326,6 +4326,23 @@ public void testTableCacheAgeDoesNotRefreshesAfterAccess() {
.isEqualTo(HALF_OF_TABLE_EXPIRATION);
}
+ @Test
+ public void testNumLoadTableCallsForMergeAppend() {
+ RESTCatalogAdapter adapter = Mockito.spy(new RESTCatalogAdapter(backendCatalog));
+
+ RESTCatalog catalog = catalog(adapter);
+
+ catalog.createNamespace(TABLE.namespace());
+
+ BaseTable table = (BaseTable) catalog.createTable(TABLE, SCHEMA);
+
+ table.newAppend().appendFile(FILE_A).commit();
Review Comment:
Before the commit, we may want to extract out the CreateSnapshotEvent and assert it's what we expect, since that's the part we're changing?
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -3271,6 +3271,11 @@ public void testCatalogWithCustomMetricsReporter() throws IOException {
assertThat(CustomMetricsReporter.SCAN_COUNTER.get()).isEqualTo(1);
// reset counter in case subclasses run this test multiple times
CustomMetricsReporter.SCAN_COUNTER.set(0);
+
+ // append file through MergeAppend and check and reset counter
+ table.newAppend().appendFile(FILE_A).commit();
+ assertThat(CustomMetricsReporter.COMMIT_COUNTER.get()).isEqualTo(1);
+ CustomMetricsReporter.COMMIT_COUNTER.set(0);
Review Comment:
Sorry not entirely following, what additional behavior related to this change are we trying test here? This just looks like it verifies the commit is performed and the metrics after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3744552314
Hey @flyrain
Do you think we can wrap this up and merge, unless there are further comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2712025933
##########
core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java:
##########
@@ -4326,6 +4326,23 @@ public void testTableCacheAgeDoesNotRefreshesAfterAccess() {
.isEqualTo(HALF_OF_TABLE_EXPIRATION);
}
+ @Test
+ public void testNumLoadTableCallsForMergeAppend() {
+ RESTCatalogAdapter adapter = Mockito.spy(new RESTCatalogAdapter(backendCatalog));
+
+ RESTCatalog catalog = catalog(adapter);
+
+ catalog.createNamespace(TABLE.namespace());
+
+ BaseTable table = (BaseTable) catalog.createTable(TABLE, SCHEMA);
+
+ table.newAppend().appendFile(FILE_A).commit();
Review Comment:
I added verification of the outgoing CommitReport on the metrics endpoint
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -3271,6 +3271,11 @@ public void testCatalogWithCustomMetricsReporter() throws IOException {
assertThat(CustomMetricsReporter.SCAN_COUNTER.get()).isEqualTo(1);
// reset counter in case subclasses run this test multiple times
CustomMetricsReporter.SCAN_COUNTER.set(0);
+
+ // append file through MergeAppend and check and reset counter
+ table.newAppend().appendFile(FILE_A).commit();
+ assertThat(CustomMetricsReporter.COMMIT_COUNTER.get()).isEqualTo(1);
+ CustomMetricsReporter.COMMIT_COUNTER.set(0);
Review Comment:
I'm not sure either :) Probably I found some test gap here, but I agree this addition is unrelated to the intent of the PR. Removing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3718792708
Hey @flyrain ,
Do you think you can take a look at this and maybe merge if there are no further comments? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3836447811
I'll go ahead and merge, thanks again for the improvement @gaborkaszab ! thanks @flyrain for reviewing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3835244340
Hey @amogh-jahagirdar,
Do you think you can take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3772449449
Resolved merge conflict.
@flyrain do you think you can take a quick look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2605941782
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
@Override
public Object updateEvent() {
long snapshotId = snapshotId();
- Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+ Snapshot justSaved = ops().current().snapshot(snapshotId);
Review Comment:
That is the other approach I was considering. I didn't want (Merging)SnapshotProducer to know about what is the underlying TableOps implementation and I preferred this more general approach. Basically, instead of asking "is this REST ops?" I ask "do we have this snapshot" that seemed more robust and general.
WDYT @flyrain ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3836557209
Thanks for the reviews @amogh-jahagirdar @flyrain !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3631536368
@flyrain @amogh-jahagirdar @nastra Would you mind taking a look at this improvement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2619984035
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
@Override
public Object updateEvent() {
long snapshotId = snapshotId();
- Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+ Snapshot justSaved = ops().current().snapshot(snapshotId);
Review Comment:
Thanks for pointing that out, @flyrain ! I updated the description to be a bit more generic, and use RESTTableOperations as an example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#issuecomment-3595503224
Hey @flyrain @amogh-jahagirdar @nastra ,
You might be interested in this change (I found you active within this area) where I found that when I do `table.newAppend().appendFile(...).commit()` then there is an excess load table request going to the REST catalog. In case I don't miss anything, I think we can skip this step because RESTTableOperations.commit() already refreshed table metadata.
Note, I also see an unconditional table refresh in `CherryPickOperation.updateEvent()` but I didn't have the time to investigate that one further. In case there is support for the current enhancement, I can take a look at the cherry pick one in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #14709:
URL: https://github.com/apache/iceberg/pull/14709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2604900299
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
@Override
public Object updateEvent() {
long snapshotId = snapshotId();
- Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+ Snapshot justSaved = ops().current().snapshot(snapshotId);
Review Comment:
Thanks a lot for the changes, @gaborkaszab ! Should we check if `ops` is an instance of `RESTTableOperations` to ensure this is REST operation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Skip unnecessary metadata refresh after merge append [iceberg]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #14709:
URL: https://github.com/apache/iceberg/pull/14709#discussion_r2616604010
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -974,7 +974,12 @@ public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
@Override
public Object updateEvent() {
long snapshotId = snapshotId();
- Snapshot justSaved = ops().refresh().snapshot(snapshotId);
+
+ Snapshot justSaved = ops().current().snapshot(snapshotId);
Review Comment:
Thanks for the explanation. To be clear, I'm all for a generic way instead of dedicating to one type of table operation. My confusion and concern came from the PR description as the following:
> With RESTTableOperations it's not required to perform a refresh after committing a merge append because the commit already refreshes table metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org