You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/05/13 15:53:41 UTC
[PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
yadavay-amzn opened a new pull request, #16319:
URL: https://github.com/apache/iceberg/pull/16319
## Summary
Adds ETag/If-None-Match support to `RESTTableOperations.refresh()` and `commit()`, so repeated refresh calls skip re-downloading unchanged table metadata.
Follow-up to #15109 (freshness-aware table loading in REST catalog).
## Changes
When `RESTTableOperations.refresh()` receives an ETag from the server, subsequent `refresh()` calls send `If-None-Match` and return the current metadata on `304 Not Modified`. The ETag is also captured from commit responses so the next refresh benefits immediately.
This reduces bandwidth and server load for repeated refresh calls (retries, transactions, reconciliation) without any API changes or cache coupling.
## Scope
This is the minimal self-contained slice of #15109. Deliberately out of scope:
- FileIO credential refresh (blocked by `final` field, needs design discussion)
- Cache callback to RESTSessionCatalog (adds coupling, can be a follow-up)
- Config/credential updates on commit (REST spec does not include them)
## Testing
5 new tests in `TestRESTTableOperationsFreshness`:
- ETag sent on second refresh
- Current metadata returned on 304
- No header sent without prior ETag
- ETag captured from commit response
- If-None-Match merged with existing read headers
## Generative AI
Generated-by: Claude Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16319:
URL: https://github.com/apache/iceberg/pull/16319#issuecomment-5008807346
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #16319:
URL: https://github.com/apache/iceberg/pull/16319#issuecomment-4479465315
Thank you for the PR @yadavay-amzn !
What makes me slightly uncomfortable is that we now have 2 different ETags. One on the cache level, one on the RESTTableOps level, and in some occasions they can diverge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16319:
URL: https://github.com/apache/iceberg/pull/16319#issuecomment-5075858108
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #16319:
URL: https://github.com/apache/iceberg/pull/16319#issuecomment-4446670817
@gaborkaszab This implements the freshness-aware loading follow-up you suggested in #15109. CI is green. Would appreciate a look when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #16319: Core: Add freshness-aware loading to RESTTableOperations
URL: https://github.com/apache/iceberg/pull/16319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #16319:
URL: https://github.com/apache/iceberg/pull/16319#issuecomment-4480044295
@gaborkaszab Thanks for the review! Good question on the two ETags.
The cache-level ETag (in `RESTCatalog`) gates whether to re-fetch the full table metadata from the server. The ops-level ETag (in `RESTTableOperations`) gates whether `refresh()` needs to re-download metadata it already has.
They serve different purposes:
- Cache ETag: "has the table changed since I last loaded it?" (catalog-level freshness)
- Ops ETag: "has the metadata changed since my last refresh?" (operation-level freshness for retry loops)
They can diverge temporarily (e.g., after a commit updates the ops ETag but before the cache is invalidated), but this is safe because both are optimistic -- a stale ETag just means an extra round-trip, never stale data.
That said, if you feel strongly about unifying them, I could propagate the ops-level ETag back to the cache on refresh. Would that address your concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Add freshness-aware loading to RESTTableOperations [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #16319:
URL: https://github.com/apache/iceberg/pull/16319#issuecomment-4544021822
Thanks for the explanation, @yadavay-amzn ! I understood the same with my first pass on the code for the 2 use-case.
I'm not sure I could argue why to keep ETags on 2 different places, and what is the use-case to maintain a table cache-level freshness-aware loading, and an ops-level freshness-aware loading. If we take a step back and don't look at how this is implemented currently, a user might expect to not do a full table load after downloading the full metadata from the REST server, unless the table changed in the meantime. With this design we might load the changed table twice, once for ops and once for the table cache. E.g.
1) first we load the table to populate the cache.
2) The table changes after this.
3) We do ops `refresh()` that does a full table load.
4) Then we do a catalog.load() that does the same full table load, getting the same ETag as in 3.
Users might rightfully say we shouldn't do a full load in step 4) because we already loaded the latest table to the client in step 3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org