You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dentiny (via GitHub)" <gi...@apache.org> on 2025/10/01 00:48:41 UTC
[PR] feat: [1/N] Add write-through cache for manifest list [iceberg-rust]
dentiny opened a new pull request, #1719:
URL: https://github.com/apache/iceberg-rust/pull/1719
## Which issue does this PR close?
- Closes https://github.com/apache/iceberg-rust/pull/1698
## What changes are included in this PR?
Context: I see huge CPU time spent on manifest list loading, especially avro deserialization (see attached PR for details), I want to leverage the object cache to avoid unnecessary IO and deser.
Discussed online with @liurenjie1024 for a bit, see
- https://github.com/apache/iceberg-rust/pull/1698#issuecomment-3332906860
- https://github.com/apache/iceberg-rust/pull/512#discussion_r2365940293
we lean towards the path that:
- Make object cache a read-through and write-through cache for manifest and manifest list
- Later loading attempts from object cache first, could be either a read-through cache, or look-aside for easier implementation
I plan to structure and split the series of PRs as follows:
- [ ] Store manifest list into object cache, if cache enabled
- [ ] Load manifest list with object cache considered, which makes object store a part of file io
- [ ] Replicate the same procedure to manifest files
## Are these changes tested?
Yes, unit test added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: [1/N] Add write-through cache for manifest list [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1719:
URL: https://github.com/apache/iceberg-rust/pull/1719#issuecomment-4008676937
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: [1/N] Add write-through cache for manifest list [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #1719: feat: [1/N] Add write-through cache for manifest list
URL: https://github.com/apache/iceberg-rust/pull/1719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: [1/N] Add write-through cache for manifest list [iceberg-rust]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #1719:
URL: https://github.com/apache/iceberg-rust/pull/1719#discussion_r2524593104
##########
crates/iceberg/src/io/object_cache.rs:
##########
@@ -118,6 +123,20 @@ impl ObjectCache {
}
}
+ /// Set a [`ManifestList`] to the cache, if duplicate key already exist, new value will be ignored.
+ /// No-op if cache disabled.
+ pub(crate) async fn set_manifest_list(
Review Comment:
```suggestion
pub(crate) async fn cache_manifest_list(
```
nit: name bike-shedding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org