[PR] perf(core): cache ROW footer and block index metadata [paimon-rust]

["XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>]

XiaoHongbo-Hope opened a new pull request, #1196:
URL: https://github.com/apache/paimon-rust/pull/1196

   A repeated ROW point read currently fetches and parses the footer and block index on every query against the same immutable file. With a one-block fixture, that makes three range reads per query (footer, index, block).
   
   Cache the parsed footer and block index within the FileIO metadata-cache context. The second read of the same file then needs only the data-block read. The cache is enabled by default when `file-format.metadata-cache.max-size` is nonzero, bounded to 32 MiB and 4,096 entries, keyed by the storage-namespaced immutable file identity and size, and bypassed when the reader has no stable key or after fork. No new option or file-format change is introduced.
   
   This is separate from #1195, which caches the ROW sidecar file size (`stat`); the two optimizations are independent. Both PRs touch the FileIO cache context, so this branch will need a trivial rebase if #1195 merges first.
   
   Validation:
   - Counted ROW reader test: first query reads footer/index/block (3); repeated query reads only block (1); disabled cache and readers without stable keys still perform all reads.
   - `cargo test --locked -p paimon --features storage-memory row_reader_ --lib` (7 passed)
   - `cargo test --locked -p paimon --features storage-memory row_metadata_cache_has_separate_type_and_bypasses_inherited_fork_state --lib` (1 passed)
   - `cargo test --locked -p paimon --test data_evolution_row_sidecar_test` (14 passed)
   - `cargo clippy --locked -p paimon --lib --features storage-memory -- -D warnings -A clippy::nonminimal_bool` (passed; the suppressed lint is present in unrelated main-branch files)
   - `cargo fmt --all`; `git diff --check`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf(core): cache ROW sidecar size and reader metadata [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #1196:
URL: https://github.com/apache/paimon-rust/pull/1196


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] perf(core): cache ROW footer and block index metadata [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on code in PR #1196:
URL: https://github.com/apache/paimon-rust/pull/1196#discussion_r4344980781


##########
crates/paimon/src/io/file_io.rs:
##########
@@ -167,6 +172,28 @@ impl FileFormatMetadataCacheContext {
             Err(_) => panic!("file-format metadata cache type mismatch"),
         }
     }
+
+    /// ROW footer and block index metadata has its own typed cache. Keep it
+    /// separate from the Parquet cache, which also uses `get_or_init`.
+    pub(crate) fn row_metadata_get_or_init<T>(
+        &self,
+        init: impl FnOnce(usize) -> T,
+    ) -> Option<Arc<T>>
+    where
+        T: Send + Sync + 'static,
+    {
+        if self.max_bytes == 0 || self.process_id != std::process::id() {
+            return None;
+        }
+        let cache = Arc::clone(self.row_metadata_cache.get_or_init(|| {

Review Comment:
   [P2] Keep ROW and Parquet within the configured metadata-cache budget
   
   This creates an additional cache with min(max_bytes, 32 MiB), while get_or_init still gives Parquet the full max_bytes. The existing file-format.metadata-cache.max-size option promises the maximum parsed file-format metadata cache size. A Data Evolution table can warm both pools through ordinary Parquet scans and sparse ROW sidecar reads in the same FileIO, so the setting no longer bounds their combined retention. A local probe with one FileIO configured to 64 KiB receives 64 KiB for each pool; at the default, their budgets total 82 MiB against the configured 50 MiB. Please share accounting or divide the configured budget between formats; an intentionally additive ROW budget needs its own explicit setting.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org