You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wangyong9999 (via GitHub)" <gi...@apache.org> on 2026/08/31 10:26:38 UTC

[PR] perf(scan): reuse index planning resources across queries [paimon-cpp]

wangyong9999 opened a new pull request, #267:
URL: https://github.com/apache/paimon-cpp/pull/267

   ### Purpose
   
   Linked issue: N/A
   
   Repeated scans in the same process rebuilt several immutable planning resources:
   
   - `DataEvolutionBatchScan` loaded the schema and rebuilt `CoreOptions` after `TableScan` had already initialized them. This also dropped the cache supplied through `ScanContext`.
   - snapshot live manifest entries were cached only when a bucket filter was present.
   - each BTree indexer created a new `CacheManager`; closing a reader invalidated its pages, and the next reader opened the index file before checking the cache.
   
   **This change reuses the `TableSchema`, `CoreOptions`, filesystem, executor, memory pool, and cache already owned by the scan when planning a DataEvolution global-index query.**
   
   **It adds a whole-table snapshot live manifest cache entry, separate from bucket-specific entries. The cache stores live entries before query filtering and applies the current predicate after a cache hit.**
   
   **It shares a bounded BTree `CacheManager` between indexers with the same cache configuration, retains immutable pages after reader destruction, and opens the input stream only on a cache miss. Retained pages use the process default memory pool so they do not outlive a caller-owned pool.**
   
   The manifest cache remains disabled for row-range scans. Cache read, deserialization, serialization, or write failures fall back to rebuilding the manifest entries. Cache-manager creation is serialized; the existing LRU remains thread-safe and bounded by `btree-index.cache-size` for each distinct cache configuration. The existing eager `BlockCache` constructor and `Close()` behavior are unchanged.
   
   On an HDFS primary-key table with 606K rows, 51 active data files, and 6.5 GB of data, a `logid` lookup returning two rows was executed twice in one process. The cold plan/read/total time was 851/163/1015 ms. The second plan/read/total time was 115/99/214 ms. BTree evaluation decreased from 621 ms to 0.55 ms. Both runs selected one indexed split, one data file, and a two-row range, and matched the result of an index-disabled scan.
   
   ### Tests
   
   - debug build: `core_test`, `common_test`, and `sst_format_test`
   - `AppendOnlyFileStoreScanTest.TestSnapshotLiveManifestCache*` (4 cases)
   - `LruCacheTest.TestForSnapshotLiveManifestEntries`
   - `BlockCacheTest.*` (7 cases)
   - `LazyFilteredBTreeReaderTest.*` (33 cases)
   - `*BTree*` (144 passed; 7 existing compatibility-data cases skipped by their fixture guard)
   - two-query native-HDFS validation: `validation status=PASS`, `path_proof status=PASS path=pk-file-local-btree`
   - `git diff --check`
   
   The local environment does not provide CMake or pre-commit executables, so the CMake and pre-commit checks are left to CI.
   
   ### API and Format
   
   Adds an overload of `CacheKey::ForSnapshotLiveManifestEntries` for a whole-table cache key. This is an additive API change. There is no object-layout, storage-format, or protocol change.
   
   ### Documentation
   
   No new user-facing feature or option.
   
   ### Generative AI tooling
   
   Generated-by: Codex (GPT-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(scan): reuse index planning resources across queries [paimon-cpp]

Posted by "wangyong9999 (via GitHub)" <gi...@apache.org>.
wangyong9999 commented on code in PR #267:
URL: https://github.com/apache/paimon-cpp/pull/267#discussion_r3897107230


##########
src/paimon/common/global_index/btree/btree_global_indexer.cpp:
##########
@@ -57,7 +91,8 @@ Result<std::unique_ptr<BTreeGlobalIndexer>> BTreeGlobalIndexer::Create(
         double high_priority_pool_ratio,
         OptionsUtils::GetValueFromMap<double>(options, BtreeDefs::kBtreeIndexHighPriorityPoolRatio,
                                               BtreeDefs::kDefaultBtreeIndexHighPriorityPoolRatio));
-    auto cache_manager = std::make_shared<CacheManager>(cache_size, high_priority_pool_ratio);
+    std::shared_ptr<CacheManager> cache_manager =
+        GetSharedCacheManager(cache_size, high_priority_pool_ratio);

Review Comment:
   Fixed in b35e9c70. Shared page keys now include an internal backend namespace. Filesystem-backed readers share by FileSystem instance; unknown readers are isolated by reader identity, and cached values retain the namespace owner until eviction.



##########
src/paimon/common/global_index/btree/btree_global_indexer.cpp:
##########
@@ -42,9 +45,40 @@
 #include "paimon/executor.h"
 #include "paimon/global_index/bitmap_global_index_result.h"
 #include "paimon/memory/bytes.h"
+#include "paimon/memory/memory_pool.h"
 #include "paimon/utils/roaring_bitmap64.h"
 
 namespace paimon {
+namespace {
+
+struct SharedCacheManager {
+    SharedCacheManager(int64_t cache_size, double high_priority_pool_ratio)
+        : cache_pool(GetDefaultPool()),
+          cache_manager(std::make_shared<CacheManager>(cache_size, high_priority_pool_ratio)) {}
+
+    // Keep the allocator alive until after cache_manager releases all cached pages.
+    std::shared_ptr<MemoryPool> cache_pool;
+    std::shared_ptr<CacheManager> cache_manager;
+};
+
+std::shared_ptr<CacheManager> GetSharedCacheManager(int64_t cache_size,
+                                                    double high_priority_pool_ratio) {
+    using CacheConfig = std::pair<int64_t, double>;
+    static std::mutex mutex;
+    static std::map<CacheConfig, SharedCacheManager> cache_managers;

Review Comment:
   Fixed in 9cd2147b. The process now retains at most four cache configurations and evicts the least recently used registry entry. An evicted manager survives only while an active reader still references it.



##########
src/paimon/core/operation/file_store_scan.cpp:
##########
@@ -150,12 +150,12 @@ Result<std::shared_ptr<FileStoreScan::RawPlan>> FileStoreScan::CreatePlan() cons
         snapshot.has_value() && scan_mode_ == ScanMode::ALL &&
         core_options_.GetScanManifestEntryCacheMaxSnapshots() > 0 &&
         core_options_.GetCache() != nullptr && !table_path_.empty() &&
-        !row_range_index_.has_value() && bucket_filter_.has_value();
+        !row_range_index_.has_value();

Review Comment:
   Fixed in 9cd2147b. Bucket cache entries now store the snapshot generation derived from the base and delta manifest-list names. A hit requires both snapshot ID and generation to match.



##########
src/paimon/core/operation/file_store_scan.cpp:
##########
@@ -347,16 +347,21 @@ Status FileStoreScan::ReadManifestEntriesWithCache(
     }
     *cache_hit = false;
 
-    // Rebuild the target snapshot bucket from all manifests and write the live entries back to the
-    // cache.
-    std::vector<ManifestFileMeta> bucket_manifest_metas;
-    for (const auto& meta : all_manifest_metas) {
-        if (MayContainBucket(meta, bucket)) {
-            bucket_manifest_metas.push_back(meta);
+    if (bucket) {
+        std::vector<ManifestFileMeta> bucket_manifest_metas;
+        for (const auto& meta : all_manifest_metas) {
+            if (MayContainBucket(meta, bucket.value())) {
+                bucket_manifest_metas.push_back(meta);
+            }
         }
+        PAIMON_RETURN_NOT_OK(
+            ReadAndMergeBucketFileEntries(bucket_manifest_metas, bucket.value(), manifest_entries));
+    } else {
+        std::vector<ManifestEntry> unmerged_entries;
+        PAIMON_RETURN_NOT_OK(
+            ReadFileEntries(all_manifest_metas, &unmerged_entries, /*apply_scan_filter=*/false));

Review Comment:
   Fixed in 9cd2147b by removing the no-bucket whole-table manifest cache path. Only bounded bucket-specific entries remain, so an oversized whole-table value cannot trigger repeated materialization and failed insertion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(scan): reuse index planning resources across queries [paimon-cpp]

Posted by "wangyong9999 (via GitHub)" <gi...@apache.org>.
wangyong9999 commented on PR #267:
URL: https://github.com/apache/paimon-cpp/pull/267#issuecomment-5479514849

   The failing integration test shared the process-wide BTree cache with an earlier case, so its file-open count can be zero on a warm cache. I updated the assertion while retaining the indexed-split, bounded-range, selected-row, and read-result checks. The new fork workflows are waiting for maintainer approval.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(scan): reuse index planning resources across queries [paimon-cpp]

Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 commented on PR #267:
URL: https://github.com/apache/paimon-cpp/pull/267#issuecomment-5491358413

   Thanks for working on the repeated Global Index lookup performance.
   
   Some downstream query engines, such as StarRocks, have a similar requirement. Their approach is to create a `GlobalIndexReader` through the public API in `global_index_scan.h` and retain/cache the reader for multiple queries. This already preserves the BTree reader's internal block cache without introducing process-wide cache sharing inside paimon-cpp.
   
   The process-wide design in this PR introduces additional concepts and hidden global state, including the shared `CacheManager` registry, cache configuration LRU, `cache_namespace`, backend lifetime retention, and cross-reader allocator management. It also makes memory accounting and cache isolation harder for downstream engines that already have their own resource management.
   
   Could we instead encourage the query engine to split the query into two explicit stages?
   
   1. **Global Index stage:** create or reuse a `GlobalIndexReader` through `GlobalIndexScan`, evaluate the index predicate, and produce a `GlobalIndexResult`. This stage can be distributed by shard.
   2. **Data Scan stage:** pass the returned `GlobalIndexResult` through `ScanContextBuilder::SetGlobalIndexResult()`, then use `TableScan` to generate the data scan plan and execute it.
   
   The engine can cache readers using a key containing the table identity, snapshot/index generation, field, index type, shard/partition, row range, and relevant options. It should also ensure that the index lookup and subsequent data scan use the same snapshot, and respect `GlobalIndexReader::IsThreadSafe()` when sharing readers.
   
   This keeps cache ownership, eviction, memory accounting, and isolation under the query engine's control while reusing the existing public API. Could we evaluate this approach before adding process-wide cache sharing to paimon-cpp?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(scan): reuse index planning resources across queries [paimon-cpp]

Posted by "wangyong9999 (via GitHub)" <gi...@apache.org>.
wangyong9999 commented on code in PR #267:
URL: https://github.com/apache/paimon-cpp/pull/267#discussion_r3895171422


##########
src/paimon/common/global_index/btree/btree_global_indexer.cpp:
##########
@@ -57,7 +91,8 @@ Result<std::unique_ptr<BTreeGlobalIndexer>> BTreeGlobalIndexer::Create(
         double high_priority_pool_ratio,
         OptionsUtils::GetValueFromMap<double>(options, BtreeDefs::kBtreeIndexHighPriorityPoolRatio,
                                               BtreeDefs::kDefaultBtreeIndexHighPriorityPoolRatio));
-    auto cache_manager = std::make_shared<CacheManager>(cache_size, high_priority_pool_ratio);
+    std::shared_ptr<CacheManager> cache_manager =
+        GetSharedCacheManager(cache_size, high_priority_pool_ratio);

Review Comment:
   This manager is shared across all indexers with the same size settings, but page keys contain only `file_path/offset/length`. Two custom `FileSystem` or reader backends can expose the same logical path with different bytes, so the second scan can reuse the first backend's pages without opening its own file and return wrong matches. Include a stable backend namespace in the key, or scope the manager by backend.



##########
src/paimon/common/global_index/btree/btree_global_indexer.cpp:
##########
@@ -42,9 +45,40 @@
 #include "paimon/executor.h"
 #include "paimon/global_index/bitmap_global_index_result.h"
 #include "paimon/memory/bytes.h"
+#include "paimon/memory/memory_pool.h"
 #include "paimon/utils/roaring_bitmap64.h"
 
 namespace paimon {
+namespace {
+
+struct SharedCacheManager {
+    SharedCacheManager(int64_t cache_size, double high_priority_pool_ratio)
+        : cache_pool(GetDefaultPool()),
+          cache_manager(std::make_shared<CacheManager>(cache_size, high_priority_pool_ratio)) {}
+
+    // Keep the allocator alive until after cache_manager releases all cached pages.
+    std::shared_ptr<MemoryPool> cache_pool;
+    std::shared_ptr<CacheManager> cache_manager;
+};
+
+std::shared_ptr<CacheManager> GetSharedCacheManager(int64_t cache_size,
+                                                    double high_priority_pool_ratio) {
+    using CacheConfig = std::pair<int64_t, double>;
+    static std::mutex mutex;
+    static std::map<CacheConfig, SharedCacheManager> cache_managers;

Review Comment:
   `cache_managers` owns a full LRU for every distinct table-level `(cache_size, ratio)` forever. Each cache is bounded, but the process total is not; reading tables with different settings keeps all previous budgets resident. Please use one process-wide budget, or bound and evict this registry.



##########
src/paimon/core/operation/file_store_scan.cpp:
##########
@@ -150,12 +150,12 @@ Result<std::shared_ptr<FileStoreScan::RawPlan>> FileStoreScan::CreatePlan() cons
         snapshot.has_value() && scan_mode_ == ScanMode::ALL &&
         core_options_.GetScanManifestEntryCacheMaxSnapshots() > 0 &&
         core_options_.GetCache() != nullptr && !table_path_.empty() &&
-        !row_range_index_.has_value() && bucket_filter_.has_value();
+        !row_range_index_.has_value();

Review Comment:
   The whole-table key is only `(table_path, branch)`, and a hit is accepted by numeric snapshot ID. After a table is dropped and recreated at the same path, snapshot IDs restart while the caller-owned cache can still hold the old entry, so an ordinary scan can return stale splits. Include a table or snapshot generation identity (for example the manifest-list name) in the cached entry/key, or invalidate it on recreation.



##########
src/paimon/core/operation/file_store_scan.cpp:
##########
@@ -347,16 +347,21 @@ Status FileStoreScan::ReadManifestEntriesWithCache(
     }
     *cache_hit = false;
 
-    // Rebuild the target snapshot bucket from all manifests and write the live entries back to the
-    // cache.
-    std::vector<ManifestFileMeta> bucket_manifest_metas;
-    for (const auto& meta : all_manifest_metas) {
-        if (MayContainBucket(meta, bucket)) {
-            bucket_manifest_metas.push_back(meta);
+    if (bucket) {
+        std::vector<ManifestFileMeta> bucket_manifest_metas;
+        for (const auto& meta : all_manifest_metas) {
+            if (MayContainBucket(meta, bucket.value())) {
+                bucket_manifest_metas.push_back(meta);
+            }
         }
+        PAIMON_RETURN_NOT_OK(
+            ReadAndMergeBucketFileEntries(bucket_manifest_metas, bucket.value(), manifest_entries));
+    } else {
+        std::vector<ManifestEntry> unmerged_entries;
+        PAIMON_RETURN_NOT_OK(
+            ReadFileEntries(all_manifest_metas, &unmerged_entries, /*apply_scan_filter=*/false));

Review Comment:
   On this no-bucket miss we materialize every manifest entry, then serialize up to `max_snapshots` full table states. If that value exceeds the cache capacity, `Put` fails silently and every later scan of the same snapshot repeats the full read, merge, copy, and serialization. Only use this path when the value can be retained; otherwise fall back to `filtered_manifest_file_metas`, or store snapshots separately so older states can be evicted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(scan): reuse loaded global index planning context [paimon-cpp]

Posted by "wangyong9999 (via GitHub)" <gi...@apache.org>.
wangyong9999 commented on PR #267:
URL: https://github.com/apache/paimon-cpp/pull/267#issuecomment-5491954741

   Agreed. I removed the process-wide reader and BTree cache registry, namespace, and allocator-lifetime changes. The query engine now owns cross-query `GlobalIndexReader` reuse through the existing public API, pins the data scan to the same snapshot, and honors `IsThreadSafe()` around shared-reader evaluation.
   
   This PR is now limited to reusing the schema, snapshot context, executor, memory pool, and caller-provided manifest cache already available inside one `TableScan`. The resulting diff is 4 files; `paimon-global-index-test` passes 132/132.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(scan): reuse loaded global index planning context [paimon-cpp]

Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 merged PR #267:
URL: https://github.com/apache/paimon-cpp/pull/267


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org