You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wangyong9999 (via GitHub)" <gi...@apache.org> on 2026/10/13 13:34:16 UTC

Re: [PR] perf(scan): prune manifest partitions before metadata decoding [paimon-cpp]

wangyong9999 commented on code in PR #430:
URL: https://github.com/apache/paimon-cpp/pull/430#discussion_r4259290296


##########
src/paimon/core/operation/file_store_scan.cpp:
##########
@@ -396,44 +388,84 @@ Status FileStoreScan::ReadManifestEntries(const std::vector<ManifestFileMeta>& m
 // bucket counts, with the current count and schema in the cache key. Exact hits
 // can be returned directly; cache misses rebuild the target snapshot bucket from the target
 // snapshot's data manifests.
-Status FileStoreScan::ReadManifestEntriesWithCache(
+Result<uint64_t> FileStoreScan::ReadManifestEntriesWithCache(
     const Snapshot& snapshot, const std::vector<ManifestFileMeta>& all_manifest_metas,
     int32_t bucket, std::vector<ManifestEntry>* manifest_entries, bool* cache_hit) const {
     Duration cache_load_duration;
-    PAIMON_ASSIGN_OR_RAISE(SnapshotLiveManifestEntries cached_entries,
-                           LoadSnapshotLiveManifestEntries(bucket));
+    const int32_t max_snapshots = core_options_.GetScanManifestEntryCacheMaxSnapshots();
+    SnapshotLiveManifestEntries cached_entries(max_snapshots);
+    auto row_filter = CreateManifestEntryRowFilter();
+    bool partition_filtered = false;
+    uint64_t candidate_entries = 0;
+    *cache_hit = false;
+    Result<std::shared_ptr<CacheValue>> value = core_options_.GetCache()->Get(
+        CreateSnapshotLiveManifestEntriesCacheKey(bucket),
+        [](const std::shared_ptr<CacheKey>&) -> Result<std::shared_ptr<CacheValue>> {
+            return std::shared_ptr<CacheValue>();
+        });
+    if (value.ok() && value.value()) {
+        PAIMON_ASSIGN_OR_RAISE(
+            std::optional<SnapshotLiveManifestEntries::Selection> selected,

Review Comment:
   This turns a bad snapshot-cache value into a sticky scan failure. Before, `LoadSnapshotLiveManifestEntries` treated a value that failed to deserialize (bad magic, unsupported entry version, truncated rows) as a miss and rebuilt the bucket, which also overwrote the bad value. Now `ReadSnapshot` returns that error straight to `CreatePlan`, and since the miss path never runs, the same value is read again on every later scan of this bucket until it happens to be evicted.
   
   Suggest keeping only predicate errors fatal: if `ReadSnapshot` fails while parsing the value itself, fall through to the rebuild path as before.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org