[PR] [core]save the runtime memory while reading all the partitions or buckets [paimon]

["weijietong (via GitHub)" <gi...@apache.org>]

weijietong opened a new pull request, #8745:
URL: https://github.com/apache/paimon/pull/8745

   ### Purpose
     Optimize readPartitionEntries and readBucketEntries in AbstractFileStoreScan to use per-entry converter during manifest reading, so ManifestEntry objects (with heavy DataFileMeta including stats, key ranges, embedded index, etc.) are immediately converted to lightweight PartitionEntry/BucketEntry and never accumulated. This reduces peak memory per manifest file from holding a full List<ManifestEntry> to only a List<PartitionEntry> — each DataFileMeta (~20 fields, several KB) is discarded after extracting the 3 needed values (rowCount, fileSize, creationTime), while PartitionEntry only retains ~6 fields (~40 bytes). For tables with millions of files the savings can be significant.
   
   ### Tests
   
     Added FileStoreScanPartitionBucketEntryTest covering single/multiple partition aggregation, multi-file merge within the same partition, partition filter pushdown, append-only table variant, and equivalent bucket entry tests — including the previously untested readBucketEntries() method which had zero coverage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core]save the runtime memory while reading all the partitions or buckets [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #8745:
URL: https://github.com/apache/paimon/pull/8745


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org