You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/18 15:09:09 UTC
[PR] [core] Optimize tag reads during snapshot expiration [paimon]
JingsongLi opened a new pull request, #9967:
URL: https://github.com/apache/paimon/pull/9967
### Purpose
Snapshot expiration previously built a full data-file index for every referenced tag. On large tables, concurrent tag reads retained memory proportional to the number of tags multiplied by all table files and could cause OOM.
This change:
- plans deletion candidates in bounded expiration batches;
- prunes tag manifests by candidate bucket ranges;
- pushes partition and bucket filtering into Avro decoding before nested file metadata;
- filters file names on a reusable projected entry before materializing ExpireFileEntry;
- keeps only matching tag entries while preserving changelog-decoupled cleanup behavior.
The filename filter is streaming rather than storage-index based; manifests still need to be scanned within matching partitions and buckets.
### Tests
- Targeted manifest streaming/filter-pushdown and tag expiration regression tests
- ExpireSnapshotsTest and FileDeletionTest
- mvn -pl paimon-core -DskipTests spotless:check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimize tag reads during snapshot expiration [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9967:
URL: https://github.com/apache/paimon/pull/9967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org