[PR] [core] Fix that latestSnapshotOfUser might list snapshot directory to find the earliest [paimon]

["yuzelin (via GitHub)" <gi...@apache.org>]

yuzelin opened a new pull request, #7572:
URL: https://github.com/apache/paimon/pull/7572

   ### Purpose
   Previously, latestSnapshotOfUser called earliestSnapshotId() to determine the loop bound before iterating snapshots. 
   
   During checkpoint, if the snapshot pointed to by the EARLIEST hint is concurrently expired, the hint misses and earliestSnapshotId() falls back to findByListFiles(), which lists the entire snapshot directory.
   
   This method is called during every checkpoint via prepareCommit → createWriterCleanChecker → latestCommittedIdentifier → latestSnapshotOfUserFromFilesystem. For jobs with high parallelism and a large number of snapshots (e.g. 10000+), this might cause massive listStatus requests, easily triggering QPS limits on object storage (e.g. OSS QpsLimitExceeded).
   
   The fix removes the earliestSnapshotId() call and instead iterates backward from the latest snapshot, stopping when a FileNotFoundException is encountered (indicating an expired snapshot). Other exceptions are thrown directly.
   
   ### Tests
   testLatestSnapshotOfUser can test the loop can break.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Fix that latestSnapshotOfUser might list snapshot directory to find the earliest [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #7572:
URL: https://github.com/apache/paimon/pull/7572


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] Fix that latestSnapshotOfUser might list snapshot directory to find the earliest [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #7572:
URL: https://github.com/apache/paimon/pull/7572#issuecomment-4168112425

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org