You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "qidian99 (via GitHub)" <gi...@apache.org> on 2023/05/17 07:09:18 UTC

[GitHub] [incubator-paimon] qidian99 opened a new pull request, #1168: [bug] Enforce snapshot expire will not expire all snapshots

qidian99 opened a new pull request, #1168:
URL: https://github.com/apache/incubator-paimon/pull/1168

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- What is the purpose of the change, or the associated issue -->
   This PR closes #1167 
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   No.
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   No.
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] qidian99 commented on pull request #1168: [bug] Enforce snapshot expire will not expire all snapshots

Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on PR #1168:
URL: https://github.com/apache/incubator-paimon/pull/1168#issuecomment-1554147659

   > Test failed due to:
   > 
   > ```
   > ConfigOptionsDocsCompletenessITCase.testCompleteness:65->compareDocumentedAndExistingOptions:209 Documentation is outdated, please regenerate it according to the instructions in paimon-docs/README.md.
   > ```
   
   @JingsongLi Done generating docs. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] yuzelin commented on a diff in pull request #1168: [bug] Enforce snapshot expire will not expire all snapshots

Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin commented on code in PR #1168:
URL: https://github.com/apache/incubator-paimon/pull/1168#discussion_r1197613314


##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreExpireImpl.java:
##########
@@ -88,6 +89,12 @@ public FileStoreExpireImpl(
             SnapshotManager snapshotManager,
             ManifestFile.Factory manifestFileFactory,
             ManifestList.Factory manifestListFactory) {
+        Preconditions.checkArgument(
+                numRetainedMin >= 1,
+                "The minimum number of completed snapshots to retain should be >= 1.");
+        Preconditions.checkArgument(
+                numRetainedMax > numRetainedMin,
+                "The maximum number of snapshots to retain should be > the minimum number.");

Review Comment:
   I think the limit should be `numRetainedMax >= numRetainedMin` because we can set `numRetainedMax == numRetainedMin` to control exactly how many snapshots are retained.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi closed pull request #1168: [bug] Enforce snapshot expire will not expire all snapshots

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #1168: [bug] Enforce snapshot expire will not expire all snapshots
URL: https://github.com/apache/incubator-paimon/pull/1168


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on pull request #1168: [bug] Enforce snapshot expire will not expire all snapshots

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #1168:
URL: https://github.com/apache/incubator-paimon/pull/1168#issuecomment-1553907392

   Test failed due to: 
   ```
   ConfigOptionsDocsCompletenessITCase.testCompleteness:65->compareDocumentedAndExistingOptions:209 Documentation is outdated, please regenerate it according to the instructions in paimon-docs/README.md.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1168: [bug] Enforce snapshot expire will not expire all snapshots

Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on code in PR #1168:
URL: https://github.com/apache/incubator-paimon/pull/1168#discussion_r1197630566


##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreExpireImpl.java:
##########
@@ -88,6 +89,12 @@ public FileStoreExpireImpl(
             SnapshotManager snapshotManager,
             ManifestFile.Factory manifestFileFactory,
             ManifestList.Factory manifestListFactory) {
+        Preconditions.checkArgument(
+                numRetainedMin >= 1,
+                "The minimum number of completed snapshots to retain should be >= 1.");
+        Preconditions.checkArgument(
+                numRetainedMax > numRetainedMin,
+                "The maximum number of snapshots to retain should be > the minimum number.");

Review Comment:
   Yes you are correct. I was confused by the following code in snapshot expiration.
   
   ```java
           // find the earliest snapshot to retain
           for (long id = Math.max(latestSnapshotId - numRetainedMax + 1, earliest);
                   id <= latestSnapshotId - numRetainedMin;
                   id++) {
               if (snapshotManager.snapshotExists(id)
                       && currentMillis - snapshotManager.snapshot(id).timeMillis()
                               <= millisRetained) {
                   // within time threshold, can assume that all snapshots after it are also within
                   // the threshold
                   expireUntil(earliest, id);
                   return;
               }
           }
   ```
   
   This segment tries to locate the first snapshot within a range of snapshots, specifically between the numRetainedMax th and numRetainedMin th latest snapshots. This snapshot needs to be preserved because it doesn't fulfill the time threshold condition for expiration.
   
   I've updated the comment to be more clear.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on pull request #1168: [bug] Enforce snapshot expire will not expire all snapshots

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #1168:
URL: https://github.com/apache/incubator-paimon/pull/1168#issuecomment-1557213106

   Can you rebase latest master?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org