You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Aitozi (via GitHub)" <gi...@apache.org> on 2024/03/05 02:55:04 UTC

[PR] [core] Introduce ExpireChangelogImpl to handle the changelog lifecycle [incubator-paimon]

Aitozi opened a new pull request, #2942:
URL: https://github.com/apache/incubator-paimon/pull/2942

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #2899
   
   In this PR, the `snapshot` will be moved to the `/changelog` directory for decouple the changelog lifecycle. We will always keep one extra metadata file in the `/changelog` directory for the conveniency of streaming scan.
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce ExpireChangelogImpl to handle the changelog lifecycle [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2942:
URL: https://github.com/apache/incubator-paimon/pull/2942#discussion_r1512862166


##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -215,10 +204,15 @@ public int expireUntil(long earliestId, long endExclusiveId) {
             }
 
             Snapshot snapshot = snapshotManager.snapshot(id);
-            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet);
+            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet, false);
 
-            // delete snapshot last
-            snapshotManager.fileIO().deleteQuietly(snapshotManager.snapshotPath(id));
+            // move snapshot metadata to changelog
+            try {
+                snapshotManager.moveToChangelog(id);
+                snapshotManager.commitLongLivedChangelogLatestHint(id);

Review Comment:
   Maybe we should not just move snapshot file.
   
   We could introduce a new JSON file for changelog.



##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -215,10 +204,15 @@ public int expireUntil(long earliestId, long endExclusiveId) {
             }
 
             Snapshot snapshot = snapshotManager.snapshot(id);
-            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet);
+            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet, false);
 
-            // delete snapshot last
-            snapshotManager.fileIO().deleteQuietly(snapshotManager.snapshotPath(id));
+            // move snapshot metadata to changelog
+            try {
+                snapshotManager.moveToChangelog(id);

Review Comment:
   Please note that it is best not to affect the original behavior. For example, here we should determine whether there is a separate configuration for the changelog lifecycle. If it is not configured or is the same as the snapshot lifecycle, we should avoid creating files in the changelog folder.



##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -186,17 +186,6 @@ public int expireUntil(long earliestId, long endExclusiveId) {
             snapshotDeletion.cleanUnusedDataFiles(snapshot, skipper);
         }
 
-        // delete changelog files
-        for (long id = beginInclusiveId; id < endExclusiveId; id++) {
-            if (LOG.isDebugEnabled()) {
-                LOG.debug("Ready to delete changelog files from snapshot #" + id);
-            }
-            Snapshot snapshot = snapshotManager.snapshot(id);
-            if (snapshot.changelogManifestList() != null) {

Review Comment:
   If possible, we should try to ensure that the delta file of the Append snapshot can be stream read when there is no changelog file present.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce ExpireChangelogImpl to handle the changelog lifecycle [incubator-paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2942:
URL: https://github.com/apache/incubator-paimon/pull/2942#discussion_r1513708306


##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -215,10 +204,15 @@ public int expireUntil(long earliestId, long endExclusiveId) {
             }
 
             Snapshot snapshot = snapshotManager.snapshot(id);
-            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet);
+            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet, false);
 
-            // delete snapshot last
-            snapshotManager.fileIO().deleteQuietly(snapshotManager.snapshotPath(id));
+            // move snapshot metadata to changelog
+            try {
+                snapshotManager.moveToChangelog(id);
+                snapshotManager.commitLongLivedChangelogLatestHint(id);

Review Comment:
   Currently, I've not seen the requirement to introduce a new metadata structure for changelog. We can rewrite the content later when we have to later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce ExpireChangelogImpl to handle the changelog lifecycle [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2942:
URL: https://github.com/apache/incubator-paimon/pull/2942#discussion_r1513838474


##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -215,10 +204,15 @@ public int expireUntil(long earliestId, long endExclusiveId) {
             }
 
             Snapshot snapshot = snapshotManager.snapshot(id);
-            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet);
+            snapshotDeletion.cleanUnusedManifests(snapshot, skippingSet, false);
 
-            // delete snapshot last
-            snapshotManager.fileIO().deleteQuietly(snapshotManager.snapshotPath(id));
+            // move snapshot metadata to changelog
+            try {
+                snapshotManager.moveToChangelog(id);
+                snapshotManager.commitLongLivedChangelogLatestHint(id);

Review Comment:
   If we have to do it later, it is hard to keep forward compatibility.
   
   It is better to provide a simple version now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce ExpireChangelogImpl to handle the changelog lifecycle [incubator-paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2942:
URL: https://github.com/apache/incubator-paimon/pull/2942#discussion_r1513709847


##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -186,17 +186,6 @@ public int expireUntil(long earliestId, long endExclusiveId) {
             snapshotDeletion.cleanUnusedDataFiles(snapshot, skipper);
         }
 
-        // delete changelog files
-        for (long id = beginInclusiveId; id < endExclusiveId; id++) {
-            if (LOG.isDebugEnabled()) {
-                LOG.debug("Ready to delete changelog files from snapshot #" + id);
-            }
-            Snapshot snapshot = snapshotManager.snapshot(id);
-            if (snapshot.changelogManifestList() != null) {

Review Comment:
   Agree, will give a try.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Introduce ExpireChangelogImpl to handle the changelog lifecycle [incubator-paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi closed pull request #2942: [core] Introduce ExpireChangelogImpl to handle the changelog lifecycle
URL: https://github.com/apache/incubator-paimon/pull/2942


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org