You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "targetoee (via GitHub)" <gi...@apache.org> on 2023/03/14 14:08:07 UTC

[GitHub] [ozone] targetoee opened a new pull request, #4398: HDDS-8061. Add SnapDiff client option to force a full diff

targetoee opened a new pull request, #4398:
URL: https://github.com/apache/ozone/pull/4398

   ## What changes were proposed in this pull request?
   By default, snapdiff would use the optimized DAG-based pruning approach. This change provides a full diff option of snapshot. 
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-8061
   
   ## How was this patch tested?
   unit test
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] targetoee commented on pull request #4398: HDDS-8061. [Snapshot] Add SnapDiff client option to force a full diff

Posted by "targetoee (via GitHub)" <gi...@apache.org>.
targetoee commented on PR #4398:
URL: https://github.com/apache/ozone/pull/4398#issuecomment-1474752161

   Done. Thanks @smengcl @sadanand48 for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sadanand48 commented on a diff in pull request #4398: HDDS-8061. [Snapshot] Add SnapDiff client option to force a full diff

Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4398:
URL: https://github.com/apache/ozone/pull/4398#discussion_r1139877030


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -389,11 +393,13 @@ private Set<String> getDeltaFiles(OmSnapshot fromSnapshot,
       // End of Workaround
     }
 
-    if (deltaFiles.isEmpty()) {
+    if (useFullDiff || deltaFiles.isEmpty()) {

Review Comment:
   If useFullDiff is true then deltaFiles.isEmpty() is always true but I guess its ok to have this .



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl merged pull request #4398: HDDS-8061. [Snapshot] Add SnapDiff client option to force a full diff

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #4398:
URL: https://github.com/apache/ozone/pull/4398


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4398: HDDS-8061. Add SnapDiff client option to force a full diff

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4398:
URL: https://github.com/apache/ozone/pull/4398#discussion_r1139642086


##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1694,6 +1694,7 @@ message SnapshotDiffRequest {
   optional string toSnapshot = 4;
   optional string token = 5;
   optional uint32 pageSize = 6;
+  optional bool fullDiff = 7;

Review Comment:
   ```suggestion
     optional bool forceFullDiff = 7;
   ```



##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotDiffHandler.java:
##########
@@ -54,6 +54,11 @@ public class SnapshotDiffHandler extends Handler {
           " (optional)")
   private int pageSize;
 
+  @CommandLine.Option(names = {"-fd", "--use-full-diff"},

Review Comment:
   ```suggestion
     @CommandLine.Option(names = {"--ffd", "--force-full-diff"},
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -389,11 +393,13 @@ private Set<String> getDeltaFiles(OmSnapshot fromSnapshot,
       // End of Workaround
     }
 
-    if (deltaFiles.isEmpty()) {
+    if (useFullDiff || deltaFiles.isEmpty()) {

Review Comment:
   This new condition looks reasonable to me.
   
   But I'd like @sadanand48 or @hemantk-12 to review this change in this class as well.



##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotDiffHandler.java:
##########
@@ -54,6 +54,11 @@ public class SnapshotDiffHandler extends Handler {
           " (optional)")
   private int pageSize;
 
+  @CommandLine.Option(names = {"-fd", "--use-full-diff"},
+      description = "perform full diff of snapshot without using" +
+          " optimised DAG based pruning approach (optional)")
+  private boolean fullDiff;

Review Comment:
   This option would be useful for debugging. But for now let's hide this option by default to regular users (still usable). We can decide whether to unhide this later.
   
   ```suggestion
         description = "perform full diff of snapshot without using" +
             " optimised DAG based pruning approach (optional)",
         hidden = true)
     private boolean fullDiff;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4398: HDDS-8061. [Snapshot] Add SnapDiff client option to force a full diff

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4398:
URL: https://github.com/apache/ozone/pull/4398#discussion_r1140909511


##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ObjectStore.java:
##########
@@ -569,6 +569,7 @@ public List<OzoneSnapshot> listSnapshot(String volumeName, String bucketName)
    * @param toSnapshot The name of the ending snapshot
    * @param token to get the index to return diff report from.
    * @param pageSize maximum entries returned to the report.
+   * @param forceFullDiff show full diff without using the optimised DAG method

Review Comment:
   nit
   ```suggestion
      * @param forceFullDiff request to force full diff, skipping DAG optimization
   ```



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java:
##########
@@ -705,6 +705,7 @@ default List<SnapshotInfo> listSnapshot(String volumeName, String bucketName)
    * @param toSnapshot The name of the ending snapshot
    * @param token to get the index to return diff report from.
    * @param pageSize maximum entries returned to the report.
+   * @param forceFullDiff perform full diff without using optimized DAG approach

Review Comment:
   ```suggestion
      * @param forceFullDiff request to force full diff, skipping DAG optimization
   ```



##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java:
##########
@@ -1059,11 +1059,13 @@ List<OzoneSnapshot> listSnapshot(String volumeName, String bucketName)
    * @param toSnapshot The name of the ending snapshot
    * @param token to get the index to return diff report from.
    * @param pageSize maximum entries returned to the report.
+   * @param forceFullDiff show full diff without using the optimised DAG method

Review Comment:
   ```suggestion
      * @param forceFullDiff request to force full diff, skipping DAG optimization
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on pull request #4398: HDDS-8061. [Snapshot] Add SnapDiff client option to force a full diff

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4398:
URL: https://github.com/apache/ozone/pull/4398#issuecomment-1474778265

   Thanks @targetoee for the patch. Thanks @sadanand48 for reviewing this as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sadanand48 commented on a diff in pull request #4398: HDDS-8061. [Snapshot] Add SnapDiff client option to force a full diff

Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4398:
URL: https://github.com/apache/ozone/pull/4398#discussion_r1139880649


##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ObjectStore.java:
##########
@@ -577,9 +578,10 @@ public SnapshotDiffResponse snapshotDiff(String volumeName,
                                            String fromSnapshot,
                                            String toSnapshot,
                                            String token,
-                                           int pageSize)
+                                           int pageSize,
+                                           boolean fullDiff)

Review Comment:
   Like @smengcl pointed out, lets change all instances of fullDiff to forceFullDiff.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org