You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "GeorgeJahad (via GitHub)" <gi...@apache.org> on 2023/06/21 22:47:22 UTC

[GitHub] [ozone] GeorgeJahad opened a new pull request, #4950: HDDS-8681. Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

GeorgeJahad opened a new pull request, #4950:
URL: https://github.com/apache/ozone/pull/4950

   ## What changes were proposed in this pull request?
   This intermittent failure is caused by a race condition with the om double-buffer flush-thread.
   
   If the test finishes before the double buffer is flushed, then it fails because the tarball doesn't contain all the expected snapshots.  The fix is to make the test wait until the double buffer is flushed.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-8681
   
   ## How was this patch tested?
   
   Without this fix, I can make this failure happen consistently by inserting a thread sleep in the double buffer thread here:  https://github.com/GeorgeJahad/ozone/blob/15d451742a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotCreateResponse.java#L84-L88
   
   
   With the fix, it no longer fails even with the thread sleep.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] adoroszlai commented on pull request #4950: HDDS-8681. [SNAPSHOT] Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4950:
URL: https://github.com/apache/ozone/pull/4950#issuecomment-1602461178

   @hemantk-12 please review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on pull request #4950: HDDS-8681. [SNAPSHOT] Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on PR #4950:
URL: https://github.com/apache/ozone/pull/4950#issuecomment-1603014486

   Had a offline discussion with @GeorgeJahad and he is going to double if `waitForSnapshotFlush` is actually needed because tarball creation waits for double buffer to flush.
   
   FYI: @smengcl  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on pull request #4950: HDDS-8681. [SNAPSHOT] Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #4950:
URL: https://github.com/apache/ozone/pull/4950#issuecomment-1608273938

   @hemantk-12 correctly pointed out that the double buffer flush in the bootstrap lock here should have been preventing this problem from happening: https://github.com/apache/ozone/blob/2fbed633e87ba455e7635e75356df3ce2dd906ac/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java#L395
   
   The reason it wasn't was because I was taking the lock after the checkpoint was created instead after, so I moved it to before and that fixed the problem.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4950: HDDS-8681. [SNAPSHOT] Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4950:
URL: https://github.com/apache/ozone/pull/4950#discussion_r1238781253


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -885,4 +886,27 @@ public void close() {
   public long getDiffCleanupServiceInterval() {
     return diffCleanupServiceInterval;
   }
+
+  // Wait for double buffer to be flushed and confirm that the
+  // snapshot directory exists.
+  public void waitForSnapshotFlush(String volumeName, String bucketName,

Review Comment:
   Looks like this is only used in tests. We can add an annotation for that.
   
   ```suggestion
     @VisibleForTesting
     public void waitForSnapshotFlush(String volumeName, String bucketName,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4950: HDDS-8681. [SNAPSHOT] Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4950:
URL: https://github.com/apache/ozone/pull/4950#discussion_r1244554370


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -145,28 +149,29 @@ private void generateSnapshotCheckpoint(HttpServletRequest request,
     }
 
     DBCheckpoint checkpoint = null;
-    try {
-      boolean flush = false;
-      String flushParam =
-          request.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH);
-      if (StringUtils.isNotEmpty(flushParam)) {
-        flush = Boolean.parseBoolean(flushParam);
-      }
 
-      List<String> receivedSstList = new ArrayList<>();
-      List<String> excludedSstList = new ArrayList<>();
-      String[] sstParam = isFormData ?
-          parseFormDataParameters(request) : request.getParameterValues(
-          OZONE_DB_CHECKPOINT_REQUEST_TO_EXCLUDE_SST);
-      if (sstParam != null) {
-        receivedSstList.addAll(
-            Arrays.stream(sstParam)
-            .filter(s -> s.endsWith(ROCKSDB_SST_SUFFIX))
-            .distinct()
-            .collect(Collectors.toList()));
-        LOG.info("Received excluding SST {}", receivedSstList);
-      }
+    boolean flush = false;
+    String flushParam =
+        request.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH);
+    if (StringUtils.isNotEmpty(flushParam)) {
+      flush = Boolean.parseBoolean(flushParam);
+    }
 
+    List<String> receivedSstList = new ArrayList<>();
+    List<String> excludedSstList = new ArrayList<>();
+    String[] sstParam = isFormData ?
+        parseFormDataParameters(request) : request.getParameterValues(
+        OZONE_DB_CHECKPOINT_REQUEST_TO_EXCLUDE_SST);
+    if (sstParam != null) {
+      receivedSstList.addAll(
+          Arrays.stream(sstParam)
+              .filter(s -> s.endsWith(ROCKSDB_SST_SUFFIX))
+              .distinct()
+              .collect(Collectors.toList()));
+      LOG.info("Received excluding SST {}", receivedSstList);
+    }
+
+    try  (BootstrapStateHandler.Lock lock = getBootstrapStateLock().lock()) {

Review Comment:
   ```suggestion
       try (BootstrapStateHandler.Lock lock = getBootstrapStateLock().lock()) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4950: HDDS-8681. [SNAPSHOT] Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4950:
URL: https://github.com/apache/ozone/pull/4950#discussion_r1244626082


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -301,4 +306,27 @@ public void writeDbDataToStream(DBCheckpoint checkpoint,
     writeDBCheckpointToStream(checkpoint, destination,
         toExcludeList, excludedList);
   }
+
+  @Override
+  public BootstrapStateHandler.Lock getBootstrapStateLock() {
+    return lock;
+  }
+
+/**

Review Comment:
   nit:
   ```suggestion
     /**
      * This lock is a no-op but can overridden by child classes.
      */
       public static class Lock extends BootstrapStateHandler.Lock {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad merged pull request #4950: HDDS-8681. [SNAPSHOT] Intermittent failure in TestOMDbCheckpointServlet#testWriteDbDataToStream

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad merged PR #4950:
URL: https://github.com/apache/ozone/pull/4950


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org