You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "GeorgeJahad (via GitHub)" <gi...@apache.org> on 2023/04/26 23:22:16 UTC

[GitHub] [ozone] GeorgeJahad opened a new pull request, #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

GeorgeJahad opened a new pull request, #4621:
URL: https://github.com/apache/ozone/pull/4621

   ## What changes were proposed in this pull request?
   
   In order to support bootstrapping followers, the om snapshot subsystem needs to make consistent copies snapshot, including the deleted keys table which can be updated at any time.
   
   This means that the background processes modifying the deletedKeys tables need to be halted, and then the bootstrapping process needs to wait for the double buffer to be flushed, (so that any remaining mods to the deletedKeys table is up to date.)
   
   This PR adds the mechanism to allow a thread to wait until the double buffer has been flushed.
   
   It is implmented by adding a set of countdown latches one for each thread that is waiting.
   
   The latches are initialized to 2, one for each of the buffers in the "double" buffer.
   
   Each time a buffer is cleared the latches are decremented once, so that when both have been cleared, the waiter will resume.
   
   The other change is to set the double buffer to periodically check for waiters when the double buffers are idle.
   
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-8499
   
   ## How was this patch tested?
   
   Unit tests added.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1182755844


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);
+      flushLatches.put(latch, latch);
+      latch.await();
+      flushLatches.remove(latch);
+    }
+
+    int notifyFlush() {
+      Set<CountDownLatch> latches = flushLatches.keySet();
+      int retval = latches.size();
+      for (CountDownLatch l: flushLatches.keySet()) {

Review Comment:
   no longer applicable



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181914364


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {

Review Comment:
   they both need to be 0, but this loop only gets called if readyBuffer size is already 0 and we are waiting for more data.  So I don't need to check it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181951876


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =

Review Comment:
   Yes, that is nicer.  I'll switch over to that.  Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1185744899


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }
       }
       return true;
     }  catch (InterruptedException ex) {

Review Comment:
   ```suggestion
       } catch (InterruptedException ex) {
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,45 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final Set<CountDownLatch> flushLatches =
+        ConcurrentHashMap.newKeySet();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);
+      flushLatches.add(latch);
+      latch.await();
+      flushLatches.remove(latch);
+    }
+
+    int notifyFlush() {
+      int retval = flushLatches.size();
+      for (CountDownLatch l: flushLatches) {

Review Comment:
   ```suggestion
         for (CountDownLatch l : flushLatches) {
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {

Review Comment:
   ```suggestion
           if (currentBuffer.size() == 0) {
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }

Review Comment:
   How is this snippet confirming both buffers are empty? Would you need to check `readyBuffer` size as well?



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -718,4 +718,7 @@ public long getTermForIndex(long transactionIndex) {
     return applyTransactionMap.get(transactionIndex);
   }
 
+  public void awaitDoubleBufferFlush() throws InterruptedException {

Review Comment:
   nit: javadoc to explain its intended usage



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181911971


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =

Review Comment:
   I don't see a ConcurrentSet defined anywhere.  Do you know of one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181935621


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();

Review Comment:
   Make sense. I just felt that `flushCurrentBuffer` would get executed even tho there is nothing to flush.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] kerneltime commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1529911486

   @hemantk-12 can you please take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181942355


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =

Review Comment:
   I had an assumption that `ConcurrentSet` is there but unfortunately it is not. 
   
   You can still create concurrent set like `ConcurrentHashMap.newKeySet()`.
   
   https://www.baeldung.com/java-concurrent-hashset-concurrenthashmap#thread-safe-hashset-using-concurrenthashmap-factory-method



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();

Review Comment:
   My bad, you are right it.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);

Review Comment:
   Yes, but you are notifying it together in line 636-637. How does it make any difference to use latch of one or two?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181916572


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -718,4 +718,7 @@ public long getTermForIndex(long transactionIndex) {
     return applyTransactionMap.get(transactionIndex);
   }
 
+  public void awaitDoubleBufferFlush() throws InterruptedException {

Review Comment:
   No, it will be used by the tarball creation code.  That code will first block the deletion service, (which is my next pr.)  Then call this method to wait until the buffers are cleared.  That way we can be sure that no moveKey operations from the deletion service are waiting in the buffers



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181934386


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);

Review Comment:
   got it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1182756413


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =

Review Comment:
   @hemantk-12  updated, please re-review



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1185824861


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }

Review Comment:
   Make sense.
   
   In this case we could just remove inner `if` right? So the while-loop would become:
   
   ```java
         while (currentBuffer.size() == 0) {
           // Both buffers are empty at this point, notify twice
           flushNotifier.notifyFlush();
           flushNotifier.notifyFlush();
   
           wait(1000L);
         }
   ```
   
   Also now that we both asked the same question. It might be worth writing that down in the code comment. :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181954401


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);

Review Comment:
   Ah, now I understand your question.  Yes that is true of 636-7, but that is a special case where we know both buffers are empty.
   
   It matters at the end of flushCurrentBuffer() here:
   https://github.com/apache/ozone/blob/9cded9de816dc80504872326d0fc2d96f48f7673/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java#L315-L317
   
   Where only one of the buffers, (the ready buffer,) is done.  Does that make more sense?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1526649964

   @hemantk-12 might also be interested in the double buffer change here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1185759009


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }

Review Comment:
   How is this snippet confirming both buffers are empty? Do you need to check `readyBuffer.size()` as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1536642519

   Thanks @smengcl and @prashantpogde if one of you can merge this, I can create the PR that depends on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1185768423


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }

Review Comment:
   https://github.com/apache/ozone/pull/4621#discussion_r1181914364
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1186238547


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }

Review Comment:
   Yes that is clearer.  I'll update.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181912843


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);

Review Comment:
   Yes, it is to handle the case where nothing is being written to the double buffer.  Right now, the wait(Long.MAX_VALUE) basically waits for ever, but we want to periodically check for waiters when there are no writes going on.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181957020


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);

Review Comment:
   Ah, I see. It makes sense completely. Somehow I missed it when I searched for it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1536461258

   @smengcl  I've made the updates you recommended.  Please take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1536780542

   > 
   
   Agreed. The same test on your dev branch CI passed: https://github.com/GeorgeJahad/ozone/actions/runs/4895130460/jobs/8740344040
   
   I will merge this based on that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181913323


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();

Review Comment:
   It wasn't before, so I figured I wouldn't do that now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181957020


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);

Review Comment:
   Ah, I see. It makes sens now. Somehow I missed it when I searched for it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181846984


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);
+      flushLatches.put(latch, latch);
+      latch.await();
+      flushLatches.remove(latch);
+    }
+
+    int notifyFlush() {
+      Set<CountDownLatch> latches = flushLatches.keySet();
+      int retval = latches.size();
+      for (CountDownLatch l: flushLatches.keySet()) {

Review Comment:
   ```suggestion
         for (CountDownLatch l: latches) {
           l.countDown();
         }
   ```
   
   to avoid any update in between 710-712.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();

Review Comment:
   Should we return `false` in this scenario?



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =

Review Comment:
   ```suggestion
       private final ConcurrentMap<CountDownLatch, Object> flushLatches =
           new ConcurrentHashMap<>();
    ```
    
    Since we are not using value from Map, can this be changed to `ConcurrentSet`?



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);

Review Comment:
   Was this intentionally changed to 1000L?



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -718,4 +718,7 @@ public long getTermForIndex(long transactionIndex) {
     return applyTransactionMap.get(transactionIndex);
   }
 
+  public void awaitDoubleBufferFlush() throws InterruptedException {

Review Comment:
   Is it supposed to be used by Deletion service?



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {

Review Comment:
   Is it supposed to be `if (readyBuffer.size() == 0)`?



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);

Review Comment:
   I'm not getting the point of using CountDownLatch of size 2 since we notify them together. Is it just to be semantically correct?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181935621


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();

Review Comment:
   Make sense. I just felt that `flushCurrentBuffer` would get executed even tho it is not needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181914364


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {

Review Comment:
   they both need to be 0, but this loop only gets if readyBuffer size is already 0 and we are waiting for more data.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1185824861


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }

Review Comment:
   Make sense.
   
   In this case we could just remove inner `if` right? So the while-loop could become:
   
   ```java
         while (currentBuffer.size() == 0) {
           // Both buffers are empty at this point, notify twice
           flushNotifier.notifyFlush();
           flushNotifier.notifyFlush();
   
           wait(1000L);
         }
   ```
   
   Am I missing anything?
   
   Also now that we both asked the same question. It might be worth writing that down in the code comment. :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1536780957

   > Looks like there is a failed test in the ci: org.apache.hadoop.hdds.scm.container.TestCloseContainerEventHandler.testCloseContainerWithDelayByLeaseManager
   > 
   > I don't believe it is related to this PR
   
   Agreed. The same test on your dev branch CI passed: https://github.com/GeorgeJahad/ozone/actions/runs/4895130460/jobs/8740344040
   
   I will merge this based on that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181915566


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -649,4 +673,46 @@ private synchronized void swapCurrentAndReadyBuffer() {
   public OzoneManagerDoubleBufferMetrics getOzoneManagerDoubleBufferMetrics() {
     return ozoneManagerDoubleBufferMetrics;
   }
+
+  @VisibleForTesting
+  int getCurrentBufferSize() {
+    return currentBuffer.size();
+  }
+
+  @VisibleForTesting
+  int getReadyBufferSize() {
+    return readyBuffer.size();
+  }
+
+  @VisibleForTesting
+  void resume() {
+    isRunning.set(true);
+  }
+
+  void awaitFlush() throws InterruptedException {
+    flushNotifier.await();
+  }
+
+  static class FlushNotifier {
+    private final ConcurrentHashMap<CountDownLatch, Object> flushLatches =
+        new ConcurrentHashMap<>();
+
+    void await() throws InterruptedException {
+
+      // Wait until both the current and ready buffers are flushed.
+      CountDownLatch latch = new CountDownLatch(2);

Review Comment:
   When this latch is created, both the ready and current buffers may contain data.  We have to wait until we are sure both are cleared.
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] hemantk-12 commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1535519667

   Thanks for updating @GeorgeJahad. It looks good to me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1185824861


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();
+        }

Review Comment:
   Make sense.
   
   In this case we could just remove inner `if` right? So the while-loop could become:
   
   ```java
         while (currentBuffer.size() == 0) {
           // Both buffers are empty at this point, notify twice
           flushNotifier.notifyFlush();
           flushNotifier.notifyFlush();
   
           wait(1000L);
         }
   ```
   
   What am I missing?
   
   Also now that we both asked the same question. It might be worth writing that down in the code comment. :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on code in PR #4621:
URL: https://github.com/apache/ozone/pull/4621#discussion_r1181941151


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java:
##########
@@ -611,7 +630,12 @@ public synchronized CompletableFuture<Void> add(OMClientResponse response,
   private synchronized boolean canFlush() {
     try {
       while (currentBuffer.size() == 0) {
-        wait(Long.MAX_VALUE);
+        wait(1000L);
+        if  (currentBuffer.size() == 0) {
+          // Both buffers are empty, so notify twice
+          flushNotifier.notifyFlush();
+          flushNotifier.notifyFlush();

Review Comment:
   >  flushCurrentBuffer would get executed even tho it is not needed.
   
   I don't believe that is true.  It goes back up to the top of the while loop and  again tests for 0.  If there is nothing ready it goes back to sleep without calling flushCurrentBuffer(), right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GeorgeJahad commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1536644891

   Looks like there is a failed test in the ci: org.apache.hadoop.hdds.scm.container.TestCloseContainerEventHandler.testCloseContainerWithDelayByLeaseManager 
   
   I don't believe it is related to this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4621:
URL: https://github.com/apache/ozone/pull/4621#issuecomment-1536781772

   Thanks @GeorgeJahad for the addition. Thanks @hemantk-12 for reviewing this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl merged pull request #4621: HDDS-8499. Add mechanism to notify threads when OM double buffer flushed.

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #4621:
URL: https://github.com/apache/ozone/pull/4621


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org