You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/07/10 19:40:04 UTC

[GitHub] [pulsar] sijie opened a new pull request #7506: [Broker] Timeout opening managed ledger operation …

sijie opened a new pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506


   *Motivation*
   
   Currently broker has a timeout mechanism on loading topics. However the underlying managed ledger library
   doesn't provide a timeout mechanism. This will get into a situation that: A TopicLoad operation times out
   after 30 seconds. But the CompletableFuture of opening a managed ledger is still kept in the cache of managed ledger
   factory. The completable future will never returns. So any sub-sequent topic lookups will fail because any
   attempts to load a topic will never attempt to re-open a managed ledger.
   
   *Modification*
   
   Introduce a timeout mechanism in managed ledger factory. If a managed ledger is not open within a given timeout
   period, the CompletableFuture will be removed. This allows any sub-sequent attempts to load topics can try to
   open the managed ledger again.
   
   *Tests*
   
   This problem can be constantly reproduced in a chaos test in kubernetes by killing k8s worker nodes. It can cause
   producer stuck forever until the owner broker pod is restarted. The change has been verified in a chaos testing environment.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] sijie commented on a change in pull request #7506: [Broker] Timeout opening managed ledger operation …

Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506#discussion_r453106816



##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
##########
@@ -320,18 +335,32 @@ public void asyncOpen(final String name, final ManagedLedgerConfig config, final
 
         // If the ledger state is bad, remove it from the map.
         CompletableFuture<ManagedLedgerImpl> existingFuture = ledgers.get(name);
-        if (existingFuture != null && existingFuture.isDone()) {
-            try {
-                ManagedLedgerImpl l = existingFuture.get();
-                if (l.getState().equals(State.Fenced.toString()) || l.getState().equals(State.Closed.toString())) {
-                    // Managed ledger is in unusable state. Recreate it.
-                    log.warn("[{}] Attempted to open ledger in {} state. Removing from the map to recreate it", name,
+        if (existingFuture != null) {
+            if (existingFuture.isDone()) {
+                try {
+                    ManagedLedgerImpl l = existingFuture.get();
+                    if (l.getState().equals(State.Fenced.toString()) || l.getState().equals(State.Closed.toString())) {
+                        // Managed ledger is in unusable state. Recreate it.
+                        log.warn("[{}] Attempted to open ledger in {} state. Removing from the map to recreate it", name,
                             l.getState());
-                    ledgers.remove(name, existingFuture);
+                        ledgers.remove(name, existingFuture);
+                    }
+                } catch (Exception e) {
+                    // Unable to get the future
+                    log.warn("[{}] Got exception while trying to retrieve ledger", name, e);
                 }
-            } catch (Exception e) {
-                // Unable to get the future
-                log.warn("[{}] Got exception while trying to retrieve ledger", name, e);
+            } else {
+                PendingInitializeManagedLedger pendingLedger = pendingInitializeLedgers.get(name);

Review comment:
       We can do that. However, I didn't go down that route because of the following reason:
   
   we need to keep the ManageLedger reference and to close it to release resources. Because the initialization involves a long pipeline including opening managed ledger and cursors. The behavior we observed is that the operation is stuck at opening cursors. So if we don't attempt to close the ManagedLedger instance, it can result in resource leaking. 
   
   With that being said, we need to keep a reference to ManagedLedgerImpl along with the CompletableFuture. Hence I chose the current implementation.
   
   Another reason is to allow checking the timestamp proactively. I fixed a couple of issues before that NPE is thrown between creating a Future and registering error handling logic. I would like to have a mechanism in place that can fix any potential bugs by proactively checking if a CompletableFuture timed out.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] merlimat commented on pull request #7506: [Broker] Timeout opening managed ledger operation …

Posted by GitBox <gi...@apache.org>.
merlimat commented on pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506#issuecomment-656888277


   > The completable future will never return. So any sub-sequent topic lookups will fail because any
   attempts to load a topic will never attempt to re-open a managed ledger.
   
   @sijie Do you know why is that never never returning? Is that for DNS error on opening the ledger?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] addisonj commented on a change in pull request #7506: [Broker] Timeout opening managed ledger operation …

Posted by GitBox <gi...@apache.org>.
addisonj commented on a change in pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506#discussion_r453069297



##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java
##########
@@ -320,18 +335,32 @@ public void asyncOpen(final String name, final ManagedLedgerConfig config, final
 
         // If the ledger state is bad, remove it from the map.
         CompletableFuture<ManagedLedgerImpl> existingFuture = ledgers.get(name);
-        if (existingFuture != null && existingFuture.isDone()) {
-            try {
-                ManagedLedgerImpl l = existingFuture.get();
-                if (l.getState().equals(State.Fenced.toString()) || l.getState().equals(State.Closed.toString())) {
-                    // Managed ledger is in unusable state. Recreate it.
-                    log.warn("[{}] Attempted to open ledger in {} state. Removing from the map to recreate it", name,
+        if (existingFuture != null) {
+            if (existingFuture.isDone()) {
+                try {
+                    ManagedLedgerImpl l = existingFuture.get();
+                    if (l.getState().equals(State.Fenced.toString()) || l.getState().equals(State.Closed.toString())) {
+                        // Managed ledger is in unusable state. Recreate it.
+                        log.warn("[{}] Attempted to open ledger in {} state. Removing from the map to recreate it", name,
                             l.getState());
-                    ledgers.remove(name, existingFuture);
+                        ledgers.remove(name, existingFuture);
+                    }
+                } catch (Exception e) {
+                    // Unable to get the future
+                    log.warn("[{}] Got exception while trying to retrieve ledger", name, e);
                 }
-            } catch (Exception e) {
-                // Unable to get the future
-                log.warn("[{}] Got exception while trying to retrieve ledger", name, e);
+            } else {
+                PendingInitializeManagedLedger pendingLedger = pendingInitializeLedgers.get(name);

Review comment:
       As opposed to requiring another call to check if the timeout has elapsed, would it make sense to instead use a CompletableFuture with a timeout? Much like we do in the BrokerService, the future created on line 370 could instead be made to have a timeout if it isn't resolved in so many milliseconds with the error handler remove the future from the cache of futures. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui merged pull request #7506: [Broker] Timeout opening managed ledger operation …

Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] sijie commented on pull request #7506: [Broker] Timeout opening managed ledger operation …

Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506#issuecomment-656909319


   @merlimat it is stuck in LedgerRecoveryOp when recovering cursors. I haven't caught the real exception. My feeling is more coming from the zookeeper side. The chaos test we did is killing the Kubernetes worker node hardly. In that worker node, it has one zookeeper pod, one bookkeeper pod, and one broker pod. This sounds like causing some zookeeper call didn't come back and the ledger recovery op stuck without triggering any callback which in return causes the problem in managed ledger library.
   
   A side note - I created an issue a while ago to separate loading cursors from loading managed ledger. The idea is that we should allow producers to produce messages once the managed ledger is ready. This would improve write availability. https://github.com/apache/pulsar/issues/7404 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] devinbost commented on pull request #7506: [Broker] Timeout opening managed ledger operation …

Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506#issuecomment-666701886


   @sijie Could this timeout issue cause a topic to seem to freeze, as reported here: https://github.com/apache/pulsar/issues/6054 ?
   If so, the odd thing about that freezing topic issue is that it occurs randomly in a baremetal docker environment while running functions without any sudden broker instance deaths.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui commented on pull request #7506: [Broker] Timeout opening managed ledger operation …

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #7506:
URL: https://github.com/apache/pulsar/pull/7506#issuecomment-656952881


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org