[GitHub] [pulsar] merlimat commented on a change in pull request #13130: [Issue 13129] [pulsar-metadata] Add watchdog thread in metadata store and track long running tasks.

[GitBox <gi...@apache.org>]

merlimat commented on a change in pull request #13130:
URL: https://github.com/apache/pulsar/pull/13130#discussion_r762164290



##########
File path: pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java
##########
@@ -317,7 +346,33 @@ public void invalidateAll() {
      */
     protected void execute(Runnable task, CompletableFuture<?> future) {
         try {
-            executor.execute(task);
+            // Wrap the original task, so we can record the thread on which it is running
+            TaskWrapper taskWrapper = new TaskWrapper(task);
+            executorWatchDog.execute(() -> {

Review comment:
       I think that having a wrapper on each task is not the best option here, since we have a single thread where we're making all the blocking calls on the future. 
   
   Also, a 5 delay in a metadata operation does not necessary indicate that there is a deadlock (eg: a particular request could be slow while there is a leader election happening).
   
   Instead, we could have the watchdog thread to execute a dummy task in the `metadata-store` executor. If that task cannot be scheduled for, say, 30sec, then we can be quite confident that there is indeed a deadlock, in which case we should dump the stack traces and kill the broker, since there's no way for it to fix itself up.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org