You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "nfsantos (via GitHub)" <gi...@apache.org> on 2024/05/14 11:27:38 UTC

[PR] OAK-10788 - Indexing job downloader: shutdown gracefully all threads in case of failure [jackrabbit-oak]

nfsantos opened a new pull request, #1456:
URL: https://github.com/apache/jackrabbit-oak/pull/1456

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OAK-10788 - Indexing job downloader: shutdown gracefully all threads in case of failure [jackrabbit-oak]

Posted by "nfsantos (via GitHub)" <gi...@apache.org>.
nfsantos commented on code in PR #1456:
URL: https://github.com/apache/jackrabbit-oak/pull/1456#discussion_r1601044885


##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedStrategy.java:
##########
@@ -545,16 +540,32 @@ public File createSortedStoreFile() throws IOException {
                 indexingReporter.addTiming("Build FFS (Dump+Merge)", FormattingUtils.formatToSeconds(elapsedSeconds));
 
                 LOG.info("[INDEXING_REPORT:BUILD_FFS]\n{}", indexingReporter.generateReport());
-            } catch (InterruptedException e) {
+            } catch (Throwable e) {
+                LOG.warn("Error dumping from MongoDB. Cancelling all tasks. Error: {}", e.toString());
+                // Cancel in order
+                cancelFuture(downloadFuture);
+                for (Future<?> transformTask : transformFutures) {
+                    cancelFuture(transformTask);
+                }
+                cancelFuture(sortBatchFuture);
+                cancelFuture(mergeSortFuture);
                 throw new RuntimeException(e);
             } finally {
                 // No longer need to monitor the size of the queues,
                 monitorFuture.cancel(true);
             }
             return flatFileStore.toFile();
         } finally {
-            threadPool.shutdown();
-            monitorThreadPool.shutdown();
+            LOG.info("Shutting down build FFS thread pool");
+            new ExecutorCloser(threadPool).close();
+            monitorThreadPool.shutdownNow();

Review Comment:
   It was a miss. I refactored the code so that the `monitorThreadPool` is no longer needed. The periodic logging that was being done by this thread pool is now done by the main thread, using a timeout on waiting for the other tasks to finish. This reduces the amount of code and eliminates one thread pool, and is also less brittle. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OAK-10788 - Indexing job downloader: shutdown gracefully all threads in case of failure [jackrabbit-oak]

Posted by "nfsantos (via GitHub)" <gi...@apache.org>.
nfsantos merged PR #1456:
URL: https://github.com/apache/jackrabbit-oak/pull/1456


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OAK-10788 - Indexing job downloader: shutdown gracefully all threads in case of failure [jackrabbit-oak]

Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #1456:
URL: https://github.com/apache/jackrabbit-oak/pull/1456#discussion_r1599864900


##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTask.java:
##########
@@ -441,23 +441,24 @@ private void downloadWithRetryOnConnectionErrors() throws InterruptedException,
                 // One of the download tasks finished with an exception. Cancel the other one. Trying to cancel the
                 // task that already failed has no effect
                 LOG.info("Error during download. Canceling download threads. Error: {}", e.toString());
-                ascendingDownloadFuture.cancel(true);
-                descendingDownloadFuture.cancel(true);
+                downloadThreadPool.shutdownNow();
                 throw new RuntimeException(e);
             } catch (InterruptedException e) {
                 LOG.info("Thread interrupted. Cancelling download threads.");
                 // The parent thread was interrupted. Shutdown the download threads.
-                ascendingDownloadFuture.cancel(true);
-                descendingDownloadFuture.cancel(true);
+                downloadThreadPool.shutdownNow();

Review Comment:
   I would reuse the existing class https://github.com/apache/jackrabbit-oak/blob/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/ExecutorCloser.java to close this executor.
   
   Sample usage can be found here https://github.com/apache/jackrabbit-oak/blob/trunk/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/MongoDocumentStoreThrottlingMetricsUpdater.java#L121



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OAK-10788 - Indexing job downloader: shutdown gracefully all threads in case of failure [jackrabbit-oak]

Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #1456:
URL: https://github.com/apache/jackrabbit-oak/pull/1456#discussion_r1600227031


##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedStrategy.java:
##########
@@ -545,16 +540,32 @@ public File createSortedStoreFile() throws IOException {
                 indexingReporter.addTiming("Build FFS (Dump+Merge)", FormattingUtils.formatToSeconds(elapsedSeconds));
 
                 LOG.info("[INDEXING_REPORT:BUILD_FFS]\n{}", indexingReporter.generateReport());
-            } catch (InterruptedException e) {
+            } catch (Throwable e) {
+                LOG.warn("Error dumping from MongoDB. Cancelling all tasks. Error: {}", e.toString());
+                // Cancel in order
+                cancelFuture(downloadFuture);
+                for (Future<?> transformTask : transformFutures) {
+                    cancelFuture(transformTask);
+                }
+                cancelFuture(sortBatchFuture);
+                cancelFuture(mergeSortFuture);
                 throw new RuntimeException(e);
             } finally {
                 // No longer need to monitor the size of the queues,
                 monitorFuture.cancel(true);
             }
             return flatFileStore.toFile();
         } finally {
-            threadPool.shutdown();
-            monitorThreadPool.shutdown();
+            LOG.info("Shutting down build FFS thread pool");
+            new ExecutorCloser(threadPool).close();
+            monitorThreadPool.shutdownNow();

Review Comment:
   Is this `monitorThreadPool` a miss or intentional?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OAK-10788 - Indexing job downloader: shutdown gracefully all threads in case of failure [jackrabbit-oak]

Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #1456:
URL: https://github.com/apache/jackrabbit-oak/pull/1456#discussion_r1599864900


##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTask.java:
##########
@@ -441,23 +441,24 @@ private void downloadWithRetryOnConnectionErrors() throws InterruptedException,
                 // One of the download tasks finished with an exception. Cancel the other one. Trying to cancel the
                 // task that already failed has no effect
                 LOG.info("Error during download. Canceling download threads. Error: {}", e.toString());
-                ascendingDownloadFuture.cancel(true);
-                descendingDownloadFuture.cancel(true);
+                downloadThreadPool.shutdownNow();
                 throw new RuntimeException(e);
             } catch (InterruptedException e) {
                 LOG.info("Thread interrupted. Cancelling download threads.");
                 // The parent thread was interrupted. Shutdown the download threads.
-                ascendingDownloadFuture.cancel(true);
-                descendingDownloadFuture.cancel(true);
+                downloadThreadPool.shutdownNow();

Review Comment:
   I would use the existing class https://github.com/apache/jackrabbit-oak/blob/trunk/oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/ExecutorCloser.java to close this executor.
   
   Sample usage can be found here https://github.com/apache/jackrabbit-oak/blob/trunk/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/mongo/MongoDocumentStoreThrottlingMetricsUpdater.java#L121



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] OAK-10788 - Indexing job downloader: shutdown gracefully all threads in case of failure [jackrabbit-oak]

Posted by "nfsantos (via GitHub)" <gi...@apache.org>.
nfsantos commented on code in PR #1456:
URL: https://github.com/apache/jackrabbit-oak/pull/1456#discussion_r1599889479


##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/indexer/document/flatfile/pipelined/PipelinedMongoDownloadTask.java:
##########
@@ -441,23 +441,24 @@ private void downloadWithRetryOnConnectionErrors() throws InterruptedException,
                 // One of the download tasks finished with an exception. Cancel the other one. Trying to cancel the
                 // task that already failed has no effect
                 LOG.info("Error during download. Canceling download threads. Error: {}", e.toString());
-                ascendingDownloadFuture.cancel(true);
-                descendingDownloadFuture.cancel(true);
+                downloadThreadPool.shutdownNow();
                 throw new RuntimeException(e);
             } catch (InterruptedException e) {
                 LOG.info("Thread interrupted. Cancelling download threads.");
                 // The parent thread was interrupted. Shutdown the download threads.
-                ascendingDownloadFuture.cancel(true);
-                descendingDownloadFuture.cancel(true);
+                downloadThreadPool.shutdownNow();

Review Comment:
   Thanks, I changed it to use ExecutorCloser. I did not knew of this utility class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org