You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by "bhabegger (via GitHub)" <gi...@apache.org> on 2026/01/09 15:05:08 UTC
[PR] Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
bhabegger opened a new pull request, #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679
Some instantiations of ThreadPoolExecutor were setting a different corePoolSize and maxPoolSize even though they were using an unbounded LinkedBlockingQueue. The pattern was repeated multiple times. This PR attempts to fix and mutualize the ThreadPoolExecutor creation code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2694191204
##########
oak-auth-ldap/pom.xml:
##########
@@ -104,7 +104,6 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
- <scope>provided</scope>
Review Comment:
Ok, I wasn't really sure about that. Just of my knowledge. What is the rationale behind "provided" here ? Where is it provided from ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697339572
##########
oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexProviderService.java:
##########
@@ -572,24 +567,7 @@ ExecutorService getExecutorService(){
}
private ExecutorService createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(5, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(INDEX_COPIER_POOL_SIZE, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
```suggestion
ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(
INDEX_COPIER_POOL_SIZE, "oak-lucene-%d",
(t, e) -> log.warn("Error occurred in asynchronous processing ", e));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2693565043
##########
oak-auth-ldap/pom.xml:
##########
@@ -104,7 +104,6 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
- <scope>provided</scope>
Review Comment:
this needs to be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697274771
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * ThreadFactory which names the threads based on a pattern. The pattern should contain a %d which is replaced by
+ * a unique number for each thread generated by this factory.
+ */
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
Added the dependency. @reschke Ok with you ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697281494
##########
oak-auth-ldap/pom.xml:
##########
@@ -104,7 +104,6 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
- <scope>provided</scope>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697338811
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,26 +171,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.onDemandSingleThreadLinkedQueueExecutor("oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
```suggestion
ThreadPoolExecutor executor = ExecutorHelper.onDemandSingleThreadLinkedQueueExecutor(
"oak-lucene-%d",
(t, e) -> log.warn("Error occurred in asynchronous processing ", e));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2698091205
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/ExecutorHelper.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.apache.commons.lang3.concurrent.BasicThreadFactory;
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Helper class complementing JDK's @see java.util.concurrent.Executors class with additional ExecutionService creation
+ */
+public class ExecutorHelper {
+
+ /**
+ * Create a ThreadPoolExecutor with an unbounded LinkedBlockingQueue as work queue.
+ * <br>
+ * NOTE: The maximum number of threads in each executor service, when using a LinkedBlockingQueue(), is corePoolSize.
+ * all other tasks are kept in the LinkedBlockingQueue, which is unbounded.
+ * <br>
+ * @see <a href="https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html">ThreadPoolExecutor javadoc</a>
+ * <br>
+ * (Using a bounded queue, such as SynchronousQueue would result in RejectedExecutionHandler if the queue would reach its capacity).
+ * We want to keep things simple and don't want to use back pressure or other mechanisms. So in summary, corePoolSize threads are used, per service.
+ * @param poolSize The size of the thread pool
+ * @param namePattern The name pattern for the threads containing %d replaced by the thread number (calculated by NamedThreadFactory).
+ * @param handler An optional handler for uncaught exceptions
+ * @return A ThreadPoolExecutor
+ */
+ public static ThreadPoolExecutor linkedQueueExecutor(int poolSize, @NotNull String namePattern, Thread.UncaughtExceptionHandler handler) {
+ return linkedQueueExecutor(poolSize, poolSize, namePattern, handler);
+ }
+
+ private static ThreadPoolExecutor linkedQueueExecutor(int minSize, int maxSize, @NotNull String namePattern, Thread.UncaughtExceptionHandler handler) {
+ // Either minSize=0 and maxSize=1 (maxSize > 1 behaves as if maxSize were 1)
+ // or minSize=maxSize (maxSize <> minSize behaves as if minSize were maxSize)
+ var factory = BasicThreadFactory.builder()
+ .daemon(true)
+ .namingPattern(namePattern);
+ if(handler != null) {
+ factory.uncaughtExceptionHandler(handler);
+ }
+ return new ThreadPoolExecutor(minSize, maxSize, 60L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(), factory.build());
+ }
+
+ /**
+ * Create a ThreadPoolExecutor with an unbounded LinkedBlockingQueue as work queue which will start a single thread
+ * in presence of work and will shut down the thread after 60 seconds of inactivity.
+ * <br>
+ * NOTE: ThreadPoolExecutor with an (unbounded) LinkedBlockingQueue does not allow setting both corePoolSize and maximumPoolSize,
+ * and thus does not allow starting more than a single thread in combination with reduction to 0.
+ * @param namePattern The name pattern for the threads containing %d replaced by the thread number (calculated by NamedThreadFactory).
+ * @param handler An optional handler for uncaught exceptions
+ * @return A ThreadPoolExecutor which will activate a single thread on demand and shut it down after 60 seconds of inactivity.
+ */
+ public static ThreadPoolExecutor onDemandSingleThreadLinkedQueueExecutor(@NotNull String namePattern, Thread.UncaughtExceptionHandler handler) {
+ var factory = BasicThreadFactory.builder()
+ .daemon(true)
+ .namingPattern(namePattern);
+ if(handler != null) {
+ factory.uncaughtExceptionHandler(handler);
+ }
+ return new ThreadPoolExecutor(0, 1, 60L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(), factory.build());
+ }
Review Comment:
Exactly! Thanks. That's why I created the internal method in the first place. 😜
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "reschke (via GitHub)" <gi...@apache.org>.
reschke commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2685992464
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.concurrent;
+
+import org.jspecify.annotations.NonNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
this needs to go into the "internal" tree (internal/concurrent)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697271384
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,25 +172,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(5, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
As suggested by @thomasmueller, in this MR, let's keep the effective behavior (rather than the maybe intended one)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697348575
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * ThreadFactory which names the threads based on a pattern. The pattern should contain a %d which is replaced by
+ * a unique number for each thread generated by this factory.
+ */
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
I probably wouldn't add the dependency. (I wouldn't add indirection or a dependency if the benefit is tiny, as it is here.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2693536728
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * ThreadFactory which names the threads based on a pattern. The pattern should contain a %d which is replaced by
+ * a unique number for each thread generated by this factory.
+ */
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
we should add `commons-lang3` as a dependency in oak-commons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2685779619
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -346,26 +344,7 @@ private synchronized void createExecutor() {
return;
}
log.debug("ExtractedTextCache createExecutor {}", this);
- ThreadPoolExecutor executor = new ThreadPoolExecutor(1, EXTRACTION_MAX_THREADS,
- 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- int index = counter.getAndIncrement();
- return "oak binary text extractor" + (index == 0 ? "" : " " + index);
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(EXTRACTION_MAX_THREADS, "oak-binary-text-extractor-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
Here we again have the same behavior as before, only 1 thread all the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697341686
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -346,26 +344,7 @@ private synchronized void createExecutor() {
return;
}
log.debug("ExtractedTextCache createExecutor {}", this);
- ThreadPoolExecutor executor = new ThreadPoolExecutor(1, EXTRACTION_MAX_THREADS,
- 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- int index = counter.getAndIncrement();
- return "oak binary text extractor" + (index == 0 ? "" : " " + index);
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(EXTRACTION_POOL_SIZE, "oak-binary-text-extractor-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
```suggestion
ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(
EXTRACTION_POOL_SIZE, "oak-binary-text-extractor-%d",
(t, e) -> log.warn("Error occurred in asynchronous processing ", e));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697339572
##########
oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexProviderService.java:
##########
@@ -572,24 +567,7 @@ ExecutorService getExecutorService(){
}
private ExecutorService createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(5, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(INDEX_COPIER_POOL_SIZE, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
```suggestion
ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(
INDEX_COPIER_POOL_SIZE,
"oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
```
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -346,26 +344,7 @@ private synchronized void createExecutor() {
return;
}
log.debug("ExtractedTextCache createExecutor {}", this);
- ThreadPoolExecutor executor = new ThreadPoolExecutor(1, EXTRACTION_MAX_THREADS,
- 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- int index = counter.getAndIncrement();
- return "oak binary text extractor" + (index == 0 ? "" : " " + index);
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(EXTRACTION_POOL_SIZE, "oak-binary-text-extractor-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
```suggestion
ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(
EXTRACTION_POOL_SIZE,
"oak-binary-text-extractor-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
```
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,26 +171,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.onDemandSingleThreadLinkedQueueExecutor("oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
```suggestion
ThreadPoolExecutor executor = ExecutorHelper.onDemandSingleThreadLinkedQueueExecutor(
"oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
```
##########
oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticIndexStatistics.java:
##########
@@ -69,12 +67,9 @@ public class ElasticIndexStatistics implements IndexStatistics {
private static final String REFRESH_SECONDS = "oak.elastic.statsRefreshSeconds";
private static final Long REFRESH_SECONDS_DEFAULT = 60L;
- private static final ExecutorService REFRESH_EXECUTOR = new ThreadPoolExecutor(
- 0, 4, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("elastic-statistics-cache-refresh-thread-%d").daemon().build()
- );
+ private static final int REFRESH_POOL_SIZE = 4;
+ private static final ExecutorService REFRESH_EXECUTOR = ExecutorHelper.linkedQueueExecutor(REFRESH_POOL_SIZE, "elastic-statistics-cache-refresh-%d");
Review Comment:
```suggestion
private static final ExecutorService REFRESH_EXECUTOR = ExecutorHelper.linkedQueueExecutor(
REFRESH_POOL_SIZE, "elastic-statistics-cache-refresh-%d");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller merged PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2685756033
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,25 +172,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(5, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
@nit0906 What was the intent here ? I'm guessing 0 threads when no load and up to 5 threads when under load. Currently we have 0 threads initially, 1 thread gets created when load arrives, it will never reach 5 (as per the doc), and it will go back down to 0 threads after when the keep alive timeout is reached.
What is more important ?
- Having 5 threads under load (but then systematically 5 threads and never 0)
- Having 0 threads when there is no load (but then only having 1 thread under load)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2693536728
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * ThreadFactory which names the threads based on a pattern. The pattern should contain a %d which is replaced by
+ * a unique number for each thread generated by this factory.
+ */
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
we should add `commons-lang3` as a dependency in oak-commons. @reschke wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2689615795
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * ThreadFactory which names the threads based on a pattern. The pattern should contain a %d which is replaced by
+ * a unique number for each thread generated by this factory.
+ */
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
Why not use the `BasicThreadFactory` from `commons-lang3` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2685767570
##########
oak-run/src/main/java/org/apache/jackrabbit/oak/run/Downloader.java:
##########
@@ -128,26 +126,8 @@ public Downloader(int concurrency, int connectTimeoutMs, int readTimeoutMs, int
this.checksumAlgorithm = null;
}
this.bufferSize = bufferSize;
-
- // The maximum number of threads in each executor service,
- // when using a LinkedBlockingQueue(), is corePoolSize.
- // all other tasks are kept in the LinkedBlockingQueue, which
- // is unbounded.
- // (Using a bounded queue, such as SynchronousQueue,
- // would result in RejectedExecutionHandler).
- // We want to keep things simple and don't want
- // to use back presssure or other mechanisms.
- // So in summary, corePoolSize threads are used, per service.
- this.executorService = new ThreadPoolExecutor(
- corePoolSize, concurrency, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("downloader-%d").daemon().build()
- );
- this.executorServiceForParts = new ThreadPoolExecutor(
- corePoolSize, concurrency, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("partDownloader-%d").daemon().build()
- );
+ this.executorService = ExecutorHelper.linkedQueueExecutor(corePoolSize, "downloader-%d");
Review Comment:
Here the behavior will be the same as in both before and after `corePoolSize` threads will be created and will never change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2685773758
##########
oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticIndexStatistics.java:
##########
@@ -69,12 +67,7 @@ public class ElasticIndexStatistics implements IndexStatistics {
private static final String REFRESH_SECONDS = "oak.elastic.statsRefreshSeconds";
private static final Long REFRESH_SECONDS_DEFAULT = 60L;
- private static final ExecutorService REFRESH_EXECUTOR = new ThreadPoolExecutor(
- 0, 4, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("elastic-statistics-cache-refresh-thread-%d").daemon().build()
- );
-
+ private static final ExecutorService REFRESH_EXECUTOR = ExecutorHelper.linkedQueueExecutor(4, "elastic-statistics-cache-refresh-%d");
Review Comment:
@fabriziofortino Same question as above: What do we want here 0 or 1 threads or always 4 threads ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2754910393
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -64,8 +62,8 @@ public class ExtractedTextCache {
Boolean.getBoolean("oak.extracted.cacheOnlySuccess");
private static final int EXTRACTION_TIMEOUT_SECONDS =
Integer.getInteger("oak.extraction.timeoutSeconds", 60);
- private static final int EXTRACTION_MAX_THREADS =
- Integer.getInteger("oak.extraction.maxThreads", 10);
+ private static final int EXTRACTION_POOL_SIZE =
+ Integer.getInteger("oak.extraction.poolSize", 1);
Review Comment:
I think that makes sense. Someone might have used the system property, but that had no effect. So best if we rename it.
##########
oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexProviderService.java:
##########
@@ -266,6 +263,7 @@ public class LuceneIndexProviderService {
}
public static final String REPOSITORY_HOME = "repository.home";
+ private static final int INDEX_COPIER_POOL_SIZE = 5;
Review Comment:
Nit: if we only use it in one place, it seems not useful to have a constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2755177581
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -64,8 +62,8 @@ public class ExtractedTextCache {
Boolean.getBoolean("oak.extracted.cacheOnlySuccess");
private static final int EXTRACTION_TIMEOUT_SECONDS =
Integer.getInteger("oak.extraction.timeoutSeconds", 60);
- private static final int EXTRACTION_MAX_THREADS =
- Integer.getInteger("oak.extraction.maxThreads", 10);
+ private static final int EXTRACTION_POOL_SIZE =
+ Integer.getInteger("oak.extraction.poolSize", 1);
Review Comment:
Is this enough ? Or can I resolve this conversation ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2755163793
##########
oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexProviderService.java:
##########
@@ -266,6 +263,7 @@ public class LuceneIndexProviderService {
}
public static final String REPOSITORY_HOME = "repository.home";
+ private static final int INDEX_COPIER_POOL_SIZE = 5;
Review Comment:
I see 2 benefits though:
- readability by having a name rather than just a number
- having the "setup" of the class at the top (these are potential candidate parameters)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "thomasmueller (via GitHub)" <gi...@apache.org>.
thomasmueller commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697341686
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -346,26 +344,7 @@ private synchronized void createExecutor() {
return;
}
log.debug("ExtractedTextCache createExecutor {}", this);
- ThreadPoolExecutor executor = new ThreadPoolExecutor(1, EXTRACTION_MAX_THREADS,
- 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- int index = counter.getAndIncrement();
- return "oak binary text extractor" + (index == 0 ? "" : " " + index);
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(EXTRACTION_POOL_SIZE, "oak-binary-text-extractor-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
```suggestion
ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(
EXTRACTION_POOL_SIZE,
"oak-binary-text-extractor-%d",
(t, e) -> log.warn("Error occurred in asynchronous processing ", e));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "joerghoh (via GitHub)" <gi...@apache.org>.
joerghoh commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2677632182
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/ExecutorHelper.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.concurrent;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public class ExecutorHelper {
+ public static ThreadPoolExecutor linkedQueueExecutor(int poolSize, String namePattern, Thread.UncaughtExceptionHandler handler) {
Review Comment:
Javadoc missing
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.concurrent;
+
+import org.jspecify.annotations.NonNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
What about having it into the ExecutorHelper class directly?
##########
oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticIndexStatistics.java:
##########
@@ -69,12 +67,7 @@ public class ElasticIndexStatistics implements IndexStatistics {
private static final String REFRESH_SECONDS = "oak.elastic.statsRefreshSeconds";
private static final Long REFRESH_SECONDS_DEFAULT = 60L;
- private static final ExecutorService REFRESH_EXECUTOR = new ThreadPoolExecutor(
- 0, 4, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("elastic-statistics-cache-refresh-thread-%d").daemon().build()
- );
-
+ private static final ExecutorService REFRESH_EXECUTOR = ExecutorHelper.linkedQueueExecutor(4, "elastic-statistics-cache-refresh-%d");
Review Comment:
same as above
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.concurrent;
+
+import org.jspecify.annotations.NonNull;
Review Comment:
IIRC for the majority of Oak we (still) use the jetbrains annotations, can you switch to that too?
(A potential migration to jspecify should be handled separately.)
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,25 +172,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(5, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
This is not identical to the code you're removing, because with the change it will always have 5 threads running, while in the original it can go down to 0 threads if no work is required.
##########
oak-run/src/main/java/org/apache/jackrabbit/oak/run/Downloader.java:
##########
@@ -128,26 +126,8 @@ public Downloader(int concurrency, int connectTimeoutMs, int readTimeoutMs, int
this.checksumAlgorithm = null;
}
this.bufferSize = bufferSize;
-
- // The maximum number of threads in each executor service,
- // when using a LinkedBlockingQueue(), is corePoolSize.
- // all other tasks are kept in the LinkedBlockingQueue, which
- // is unbounded.
- // (Using a bounded queue, such as SynchronousQueue,
- // would result in RejectedExecutionHandler).
- // We want to keep things simple and don't want
- // to use back presssure or other mechanisms.
- // So in summary, corePoolSize threads are used, per service.
- this.executorService = new ThreadPoolExecutor(
- corePoolSize, concurrency, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("downloader-%d").daemon().build()
- );
- this.executorServiceForParts = new ThreadPoolExecutor(
- corePoolSize, concurrency, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("partDownloader-%d").daemon().build()
- );
+ this.executorService = ExecutorHelper.linkedQueueExecutor(corePoolSize, "downloader-%d");
Review Comment:
same as above, the number of threads running consistently is different than before.
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/ExecutorHelper.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.concurrent;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public class ExecutorHelper {
+ public static ThreadPoolExecutor linkedQueueExecutor(int poolSize, String namePattern, Thread.UncaughtExceptionHandler handler) {
+ // The maximum number of threads in each executor service,
+ // when using a LinkedBlockingQueue(), is corePoolSize.
+ // all other tasks are kept in the LinkedBlockingQueue, which
+ // is unbounded.
+ // (Using a bounded queue, such as SynchronousQueue,
+ // would result in RejectedExecutionHandler).
+ // We want to keep things simple and don't want
+ // to use back pressure or other mechanisms.
+ // So in summary, corePoolSize threads are used, per service.
+ return new ThreadPoolExecutor(poolSize, poolSize, 60L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(), new NamedThreadFactory(namePattern, handler));
+ }
+
+ public static ExecutorService linkedQueueExecutor(int poolSize, String namePattern) {
Review Comment:
missing Javadoc
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -346,26 +344,7 @@ private synchronized void createExecutor() {
return;
}
log.debug("ExtractedTextCache createExecutor {}", this);
- ThreadPoolExecutor executor = new ThreadPoolExecutor(1, EXTRACTION_MAX_THREADS,
- 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- int index = counter.getAndIncrement();
- return "oak binary text extractor" + (index == 0 ? "" : " " + index);
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(EXTRACTION_MAX_THREADS, "oak-binary-text-extractor-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2681130056
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,25 +172,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(5, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
In fact, this is part of the issue. In the original code the number of threads is either 0 or 1. It will never go above 1 as the maximumPoolSize is ignored in case of an unbounded queue (as per https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html):
> 2. Unbounded queues. Using an unbounded queue (for example a [LinkedBlockingQueue](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/LinkedBlockingQueue.html) without a predefined capacity) will cause new tasks to wait in the queue when all corePoolSize threads are busy. Thus, no more than corePoolSize threads will ever be created. (And _**the value of the maximumPoolSize therefore doesn't have any effect.**_) This may be appropriate when each task is completely independent of others, so tasks cannot affect each others execution; for example, in a web page server. While this style of queuing can be useful in smoothing out transient bursts of requests, it admits the possibility of unbounded work queue growth when commands continue to arrive on average faster than they can be processed.
(If fact in this particular case corePoolSize 0 it will go above but only to 1 thread).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#issuecomment-3737300560
> @bhabegger good catch, thanks for the PR; can you create a dedicated OAK ticket for it?
Well, in fact it was hinted to me by @thomasmueller as a pending task that I took for my onboarding :) So, I don't have much credit for catching this;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2755165602
##########
oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexProviderService.java:
##########
@@ -266,6 +263,7 @@ public class LuceneIndexProviderService {
}
public static final String REPOSITORY_HOME = "repository.home";
+ private static final int INDEX_COPIER_POOL_SIZE = 5;
Review Comment:
(But I won't fight over it :))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697279766
##########
oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticIndexStatistics.java:
##########
@@ -69,12 +67,7 @@ public class ElasticIndexStatistics implements IndexStatistics {
private static final String REFRESH_SECONDS = "oak.elastic.statsRefreshSeconds";
private static final Long REFRESH_SECONDS_DEFAULT = 60L;
- private static final ExecutorService REFRESH_EXECUTOR = new ThreadPoolExecutor(
- 0, 4, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("elastic-statistics-cache-refresh-thread-%d").daemon().build()
- );
-
+ private static final ExecutorService REFRESH_EXECUTOR = ExecutorHelper.linkedQueueExecutor(4, "elastic-statistics-cache-refresh-%d");
Review Comment:
As before, for the time being, same effective behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2693216661
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * ThreadFactory which names the threads based on a pattern. The pattern should contain a %d which is replaced by
+ * a unique number for each thread generated by this factory.
+ */
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
We could indeed. However, the dependency to commons-lang3 is not yet in oak-commons module and this would pull it in transitively in other modules where in many of them have it marked as "provided". Being new to this project, I'm not clear on why that could be and if it would be a problem to add the dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697745982
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * ThreadFactory which names the threads based on a pattern. The pattern should contain a %d which is replaced by
+ * a unique number for each thread generated by this factory.
+ */
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
@thomasmueller `commons-lang3` is already part of oak-commons but as a test dependency. So we are already using it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OAK-12054: Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "rishabhdaim (via GitHub)" <gi...@apache.org>.
rishabhdaim commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2697752428
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/internal/concurrent/ExecutorHelper.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.internal.concurrent;
+
+import org.apache.commons.lang3.concurrent.BasicThreadFactory;
+import org.jetbrains.annotations.NotNull;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Helper class complementing JDK's @see java.util.concurrent.Executors class with additional ExecutionService creation
+ */
+public class ExecutorHelper {
+
+ /**
+ * Create a ThreadPoolExecutor with an unbounded LinkedBlockingQueue as work queue.
+ * <br>
+ * NOTE: The maximum number of threads in each executor service, when using a LinkedBlockingQueue(), is corePoolSize.
+ * all other tasks are kept in the LinkedBlockingQueue, which is unbounded.
+ * <br>
+ * @see <a href="https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html">ThreadPoolExecutor javadoc</a>
+ * <br>
+ * (Using a bounded queue, such as SynchronousQueue would result in RejectedExecutionHandler if the queue would reach its capacity).
+ * We want to keep things simple and don't want to use back pressure or other mechanisms. So in summary, corePoolSize threads are used, per service.
+ * @param poolSize The size of the thread pool
+ * @param namePattern The name pattern for the threads containing %d replaced by the thread number (calculated by NamedThreadFactory).
+ * @param handler An optional handler for uncaught exceptions
+ * @return A ThreadPoolExecutor
+ */
+ public static ThreadPoolExecutor linkedQueueExecutor(int poolSize, @NotNull String namePattern, Thread.UncaughtExceptionHandler handler) {
+ return linkedQueueExecutor(poolSize, poolSize, namePattern, handler);
+ }
+
+ private static ThreadPoolExecutor linkedQueueExecutor(int minSize, int maxSize, @NotNull String namePattern, Thread.UncaughtExceptionHandler handler) {
+ // Either minSize=0 and maxSize=1 (maxSize > 1 behaves as if maxSize were 1)
+ // or minSize=maxSize (maxSize <> minSize behaves as if minSize were maxSize)
+ var factory = BasicThreadFactory.builder()
+ .daemon(true)
+ .namingPattern(namePattern);
+ if(handler != null) {
+ factory.uncaughtExceptionHandler(handler);
+ }
+ return new ThreadPoolExecutor(minSize, maxSize, 60L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(), factory.build());
+ }
+
+ /**
+ * Create a ThreadPoolExecutor with an unbounded LinkedBlockingQueue as work queue which will start a single thread
+ * in presence of work and will shut down the thread after 60 seconds of inactivity.
+ * <br>
+ * NOTE: ThreadPoolExecutor with an (unbounded) LinkedBlockingQueue does not allow setting both corePoolSize and maximumPoolSize,
+ * and thus does not allow starting more than a single thread in combination with reduction to 0.
+ * @param namePattern The name pattern for the threads containing %d replaced by the thread number (calculated by NamedThreadFactory).
+ * @param handler An optional handler for uncaught exceptions
+ * @return A ThreadPoolExecutor which will activate a single thread on demand and shut it down after 60 seconds of inactivity.
+ */
+ public static ThreadPoolExecutor onDemandSingleThreadLinkedQueueExecutor(@NotNull String namePattern, Thread.UncaughtExceptionHandler handler) {
+ var factory = BasicThreadFactory.builder()
+ .daemon(true)
+ .namingPattern(namePattern);
+ if(handler != null) {
+ factory.uncaughtExceptionHandler(handler);
+ }
+ return new ThreadPoolExecutor(0, 1, 60L, TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(), factory.build());
+ }
Review Comment:
It should call `linkedQueueExecutor` with min/max as 0/1 instead of duplicating the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "joerghoh (via GitHub)" <gi...@apache.org>.
joerghoh commented on PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#issuecomment-3730602594
@bhabegger good catch, thanks for the PR; can you create a dedicated OAK ticket for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor creation of ThreadPoolExecutors [jackrabbit-oak]
Posted by "bhabegger (via GitHub)" <gi...@apache.org>.
bhabegger commented on code in PR #2679:
URL: https://github.com/apache/jackrabbit-oak/pull/2679#discussion_r2681137910
##########
oak-commons/src/main/java/org/apache/jackrabbit/oak/commons/concurrent/NamedThreadFactory.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.jackrabbit.oak.commons.concurrent;
+
+import org.jspecify.annotations.NonNull;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class NamedThreadFactory implements ThreadFactory {
Review Comment:
I thought about it and came to the conclusion that the class could be useful by itself. But either way is fine by me.
##########
oak-search/src/main/java/org/apache/jackrabbit/oak/plugins/index/search/ExtractedTextCache.java:
##########
@@ -346,26 +344,7 @@ private synchronized void createExecutor() {
return;
}
log.debug("ExtractedTextCache createExecutor {}", this);
- ThreadPoolExecutor executor = new ThreadPoolExecutor(1, EXTRACTION_MAX_THREADS,
- 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler = (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- int index = counter.getAndIncrement();
- return "oak binary text extractor" + (index == 0 ? "" : " " + index);
- }
- });
- executor.setKeepAliveTime(1, TimeUnit.MINUTES);
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(EXTRACTION_MAX_THREADS, "oak-binary-text-extractor-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
same :)
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,25 +172,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(5, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
In fat, this is part of the issue. In the original code the number of threads is either 0 or 1. It will never go above 1 as the maximumPoolSize is ignored in case of an unbounded queue (as per https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html):
> 2. Unbounded queues. Using an unbounded queue (for example a [LinkedBlockingQueue](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/LinkedBlockingQueue.html) without a predefined capacity) will cause new tasks to wait in the queue when all corePoolSize threads are busy. Thus, no more than corePoolSize threads will ever be created. (And _**the value of the maximumPoolSize therefore doesn't have any effect.**_) This may be appropriate when each task is completely independent of others, so tasks cannot affect each others execution; for example, in a web page server. While this style of queuing can be useful in smoothing out transient bursts of requests, it admits the possibility of unbounded work queue growth when commands continue to arrive on average faster than they can be processed.
(If fact in this particular case corePoolSize 0 it will go above but only to 1 thread).
##########
oak-run/src/main/java/org/apache/jackrabbit/oak/run/Downloader.java:
##########
@@ -128,26 +126,8 @@ public Downloader(int concurrency, int connectTimeoutMs, int readTimeoutMs, int
this.checksumAlgorithm = null;
}
this.bufferSize = bufferSize;
-
- // The maximum number of threads in each executor service,
- // when using a LinkedBlockingQueue(), is corePoolSize.
- // all other tasks are kept in the LinkedBlockingQueue, which
- // is unbounded.
- // (Using a bounded queue, such as SynchronousQueue,
- // would result in RejectedExecutionHandler).
- // We want to keep things simple and don't want
- // to use back presssure or other mechanisms.
- // So in summary, corePoolSize threads are used, per service.
- this.executorService = new ThreadPoolExecutor(
- corePoolSize, concurrency, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("downloader-%d").daemon().build()
- );
- this.executorServiceForParts = new ThreadPoolExecutor(
- corePoolSize, concurrency, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("partDownloader-%d").daemon().build()
- );
+ this.executorService = ExecutorHelper.linkedQueueExecutor(corePoolSize, "downloader-%d");
Review Comment:
same :)
##########
oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elastic/ElasticIndexStatistics.java:
##########
@@ -69,12 +67,7 @@ public class ElasticIndexStatistics implements IndexStatistics {
private static final String REFRESH_SECONDS = "oak.elastic.statsRefreshSeconds";
private static final Long REFRESH_SECONDS_DEFAULT = 60L;
- private static final ExecutorService REFRESH_EXECUTOR = new ThreadPoolExecutor(
- 0, 4, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- BasicThreadFactory.builder().namingPattern("elastic-statistics-cache-refresh-thread-%d").daemon().build()
- );
-
+ private static final ExecutorService REFRESH_EXECUTOR = ExecutorHelper.linkedQueueExecutor(4, "elastic-statistics-cache-refresh-%d");
Review Comment:
same :)
##########
oak-run-commons/src/main/java/org/apache/jackrabbit/oak/index/IndexHelper.java:
##########
@@ -174,25 +172,7 @@ protected void bindIndexInfoProviders(IndexInfoServiceImpl indexInfoService) {
}
private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 5, 60L, TimeUnit.SECONDS,
- new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
- private final AtomicInteger counter = new AtomicInteger();
- private final Thread.UncaughtExceptionHandler handler =
- (t, e) -> log.warn("Error occurred in asynchronous processing ", e);
-
- @Override
- public Thread newThread(@NotNull Runnable r) {
- Thread thread = new Thread(r, createName());
- thread.setDaemon(true);
- thread.setPriority(Thread.MIN_PRIORITY);
- thread.setUncaughtExceptionHandler(handler);
- return thread;
- }
-
- private String createName() {
- return "oak-lucene-" + counter.getAndIncrement();
- }
- });
+ ThreadPoolExecutor executor = ExecutorHelper.linkedQueueExecutor(5, "oak-lucene-%d", (t, e) -> log.warn("Error occurred in asynchronous processing ", e));
Review Comment:
So the question is: do we want 1 thread (this would have the code running as it currently is) or 5 threads (which seems to have been the intent in case of load)....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: oak-dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org