You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@solr.apache.org by ds...@apache.org on 2024/02/04 00:26:19 UTC
(solr) branch branch_9x updated (0ce7a6690ca -> 563acd2cac8)
This is an automated email from the ASF dual-hosted git repository.
dsmiley pushed a change to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/solr.git
from 0ce7a6690ca SOLR-17068: Fix tidy/check
new d17808347aa SOLR-17144: Do not keep an active thread per core (#2236)
new 563acd2cac8 Optimize ZkStateReader.refreshCollectionList (#2217)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
solr/CHANGES.txt | 7 +++++--
solr/core/src/java/org/apache/solr/core/SolrCore.java | 4 +++-
.../java/org/apache/solr/common/cloud/ZkStateReader.java | 2 +-
.../java/org/apache/solr/common/util/ExecutorUtil.java | 15 ++++++++++++++-
4 files changed, 23 insertions(+), 5 deletions(-)
(solr) 01/02: SOLR-17144: Do not keep an active thread per core (#2236)
Posted by ds...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dsmiley pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/solr.git
commit d17808347aa70ef9838e53a6f430220c598edf53
Author: Pierre Salagnac <82...@users.noreply.github.com>
AuthorDate: Sat Feb 3 22:23:46 2024 +0100
SOLR-17144: Do not keep an active thread per core (#2236)
Co-authored-by: Pierre Salagnac <ps...@salesforce.com>
---
solr/CHANGES.txt | 2 +-
solr/core/src/java/org/apache/solr/core/SolrCore.java | 4 +++-
.../java/org/apache/solr/common/util/ExecutorUtil.java | 15 ++++++++++++++-
3 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 5d807dc036f..a34cfdbcff2 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -19,7 +19,7 @@ Improvements
Optimizations
---------------------
-(No changes)
+* SOLR-17144: Close searcherExecutor thread per core after 1 minute (Pierre Salagnac, Christine Poerschke)
Bug Fixes
---------------------
diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.java
index be6f7526b3d..c72d9debc55 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java
@@ -2056,8 +2056,10 @@ public class SolrCore implements SolrInfoBean, Closeable {
private final ArrayDeque<RefCounted<SolrIndexSearcher>> _searchers = new ArrayDeque<>();
private final ArrayDeque<RefCounted<SolrIndexSearcher>> _realtimeSearchers = new ArrayDeque<>();
+ // Single threaded pool, with a time-to-keep of 60 seconds
final ExecutorService searcherExecutor =
- ExecutorUtil.newMDCAwareSingleThreadExecutor(new SolrNamedThreadFactory("searcherExecutor"));
+ ExecutorUtil.newMDCAwareSingleLazyThreadExecutor(
+ new SolrNamedThreadFactory("searcherExecutor"), 60L, TimeUnit.SECONDS);
private int onDeckSearchers; // number of searchers preparing
// Lock ordering: one can acquire the openSearcherLock and then the searcherLock, but not
// vice-versa.
diff --git a/solr/solrj/src/java/org/apache/solr/common/util/ExecutorUtil.java b/solr/solrj/src/java/org/apache/solr/common/util/ExecutorUtil.java
index cfa7b176d50..f0d1f8bc698 100644
--- a/solr/solrj/src/java/org/apache/solr/common/util/ExecutorUtil.java
+++ b/solr/solrj/src/java/org/apache/solr/common/util/ExecutorUtil.java
@@ -150,12 +150,25 @@ public class ExecutorUtil {
nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(), threadFactory);
}
- /** See {@link java.util.concurrent.Executors#newSingleThreadExecutor(ThreadFactory)} */
+ /**
+ * See {@link java.util.concurrent.Executors#newSingleThreadExecutor(ThreadFactory)}. Note the
+ * thread is always active, even if no tasks are submitted to the executor.
+ */
public static ExecutorService newMDCAwareSingleThreadExecutor(ThreadFactory threadFactory) {
return new MDCAwareThreadPoolExecutor(
1, 1, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(), threadFactory);
}
+ /**
+ * Similar to {@link #newMDCAwareSingleThreadExecutor(ThreadFactory)}, but the thread will not be
+ * kept active after the specified time if no task is submitted to the executor.
+ */
+ public static ExecutorService newMDCAwareSingleLazyThreadExecutor(
+ ThreadFactory threadFactory, long keepAliveTime, TimeUnit unit) {
+ return new MDCAwareThreadPoolExecutor(
+ 0, 1, keepAliveTime, unit, new LinkedBlockingQueue<>(), threadFactory);
+ }
+
/** Create a cached thread pool using a named thread factory */
public static ExecutorService newMDCAwareCachedThreadPool(String name) {
return newMDCAwareCachedThreadPool(new SolrNamedThreadFactory(name));
(solr) 02/02: Optimize ZkStateReader.refreshCollectionList (#2217)
Posted by ds...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dsmiley pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/solr.git
commit 563acd2cac85dbdb5ad58ea3ceb2630720dadf8d
Author: David Smiley <ds...@apache.org>
AuthorDate: Sat Feb 3 16:28:11 2024 -0500
Optimize ZkStateReader.refreshCollectionList (#2217)
Avoid O(N^2) for many collections.
The arg to retain(col) should ideally always be a Set.
---
solr/CHANGES.txt | 5 ++++-
.../src/java/org/apache/solr/common/cloud/ZkStateReader.java | 2 +-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index a34cfdbcff2..dc99b116d59 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -21,6 +21,9 @@ Optimizations
---------------------
* SOLR-17144: Close searcherExecutor thread per core after 1 minute (Pierre Salagnac, Christine Poerschke)
+* GITHUB#2217: Scale to 10K+ collections better in ZkStateReader.refreshCollectionsList (David Smiley)
+
+
Bug Fixes
---------------------
(No changes)
@@ -103,7 +106,7 @@ Improvements
Now available at `GET /api/node/commands/someRequestId` (Sanjay Dutt via Jason Gerlowski)
* SOLR-17068: bin/solr post CLI use of options is now aligned closely with bin/post CLI tool, and is consistently
- referenced throughout the Ref Guide and source code, and is used through out our tests. The bin/post tool
+ referenced throughout the Ref Guide and source code, and is used through out our tests. The bin/post tool
remains and has been tested to work. (Eric Pugh)
Optimizations
diff --git a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
index d3f27730139..ffb19c3a32f 100644
--- a/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
+++ b/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ZkStateReader.java
@@ -676,7 +676,7 @@ public class ZkStateReader implements SolrCloseable {
// Don't mess with watchedCollections, they should self-manage.
// First, drop any children that disappeared.
- this.lazyCollectionStates.keySet().retainAll(children);
+ this.lazyCollectionStates.keySet().retainAll(new HashSet<>(children)); // Set avoids O(N^2)
for (String coll : children) {
// We will create an eager collection for any interesting collections, so don't add to lazy.
if (!collectionWatches.watchedCollections().contains(coll)) {