You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "N624-debu (via GitHub)" <gi...@apache.org> on 2025/07/05 18:40:44 UTC
[PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
N624-debu opened a new pull request, #14900:
URL: https://github.com/apache/lucene/pull/14900
# [Draft] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling
This draft PR introduces a prototype `SharedMergeScheduler`, which extends `MergeScheduler` and routes all merge tasks through a shared thread pool across `IndexWriter` instances.
## Motivation
In multi-tenant environments (e.g., Solr, Elasticsearch), many `IndexWriter`s may coexist in the same JVM. The default Lucene behavior assigns each writer its own `ConcurrentMergeScheduler` and thread pool, which can lead to resource oversubscription and inefficient coordination.
This implementation introduces a new merge scheduler that centralizes merge execution via a shared thread pool. This idea builds on feedback from [GitHub issue #13883](https://github.com/apache/lucene/issues/13883), where contributors suggested exploring a dedicated scheduler based on Java's executor framework, rather than modifying the existing `ConcurrentMergeScheduler`.
## Implementation Highlights
- Adds a new class `SharedMergeScheduler` in `lucene.index`
- Implements the `merge(MergeSource, MergeTrigger)` method using the public `MergeSource` API
- Uses a static `Executors.newFixedThreadPool(4)` as a prototype shared executor
- Keeps `ConcurrentMergeScheduler` unchanged for easier evaluation and iteration
## Next Steps
- Add lifecycle management and graceful shutdown to the executor
- Evaluate fairness or throttling strategies across writers
- Potentially combine with a centralized merge manager
- Seek feedback on architecture fit and maintainability
---
This PR is opened to propose and evaluate a shared-thread-pool merge scheduler design, and to gather feedback for further development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3135188800
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3042617156
@vigyasharma what would you think of the following:
- Instead of making it a singleton, add the executor as a constructor argument so that the same executor can be used for indexing and merging? (plus add javadocs to recommend using a fixed thread pool and sharing it with indexing)
- Tasks get wrapped when submitted to the executor, similarly to what `TaskExecutor` does with `TaskExecutor.Task`, to allow merges to run in the current thread if they don't get picked up by the executor in a timely fashion.
- To mimic CMS, maybe this Task wrapper should keep pulling pending merges that are below MIN_BIG_MERGE_MB=50MB without waiting for them to go through the executor queue so that they always run almost immediately? (this requires tracking pending merges in a separate queue)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3186265199
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "vigyasharma (via GitHub)" <gi...@apache.org>.
vigyasharma commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3046266969
Based on all this, I think we get the following structure:
1. A merge scheduler that accepts an `ExecutorService` in its ctor. It will offload most of the scheduling / queueing / throttling logic to the executor service and its backing queue implementation. We will recommend using `MergeTasksExecutorService` or `IndexingMergeSharedExecutorService` but users are free to pass in their own executors.
2. A `MergeTasksExecutorService` which does CMS like scheduling but for merge tasks only. The fixed thread pool is shared across all submitted merges across all writers. This executor service ~~assumes~~ requires a separate indexing thread pool, and puts back-pressure on indexing when merges accumulate e.g. by making the merge run on the indexing thread which tries to submit it.
3. A `IndexingMergeSharedExecutorService` that can effectively handle sharing threads b/w indexing and merge tasks.
FWIW, I'm not really sure if sharing a thread pool b/w indexing and merging would be simpler than having separate thread pools and applying backpressure on indexing. I still need to grok all the details. But if we think backpressure is the way to go, and we'll always only use the `MergeTasksExecutorService`, then writing this executor as part of the scheduler and making it a singleton might make the code simpler?
OTOH, if we want a shared indexing/merging thread pool, or foresee a need to have different thread pools for sets of writers (shards in Elasticsearch / OpenSearch), then the ctor arg for executor service makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "vigyasharma (via GitHub)" <gi...@apache.org>.
vigyasharma commented on code in PR #14900:
URL: https://github.com/apache/lucene/pull/14900#discussion_r2227686675
##########
lucene/core/src/java/org/apache/lucene/index/SharedMergeScheduler.java:
##########
@@ -0,0 +1,84 @@
+package org.apache.lucene.index;
+
+import java.io.IOException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArraySet;
+
+/**
+ * SharedMergeScheduler is an experimental MergeScheduler that submits merge tasks to a shared
+ * thread pool across IndexWriters.
+ */
+public class SharedMergeScheduler extends MergeScheduler {
+
+ // Tracks submitted merges per writer
+ private final ConcurrentHashMap<IndexWriter, Set<MergeTaskWrapper>> writerToMerges = new ConcurrentHashMap<>();
+
+ /**
+ * Executor service provided externally to handle merge tasks.
+ * Allows sharing a thread pool across IndexWriters if configured that way.
+ */
+ private final ExecutorService executor;
+
+ public SharedMergeScheduler(ExecutorService executor) {
+ this.executor = executor;
+ }
+
+ /**
+ * Retrieves pending merge tasks from the given {@link MergeSource} and submits them to the shared
+ * thread pool for execution.
+ *
+ * @param mergeSource the source of merge tasks (typically an IndexWriter)
+ * @param trigger the event that triggered the merge (e.g., SEGMENT_FLUSH, EXPLICIT)
+ * @throws IOException if merging fails
+ */
+ @Override
+ public void merge(MergeSource mergeSource, MergeTrigger trigger) throws IOException {
+ while (true) {
+ final MergePolicy.OneMerge merge = mergeSource.getNextMerge();
+ if (merge == null) break;
+
+ Runnable mergeRunnable = () -> {
+ try {
+ mergeSource.merge(merge);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ };
+
+
+ MergeTaskWrapper wrappedTask = new MergeTaskWrapper(mergeRunnable, (IndexWriter) mergeSource, merge.totalBytesSize());
+
+ // Registering this task under the writer
+ writerToMerges.computeIfAbsent((IndexWriter) mergeSource, k -> new CopyOnWriteArraySet<>()).add(wrappedTask);
Review Comment:
I don't think you can cast `mergeSource` as an `IndexWriter`. However, you might not need to. This CMS is no longer a singleton, only the executor is shared. The CMS is actually owned by an indexWriter, which means all calls to `merge()` come from the same IW.
So instead of a `writer -> merges` mapping that you'd need in a singleton, all you need here is the set of pending and running merges submitted to this CMS. They are all from the same writer! When writer closes, instead of shutting down the executor, you could update the merge objects in the set (MergeTaskWrappers), and set an aborted flag. And update your runnable to skip the merge if "aborted" flag has been set.
##########
lucene/core/src/java/org/apache/lucene/index/SharedMergeScheduler.java:
##########
@@ -0,0 +1,84 @@
+package org.apache.lucene.index;
+
+import java.io.IOException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArraySet;
+
+/**
+ * SharedMergeScheduler is an experimental MergeScheduler that submits merge tasks to a shared
+ * thread pool across IndexWriters.
+ */
+public class SharedMergeScheduler extends MergeScheduler {
+
+ // Tracks submitted merges per writer
+ private final ConcurrentHashMap<IndexWriter, Set<MergeTaskWrapper>> writerToMerges = new ConcurrentHashMap<>();
+
+ /**
+ * Executor service provided externally to handle merge tasks.
+ * Allows sharing a thread pool across IndexWriters if configured that way.
+ */
+ private final ExecutorService executor;
+
+ public SharedMergeScheduler(ExecutorService executor) {
+ this.executor = executor;
+ }
+
+ /**
+ * Retrieves pending merge tasks from the given {@link MergeSource} and submits them to the shared
+ * thread pool for execution.
+ *
+ * @param mergeSource the source of merge tasks (typically an IndexWriter)
+ * @param trigger the event that triggered the merge (e.g., SEGMENT_FLUSH, EXPLICIT)
+ * @throws IOException if merging fails
+ */
+ @Override
+ public void merge(MergeSource mergeSource, MergeTrigger trigger) throws IOException {
+ while (true) {
Review Comment:
can we use `hasPendingMerges()` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3069935110
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3090537006
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3070054034
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3039832145
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3039618622
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3084596936
> FWIW, I'm not really sure if sharing a thread pool b/w indexing and merging would be simpler than having separate thread pools and applying backpressure on indexing
To me the reason for sharing the thread pools is not to make things simpler but rather to make it easier to control the overall number of active indexing+merging threads. For CPU-bound workloads (which indexing tends to be), the best approach is to create a thread pool sized based on the number of cores of the machine. You can't do this if indexing and merging don't use the same thread pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3069168872
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "vigyasharma (via GitHub)" <gi...@apache.org>.
vigyasharma commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3046261414
@jpountz I feel the scheduler should have some control on how the executor is created, its backing queue etc, so that it can prioritize how merges get scheduled, e.g. pick smaller merges before large ones, or have merges smaller than `MIN_BIG_MERGE_MB` so straight through without waiting. Otherwise, all the "merge scheduling" logic really gets offloaded to how the thread pool manages its task queue. That could be hard to get right, but maybe it's okay for expert users?
We could have the multi-tenant CMS accept a constructor argument for a custom `MergeTasksExecutorService` defined in Lucene. It will maintain a fixed thread pool, use a fixed size priority blocking queue that prefers small merges over big ones, and add custom logic to directly execute small merges, have bounded wait time on merges, use calling thread when needed etc. Something on the lines of Lucene's `TaskExecutor` that you mentioned. On similar lines, we can also create an `IndexingMergeSharedExecutorService` to share threads across indexing and merge tasks. We'll probably need to write a custom executor service; I'm not sure if wrapping an executor will give us the hooks we need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "N624-debu (via GitHub)" <gi...@apache.org>.
N624-debu commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3135258237
> This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
This PR introduces internal improvements to merge task tracking and does not affect user-facing functionality. I believe a changelog entry is not necessary, so adding the skip-changelog label would be appropriate. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "vigyasharma (via GitHub)" <gi...@apache.org>.
vigyasharma commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3042311864
This seems to be in the right direction. Let's make it a singleton? As a next step, should we add support for prioritising small merges over larger ones (by backing the thread pool with a priority blocking queue)?
Also, since this is a draft PR, let's set it as "draft" on GitHub. You can also add a comment anywhere in your PR that says `// no commit`, which will ensure that the PR is not merged until it's ready (some GitHub checks will fail).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3408720449
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3070000699
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "vigyasharma (via GitHub)" <gi...@apache.org>.
vigyasharma commented on code in PR #14900:
URL: https://github.com/apache/lucene/pull/14900#discussion_r2205707984
##########
lucene/core/src/java/org/apache/lucene/index/SharedMergeScheduler.java:
##########
@@ -0,0 +1,68 @@
+package org.apache.lucene.index;
+
+import java.io.IOException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * SharedMergeScheduler is an experimental MergeScheduler that submits merge tasks to a shared
+ * thread pool across IndexWriters.
+ */
+public class SharedMergeScheduler extends MergeScheduler {
+
+ /**
+ * Executor service provided externally to handle merge tasks.
+ * Allows sharing a thread pool across IndexWriters if configured that way.
+ */
+ private final ExecutorService sharedExecutor;
+
+ public SharedMergeScheduler(ExecutorService sharedExecutor) {
+ this.sharedExecutor = sharedExecutor;
+ }
+
+ /**
+ * Retrieves pending merge tasks from the given {@link MergeSource} and submits them to the shared
+ * thread pool for execution.
+ *
+ * @param mergeSource the source of merge tasks (typically an IndexWriter)
+ * @param trigger the event that triggered the merge (e.g., SEGMENT_FLUSH, EXPLICIT)
+ * @throws IOException if merging fails
+ */
+ @Override
+ public void merge(MergeSource mergeSource, MergeTrigger trigger) throws IOException {
+ while (true) {
+ final MergePolicy.OneMerge merge = mergeSource.getNextMerge();
+ if (merge == null) break;
+
+ Runnable mergeRunnable = () -> {
+ try {
+ mergeSource.merge(merge);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ };
+
+ MergeTaskWrapper wrappedTask = new MergeTaskWrapper(mergeRunnable, mergeSource, merge.totalBytesSize());
+ sharedExecutor.submit(wrappedTask.getMergeTask());
+ }
+ }
+
+ /**
+ * Closes the merge scheduler. This implementation is currently a no-op. In a production-ready
+ * version, the shared executor should be properly shut down.
+ */
+ @Override
+ public void close() {
Review Comment:
This impl. for `close()` will shut down the entire executor. Since this is a shared merge scheduler, you only want to address merges related to the closing IndexWriter, like cancelling all pending merges and waiting for running merges to complete. (See[ close()](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L470) in CMS as well).
##########
lucene/core/src/java/org/apache/lucene/index/MergeTaskWrapper.java:
##########
@@ -0,0 +1,25 @@
+package org.apache.lucene.index;
+
+public class MergeTaskWrapper {
+ private final Runnable mergeTask;
+ private final Object sourceWriter; // Can be IndexWriter or its ID
+ private final long mergeSize; // For priority-based scheduling
+
+ public MergeTaskWrapper(Runnable mergeTask, Object sourceWriter, long mergeSize) {
+ this.mergeTask = mergeTask;
+ this.sourceWriter = sourceWriter;
+ this.mergeSize = mergeSize;
+ }
+
+ public Runnable getMergeTask() {
+ return mergeTask;
+ }
+
+ public Object getSourceWriter() {
+ return sourceWriter;
+ }
+
+ public long getMergeSize() {
+ return mergeSize;
+ }
+}
Review Comment:
nit: we keep an empty line at the end of file. There might be an editor/formatter setting you want to add that brings it in automatically.
##########
lucene/core/src/java/org/apache/lucene/index/MergeTaskWrapper.java:
##########
@@ -0,0 +1,25 @@
+package org.apache.lucene.index;
+
+public class MergeTaskWrapper {
+ private final Runnable mergeTask;
+ private final Object sourceWriter; // Can be IndexWriter or its ID
Review Comment:
You can directly use `IndexWriter` or `String` for ID instead of using Object. Keeping the type as specific as possible helps catch a lot of bugs before runtime.
##########
lucene/core/src/java/org/apache/lucene/index/MergeTaskWrapper.java:
##########
@@ -0,0 +1,25 @@
+package org.apache.lucene.index;
+
+public class MergeTaskWrapper {
Review Comment:
I'm curious how we intend to use this wrapper, looking forward to the next iteration of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] SharedMergeScheduler using shared thread pool for multi-tenant merge scheduling [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14900:
URL: https://github.com/apache/lucene/pull/14900#issuecomment-3089718090
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org