You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "jpountz (via GitHub)" <gi...@apache.org> on 2023/07/13 13:36:56 UTC
[GitHub] [lucene] jpountz opened a new issue, #12438: Provide a recommended Executor implementation for concurrent search
jpountz opened a new issue, #12438:
URL: https://github.com/apache/lucene/issues/12438
### Description
Currently, IndexSearcher accepts an executor and doesn't give guidelines about the expected behavior of the executor. I think it would help users if we had a recommended executor for concurrent search. I think it needs the following properties:
### Fixed number of threads
I don't think it makes sense to use a cached threadpool for this use-case.
### Run tasks in the current thread when already in the threadpool
As we're making more things concurrent (`collection`, `rewrite`, `createWeight`) it's becoming increasingly likely to create deadlocks, e.g. by calling `createWeight` from a collector and a fixed number of threads. Our testing framework does this for instance. We could protect from this scenario by running tasks in the current thread when executing from a thread of the thread pool.
### Empty queue, blocking submission and no rejections
Given how we use this executor for fork/join, we don't need an actual queue and could use a `SynchronousQueue`, plus submission should wait until a thread is free instead of rejecting.
One assumption I am making here is that we would want to always run tasks in the executor rather than in the current thread, as this makes it easier to reason about the number of threads that can contribute to search. This is not how it works today, e.g. we run the last slice in the current thread, so we'd have to change this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org