ExecutorService support in SolrIndexSearcher

[Aghasi Ghazaryan <ag...@picsart.com.INVALID>]

Hi,

Lucene's IndexSearcher
<http://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/IndexSearcher.html#IndexSearcher-org.apache.lucene.index.IndexReaderContext-java.util.concurrent.ExecutorService->
supports
running searches for each segment separately, using the provided
ExecutorService.
I wonder why SolrIndexSearcher does not support the same as it may improve
queries performance a lot?

Thanks, looking forward to hearing from you.

Regards
Aghasi Ghazaryan

Re: ExecutorService support in SolrIndexSearcher

[Michael McCandless <lu...@mikemccandless.com>]

We pass ExecutorService to Lucene's IndexSearcher at Amazon (for customer
facing product search) and it's a big win on long-pole query latencies, but
hurts red-line QPS (cluster capacity) a bit, due to less efficient
collection across segments and thread context switching.

I'm surprised it's not an option for Solr and Elasticsearch ... for certain
applications it's a huge win.

And yes as David points out -- Collectors (CollectorManagers) need to
support this "gather results for each segment separately then reduce in the
end" mode...

Mike McCandless

http://blog.mikemccandless.com


On Fri, Aug 30, 2019 at 4:45 PM David Smiley <da...@gmail.com>
wrote:

> It'd take some work to do that.  Years ago I recall Etsy did a POC and
> shared their experience at Lucene/Solr Revolution in Washington DC; I
> attended the presentation with great interest.  One of the major obstacles,
> if I recall, was the Collector needs to support this mode of operation, and
> in particular Solr's means of flipping bits in a big bitset to accumulate
> the DocSet had to be careful so that multiple threads don't try to
> overwrite the same underlying "long" in the long[].
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Aug 26, 2019 at 7:02 AM Aghasi Ghazaryan
> <ag...@picsart.com.invalid> wrote:
>
> > Hi,
> >
> > Lucene's IndexSearcher
> > <
> >
> http://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/IndexSearcher.html#IndexSearcher-org.apache.lucene.index.IndexReaderContext-java.util.concurrent.ExecutorService-
> > >
> > supports
> > running searches for each segment separately, using the provided
> > ExecutorService.
> > I wonder why SolrIndexSearcher does not support the same as it may
> improve
> > queries performance a lot?
> >
> > Thanks, looking forward to hearing from you.
> >
> > Regards
> > Aghasi Ghazaryan
> >
>

Re: ExecutorService support in SolrIndexSearcher

[David Smiley <da...@gmail.com>]

It'd take some work to do that.  Years ago I recall Etsy did a POC and
shared their experience at Lucene/Solr Revolution in Washington DC; I
attended the presentation with great interest.  One of the major obstacles,
if I recall, was the Collector needs to support this mode of operation, and
in particular Solr's means of flipping bits in a big bitset to accumulate
the DocSet had to be careful so that multiple threads don't try to
overwrite the same underlying "long" in the long[].

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Aug 26, 2019 at 7:02 AM Aghasi Ghazaryan
<ag...@picsart.com.invalid> wrote:

> Hi,
>
> Lucene's IndexSearcher
> <
> http://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/IndexSearcher.html#IndexSearcher-org.apache.lucene.index.IndexReaderContext-java.util.concurrent.ExecutorService-
> >
> supports
> running searches for each segment separately, using the provided
> ExecutorService.
> I wonder why SolrIndexSearcher does not support the same as it may improve
> queries performance a lot?
>
> Thanks, looking forward to hearing from you.
>
> Regards
> Aghasi Ghazaryan
>