Doing concurrent searches efficiently

[Roberto Ragusa <ma...@robertoragusa.it>]

Hi,

I would like an expert opinion about how to optimally do concurrent
searches on the same index (let's suppose there are several threads
doing searches). Consider these options:

a) one IndexReader, all threads use it
b) cloned IndexReader's, each thread uses a clone
c) opened IndexReader's, each thread open and keeps its own IndexReader

What I would like to achieve is:

x) maximum CPU and memory efficiency (e.g. only one copy of norms vectors,
only one copy of FieldCache for a segment)
y) avoid performance issues related to locks
z) let each thread have an atomic view of the index, i.e. do a reopen when it wants
(and with no coordination with other threads)

I think z) is impossible to achieve with a).
Is it best to do b) or c) ?
I'm afraid c) implies multiple copy of the norms (!), while I'm not sure
about the FieldCache.

I'm talking about version 3.0, but I would be really interested to hear if
newer versions have different characteristics.

Thanks.

-- 
   Roberto Ragusa    mail at robertoragusa.it

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Doing concurrent searches efficiently

[Adrien Grand <jp...@gmail.com>]

Hi Roberto,

On Wed, Jun 19, 2013 at 12:57 PM, Roberto Ragusa <ma...@robertoragusa.it> wrote:
> Hi,
>
> I would like an expert opinion about how to optimally do concurrent
> searches on the same index (let's suppose there are several threads
> doing searches). Consider these options:
>
> a) one IndexReader, all threads use it
> b) cloned IndexReader's, each thread uses a clone
> c) opened IndexReader's, each thread open and keeps its own IndexReader

Best option is a), Lucene will internally clone per-thread
data-structure that don't support concurrent access.

> What I would like to achieve is:
> z) let each thread have an atomic view of the index, i.e. do a reopen when it wants
> (and with no coordination with other threads)
>
> I think z) is impossible to achieve with a).

The best way to achieve this is to use SearcherManager[1][2].

[1] http://lucene.apache.org/core/4_3_0/core/org/apache/lucene/search/SearcherManager.html
[2] http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html

--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org