You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Gimantha Bandara <gi...@wso2.com> on 2015/04/07 14:09:26 UTC
How to read multiple indices in parallel.
Hi all,
As I can see the Multireader is reading the multiple indices sequentially
(correct me if I am wrong). So using a IndexSearcher on a multireader will
also perform sequential searches right? Is there a lucene-built-in class to
search several indices parallely?
--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919
Re: How to read multiple indices in parallel.
Posted by Gimantha Bandara <gi...@wso2.com>.
That was really helpful. Thanks a lot Terry!
On Tue, Apr 7, 2015 at 8:17 PM, Terry Smith <sh...@gmail.com> wrote:
> Gimantha,
>
> Search will run in parallel even across indices.
>
> This happens because IndexSearcher searches by LeafReader and it doesn't
> matter where those LeafReaders come from (DirectoryReader or MultiReader)
> they are all treated equally.
>
> Example:
>
> DirectoryReader(A):
> LeafReader(B), LeafReader(C)
>
> DirectoryReader(D):
> LeafReader(E), LeafReader(F)
>
> Searching over A would use leaves B, C.
> Searching over B would use leaves E, F.
> Searching over a MultiReader on (A, B) would use leaves B, C, E, F.
>
> This runs serially by default but can run in parallel if you provide an
> ExecutorService to the IndexSearcher and use a compatible search() method
> on it.
>
> --Terry
>
>
> On Tue, Apr 7, 2015 at 10:27 AM, Gimantha Bandara <gi...@wso2.com>
> wrote:
>
> > Hi Terry,
> >
> > I have multiple indices in separate locations. If I used multireader and
> > used an executorservice with the indexSearcher It will go thru the
> segments
> > in parallel and search right? But still searching between different
> indices
> > will happen sequentially..Isnt it?
> >
> > On Tue, Apr 7, 2015 at 7:15 PM, Terry Smith <sh...@gmail.com> wrote:
> >
> > > Gimantha,
> > >
> > > With Lucene 5.0 you can pass in an ExecutorService to the constructor
> of
> > > your IndexSearcher and it will search the segments in parallel if you
> use
> > > one of the IndexSearcher.search() methods that returns a TopDocs (and
> > don't
> > > supply your own Collector).
> > >
> > > The not-yet-released Lucene 5.1 includes some changes (LUCENE-6294
> > > <https://issues.apache.org/jira/browse/LUCENE-6294>) that enable
> better
> > > parallel query support.
> > >
> > > --Terry
> > >
> > >
> > > On Tue, Apr 7, 2015 at 8:09 AM, Gimantha Bandara <gi...@wso2.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > As I can see the Multireader is reading the multiple indices
> > sequentially
> > > > (correct me if I am wrong). So using a IndexSearcher on a multireader
> > > will
> > > > also perform sequential searches right? Is there a lucene-built-in
> > class
> > > to
> > > > search several indices parallely?
> > > >
> > > > --
> > > > Gimantha Bandara
> > > > Software Engineer
> > > > WSO2. Inc : http://wso2.com
> > > > Mobile : +94714961919
> > > >
> > >
> >
> >
> >
> > --
> > Gimantha Bandara
> > Software Engineer
> > WSO2. Inc : http://wso2.com
> > Mobile : +94714961919
> >
>
--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919
Re: How to read multiple indices in parallel.
Posted by Terry Smith <sh...@gmail.com>.
Gimantha,
Search will run in parallel even across indices.
This happens because IndexSearcher searches by LeafReader and it doesn't
matter where those LeafReaders come from (DirectoryReader or MultiReader)
they are all treated equally.
Example:
DirectoryReader(A):
LeafReader(B), LeafReader(C)
DirectoryReader(D):
LeafReader(E), LeafReader(F)
Searching over A would use leaves B, C.
Searching over B would use leaves E, F.
Searching over a MultiReader on (A, B) would use leaves B, C, E, F.
This runs serially by default but can run in parallel if you provide an
ExecutorService to the IndexSearcher and use a compatible search() method
on it.
--Terry
On Tue, Apr 7, 2015 at 10:27 AM, Gimantha Bandara <gi...@wso2.com> wrote:
> Hi Terry,
>
> I have multiple indices in separate locations. If I used multireader and
> used an executorservice with the indexSearcher It will go thru the segments
> in parallel and search right? But still searching between different indices
> will happen sequentially..Isnt it?
>
> On Tue, Apr 7, 2015 at 7:15 PM, Terry Smith <sh...@gmail.com> wrote:
>
> > Gimantha,
> >
> > With Lucene 5.0 you can pass in an ExecutorService to the constructor of
> > your IndexSearcher and it will search the segments in parallel if you use
> > one of the IndexSearcher.search() methods that returns a TopDocs (and
> don't
> > supply your own Collector).
> >
> > The not-yet-released Lucene 5.1 includes some changes (LUCENE-6294
> > <https://issues.apache.org/jira/browse/LUCENE-6294>) that enable better
> > parallel query support.
> >
> > --Terry
> >
> >
> > On Tue, Apr 7, 2015 at 8:09 AM, Gimantha Bandara <gi...@wso2.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > As I can see the Multireader is reading the multiple indices
> sequentially
> > > (correct me if I am wrong). So using a IndexSearcher on a multireader
> > will
> > > also perform sequential searches right? Is there a lucene-built-in
> class
> > to
> > > search several indices parallely?
> > >
> > > --
> > > Gimantha Bandara
> > > Software Engineer
> > > WSO2. Inc : http://wso2.com
> > > Mobile : +94714961919
> > >
> >
>
>
>
> --
> Gimantha Bandara
> Software Engineer
> WSO2. Inc : http://wso2.com
> Mobile : +94714961919
>
Re: How to read multiple indices in parallel.
Posted by Gimantha Bandara <gi...@wso2.com>.
Hi Terry,
I have multiple indices in separate locations. If I used multireader and
used an executorservice with the indexSearcher It will go thru the segments
in parallel and search right? But still searching between different indices
will happen sequentially..Isnt it?
On Tue, Apr 7, 2015 at 7:15 PM, Terry Smith <sh...@gmail.com> wrote:
> Gimantha,
>
> With Lucene 5.0 you can pass in an ExecutorService to the constructor of
> your IndexSearcher and it will search the segments in parallel if you use
> one of the IndexSearcher.search() methods that returns a TopDocs (and don't
> supply your own Collector).
>
> The not-yet-released Lucene 5.1 includes some changes (LUCENE-6294
> <https://issues.apache.org/jira/browse/LUCENE-6294>) that enable better
> parallel query support.
>
> --Terry
>
>
> On Tue, Apr 7, 2015 at 8:09 AM, Gimantha Bandara <gi...@wso2.com>
> wrote:
>
> > Hi all,
> >
> > As I can see the Multireader is reading the multiple indices sequentially
> > (correct me if I am wrong). So using a IndexSearcher on a multireader
> will
> > also perform sequential searches right? Is there a lucene-built-in class
> to
> > search several indices parallely?
> >
> > --
> > Gimantha Bandara
> > Software Engineer
> > WSO2. Inc : http://wso2.com
> > Mobile : +94714961919
> >
>
--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919
Re: How to read multiple indices in parallel.
Posted by Terry Smith <sh...@gmail.com>.
Gimantha,
With Lucene 5.0 you can pass in an ExecutorService to the constructor of
your IndexSearcher and it will search the segments in parallel if you use
one of the IndexSearcher.search() methods that returns a TopDocs (and don't
supply your own Collector).
The not-yet-released Lucene 5.1 includes some changes (LUCENE-6294
<https://issues.apache.org/jira/browse/LUCENE-6294>) that enable better
parallel query support.
--Terry
On Tue, Apr 7, 2015 at 8:09 AM, Gimantha Bandara <gi...@wso2.com> wrote:
> Hi all,
>
> As I can see the Multireader is reading the multiple indices sequentially
> (correct me if I am wrong). So using a IndexSearcher on a multireader will
> also perform sequential searches right? Is there a lucene-built-in class to
> search several indices parallely?
>
> --
> Gimantha Bandara
> Software Engineer
> WSO2. Inc : http://wso2.com
> Mobile : +94714961919
>