You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Nicola Buso <nb...@ebi.ac.uk> on 2014/05/22 17:12:48 UTC
IndexWriter.addIndexes() multithread correct?
Hi,
I was wondering if it's correct to call the method:
IndexWriter.addIndexes(...)
from different threads concurrently.
nicola.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.addIndexes() multithread correct?
Posted by Erick Erickson <er...@gmail.com>.
right, for docs with the same score, ties are broken by the internal Lucene ID.
This may even change _on the same node_ due to merges!
If you want to control this, consider always specifying a secondary sort by,
say, your id field if you have one, or date stamp or......
Best,
Erick
On Thu, May 22, 2014 at 8:58 AM, Nicola Buso <nb...@ebi.ac.uk> wrote:
> Hi Uwe,
>
> thanks for the reply.
> I see slightly different order in the results, and I was thinking are
> docs with the same score and the difference between the two index (one
> with "addIndexes" multithreaded and one no) is because of the order of
> insert in the merged index; could be this the difference I see?
>
>
> nicola.
>
>
> On Thu, 2014-05-22 at 17:49 +0200, Uwe Schindler wrote:
>> Hi Nicola,
>>
>> Yes, it is thread safe, like all other methods in IndexWriter. In the case of the one taking IndexReaders, Lucene will also do the merging concurrently.
>>
>> Say hello to Jo McEntyre! :-)
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Nicola Buso [mailto:nbuso@ebi.ac.uk]
>> > Sent: Thursday, May 22, 2014 5:13 PM
>> > To: java-user@lucene.apache.org
>> > Subject: IndexWriter.addIndexes() multithread correct?
>> >
>> > Hi,
>> >
>> > I was wondering if it's correct to call the method:
>> > IndexWriter.addIndexes(...)
>> >
>> > from different threads concurrently.
>> >
>> >
>> > nicola.
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: IndexWriter.addIndexes() multithread correct?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
Yes, that is all fine. Because the index merging is concurrent, the order of documents in the resulting index may be different. This is not different than with relational database systems if you insert tables into others (the order of rows is also undefined).
About your question: The different order of results is caused by the scoring algorithm : If 2 documents have exact same score, the second sort order is "index order", which is not stable as explained above. So slightly different result order is perfectly fine. To change this, add a second sort criteria after the score (but this slows down searching for no reason).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Nicola Buso [mailto:nbuso@ebi.ac.uk]
> Sent: Thursday, May 22, 2014 5:58 PM
> To: Uwe Schindler
> Cc: java-user@lucene.apache.org
> Subject: Re: IndexWriter.addIndexes() multithread correct?
>
> Hi Uwe,
>
> thanks for the reply.
> I see slightly different order in the results, and I was thinking are docs with
> the same score and the difference between the two index (one with
> "addIndexes" multithreaded and one no) is because of the order of insert in
> the merged index; could be this the difference I see?
>
>
> nicola.
>
>
> On Thu, 2014-05-22 at 17:49 +0200, Uwe Schindler wrote:
> > Hi Nicola,
> >
> > Yes, it is thread safe, like all other methods in IndexWriter. In the case of
> the one taking IndexReaders, Lucene will also do the merging concurrently.
> >
> > Say hello to Jo McEntyre! :-)
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Nicola Buso [mailto:nbuso@ebi.ac.uk]
> > > Sent: Thursday, May 22, 2014 5:13 PM
> > > To: java-user@lucene.apache.org
> > > Subject: IndexWriter.addIndexes() multithread correct?
> > >
> > > Hi,
> > >
> > > I was wondering if it's correct to call the method:
> > > IndexWriter.addIndexes(...)
> > >
> > > from different threads concurrently.
> > >
> > >
> > > nicola.
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter.addIndexes() multithread correct?
Posted by Nicola Buso <nb...@ebi.ac.uk>.
Hi Uwe,
thanks for the reply.
I see slightly different order in the results, and I was thinking are
docs with the same score and the difference between the two index (one
with "addIndexes" multithreaded and one no) is because of the order of
insert in the merged index; could be this the difference I see?
nicola.
On Thu, 2014-05-22 at 17:49 +0200, Uwe Schindler wrote:
> Hi Nicola,
>
> Yes, it is thread safe, like all other methods in IndexWriter. In the case of the one taking IndexReaders, Lucene will also do the merging concurrently.
>
> Say hello to Jo McEntyre! :-)
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Nicola Buso [mailto:nbuso@ebi.ac.uk]
> > Sent: Thursday, May 22, 2014 5:13 PM
> > To: java-user@lucene.apache.org
> > Subject: IndexWriter.addIndexes() multithread correct?
> >
> > Hi,
> >
> > I was wondering if it's correct to call the method:
> > IndexWriter.addIndexes(...)
> >
> > from different threads concurrently.
> >
> >
> > nicola.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: IndexWriter.addIndexes() multithread correct?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Nicola,
Yes, it is thread safe, like all other methods in IndexWriter. In the case of the one taking IndexReaders, Lucene will also do the merging concurrently.
Say hello to Jo McEntyre! :-)
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Nicola Buso [mailto:nbuso@ebi.ac.uk]
> Sent: Thursday, May 22, 2014 5:13 PM
> To: java-user@lucene.apache.org
> Subject: IndexWriter.addIndexes() multithread correct?
>
> Hi,
>
> I was wondering if it's correct to call the method:
> IndexWriter.addIndexes(...)
>
> from different threads concurrently.
>
>
> nicola.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org