You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Tom Emerson <tr...@gmail.com> on 2006/05/18 18:10:57 UTC
How are results merged from a multisearcher?
Greetings,
Could someone describe how the results from multiple indices are merged when
using a MultiSearcher? My naive intuition is that the scores for documents
found in each index could be wildly different, so what criteria is used to
merge the scored docs?
Many thanks in advance,
-tree
--
Tom Emerson
tremerson@gmail.com
http://www.dreamersrealm.net/~tree
Re: How are results merged from a multisearcher?
Posted by Tom Emerson <tr...@gmail.com>.
Doug,
Thanks much for the clarification. That helps put everything in the right
frame for me!
-tree
On 5/22/06, Doug Cutting <cu...@apache.org> wrote:
>
> Tom Emerson wrote:
> > Thanks for the clarification. What then is the difference between a
> > MultiSearcher and using an IndexSearcher on a MultiReader?
>
> The results should be identical. A MultiSearcher permits use of
> ParallelMultiSearcher and RemoteSearchable, for parallel and/or
> distributed operation. But, for single-threaded searching, a
> MultiReader is probably fastest.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Tom Emerson
tremerson@gmail.com
http://www.dreamersrealm.net/~tree
Re: How are results merged from a multisearcher?
Posted by Doug Cutting <cu...@apache.org>.
Tom Emerson wrote:
> Thanks for the clarification. What then is the difference between a
> MultiSearcher and using an IndexSearcher on a MultiReader?
The results should be identical. A MultiSearcher permits use of
ParallelMultiSearcher and RemoteSearchable, for parallel and/or
distributed operation. But, for single-threaded searching, a
MultiReader is probably fastest.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How are results merged from a multisearcher?
Posted by Tom Emerson <tr...@gmail.com>.
Daniel,
Thanks for the clarification. What then is the difference between a
MultiSearcher and using an IndexSearcher on a MultiReader?
On 5/18/06, Daniel Naber <lu...@danielnaber.de> wrote:
>
> On Donnerstag 18 Mai 2006 23:26, Tom Emerson wrote:
>
> > OK, but what does "merged correctly" mean?
>
> I assume it means: querying over several indices gives the same ranking as
> if the documents were in one index.
>
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Tom Emerson
tremerson@gmail.com
http://www.dreamersrealm.net/~tree
Re: How are results merged from a multisearcher?
Posted by Daniel Naber <lu...@danielnaber.de>.
On Donnerstag 18 Mai 2006 23:26, Tom Emerson wrote:
> OK, but what does "merged correctly" mean?
I assume it means: querying over several indices gives the same ranking as
if the documents were in one index.
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How are results merged from a multisearcher?
Posted by Tom Emerson <tr...@gmail.com>.
OK, but what does "merged correctly" mean? That is really the crux of my
question: what is the merging semantics across indices with possibly
divergent IDFs.
On 5/18/06, Daniel Naber <lu...@danielnaber.de> wrote:
>
> On Donnerstag 18 Mai 2006 18:36, Ken Krugler wrote:
>
> > >Could someone describe how the results from multiple indices are merged
> > > when using a MultiSearcher? My naive intuition is that the scores for
> > > documents found in each index could be wildly different, so what
> > > criteria is used to merge the scored docs?
> >
> > I believe they are blindly merged.
> >
> > Which means that the IDFs for terms between multiple indices must be
> > relatively equal, otherwise the results will be skewed.
>
> That was the case in Lucene <= 1.4, in 1.9 it should be fixed, i.e.
> merging
> is supposed to work correctly.
>
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Tom Emerson
tremerson@gmail.com
http://www.dreamersrealm.net/~tree
Re: How are results merged from a multisearcher?
Posted by Ken Krugler <kk...@transpac.com>.
>On Donnerstag 18 Mai 2006 18:36, Ken Krugler wrote:
>
> > >Could someone describe how the results from multiple indices are merged
>> > when using a MultiSearcher? My naive intuition is that the scores for
>> > documents found in each index could be wildly different, so what
>> > criteria is used to merge the scored docs?
>>
>> I believe they are blindly merged.
>>
>> Which means that the IDFs for terms between multiple indices must be
> > relatively equal, otherwise the results will be skewed.
>
>That was the case in Lucene <= 1.4, in 1.9 it should be fixed, i.e. merging
>is supposed to work correctly.
Sorry, you're right. In Lucene 1.9 the MultiSearcher has been
improved. In Nutch, which uses its own implementation, this is still
a problem as per:
http://issues.apache.org/jira/browse/NUTCH-92
-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How are results merged from a multisearcher?
Posted by Daniel Naber <lu...@danielnaber.de>.
On Donnerstag 18 Mai 2006 18:36, Ken Krugler wrote:
> >Could someone describe how the results from multiple indices are merged
> > when using a MultiSearcher? My naive intuition is that the scores for
> > documents found in each index could be wildly different, so what
> > criteria is used to merge the scored docs?
>
> I believe they are blindly merged.
>
> Which means that the IDFs for terms between multiple indices must be
> relatively equal, otherwise the results will be skewed.
That was the case in Lucene <= 1.4, in 1.9 it should be fixed, i.e. merging
is supposed to work correctly.
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How are results merged from a multisearcher?
Posted by Ken Krugler <kk...@transpac.com>.
>Greetings,
>
>Could someone describe how the results from multiple indices are merged when
>using a MultiSearcher? My naive intuition is that the scores for documents
>found in each index could be wildly different, so what criteria is used to
>merge the scored docs?
I believe they are blindly merged.
Which means that the IDFs for terms between multiple indices must be
relatively equal, otherwise the results will be skewed.
The simple approach that most people take when dealing with this
issue is to generate a larger set of smaller indices from the total
data set, then randomize the selection of indices that get merged to
form the N final indices. This randomization helps avoid the IDF skew
problem.
There's an Jira issue on the Nutch side (see NUTCH-92) around this
same problem.
-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org