You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Christian Reuschling <ch...@gmail.com> on 2009/08/04 11:50:16 UTC
ParallelMultiSearcher and idf
Hello,
when searching over multiple indices, we create one IndexReader for each index,
and wrap them into a MultiReader, that we use for IndexSearcher creation.
This is fine for searching multiple indices on one machine, but in the case the
indices are distributed over the (intra)net, this scenario has several lacks:
- searching/scoring/sorting is 100% on the client machine, so you need all the
ram and cpu power at every client.
- all the data necessary for scoring must go over the net - so the traffic
should be significantly higher
- thus, there is a lack of overall performance
Nevertheless, creating a MultiReader and making a searcher out of it has one
advantage (at least can be an advantage depending on the scenario): The
document freqiencies of a term will be summed up, and thus it is 100%
transparent for scoring whether the indices are splittet or not.
I'm wondering whether there is the possibility to get the advantages of both
scenarios, e.g. by first summing up the query terms-related document
frequencies, and sending them together with the query to every (remote)searcher
of ParallelMultiSearcher, for scoring.
Maybe this is exactly what ParallelMultiSearcher does, and I haven't seen it?
Thanks for clarification!
Chris
Re: ParallelMultiSearcher and idf
Posted by Simon Willnauer <si...@googlemail.com>.
Hey Christian,
you might wanna look at distributed solr
(http://wiki.apache.org/solr/DistributedSearch) or if you haven't done
so have a look at the Katta project
(http://katta.sourceforge.net/documentation/how-katta-works) maybe
this can help you out.
About distributed IDF and Scoring have a look at this link:
http://wunderwood.org/most_casual_observer/2007/04/progressive_reranking.html
Simon
On Tue, Aug 4, 2009 at 5:00 PM, Christian
Reuschling<ch...@gmail.com> wrote:
> Hi Otis,
>
> thanks for the answer - I'm aware of Solr, but it seems this is - according to
> its abstraction level - too generalized for us. Solr seems to be nice in the
> case you want to use the black box, and won't be aware of 'what is under the
> hood'.
> But maybe I'm totaly wrong. At least, it would be from interest how Solr
> realizes its distributed search, in the case it makes something different
> than using the core-Lucene ParallelMultiSearcher with RemoteSearchables. Maybe
> on this list somebody knows the answer.
>
>
>
>
>
> On Tue, 4 Aug 2009 07:20:23 -0700 (PDT)
> Otis Gospodnetic <ot...@yahoo.com> wrote:
>
>> Hi Christian,
>>
>> You didn't mention Solr, so I'm not sure if you are aware of it. Maybe Solr
>> meets your needs?
>>
>> Otis
>> --
>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>
>>
>>
>> ----- Original Message ----
>> > From: Christian Reuschling <ch...@gmail.com>
>> > To: java-user@lucene.apache.org
>> > Sent: Tuesday, August 4, 2009 5:50:16 AM
>> > Subject: ParallelMultiSearcher and idf
>> >
>> > Hello,
>> >
>> > when searching over multiple indices, we create one IndexReader for each
>> > index, and wrap them into a MultiReader, that we use for IndexSearcher
>> > creation.
>> >
>> > This is fine for searching multiple indices on one machine, but in the case
>> > the indices are distributed over the (intra)net, this scenario has several
>> > lacks:
>> >
>> > - searching/scoring/sorting is 100% on the client machine, so you need all
>> > the ram and cpu power at every client.
>> > - all the data necessary for scoring must go over the net - so the traffic
>> > should be significantly higher
>> > - thus, there is a lack of overall performance
>> >
>> > Nevertheless, creating a MultiReader and making a searcher out of it has one
>> > advantage (at least can be an advantage depending on the scenario): The
>> > document freqiencies of a term will be summed up, and thus it is 100%
>> > transparent for scoring whether the indices are splittet or not.
>> >
>> > I'm wondering whether there is the possibility to get the advantages of both
>> > scenarios, e.g. by first summing up the query terms-related document
>> > frequencies, and sending them together with the query to every
>> > (remote)searcher of ParallelMultiSearcher, for scoring.
>> >
>> > Maybe this is exactly what ParallelMultiSearcher does, and I haven't seen
>> > it?
>> >
>> >
>> > Thanks for clarification!
>> >
>> > Chris
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ParallelMultiSearcher and idf
Posted by Christian Reuschling <ch...@gmail.com>.
Hi Otis,
thanks for the answer - I'm aware of Solr, but it seems this is - according to
its abstraction level - too generalized for us. Solr seems to be nice in the
case you want to use the black box, and won't be aware of 'what is under the
hood'.
But maybe I'm totaly wrong. At least, it would be from interest how Solr
realizes its distributed search, in the case it makes something different
than using the core-Lucene ParallelMultiSearcher with RemoteSearchables. Maybe
on this list somebody knows the answer.
On Tue, 4 Aug 2009 07:20:23 -0700 (PDT)
Otis Gospodnetic <ot...@yahoo.com> wrote:
> Hi Christian,
>
> You didn't mention Solr, so I'm not sure if you are aware of it. Maybe Solr
> meets your needs?
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
> > From: Christian Reuschling <ch...@gmail.com>
> > To: java-user@lucene.apache.org
> > Sent: Tuesday, August 4, 2009 5:50:16 AM
> > Subject: ParallelMultiSearcher and idf
> >
> > Hello,
> >
> > when searching over multiple indices, we create one IndexReader for each
> > index, and wrap them into a MultiReader, that we use for IndexSearcher
> > creation.
> >
> > This is fine for searching multiple indices on one machine, but in the case
> > the indices are distributed over the (intra)net, this scenario has several
> > lacks:
> >
> > - searching/scoring/sorting is 100% on the client machine, so you need all
> > the ram and cpu power at every client.
> > - all the data necessary for scoring must go over the net - so the traffic
> > should be significantly higher
> > - thus, there is a lack of overall performance
> >
> > Nevertheless, creating a MultiReader and making a searcher out of it has one
> > advantage (at least can be an advantage depending on the scenario): The
> > document freqiencies of a term will be summed up, and thus it is 100%
> > transparent for scoring whether the indices are splittet or not.
> >
> > I'm wondering whether there is the possibility to get the advantages of both
> > scenarios, e.g. by first summing up the query terms-related document
> > frequencies, and sending them together with the query to every
> > (remote)searcher of ParallelMultiSearcher, for scoring.
> >
> > Maybe this is exactly what ParallelMultiSearcher does, and I haven't seen
> > it?
> >
> >
> > Thanks for clarification!
> >
> > Chris
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: ParallelMultiSearcher and idf
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Christian,
You didn't mention Solr, so I'm not sure if you are aware of it. Maybe Solr meets your needs?
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: Christian Reuschling <ch...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Tuesday, August 4, 2009 5:50:16 AM
> Subject: ParallelMultiSearcher and idf
>
> Hello,
>
> when searching over multiple indices, we create one IndexReader for each index,
> and wrap them into a MultiReader, that we use for IndexSearcher creation.
>
> This is fine for searching multiple indices on one machine, but in the case the
> indices are distributed over the (intra)net, this scenario has several lacks:
>
> - searching/scoring/sorting is 100% on the client machine, so you need all the
> ram and cpu power at every client.
> - all the data necessary for scoring must go over the net - so the traffic
> should be significantly higher
> - thus, there is a lack of overall performance
>
> Nevertheless, creating a MultiReader and making a searcher out of it has one
> advantage (at least can be an advantage depending on the scenario): The
> document freqiencies of a term will be summed up, and thus it is 100%
> transparent for scoring whether the indices are splittet or not.
>
> I'm wondering whether there is the possibility to get the advantages of both
> scenarios, e.g. by first summing up the query terms-related document
> frequencies, and sending them together with the query to every (remote)searcher
> of ParallelMultiSearcher, for scoring.
>
> Maybe this is exactly what ParallelMultiSearcher does, and I haven't seen it?
>
>
> Thanks for clarification!
>
> Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org