You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Yandong Yao <yy...@gmail.com> on 2013/01/09 08:24:38 UTC
How to run many MoreLikeThis request efficiently?
Hi Solr Guru,
I have two set of documents in one SolrCore, each set has about 1M
documents with different document type, say 'type1' and 'type2'.
Many documents in first set are very similar with 1 or 2 documents in the
second set, What I want to get is: for each document in set 2, return the
most similar document in set 1 using either 'MoreLikeThisHandler' or
'MoreLikeThisComponent'.
Currently I use following code to get the result, while it will send far
too many request to Solr server serially. Is there any way to enhance this
besides using multi-threading? Thanks very much!
for each document in set 2 whose type is 'type2'
run MoreLikeThis request against Solr server and get the most similar
document
end.
Regards,
Yandong
Re: How to run many MoreLikeThis request efficiently?
Posted by Yandong Yao <yy...@gmail.com>.
Hi Otis,
Really appreciate your help on this!! Will go with multi-thread firstly,
and then provide a custom component when performance is not good enough.
Regards,
Yandong
2013/1/10 Otis Gospodnetic <ot...@gmail.com>
> Patience, young Yandong :)
>
> Multi-threading *in your application* is the way to go. Alternatively, one
> could write a custom SearchComponent that is called once and inside of
> which the whole work is done after just one call to it. This component
> could then write the output somewhere, like in a new index since making a
> blocking call to it may time out.
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Jan 9, 2013 6:07 PM, "Yandong Yao" <yy...@gmail.com> wrote:
>
> > Any comments on this? Thanks very much in advance!
> >
> > 2013/1/9 Yandong Yao <yy...@gmail.com>
> >
> > > Hi Solr Guru,
> > >
> > > I have two set of documents in one SolrCore, each set has about 1M
> > > documents with different document type, say 'type1' and 'type2'.
> > >
> > > Many documents in first set are very similar with 1 or 2 documents in
> the
> > > second set, What I want to get is: for each document in set 2, return
> > the
> > > most similar document in set 1 using either 'MoreLikeThisHandler' or
> > > 'MoreLikeThisComponent'.
> > >
> > > Currently I use following code to get the result, while it will send
> far
> > > too many request to Solr server serially. Is there any way to enhance
> > this
> > > besides using multi-threading? Thanks very much!
> > >
> > > for each document in set 2 whose type is 'type2'
> > > run MoreLikeThis request against Solr server and get the most
> similar
> > > document
> > > end.
> > >
> > > Regards,
> > > Yandong
> > >
> >
>
Re: How to run many MoreLikeThis request efficiently?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Patience, young Yandong :)
Multi-threading *in your application* is the way to go. Alternatively, one
could write a custom SearchComponent that is called once and inside of
which the whole work is done after just one call to it. This component
could then write the output somewhere, like in a new index since making a
blocking call to it may time out.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jan 9, 2013 6:07 PM, "Yandong Yao" <yy...@gmail.com> wrote:
> Any comments on this? Thanks very much in advance!
>
> 2013/1/9 Yandong Yao <yy...@gmail.com>
>
> > Hi Solr Guru,
> >
> > I have two set of documents in one SolrCore, each set has about 1M
> > documents with different document type, say 'type1' and 'type2'.
> >
> > Many documents in first set are very similar with 1 or 2 documents in the
> > second set, What I want to get is: for each document in set 2, return
> the
> > most similar document in set 1 using either 'MoreLikeThisHandler' or
> > 'MoreLikeThisComponent'.
> >
> > Currently I use following code to get the result, while it will send far
> > too many request to Solr server serially. Is there any way to enhance
> this
> > besides using multi-threading? Thanks very much!
> >
> > for each document in set 2 whose type is 'type2'
> > run MoreLikeThis request against Solr server and get the most similar
> > document
> > end.
> >
> > Regards,
> > Yandong
> >
>
Re: How to run many MoreLikeThis request efficiently?
Posted by Yandong Yao <yy...@gmail.com>.
Any comments on this? Thanks very much in advance!
2013/1/9 Yandong Yao <yy...@gmail.com>
> Hi Solr Guru,
>
> I have two set of documents in one SolrCore, each set has about 1M
> documents with different document type, say 'type1' and 'type2'.
>
> Many documents in first set are very similar with 1 or 2 documents in the
> second set, What I want to get is: for each document in set 2, return the
> most similar document in set 1 using either 'MoreLikeThisHandler' or
> 'MoreLikeThisComponent'.
>
> Currently I use following code to get the result, while it will send far
> too many request to Solr server serially. Is there any way to enhance this
> besides using multi-threading? Thanks very much!
>
> for each document in set 2 whose type is 'type2'
> run MoreLikeThis request against Solr server and get the most similar
> document
> end.
>
> Regards,
> Yandong
>