You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by ravicv <ra...@gmail.com> on 2012/08/23 14:49:13 UTC
Query regarding multi core search
Hi,
How sorting is done in SOLR with multiple cores .. say 20 cores.. because in
multi core search it should search in all cores then on complete results it
should sort... please correct me if i am wrong.
In our scenario we are executing same query on 4 cores and finally sorting
the results based on one field. It works good. But i want to implement
similar thing with in SOLR . can any one suggest me some code or blog
regarding this?
I have tried some approaches , but it takes more memory.... :(
Thanks,
Ravi
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-regarding-multi-core-search-tp4002847.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query regarding multi core search
Posted by Erick Erickson <er...@gmail.com>.
OK, let's back up. _What_ is your memory error? Where
is it coming from? Solr sorting or your custom code?
How many documents are you returning to your custom code
to sort, all of them? Some subset?
Consider re-thinking how you're doing this. In Solr 4 Beta,
there's the possibility of storing all your fields and doing
partial updates, that is updating just a few fields in a doc.
Under the covers this stores all the data and re-indexes the
entire document. That may be worth exploring...
And saying "my index is 30G" isn't actually than informative,
the stored data (in the *.fdt and *.fdx files in your data/index dir)
store the verbatim copy of the data for any fields where ou
specify 'stored="true" '. They're almost totally irrelevant for
searching requirements.
Solr reads all of the values for a field for _all_ the
documents into memory in order to sort by that field, plus
some overhead, more or less depending on what version
of Solr you use (which is what?). Until you show us the
stack trace, it's very hard to know much....
Best
Erick
On Mon, Aug 27, 2012 at 12:51 AM, ravicv <ra...@gmail.com> wrote:
> Hi Erick
>
> Thanks For reply.
>
> My use case is I have 50 fields to index , but at the time of index all 50
> fields data is unavailable.
> So we have divided them in to 24 separate cores and indexing . we are using
> shards for querying. But the problem is each core is having huge data. Total
> size of all 24 cores is 30 GB.
> So we wrote a custom request handler to query each core and finally merge
> the results and to give final response. But some search queries responses
> should be sorted based on a field. Since the response is huge while sorting
> we are getting heap space errors sometimes.
>
> Is their any better way to handle this scenario other than this? Id so
> please suggest me ...
> Also how SOLR is sorting the results...
>
> Thanks,
> Ravi
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Query-regarding-multi-core-search-tp4002847p4003400.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query regarding multi core search
Posted by ravicv <ra...@gmail.com>.
Hi Erick
Thanks For reply.
My use case is I have 50 fields to index , but at the time of index all 50
fields data is unavailable.
So we have divided them in to 24 separate cores and indexing . we are using
shards for querying. But the problem is each core is having huge data. Total
size of all 24 cores is 30 GB.
So we wrote a custom request handler to query each core and finally merge
the results and to give final response. But some search queries responses
should be sorted based on a field. Since the response is huge while sorting
we are getting heap space errors sometimes.
Is their any better way to handle this scenario other than this? Id so
please suggest me ...
Also how SOLR is sorting the results...
Thanks,
Ravi
--
View this message in context: http://lucene.472066.n3.nabble.com/Query-regarding-multi-core-search-tp4002847p4003400.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query regarding multi core search
Posted by Erick Erickson <er...@gmail.com>.
Why do you have 4 cores in the first place? The usual use-case
is that cores aren't for similar documents....
But the easiest thing to do would be to include the sort field
in the response and have the app (or whatever is aggregating
the 4 responses) sort the responses, essentially merging the
4 sets of documents. Paging will be a problem here though...
If the 4 cores are identical, why not
1> put them all in the same core
or
2> make them shards?
Best
Erick
On Thu, Aug 23, 2012 at 8:49 AM, ravicv <ra...@gmail.com> wrote:
> Hi,
>
> How sorting is done in SOLR with multiple cores .. say 20 cores.. because in
> multi core search it should search in all cores then on complete results it
> should sort... please correct me if i am wrong.
>
> In our scenario we are executing same query on 4 cores and finally sorting
> the results based on one field. It works good. But i want to implement
> similar thing with in SOLR . can any one suggest me some code or blog
> regarding this?
>
> I have tried some approaches , but it takes more memory.... :(
>
> Thanks,
> Ravi
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Query-regarding-multi-core-search-tp4002847.html
> Sent from the Solr - User mailing list archive at Nabble.com.