You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alok Bhandari <al...@gmail.com> on 2013/10/03 12:31:16 UTC

Solr grouping performace

Hello ,
I am using solr 4.0 , I want to group entries on one of the int field , I
need all the groups and group.limit is 1. I am getting very slow performance
and some times I am getting OutOfMemory also. 

My index is having 20 million records and out of which my search result
returns 1 million document and I do grouping on these 1 million docs. The
side of data on disk is approx 2GB and I am having Xmx 2GB. Please can
anyone help me out in this. performance is very slow it takes 10-12 seconds.

Thanks ,

Alok



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-grouping-performace-tp4093300.html
Sent from the Solr - User mailing list archive at Nabble.com.

AW: AW: Solr grouping performace

Posted by Sandro Zbinden <zb...@imagic.ch>.
About the performance I can't say too much but what I would recommend to do is that you loop over the results.

The first query gives you the number of groups  (ngroups)

q=*:*&groups=true&group.ngroups=true&group.field=myfield&start=0&rows=10000

And after that you execute the other queries in a simply change the start param

q=*:*&groups=true&group.field=myfield&start=10000&rows=10000
q=*:*&groups=true&group.field=myfield&start=20000&rows=10000
q=*:*&groups=true&group.field=myfield&start=30000&rows=10000

and so on...

I think like this you can avoid the solr out of memory exception.
PS: But be carefull if you store all the rows on client side you need memory too :-)

Bests Sandro


-----Ursprüngliche Nachricht-----
Von: Alok Bhandari [mailto:alokomprakashbhandari@gmail.com] 
Gesendet: Donnerstag, 3. Oktober 2013 13:02
An: solr-user@lucene.apache.org
Betreff: Re: AW: Solr grouping performace

Thanks for reply Sandro.

My requirement is that I need all groups and then build compact data from it to send to server. I am not sure about how much RAM should be allocated to JVM instance to make it serve requests faster , any inputs on that are welcome.



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-grouping-performace-tp4093300p4093311.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: AW: Solr grouping performace

Posted by Alok Bhandari <al...@gmail.com>.
Thanks for reply Sandro.

My requirement is that I need all groups and then build compact data from it
to send to server. I am not sure about how much RAM should be allocated to
JVM instance to make it serve requests faster , any inputs on that are
welcome.



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-grouping-performace-tp4093300p4093311.html
Sent from the Solr - User mailing list archive at Nabble.com.

AW: Solr grouping performace

Posted by Sandro Zbinden <zb...@imagic.ch>.
Hey Alok

I don't think that the group performance is bad on integer fields. But try to load all results in one query is very memory consuming.

Did you try to only load the first 1000 results with rows=1000 and start=0 ?

To get the total  group count you can use the group.ngroups=true param. 

Bests

Sandro Zbinden

-----Ursprüngliche Nachricht-----
Von: Alok Bhandari [mailto:alokomprakashbhandari@gmail.com] 
Gesendet: Donnerstag, 3. Oktober 2013 12:31
An: solr-user@lucene.apache.org
Betreff: Solr grouping performace

Hello ,
I am using solr 4.0 , I want to group entries on one of the int field , I need all the groups and group.limit is 1. I am getting very slow performance and some times I am getting OutOfMemory also. 

My index is having 20 million records and out of which my search result returns 1 million document and I do grouping on these 1 million docs. The side of data on disk is approx 2GB and I am having Xmx 2GB. Please can anyone help me out in this. performance is very slow it takes 10-12 seconds.

Thanks ,

Alok



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-grouping-performace-tp4093300.html
Sent from the Solr - User mailing list archive at Nabble.com.