You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by rahul <as...@gmail.com> on 2011/03/15 13:12:48 UTC
Solrj performance bottleneck
Hi,
I am using Solrj as a Solr client in my project.
While searching, for a few words, it seems Solrj takes more time to send
response, for eg (8 - 12 sec). While searching most of the other words it
seems Solrj take less amount of time only.
For eg, if I post a search url in browser, it shows the QTime in
milliseconds only.
http://serverName/solr/mydata/select?q=computing&qt=myhandler&fq=category:1
But, if I query the same using Solrj from my project like below, it takes
long time(8 - 12 sec) to produce the same results. Hence, I suspect whether
Solrj takes such long time to produce results.
SolrServer server = new CommonsHttpSolrServer(url);
SolrQuery query = new SolrQuery("computing");
query.setParam("qt", "myhandler");
query.setFilterQueries("category:1");
query.setHighlight(false);
QueryResponse rsp = server.query( query );
I have tried both POTH and GET method. But, both are taking much time.
Any idea why Solrj takes such long time for particular words. It returns
around 40 doc list as a search result. I have even comment out highlighting
for that.
And any way to speed it up.
Note: I am using Tomcat and set heap size as around 1024 mb.
Thanks,
--
View this message in context: http://lucene.472066.n3.nabble.com/Solrj-performance-bottleneck-tp2681294p2681294.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj performance bottleneck
Posted by Lance Norskog <go...@gmail.com>.
There is a separate auto-suggest tool that creates a simple in-memory
database outside of the Lucene index. This is called TST.
On Tue, Apr 5, 2011 at 3:36 AM, rahul <as...@gmail.com> wrote:
> Thanks Stefan and Victor ! we are using GWT for front end. We stopped issuing
> multiple asynchronous queries and issue a request and fetch results and then
> filter the results based on what has
> been typed subsequent to the request and then re trigger the request only if
> we don't get the expected results.
>
> Thanks Victor, I appreciate the link to the Jquery example and we will look
> into it as a reference.
>
> Regards,
> Rahul.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solrj-performance-bottleneck-tp2682797p2779387.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Lance Norskog
goksron@gmail.com
Re: Solrj performance bottleneck
Posted by rahul <as...@gmail.com>.
Thanks Stefan and Victor ! we are using GWT for front end. We stopped issuing
multiple asynchronous queries and issue a request and fetch results and then
filter the results based on what has
been typed subsequent to the request and then re trigger the request only if
we don't get the expected results.
Thanks Victor, I appreciate the link to the Jquery example and we will look
into it as a reference.
Regards,
Rahul.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solrj-performance-bottleneck-tp2682797p2779387.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj performance bottleneck
Posted by openvictor Open <op...@gmail.com>.
Dear Rahul,
Stefan has the right solution. the autosuggest must be checked both from
Javascript and your backend. For javascript there are some really nice tools
to do that such as Jquery which implements a auto-suggest with a tunable
delay. It has also highlighting, you can add additional information etc...
It is actually quite impressive. Here is the address :
http://jqueryui.com/demos/autocomplete/#remote-jsonp. It's open source so
you can just copy what they have done or see the method they used.
For backend limit the number of request / second per ip or session and / or
cache result. As for cache normally solr caches the common request but I
don't know for term components.
Hope this helps you !
Victor
2011/4/4 Stefan Matheis <ma...@googlemail.com>
> rahul,
>
> On Mon, Apr 4, 2011 at 4:18 PM, rahul <as...@gmail.com> wrote:
> > if anybody has some suggestions/experience on how to leverage
> autosuggestion
> > without affecting search performance much, please do share them.
>
> we use javascript intervals for autosuggestion. regularly check the
> value of the monitored input field and if changed, trigger a new
> request. this will cover both cases, slow-typing users and also
> ten-finger-guys (which will type much faster). a new request for every
> added character is indeed too much, even if your backend is responding
> within a few ms.
>
> Regards
> Stefan
>
Re: Solrj performance bottleneck
Posted by Stefan Matheis <ma...@googlemail.com>.
rahul,
On Mon, Apr 4, 2011 at 4:18 PM, rahul <as...@gmail.com> wrote:
> if anybody has some suggestions/experience on how to leverage autosuggestion
> without affecting search performance much, please do share them.
we use javascript intervals for autosuggestion. regularly check the
value of the monitored input field and if changed, trigger a new
request. this will cover both cases, slow-typing users and also
ten-finger-guys (which will type much faster). a new request for every
added character is indeed too much, even if your backend is responding
within a few ms.
Regards
Stefan
Re: Solrj performance bottleneck
Posted by rahul <as...@gmail.com>.
Hi All,
I just to want to share some findings which clearly identified the reason
for our performance bottleneck. we had looked into several areas for
optimization mostly directed at Solr configurations, stored fields,
highlighting, JVM, OS cache etc. But it turned out that the "main" culprit
was elsewhere. We were using the terms component for auto suggestion and
while examining the firebug outputs for time taken during the searches, we
detected that multiple requests were being spawned for autosuggestion as we
typed in the keyword to search (1 request per each character typed) and this
in turn cost us great delay in getting the search results. Once we turned
auto suggestion off, the performance was remarkably better and came down to
a second or so (compared to 8-10 seconds registered earlier).
if anybody has some suggestions/experience on how to leverage autosuggestion
without affecting search performance much, please do share them.
Once again, thanks for your inputs in analyzing our issues.
Thanks,
--
View this message in context: http://lucene.472066.n3.nabble.com/Solrj-performance-bottleneck-tp2682797p2775245.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj performance bottleneck
Posted by rahul <as...@gmail.com>.
thanks for all your info.
I will try increase the RAM and check it.
thanks,
--
View this message in context: http://lucene.472066.n3.nabble.com/Solrj-performance-bottleneck-tp2682797p2692503.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj performance bottleneck
Posted by Bill Bell <bi...@gmail.com>.
Try give Solr like 1.5gb by setting Jave params. Solr is usually CPU bound. So medium or large instances are good.
Bill Bell
Sent from mobile
On Mar 16, 2011, at 10:56 AM, Asharudeen <as...@gmail.com> wrote:
> Hi
>
> Thanks for your info.
>
> Currently my index size is around 4GB. Normally in small instances total
> available memory will be 1.6GB. In my setup, I allocated around 1GB as a
> heap size for tomcat. Hence I believe, remaining 600 MB will be used for OS
> cache.
>
> I believe, I need to migrate my Solr instance from small instance to large.
> So that some more memory will be allotted for OS cache. But initially I
> suspect, since I call Solrj code from another instance, I need to increase
> the memory in the instance from where I run the Solrj. But you said I need
> to increase the memory in Solr instance only. Here, just I want to double
> check this case only. sorry for that.
>
> Once again thanks for your replies.
>
> Regards,
>
>
> On Wed, Mar 16, 2011 at 7:02 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
>
>> On Wed, Mar 16, 2011 at 7:25 AM, rahul <as...@gmail.com> wrote:
>>> In our setup, we are having Solr index in one machine. And Solrj client
>> part
>>> (java code) in another machine. Currently as you suggest, if it may be a
>>> 'not enough free RAM for the OS to cache' then whether I need to increase
>>> the RAM in the machine in which Solrj query part is there.??? Or need to
>>> increase RAM for Solr instance for the OS cache?
>>
>> That would be RAM for the Solr instance. If there is not enough free
>> memory for the OS to cache, then each document retrieved will be a
>> disk seek + read.
>>
>>> Since both the system are in local Amazon network (Linux EC2 small
>>> instances), I believe the network wont be a issue.
>>
>> Ah, how big is your index?
>>
>>> Another thing, in the reply you have mentioned 'client not reading fast
>>> enough'. Whether it is related to network or Solrj.
>>
>> That was a general issue - it *can* be the client, but since you're
>> using SolrJ it would be the network.
>>
>> -Yonik
>> http://lucidimagination.com
>>
Re: Solrj performance bottleneck
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Mar 16, 2011 at 12:56 PM, Asharudeen <as...@gmail.com> wrote:
> Currently my index size is around 4GB. Normally in small instances total
> available memory will be 1.6GB. In my setup, I allocated around 1GB as a
> heap size for tomcat. Hence I believe, remaining 600 MB will be used for OS
> cache.
Actually, even less. A JVM with a 1.6GB heap size will take up even
more memory (since the heap size does not count stuff not on the heap,
like the JVM code itself). This is definitely your problem.
> I believe, I need to migrate my Solr instance from small instance to large.
> So that some more memory will be allotted for OS cache. But initially I
> suspect, since I call Solrj code from another instance, I need to increase
> the memory in the instance from where I run the Solrj. But you said I need
> to increase the memory in Solr instance only. Here, just I want to double
> check this case only. sorry for that.
SolrJ itself won't take up much memory. It depends on what else your
client app is doing, but a small instance may be fine.
-Yonik
http://lucidimagination.com
Re: Solrj performance bottleneck
Posted by Asharudeen <as...@gmail.com>.
Hi
Thanks for your info.
Currently my index size is around 4GB. Normally in small instances total
available memory will be 1.6GB. In my setup, I allocated around 1GB as a
heap size for tomcat. Hence I believe, remaining 600 MB will be used for OS
cache.
I believe, I need to migrate my Solr instance from small instance to large.
So that some more memory will be allotted for OS cache. But initially I
suspect, since I call Solrj code from another instance, I need to increase
the memory in the instance from where I run the Solrj. But you said I need
to increase the memory in Solr instance only. Here, just I want to double
check this case only. sorry for that.
Once again thanks for your replies.
Regards,
On Wed, Mar 16, 2011 at 7:02 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Wed, Mar 16, 2011 at 7:25 AM, rahul <as...@gmail.com> wrote:
> > In our setup, we are having Solr index in one machine. And Solrj client
> part
> > (java code) in another machine. Currently as you suggest, if it may be a
> > 'not enough free RAM for the OS to cache' then whether I need to increase
> > the RAM in the machine in which Solrj query part is there.??? Or need to
> > increase RAM for Solr instance for the OS cache?
>
> That would be RAM for the Solr instance. If there is not enough free
> memory for the OS to cache, then each document retrieved will be a
> disk seek + read.
>
> > Since both the system are in local Amazon network (Linux EC2 small
> > instances), I believe the network wont be a issue.
>
> Ah, how big is your index?
>
> > Another thing, in the reply you have mentioned 'client not reading fast
> > enough'. Whether it is related to network or Solrj.
>
> That was a general issue - it *can* be the client, but since you're
> using SolrJ it would be the network.
>
> -Yonik
> http://lucidimagination.com
>
Re: Solrj performance bottleneck
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Mar 16, 2011 at 7:25 AM, rahul <as...@gmail.com> wrote:
> In our setup, we are having Solr index in one machine. And Solrj client part
> (java code) in another machine. Currently as you suggest, if it may be a
> 'not enough free RAM for the OS to cache' then whether I need to increase
> the RAM in the machine in which Solrj query part is there.??? Or need to
> increase RAM for Solr instance for the OS cache?
That would be RAM for the Solr instance. If there is not enough free
memory for the OS to cache, then each document retrieved will be a
disk seek + read.
> Since both the system are in local Amazon network (Linux EC2 small
> instances), I believe the network wont be a issue.
Ah, how big is your index?
> Another thing, in the reply you have mentioned 'client not reading fast
> enough'. Whether it is related to network or Solrj.
That was a general issue - it *can* be the client, but since you're
using SolrJ it would be the network.
-Yonik
http://lucidimagination.com
Re: Solrj performance bottleneck
Posted by rahul <as...@gmail.com>.
Hi,
Thanks for your information.
One simple question. Please clarify me.
In our setup, we are having Solr index in one machine. And Solrj client part
(java code) in another machine. Currently as you suggest, if it may be a
'not enough free RAM for the OS to cache' then whether I need to increase
the RAM in the machine in which Solrj query part is there.??? Or need to
increase RAM for Solr instance for the OS cache?
Since both the system are in local Amazon network (Linux EC2 small
instances), I believe the network wont be a issue.
Another thing, in the reply you have mentioned 'client not reading fast
enough'. Whether it is related to network or Solrj.
Thanks in advance for your info.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solrj-performance-bottleneck-tp2682797p2687448.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj performance bottleneck
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Tue, Mar 15, 2011 at 8:12 AM, rahul <as...@gmail.com> wrote:
> I am using Solrj as a Solr client in my project.
>
> While searching, for a few words, it seems Solrj takes more time to send
> response, for eg (8 - 12 sec). While searching most of the other words it
> seems Solrj take less amount of time only.
>
> For eg, if I post a search url in browser, it shows the QTime in
> milliseconds only.
The QTime does not measure the time to stream back the response (which
includes loading stored fields).
Since the response is streamed, it's not possible to include this
time. The difference normally isn't that
large unless you have a network issue, a client that is taking a long
time to read the response,
or a very large index and not enough free RAM for the OS to cache all the files.
Check the solr logs and make sure that equivalent queries are being received.
The QTime is also logged.
-Yonik
http://lucidimagination.com