Solr CPU usage

[Girish Redekar <gi...@aplopio.com>]

Hi

I'm testing my Solr instance with multiple simultaneous requests. Here's my
test.

For an index of ~200K docs, I query Solr with 10 simultaneous threads. Can
someone help me explain/improve the following observations:

1) Solr doesn't seem to use all the available CPU to improve response times
(query times are good, but the time required to return documents aren't so
good). My CPU seems to be running at ~30%
2) As expected, time for response increases as the num of requested results
increase. What's surprising (and perplexing) is that Solr seems to use
*more* of the CPU when I'm requesting *fewer* docs. Consequently, its
performance in returning a larger result set is very bad
3) To counter 1, is there a way to make two Solr instances search on the
same index (so that concurrent requests are served faster)

Any help in this regard would be very useful.

Thanks !
Girish Redekar
http://girishredekar.net

Re: Solr CPU usage

[Yonik Seeley <yo...@lucidimagination.com>]

On Fri, Nov 27, 2009 at 12:09 PM, Girish Redekar
<gi...@aplopio.com> wrote:
> Am running both my server and client on ubuntu machines. The client is on a
> different box. The server CPU and RAM are both well below 50%.

OK, then the obvious thing to try would be to move the client to the
server machine and see if you can max out the CPUs.  Sould be doable
on Solr 1.4

-Yonik
http://www.lucidimagination.com

Re: Solr CPU usage

[Girish Redekar <gi...@aplopio.com>]

Yonik,
Am running both my server and client on ubuntu machines. The client is on a
different box. The server CPU and RAM are both well below 50%.

Girish Redekar
http://girishredekar.net


On Fri, Nov 27, 2009 at 10:07 PM, Yonik Seeley
<yo...@lucidimagination.com>wrote:

> On Fri, Nov 27, 2009 at 9:30 AM, Girish Redekar
> <gi...@aplopio.com> wrote:
> > Hi
> >
> > I'm testing my Solr instance with multiple simultaneous requests. Here's
> my
> > test.
> >
> > For an index of ~200K docs, I query Solr with 10 simultaneous threads.
> Can
> > someone help me explain/improve the following observations:
> >
> > 1) Solr doesn't seem to use all the available CPU to improve response
> times
> > (query times are good, but the time required to return documents aren't
> so
> > good). My CPU seems to be running at ~30%
> > 2) As expected, time for response increases as the num of requested
> results
> > increase. What's surprising (and perplexing) is that Solr seems to use
> > *more* of the CPU when I'm requesting *fewer* docs. Consequently, its
> > performance in returning a larger result set is very bad
>
> This may point to another bottleneck - if the OS is low on free RAM,
> it could be disk IO.
> If this is on Windows, you could have contention reading the index files.
> Otherwise, you may have a bottleneck in network IO.  Is the client you
> are testing with on the same box?
> Is this Solr 1.4?
>
> -Yonik
> http://www.lucidimagination.com
>

Re: Solr CPU usage

[Yonik Seeley <yo...@lucidimagination.com>]

On Fri, Nov 27, 2009 at 9:30 AM, Girish Redekar
<gi...@aplopio.com> wrote:
> Hi
>
> I'm testing my Solr instance with multiple simultaneous requests. Here's my
> test.
>
> For an index of ~200K docs, I query Solr with 10 simultaneous threads. Can
> someone help me explain/improve the following observations:
>
> 1) Solr doesn't seem to use all the available CPU to improve response times
> (query times are good, but the time required to return documents aren't so
> good). My CPU seems to be running at ~30%
> 2) As expected, time for response increases as the num of requested results
> increase. What's surprising (and perplexing) is that Solr seems to use
> *more* of the CPU when I'm requesting *fewer* docs. Consequently, its
> performance in returning a larger result set is very bad

This may point to another bottleneck - if the OS is low on free RAM,
it could be disk IO.
If this is on Windows, you could have contention reading the index files.
Otherwise, you may have a bottleneck in network IO.  Is the client you
are testing with on the same box?
Is this Solr 1.4?

-Yonik
http://www.lucidimagination.com