You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by wojtekpia <wo...@hotmail.com> on 2008/11/05 03:01:34 UTC
Throughput Optimization
I've been running load tests over the past week or 2, and I can't figure out
my system's bottle neck that prevents me from increasing throughput. First
I'll describe my Solr setup, then what I've tried to optimize the system.
I have 10 million records and 59 fields (all are indexed, 37 are stored, 17
have termVectors, 33 are multi-valued) which takes about 15GB of disk space.
Most field values are very short (single word or number), and usually about
half the fields have any data at all. I'm running on an 8-core, 64-bit, 32GB
RAM Redhat box. I allocate about 24GB of memory to the java process, and my
filterCache size is 700,000. I'm using a version of Solr between 1.3 and the
current trunk (including the latest SOLR-667 (FastLRUCache) patch), and
Tomcat 6.0.
I'm running a ramp-test, increasing the number of users every few minutes. I
measure the maximum number of requests that Solr can handle per second with
a fixed response time, and call that my throughput. I'd like to see a single
physical resource be maxed out at some point during my test so I know it is
my bottle neck. I generated random queries for my dataset representing a
more or less realistic scenario. The queries include faceting by up to 6
fields, and quering by up to 8 fields.
I ran a baseline on the un-optimized setup, and saw peak CPU usage of about
50%, IO usage around 5%, and negligible network traffic. Interestingly, the
CPU peaked when I had 8 concurrent users, and actually dropped down to about
40% when I increased the users beyond 8. Is that because I have 8 cores?
I changed a few settings and observed the effect on throughput:
1. Increased filterCache size, and throughput increased by about 50%, but it
seems to peak.
2. Put the entire index on a RAM disk, and significantly reduced the average
response time, but my throughput didn't change (i.e. even though my response
time was 10X faster, the maximum number of requests I could make per second
didn't increase). This makes no sense to me, unless there is another bottle
neck somewhere.
3. Reduced the number of records in my index. The throughput increased, but
the shape of all my graphs stayed the same, and my CPU usage was identical.
I have a few questions:
1. Can I get more than 50% CPU utilization?
2. Why does CPU utilization fall when I make more than 8 concurrent
requests?
3. Is there an obvious bottleneck that I'm missing?
4. Does Tomcat have any settings that affect Solr performance?
Any input is greatly appreciated.
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by christophe <ch...@lemoine-fr.com>.
Does the number of searcher affect CPU usage ?
Not totally sure about it but I think some versions of Tomcat were not
totally scalable over 4 CPUs (or 4 cores).
C.
wojtekpia wrote:
> Yes, I am seeing evictions. I've tried setting my filterCache higher, but
> then I start getting Out Of Memory exceptions. My filterCache hit ratio is >
> .99. It looks like I've hit a RAM bound here.
>
> I ran a test without faceting. The response times / throughput were both
> significantly higher, there were no evictions from the filter cache, but I
> still wasn't getting > 50% CPU utilization. Any thoughts on what physical
> bound I've hit in this case?
>
>
>
> Erik Hatcher wrote:
>
>> One quick question.... are you seeing any evictions from your
>> filterCache? If so, it isn't set large enough to handle the faceting
>> you're doing.
>>
>> Erik
>>
>>
>> On Nov 4, 2008, at 8:01 PM, wojtekpia wrote:
>>
>>
>>> I've been running load tests over the past week or 2, and I can't
>>> figure out
>>> my system's bottle neck that prevents me from increasing throughput.
>>> First
>>> I'll describe my Solr setup, then what I've tried to optimize the
>>> system.
>>>
>>> I have 10 million records and 59 fields (all are indexed, 37 are
>>> stored, 17
>>> have termVectors, 33 are multi-valued) which takes about 15GB of
>>> disk space.
>>> Most field values are very short (single word or number), and
>>> usually about
>>> half the fields have any data at all. I'm running on an 8-core, 64-
>>> bit, 32GB
>>> RAM Redhat box. I allocate about 24GB of memory to the java process,
>>> and my
>>> filterCache size is 700,000. I'm using a version of Solr between 1.3
>>> and the
>>> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
>>> and
>>> Tomcat 6.0.
>>>
>>> I'm running a ramp-test, increasing the number of users every few
>>> minutes. I
>>> measure the maximum number of requests that Solr can handle per
>>> second with
>>> a fixed response time, and call that my throughput. I'd like to see
>>> a single
>>> physical resource be maxed out at some point during my test so I
>>> know it is
>>> my bottle neck. I generated random queries for my dataset
>>> representing a
>>> more or less realistic scenario. The queries include faceting by up
>>> to 6
>>> fields, and quering by up to 8 fields.
>>>
>>> I ran a baseline on the un-optimized setup, and saw peak CPU usage
>>> of about
>>> 50%, IO usage around 5%, and negligible network traffic.
>>> Interestingly, the
>>> CPU peaked when I had 8 concurrent users, and actually dropped down
>>> to about
>>> 40% when I increased the users beyond 8. Is that because I have 8
>>> cores?
>>>
>>> I changed a few settings and observed the effect on throughput:
>>>
>>> 1. Increased filterCache size, and throughput increased by about
>>> 50%, but it
>>> seems to peak.
>>> 2. Put the entire index on a RAM disk, and significantly reduced the
>>> average
>>> response time, but my throughput didn't change (i.e. even though my
>>> response
>>> time was 10X faster, the maximum number of requests I could make per
>>> second
>>> didn't increase). This makes no sense to me, unless there is another
>>> bottle
>>> neck somewhere.
>>> 3. Reduced the number of records in my index. The throughput
>>> increased, but
>>> the shape of all my graphs stayed the same, and my CPU usage was
>>> identical.
>>>
>>> I have a few questions:
>>> 1. Can I get more than 50% CPU utilization?
>>> 2. Why does CPU utilization fall when I make more than 8 concurrent
>>> requests?
>>> 3. Is there an obvious bottleneck that I'm missing?
>>> 4. Does Tomcat have any settings that affect Solr performance?
>>>
>>> Any input is greatly appreciated.
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>
>
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Nov 5, 2008 at 2:44 PM, wojtekpia <wo...@hotmail.com> wrote:
> I'll try changing my other caches to LRUCache and observe performance.
> Interestingly, the FastLRUCache has given me a ~10% increase in performance,
> much lower than I've read on the SOLR-667 thread.
That's better than I would have expected for total system throughput -
there's a lot of other stuff besides cache lookups happening.
-Yonik
RE: Throughput Optimization
Posted by wojtekpia <wo...@hotmail.com>.
I'll try changing my other caches to LRUCache and observe performance.
Interestingly, the FastLRUCache has given me a ~10% increase in performance,
much lower than I've read on the SOLR-667 thread.
Would compressing some of my stored fields significantly improve
performance? Most of my stored fields contain single words or numbers, but I
do have one relatively large stored field that contains up to a couple
paragraphs of text.
I agree that my 3% query cache hit rate is quite low (probably
unrealistically low). I'm treating these results as the worst-case.
Feak, Todd wrote:
>
> Yonik said something about the FastLRUCache giving the most gain for
> high hit-rates and the LRUCache being faster for low hit-rates. It's in
> his Nov 1 comment on SOLR-667. I'm not sure if anything changed since
> then, as it's an active issue, but you may want to try the LRUCache for
> your query cache.
>
> It sounds like you are memory bound already, but you may want to
> investigate the tradeoffs of your filter cache vs. document cache. High
> document hit-rate was a big performance boost for us, as document
> garbage collection is a lot of overhead. I believe that would show up as
> CPU usage though, so it may not be your bottleneck.
>
> This also brings up an interesting question. 3% hit rate on your query
> cache seems low to me. Are you sure your load test is mimicking
> realistic query patterns from your user base? I realize this probably
> isn't part of your bottleneck, just curious.
>
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20348749.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Nov 5, 2008 at 2:29 PM, Feak, Todd <To...@smss.sony.com> wrote:
> Yonik said something about the FastLRUCache giving the most gain for
> high hit-rates and the LRUCache being faster for low hit-rates.
Right, for single-threaded requests. FastLRUCache has faster gets and
slower puts (on average). How multi-
The queryCache, which is typically only checked once per request, is
likely to see little real contention under realistic scenarios.
The documentCache could perhaps benefit from FastLRUCache, even with a
lower hit ratio, when concurrency is higher. This is because it's
checked more per request - the number of documents to return, and thus
increases in concurrency may outweigh the slower put(). I have no
idea what the crossover point would be though.
aside: I just committed a little fix to SOLR_667 in the setting of the
acceptableSize (it should normally be between the min and max).
-Yonik
RE: Throughput Optimization
Posted by "Feak, Todd" <To...@smss.sony.com>.
Yonik said something about the FastLRUCache giving the most gain for
high hit-rates and the LRUCache being faster for low hit-rates. It's in
his Nov 1 comment on SOLR-667. I'm not sure if anything changed since
then, as it's an active issue, but you may want to try the LRUCache for
your query cache.
It sounds like you are memory bound already, but you may want to
investigate the tradeoffs of your filter cache vs. document cache. High
document hit-rate was a big performance boost for us, as document
garbage collection is a lot of overhead. I believe that would show up as
CPU usage though, so it may not be your bottleneck.
This also brings up an interesting question. 3% hit rate on your query
cache seems low to me. Are you sure your load test is mimicking
realistic query patterns from your user base? I realize this probably
isn't part of your bottleneck, just curious.
-Todd Feak
-----Original Message-----
From: wojtekpia [mailto:wojtek_p@hotmail.com]
Sent: Wednesday, November 05, 2008 11:08 AM
To: solr-user@lucene.apache.org
Subject: RE: Throughput Optimization
My documentCache hit rate is ~.7, and my queryCache is ~.03. I'm using
FastLRUCache on all 3 of the caches.
Feak, Todd wrote:
>
> What are your other cache hit rates looking like?
> Which caches are you using the FastLRUCache on?
>
> -Todd Feak
>
> -----Original Message-----
> From: wojtekpia [mailto:wojtek_p@hotmail.com]
> Sent: Wednesday, November 05, 2008 8:15 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Throughput Optimization
>
>
> Yes, I am seeing evictions. I've tried setting my filterCache higher,
> but
> then I start getting Out Of Memory exceptions. My filterCache hit
ratio
> is >
> .99. It looks like I've hit a RAM bound here.
>
> I ran a test without faceting. The response times / throughput were
both
> significantly higher, there were no evictions from the filter cache,
but
> I
> still wasn't getting > 50% CPU utilization. Any thoughts on what
> physical
> bound I've hit in this case?
>
>
>
> Erik Hatcher wrote:
>>
>> One quick question.... are you seeing any evictions from your
>> filterCache? If so, it isn't set large enough to handle the faceting
>
>> you're doing.
>>
>> Erik
>>
>>
>> On Nov 4, 2008, at 8:01 PM, wojtekpia wrote:
>>
>>>
>>> I've been running load tests over the past week or 2, and I can't
>>> figure out
>>> my system's bottle neck that prevents me from increasing throughput.
>
>>> First
>>> I'll describe my Solr setup, then what I've tried to optimize the
>>> system.
>>>
>>> I have 10 million records and 59 fields (all are indexed, 37 are
>>> stored, 17
>>> have termVectors, 33 are multi-valued) which takes about 15GB of
>>> disk space.
>>> Most field values are very short (single word or number), and
>>> usually about
>>> half the fields have any data at all. I'm running on an 8-core, 64-
>>> bit, 32GB
>>> RAM Redhat box. I allocate about 24GB of memory to the java process,
>
>>> and my
>>> filterCache size is 700,000. I'm using a version of Solr between 1.3
>
>>> and the
>>> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
>>> and
>>> Tomcat 6.0.
>>>
>>> I'm running a ramp-test, increasing the number of users every few
>>> minutes. I
>>> measure the maximum number of requests that Solr can handle per
>>> second with
>>> a fixed response time, and call that my throughput. I'd like to see
>>> a single
>>> physical resource be maxed out at some point during my test so I
>>> know it is
>>> my bottle neck. I generated random queries for my dataset
>>> representing a
>>> more or less realistic scenario. The queries include faceting by up
>>> to 6
>>> fields, and quering by up to 8 fields.
>>>
>>> I ran a baseline on the un-optimized setup, and saw peak CPU usage
>>> of about
>>> 50%, IO usage around 5%, and negligible network traffic.
>>> Interestingly, the
>>> CPU peaked when I had 8 concurrent users, and actually dropped down
>>> to about
>>> 40% when I increased the users beyond 8. Is that because I have 8
>>> cores?
>>>
>>> I changed a few settings and observed the effect on throughput:
>>>
>>> 1. Increased filterCache size, and throughput increased by about
>>> 50%, but it
>>> seems to peak.
>>> 2. Put the entire index on a RAM disk, and significantly reduced the
>
>>> average
>>> response time, but my throughput didn't change (i.e. even though my
>>> response
>>> time was 10X faster, the maximum number of requests I could make per
>
>>> second
>>> didn't increase). This makes no sense to me, unless there is another
>
>>> bottle
>>> neck somewhere.
>>> 3. Reduced the number of records in my index. The throughput
>>> increased, but
>>> the shape of all my graphs stayed the same, and my CPU usage was
>>> identical.
>>>
>>> I have a few questions:
>>> 1. Can I get more than 50% CPU utilization?
>>> 2. Why does CPU utilization fall when I make more than 8 concurrent
>>> requests?
>>> 3. Is there an obvious bottleneck that I'm missing?
>>> 4. Does Tomcat have any settings that affect Solr performance?
>>>
>>> Any input is greatly appreciated.
>>>
>>> --
>>> View this message in context:
>>>
> http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Throughput-Optimization-tp20335132p20343425.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context:
http://www.nabble.com/Throughput-Optimization-tp20335132p20346663.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Throughput Optimization
Posted by wojtekpia <wo...@hotmail.com>.
My documentCache hit rate is ~.7, and my queryCache is ~.03. I'm using
FastLRUCache on all 3 of the caches.
Feak, Todd wrote:
>
> What are your other cache hit rates looking like?
> Which caches are you using the FastLRUCache on?
>
> -Todd Feak
>
> -----Original Message-----
> From: wojtekpia [mailto:wojtek_p@hotmail.com]
> Sent: Wednesday, November 05, 2008 8:15 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Throughput Optimization
>
>
> Yes, I am seeing evictions. I've tried setting my filterCache higher,
> but
> then I start getting Out Of Memory exceptions. My filterCache hit ratio
> is >
> .99. It looks like I've hit a RAM bound here.
>
> I ran a test without faceting. The response times / throughput were both
> significantly higher, there were no evictions from the filter cache, but
> I
> still wasn't getting > 50% CPU utilization. Any thoughts on what
> physical
> bound I've hit in this case?
>
>
>
> Erik Hatcher wrote:
>>
>> One quick question.... are you seeing any evictions from your
>> filterCache? If so, it isn't set large enough to handle the faceting
>
>> you're doing.
>>
>> Erik
>>
>>
>> On Nov 4, 2008, at 8:01 PM, wojtekpia wrote:
>>
>>>
>>> I've been running load tests over the past week or 2, and I can't
>>> figure out
>>> my system's bottle neck that prevents me from increasing throughput.
>
>>> First
>>> I'll describe my Solr setup, then what I've tried to optimize the
>>> system.
>>>
>>> I have 10 million records and 59 fields (all are indexed, 37 are
>>> stored, 17
>>> have termVectors, 33 are multi-valued) which takes about 15GB of
>>> disk space.
>>> Most field values are very short (single word or number), and
>>> usually about
>>> half the fields have any data at all. I'm running on an 8-core, 64-
>>> bit, 32GB
>>> RAM Redhat box. I allocate about 24GB of memory to the java process,
>
>>> and my
>>> filterCache size is 700,000. I'm using a version of Solr between 1.3
>
>>> and the
>>> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
>>> and
>>> Tomcat 6.0.
>>>
>>> I'm running a ramp-test, increasing the number of users every few
>>> minutes. I
>>> measure the maximum number of requests that Solr can handle per
>>> second with
>>> a fixed response time, and call that my throughput. I'd like to see
>>> a single
>>> physical resource be maxed out at some point during my test so I
>>> know it is
>>> my bottle neck. I generated random queries for my dataset
>>> representing a
>>> more or less realistic scenario. The queries include faceting by up
>>> to 6
>>> fields, and quering by up to 8 fields.
>>>
>>> I ran a baseline on the un-optimized setup, and saw peak CPU usage
>>> of about
>>> 50%, IO usage around 5%, and negligible network traffic.
>>> Interestingly, the
>>> CPU peaked when I had 8 concurrent users, and actually dropped down
>>> to about
>>> 40% when I increased the users beyond 8. Is that because I have 8
>>> cores?
>>>
>>> I changed a few settings and observed the effect on throughput:
>>>
>>> 1. Increased filterCache size, and throughput increased by about
>>> 50%, but it
>>> seems to peak.
>>> 2. Put the entire index on a RAM disk, and significantly reduced the
>
>>> average
>>> response time, but my throughput didn't change (i.e. even though my
>>> response
>>> time was 10X faster, the maximum number of requests I could make per
>
>>> second
>>> didn't increase). This makes no sense to me, unless there is another
>
>>> bottle
>>> neck somewhere.
>>> 3. Reduced the number of records in my index. The throughput
>>> increased, but
>>> the shape of all my graphs stayed the same, and my CPU usage was
>>> identical.
>>>
>>> I have a few questions:
>>> 1. Can I get more than 50% CPU utilization?
>>> 2. Why does CPU utilization fall when I make more than 8 concurrent
>>> requests?
>>> 3. Is there an obvious bottleneck that I'm missing?
>>> 4. Does Tomcat have any settings that affect Solr performance?
>>>
>>> Any input is greatly appreciated.
>>>
>>> --
>>> View this message in context:
>>>
> http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Throughput-Optimization-tp20335132p20343425.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20346663.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Throughput Optimization
Posted by "Feak, Todd" <To...@smss.sony.com>.
What are your other cache hit rates looking like?
Which caches are you using the FastLRUCache on?
-Todd Feak
-----Original Message-----
From: wojtekpia [mailto:wojtek_p@hotmail.com]
Sent: Wednesday, November 05, 2008 8:15 AM
To: solr-user@lucene.apache.org
Subject: Re: Throughput Optimization
Yes, I am seeing evictions. I've tried setting my filterCache higher,
but
then I start getting Out Of Memory exceptions. My filterCache hit ratio
is >
.99. It looks like I've hit a RAM bound here.
I ran a test without faceting. The response times / throughput were both
significantly higher, there were no evictions from the filter cache, but
I
still wasn't getting > 50% CPU utilization. Any thoughts on what
physical
bound I've hit in this case?
Erik Hatcher wrote:
>
> One quick question.... are you seeing any evictions from your
> filterCache? If so, it isn't set large enough to handle the faceting
> you're doing.
>
> Erik
>
>
> On Nov 4, 2008, at 8:01 PM, wojtekpia wrote:
>
>>
>> I've been running load tests over the past week or 2, and I can't
>> figure out
>> my system's bottle neck that prevents me from increasing throughput.
>> First
>> I'll describe my Solr setup, then what I've tried to optimize the
>> system.
>>
>> I have 10 million records and 59 fields (all are indexed, 37 are
>> stored, 17
>> have termVectors, 33 are multi-valued) which takes about 15GB of
>> disk space.
>> Most field values are very short (single word or number), and
>> usually about
>> half the fields have any data at all. I'm running on an 8-core, 64-
>> bit, 32GB
>> RAM Redhat box. I allocate about 24GB of memory to the java process,
>> and my
>> filterCache size is 700,000. I'm using a version of Solr between 1.3
>> and the
>> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
>> and
>> Tomcat 6.0.
>>
>> I'm running a ramp-test, increasing the number of users every few
>> minutes. I
>> measure the maximum number of requests that Solr can handle per
>> second with
>> a fixed response time, and call that my throughput. I'd like to see
>> a single
>> physical resource be maxed out at some point during my test so I
>> know it is
>> my bottle neck. I generated random queries for my dataset
>> representing a
>> more or less realistic scenario. The queries include faceting by up
>> to 6
>> fields, and quering by up to 8 fields.
>>
>> I ran a baseline on the un-optimized setup, and saw peak CPU usage
>> of about
>> 50%, IO usage around 5%, and negligible network traffic.
>> Interestingly, the
>> CPU peaked when I had 8 concurrent users, and actually dropped down
>> to about
>> 40% when I increased the users beyond 8. Is that because I have 8
>> cores?
>>
>> I changed a few settings and observed the effect on throughput:
>>
>> 1. Increased filterCache size, and throughput increased by about
>> 50%, but it
>> seems to peak.
>> 2. Put the entire index on a RAM disk, and significantly reduced the
>> average
>> response time, but my throughput didn't change (i.e. even though my
>> response
>> time was 10X faster, the maximum number of requests I could make per
>> second
>> didn't increase). This makes no sense to me, unless there is another
>> bottle
>> neck somewhere.
>> 3. Reduced the number of records in my index. The throughput
>> increased, but
>> the shape of all my graphs stayed the same, and my CPU usage was
>> identical.
>>
>> I have a few questions:
>> 1. Can I get more than 50% CPU utilization?
>> 2. Why does CPU utilization fall when I make more than 8 concurrent
>> requests?
>> 3. Is there an obvious bottleneck that I'm missing?
>> 4. Does Tomcat have any settings that affect Solr performance?
>>
>> Any input is greatly appreciated.
>>
>> --
>> View this message in context:
>>
http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context:
http://www.nabble.com/Throughput-Optimization-tp20335132p20343425.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Nov 5, 2008 at 11:14 AM, wojtekpia <wo...@hotmail.com> wrote:
> Yes, I am seeing evictions. I've tried setting my filterCache higher, but
> then I start getting Out Of Memory exceptions. My filterCache hit ratio is >
> .99. It looks like I've hit a RAM bound here.
Evictions on the filterCache are fine, as long as it's on the order of
a few per request.
The filter cache is checked for the set of documents that matches the
base query (and filters), and also for unique terms in the field while
faceting (which can be many). As long as the evictions aren't from
the latter category, you should be fine.
-Yonik
RE: Throughput Optimization
Posted by "Feak, Todd" <To...@smss.sony.com>.
If you are seeing < 90% CPU usage and are not IO (File or Network)
bound, then you are most probably bound by lock contention. If your CPU
usage goes down as you throw more threads at the box, that's an even
bigger indication that that is the issue.
A good profiling tool should help you locate this. I'm not endorsing it
in any way, but I've use YourKit locally and have been able to see where
the actual contention was coming from. That lead to my interest in the
SOLR-667 cache fixes which provided enormous benefit.
-Todd
-----Original Message-----
From: wojtekpia [mailto:wojtek_p@hotmail.com]
Sent: Wednesday, November 05, 2008 8:15 AM
To: solr-user@lucene.apache.org
Subject: Re: Throughput Optimization
Yes, I am seeing evictions. I've tried setting my filterCache higher,
but
then I start getting Out Of Memory exceptions. My filterCache hit ratio
is >
.99. It looks like I've hit a RAM bound here.
I ran a test without faceting. The response times / throughput were both
significantly higher, there were no evictions from the filter cache, but
I
still wasn't getting > 50% CPU utilization. Any thoughts on what
physical
bound I've hit in this case?
Erik Hatcher wrote:
>
> One quick question.... are you seeing any evictions from your
> filterCache? If so, it isn't set large enough to handle the faceting
> you're doing.
>
> Erik
>
>
> On Nov 4, 2008, at 8:01 PM, wojtekpia wrote:
>
>>
>> I've been running load tests over the past week or 2, and I can't
>> figure out
>> my system's bottle neck that prevents me from increasing throughput.
>> First
>> I'll describe my Solr setup, then what I've tried to optimize the
>> system.
>>
>> I have 10 million records and 59 fields (all are indexed, 37 are
>> stored, 17
>> have termVectors, 33 are multi-valued) which takes about 15GB of
>> disk space.
>> Most field values are very short (single word or number), and
>> usually about
>> half the fields have any data at all. I'm running on an 8-core, 64-
>> bit, 32GB
>> RAM Redhat box. I allocate about 24GB of memory to the java process,
>> and my
>> filterCache size is 700,000. I'm using a version of Solr between 1.3
>> and the
>> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
>> and
>> Tomcat 6.0.
>>
>> I'm running a ramp-test, increasing the number of users every few
>> minutes. I
>> measure the maximum number of requests that Solr can handle per
>> second with
>> a fixed response time, and call that my throughput. I'd like to see
>> a single
>> physical resource be maxed out at some point during my test so I
>> know it is
>> my bottle neck. I generated random queries for my dataset
>> representing a
>> more or less realistic scenario. The queries include faceting by up
>> to 6
>> fields, and quering by up to 8 fields.
>>
>> I ran a baseline on the un-optimized setup, and saw peak CPU usage
>> of about
>> 50%, IO usage around 5%, and negligible network traffic.
>> Interestingly, the
>> CPU peaked when I had 8 concurrent users, and actually dropped down
>> to about
>> 40% when I increased the users beyond 8. Is that because I have 8
>> cores?
>>
>> I changed a few settings and observed the effect on throughput:
>>
>> 1. Increased filterCache size, and throughput increased by about
>> 50%, but it
>> seems to peak.
>> 2. Put the entire index on a RAM disk, and significantly reduced the
>> average
>> response time, but my throughput didn't change (i.e. even though my
>> response
>> time was 10X faster, the maximum number of requests I could make per
>> second
>> didn't increase). This makes no sense to me, unless there is another
>> bottle
>> neck somewhere.
>> 3. Reduced the number of records in my index. The throughput
>> increased, but
>> the shape of all my graphs stayed the same, and my CPU usage was
>> identical.
>>
>> I have a few questions:
>> 1. Can I get more than 50% CPU utilization?
>> 2. Why does CPU utilization fall when I make more than 8 concurrent
>> requests?
>> 3. Is there an obvious bottleneck that I'm missing?
>> 4. Does Tomcat have any settings that affect Solr performance?
>>
>> Any input is greatly appreciated.
>>
>> --
>> View this message in context:
>>
http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context:
http://www.nabble.com/Throughput-Optimization-tp20335132p20343425.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by wojtekpia <wo...@hotmail.com>.
Yes, I am seeing evictions. I've tried setting my filterCache higher, but
then I start getting Out Of Memory exceptions. My filterCache hit ratio is >
.99. It looks like I've hit a RAM bound here.
I ran a test without faceting. The response times / throughput were both
significantly higher, there were no evictions from the filter cache, but I
still wasn't getting > 50% CPU utilization. Any thoughts on what physical
bound I've hit in this case?
Erik Hatcher wrote:
>
> One quick question.... are you seeing any evictions from your
> filterCache? If so, it isn't set large enough to handle the faceting
> you're doing.
>
> Erik
>
>
> On Nov 4, 2008, at 8:01 PM, wojtekpia wrote:
>
>>
>> I've been running load tests over the past week or 2, and I can't
>> figure out
>> my system's bottle neck that prevents me from increasing throughput.
>> First
>> I'll describe my Solr setup, then what I've tried to optimize the
>> system.
>>
>> I have 10 million records and 59 fields (all are indexed, 37 are
>> stored, 17
>> have termVectors, 33 are multi-valued) which takes about 15GB of
>> disk space.
>> Most field values are very short (single word or number), and
>> usually about
>> half the fields have any data at all. I'm running on an 8-core, 64-
>> bit, 32GB
>> RAM Redhat box. I allocate about 24GB of memory to the java process,
>> and my
>> filterCache size is 700,000. I'm using a version of Solr between 1.3
>> and the
>> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
>> and
>> Tomcat 6.0.
>>
>> I'm running a ramp-test, increasing the number of users every few
>> minutes. I
>> measure the maximum number of requests that Solr can handle per
>> second with
>> a fixed response time, and call that my throughput. I'd like to see
>> a single
>> physical resource be maxed out at some point during my test so I
>> know it is
>> my bottle neck. I generated random queries for my dataset
>> representing a
>> more or less realistic scenario. The queries include faceting by up
>> to 6
>> fields, and quering by up to 8 fields.
>>
>> I ran a baseline on the un-optimized setup, and saw peak CPU usage
>> of about
>> 50%, IO usage around 5%, and negligible network traffic.
>> Interestingly, the
>> CPU peaked when I had 8 concurrent users, and actually dropped down
>> to about
>> 40% when I increased the users beyond 8. Is that because I have 8
>> cores?
>>
>> I changed a few settings and observed the effect on throughput:
>>
>> 1. Increased filterCache size, and throughput increased by about
>> 50%, but it
>> seems to peak.
>> 2. Put the entire index on a RAM disk, and significantly reduced the
>> average
>> response time, but my throughput didn't change (i.e. even though my
>> response
>> time was 10X faster, the maximum number of requests I could make per
>> second
>> didn't increase). This makes no sense to me, unless there is another
>> bottle
>> neck somewhere.
>> 3. Reduced the number of records in my index. The throughput
>> increased, but
>> the shape of all my graphs stayed the same, and my CPU usage was
>> identical.
>>
>> I have a few questions:
>> 1. Can I get more than 50% CPU utilization?
>> 2. Why does CPU utilization fall when I make more than 8 concurrent
>> requests?
>> 3. Is there an obvious bottleneck that I'm missing?
>> 4. Does Tomcat have any settings that affect Solr performance?
>>
>> Any input is greatly appreciated.
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20343425.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
One quick question.... are you seeing any evictions from your
filterCache? If so, it isn't set large enough to handle the faceting
you're doing.
Erik
On Nov 4, 2008, at 8:01 PM, wojtekpia wrote:
>
> I've been running load tests over the past week or 2, and I can't
> figure out
> my system's bottle neck that prevents me from increasing throughput.
> First
> I'll describe my Solr setup, then what I've tried to optimize the
> system.
>
> I have 10 million records and 59 fields (all are indexed, 37 are
> stored, 17
> have termVectors, 33 are multi-valued) which takes about 15GB of
> disk space.
> Most field values are very short (single word or number), and
> usually about
> half the fields have any data at all. I'm running on an 8-core, 64-
> bit, 32GB
> RAM Redhat box. I allocate about 24GB of memory to the java process,
> and my
> filterCache size is 700,000. I'm using a version of Solr between 1.3
> and the
> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
> and
> Tomcat 6.0.
>
> I'm running a ramp-test, increasing the number of users every few
> minutes. I
> measure the maximum number of requests that Solr can handle per
> second with
> a fixed response time, and call that my throughput. I'd like to see
> a single
> physical resource be maxed out at some point during my test so I
> know it is
> my bottle neck. I generated random queries for my dataset
> representing a
> more or less realistic scenario. The queries include faceting by up
> to 6
> fields, and quering by up to 8 fields.
>
> I ran a baseline on the un-optimized setup, and saw peak CPU usage
> of about
> 50%, IO usage around 5%, and negligible network traffic.
> Interestingly, the
> CPU peaked when I had 8 concurrent users, and actually dropped down
> to about
> 40% when I increased the users beyond 8. Is that because I have 8
> cores?
>
> I changed a few settings and observed the effect on throughput:
>
> 1. Increased filterCache size, and throughput increased by about
> 50%, but it
> seems to peak.
> 2. Put the entire index on a RAM disk, and significantly reduced the
> average
> response time, but my throughput didn't change (i.e. even though my
> response
> time was 10X faster, the maximum number of requests I could make per
> second
> didn't increase). This makes no sense to me, unless there is another
> bottle
> neck somewhere.
> 3. Reduced the number of records in my index. The throughput
> increased, but
> the shape of all my graphs stayed the same, and my CPU usage was
> identical.
>
> I have a few questions:
> 1. Can I get more than 50% CPU utilization?
> 2. Why does CPU utilization fall when I make more than 8 concurrent
> requests?
> 3. Is there an obvious bottleneck that I'm missing?
> 4. Does Tomcat have any settings that affect Solr performance?
>
> Any input is greatly appreciated.
>
> --
> View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20335132.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Dec 4, 2008 at 2:30 PM, wojtekpia <wo...@hotmail.com> wrote:
>
> New faceting stuff off because I'm encountering some problems when I turn it
> on, I posted the details:
> http://www.nabble.com/new-faceting-algorithm-td20674902.html#a20840622
Missed that, thanks... will respond there.
-Yonik
>
> Yonik Seeley wrote:
>>
>> On Thu, Dec 4, 2008 at 1:54 PM, wojtekpia <wo...@hotmail.com> wrote:
>>> It looks like file locking was the bottleneck - CPU usage is up to ~98%
>>> (from
>>> the previous peak of ~50%).
>>
>> Great to hear it!
>>
>>> I'm running the trunk code from Dec 2 with the
>>> faceting improvement (SOLR-475) turned off. Thanks for all the help!
>>
>> new faceting stuff off because it didn't improve things in your case,
>> or because you didn't want to change that variable just now?
>>
>> -Yonik
>>
>
> --
> View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20840668.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Throughput Optimization
Posted by wojtekpia <wo...@hotmail.com>.
New faceting stuff off because I'm encountering some problems when I turn it
on, I posted the details:
http://www.nabble.com/new-faceting-algorithm-td20674902.html#a20840622
Yonik Seeley wrote:
>
> On Thu, Dec 4, 2008 at 1:54 PM, wojtekpia <wo...@hotmail.com> wrote:
>> It looks like file locking was the bottleneck - CPU usage is up to ~98%
>> (from
>> the previous peak of ~50%).
>
> Great to hear it!
>
>> I'm running the trunk code from Dec 2 with the
>> faceting improvement (SOLR-475) turned off. Thanks for all the help!
>
> new faceting stuff off because it didn't improve things in your case,
> or because you didn't want to change that variable just now?
>
> -Yonik
>
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20840668.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Dec 4, 2008 at 1:54 PM, wojtekpia <wo...@hotmail.com> wrote:
> It looks like file locking was the bottleneck - CPU usage is up to ~98% (from
> the previous peak of ~50%).
Great to hear it!
> I'm running the trunk code from Dec 2 with the
> faceting improvement (SOLR-475) turned off. Thanks for all the help!
new faceting stuff off because it didn't improve things in your case,
or because you didn't want to change that variable just now?
-Yonik
>
> Yonik Seeley wrote:
>>
>> FYI, SOLR-465 has been committed. Let us know if it improves your
>> scenario.
>>
>> -Yonik
>>
>
> --
> View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20840017.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Throughput Optimization
Posted by wojtekpia <wo...@hotmail.com>.
It looks like file locking was the bottleneck - CPU usage is up to ~98% (from
the previous peak of ~50%). I'm running the trunk code from Dec 2 with the
faceting improvement (SOLR-475) turned off. Thanks for all the help!
Yonik Seeley wrote:
>
> FYI, SOLR-465 has been committed. Let us know if it improves your
> scenario.
>
> -Yonik
>
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20840017.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
FYI, SOLR-465 has been committed. Let us know if it improves your scenario.
-Yonik
On Wed, Nov 5, 2008 at 5:39 PM, Yonik Seeley <yo...@apache.org> wrote:
> On Wed, Nov 5, 2008 at 5:18 PM, wojtekpia <wo...@hotmail.com> wrote:
>> I'd like to integrate this improvement into my deployment. Is it just a
>> matter of getting the latest Lucene jars (Lucene nightly build)?
>
> You need to apply this source code patch to Solr:
> https://issues.apache.org/jira/browse/SOLR-465
>
> Or you can wait for me to commit it (which I'll probably do in a day
> unless someone else sees problems with it) and then just use a recent
> Solr nightly build.
>
> -Yonik
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Nov 5, 2008 at 5:18 PM, wojtekpia <wo...@hotmail.com> wrote:
> I'd like to integrate this improvement into my deployment. Is it just a
> matter of getting the latest Lucene jars (Lucene nightly build)?
You need to apply this source code patch to Solr:
https://issues.apache.org/jira/browse/SOLR-465
Or you can wait for me to commit it (which I'll probably do in a day
unless someone else sees problems with it) and then just use a recent
Solr nightly build.
-Yonik
Re: Throughput Optimization
Posted by wojtekpia <wo...@hotmail.com>.
I'd like to integrate this improvement into my deployment. Is it just a
matter of getting the latest Lucene jars (Lucene nightly build)?
Yonik Seeley wrote:
>
> You're probably hitting some contention with the locking around the
> reading of index files... this has been recently improved in Lucene
> for non-Windows boxes, and we're integrating that into Solr (should
> def be in the next release).
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20349247.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by wojtekpia <wo...@hotmail.com>.
Where is the alt directory in the source tree (or what is the JIRA issue
number)? I'd like to apply this patch and re-run my tests.
Does changing the lockType in solrconfig.xml address this issue? (My
lockType is the default - single).
markrmiller wrote:
>
> The latest alt directory patch uses It.
>
> - Mark
>
>
--
View this message in context: http://www.nabble.com/Throughput-Optimization-tp20335132p20345965.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Throughput Optimization
Posted by Mark Miller <ma...@gmail.com>.
The latest alt directory patch uses It.
- Mark
On Nov 5, 2008, at 9:25 AM, "Yonik Seeley" <yo...@apache.org> wrote:
> You're probably hitting some contention with the locking around the
> reading of index files... this has been recently improved in Lucene
> for non-Windows boxes, and we're integrating that into Solr (should
> def be in the next release).
>
> -Yonik
>
> On Tue, Nov 4, 2008 at 9:01 PM, wojtekpia <wo...@hotmail.com>
> wrote:
>>
>> I've been running load tests over the past week or 2, and I can't
>> figure out
>> my system's bottle neck that prevents me from increasing
>> throughput. First
>> I'll describe my Solr setup, then what I've tried to optimize the
>> system.
>>
>> I have 10 million records and 59 fields (all are indexed, 37 are
>> stored, 17
>> have termVectors, 33 are multi-valued) which takes about 15GB of
>> disk space.
>> Most field values are very short (single word or number), and
>> usually about
>> half the fields have any data at all. I'm running on an 8-core, 64-
>> bit, 32GB
>> RAM Redhat box. I allocate about 24GB of memory to the java
>> process, and my
>> filterCache size is 700,000. I'm using a version of Solr between
>> 1.3 and the
>> current trunk (including the latest SOLR-667 (FastLRUCache) patch),
>> and
>> Tomcat 6.0.
>>
>> I'm running a ramp-test, increasing the number of users every few
>> minutes. I
>> measure the maximum number of requests that Solr can handle per
>> second with
>> a fixed response time, and call that my throughput. I'd like to see
>> a single
>> physical resource be maxed out at some point during my test so I
>> know it is
>> my bottle neck. I generated random queries for my dataset
>> representing a
>> more or less realistic scenario. The queries include faceting by up
>> to 6
>> fields, and quering by up to 8 fields.
>>
>> I ran a baseline on the un-optimized setup, and saw peak CPU usage
>> of about
>> 50%, IO usage around 5%, and negligible network traffic.
>> Interestingly, the
>> CPU peaked when I had 8 concurrent users, and actually dropped down
>> to about
>> 40% when I increased the users beyond 8. Is that because I have 8
>> cores?
>>
>> I changed a few settings and observed the effect on throughput:
>>
>> 1. Increased filterCache size, and throughput increased by about
>> 50%, but it
>> seems to peak.
>> 2. Put the entire index on a RAM disk, and significantly reduced
>> the average
>> response time, but my throughput didn't change (i.e. even though my
>> response
>> time was 10X faster, the maximum number of requests I could make
>> per second
>> didn't increase). This makes no sense to me, unless there is
>> another bottle
>> neck somewhere.
>> 3. Reduced the number of records in my index. The throughput
>> increased, but
>> the shape of all my graphs stayed the same, and my CPU usage was
>> identical.
>>
>> I have a few questions:
>> 1. Can I get more than 50% CPU utilization?
>> 2. Why does CPU utilization fall when I make more than 8 concurrent
>> requests?
>> 3. Is there an obvious bottleneck that I'm missing?
>> 4. Does Tomcat have any settings that affect Solr performance?
>>
>> Any input is greatly appreciated.
Re: Throughput Optimization
Posted by Yonik Seeley <yo...@apache.org>.
You're probably hitting some contention with the locking around the
reading of index files... this has been recently improved in Lucene
for non-Windows boxes, and we're integrating that into Solr (should
def be in the next release).
-Yonik
On Tue, Nov 4, 2008 at 9:01 PM, wojtekpia <wo...@hotmail.com> wrote:
>
> I've been running load tests over the past week or 2, and I can't figure out
> my system's bottle neck that prevents me from increasing throughput. First
> I'll describe my Solr setup, then what I've tried to optimize the system.
>
> I have 10 million records and 59 fields (all are indexed, 37 are stored, 17
> have termVectors, 33 are multi-valued) which takes about 15GB of disk space.
> Most field values are very short (single word or number), and usually about
> half the fields have any data at all. I'm running on an 8-core, 64-bit, 32GB
> RAM Redhat box. I allocate about 24GB of memory to the java process, and my
> filterCache size is 700,000. I'm using a version of Solr between 1.3 and the
> current trunk (including the latest SOLR-667 (FastLRUCache) patch), and
> Tomcat 6.0.
>
> I'm running a ramp-test, increasing the number of users every few minutes. I
> measure the maximum number of requests that Solr can handle per second with
> a fixed response time, and call that my throughput. I'd like to see a single
> physical resource be maxed out at some point during my test so I know it is
> my bottle neck. I generated random queries for my dataset representing a
> more or less realistic scenario. The queries include faceting by up to 6
> fields, and quering by up to 8 fields.
>
> I ran a baseline on the un-optimized setup, and saw peak CPU usage of about
> 50%, IO usage around 5%, and negligible network traffic. Interestingly, the
> CPU peaked when I had 8 concurrent users, and actually dropped down to about
> 40% when I increased the users beyond 8. Is that because I have 8 cores?
>
> I changed a few settings and observed the effect on throughput:
>
> 1. Increased filterCache size, and throughput increased by about 50%, but it
> seems to peak.
> 2. Put the entire index on a RAM disk, and significantly reduced the average
> response time, but my throughput didn't change (i.e. even though my response
> time was 10X faster, the maximum number of requests I could make per second
> didn't increase). This makes no sense to me, unless there is another bottle
> neck somewhere.
> 3. Reduced the number of records in my index. The throughput increased, but
> the shape of all my graphs stayed the same, and my CPU usage was identical.
>
> I have a few questions:
> 1. Can I get more than 50% CPU utilization?
> 2. Why does CPU utilization fall when I make more than 8 concurrent
> requests?
> 3. Is there an obvious bottleneck that I'm missing?
> 4. Does Tomcat have any settings that affect Solr performance?
>
> Any input is greatly appreciated.