You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Max Bridgewater <ma...@gmail.com> on 2016/11/22 13:54:20 UTC

Solr 6 Performance Suggestions

I migrated an application from Solr 4 to Solr 6.  solrconfig.xml  and
schema.xml are sensibly the same. The JVM params are also pretty much
similar.  The indicces have each about 2 million documents. No particular
tuning was done to Solr 6 beyond the default settings. Solr 4 is running in
Tomcat 7.

Early results seem to show Solr 4 outperforming Solr 6. The first shows an
average response time of 280 ms while the second averages at 430 ms. The
test cases were exactly the same, the machines where exactly the same and
heap settings exactly the same (Xms24g, Xmx24g). Requests were sent with
Jmeter with 50 concurrent threads for 2h.

I know that this is not enough information to claim that Solr 4 generally
outperforms Solr 6. I also know that this pretty much depends on what the
application does. So I am not claiming anything general. All I want to do
is get some input before I start digging.

What are some things I could tune to improve the numbers for Solr 6? Have
you guys experienced such discrepancies?

Thanks,
Max.

Re: Solr 6 Performance Suggestions

Posted by Walter Underwood <wu...@wunderwood.org>.
We had some serious slowness at startup before I set Xms to be the same as Xmx.

We run with an 8G heap. We have multiple collections but don’t use faceting.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 28, 2016, at 8:40 AM, Max Bridgewater <ma...@gmail.com> wrote:
> 
> Thanks again Folks. I tried each suggestion and none made any difference. I
> am setting up a lab for performance monitoring using App Dynamics.
> Hopefully I am able to figure out something.
> 
> On Mon, Nov 28, 2016 at 11:20 AM, Erick Erickson <er...@gmail.com>
> wrote:
> 
>> bq: If you know the maximum size you ever will need, setting Xmx is good.
>> 
>> Not quite sure what you're getting at here. I pretty much guarantee that a
>> production system will eat up the default heap size, so not setting Xmx
>> will
>> cause OOM errors pretty soon. Or did you mean Xms?
>> 
>> As far as setting Xms, there are differing opinions, mostly though since
>> Solr
>> likes memory so much there's a lot of tuning to try to determine Xmx and
>> it's pretty much guaranteed that Java will need close to that amount of
>> memory.
>> So setting Xms=Xmx is a minor optimization if that assumption is true.
>> It's arguable
>> how much practical difference it makes though.
>> 
>> Best,
>> Erick
>> 
>> On Mon, Nov 28, 2016 at 2:14 AM, Florian Gleixner <fl...@redflo.de> wrote:
>>> Am 28.11.2016 um 00:00 schrieb Shawn Heisey:
>>>> 
>>>> On 11/27/2016 12:51 PM, Florian Gleixner wrote:
>>>>> 
>>>>> On 22.11.2016 14:54, Max Bridgewater wrote:
>>>>>> 
>>>>>> test cases were exactly the same, the machines where exactly the same
>>>>>> and heap settings exactly the same (Xms24g, Xmx24g). Requests were
>>>>>> sent with
>>>>> 
>>>>> Setting heap too large is a common error. Recent Solr use the
>>>>> filesystem cache, so you don't have to set heap to the size of the
>>>>> index. The avalible RAM has to be able to run the OS, run the jvm and
>>>>> hold most of the index data in filesystem cache. If you have 32GB RAM
>>>>> and a 20GB Index, then set -Xms never higher than 10GB. I personally
>>>>> would set -Xms to 4GB and omit -Xmx
>>>> 
>>>> 
>>>> In my mind, the Xmx setting is much more important than Xms.  Setting
>>>> both to the same number avoids any need for Java to detect memory
>>>> pressure before increasing the heap size, which can be helpful.
>>>> 
>>> 
>>> From https://cwiki.apache.org/confluence/display/solr/JVM+Settings
>>> 
>>> "The maximum heap size, set with -Xmx, is more critical. If the memory
>> heap
>>> grows to this size, object creation may begin to fail and throw
>>> OutOfMemoryException. Setting this limit too low can cause spurious
>> errors
>>> in your application, but setting it too high can be detrimental as well."
>>> 
>>> you are right, Xmx is more important. But setting Xms to Xmx will waste
>> RAM,
>>> that the OS can use to cache your index data. Setting Xmx can avoid
>> problems
>>> in some situations where solr can eat up your filesystem cache until the
>>> next GC has been finished.
>>> 
>>>> Without Xmx, Java is in control of the max heap size, and it may not
>>>> make the correct choice.  It's important to know what your max heap is,
>>>> because chances are excellent that the max heap *will* be reached.  Solr
>>>> allocates a lot of memory to do its job.
>>>> 
>>> 
>>> If you know the maximum size you ever will need, setting Xmx is good.
>>> 
>>> 
>>> 
>>> 
>> 


Re: Solr 6 Performance Suggestions

Posted by Max Bridgewater <ma...@gmail.com>.
Thanks again Folks. I tried each suggestion and none made any difference. I
am setting up a lab for performance monitoring using App Dynamics.
Hopefully I am able to figure out something.

On Mon, Nov 28, 2016 at 11:20 AM, Erick Erickson <er...@gmail.com>
wrote:

> bq: If you know the maximum size you ever will need, setting Xmx is good.
>
> Not quite sure what you're getting at here. I pretty much guarantee that a
> production system will eat up the default heap size, so not setting Xmx
> will
> cause OOM errors pretty soon. Or did you mean Xms?
>
> As far as setting Xms, there are differing opinions, mostly though since
> Solr
> likes memory so much there's a lot of tuning to try to determine Xmx and
> it's pretty much guaranteed that Java will need close to that amount of
> memory.
> So setting Xms=Xmx is a minor optimization if that assumption is true.
> It's arguable
> how much practical difference it makes though.
>
> Best,
> Erick
>
> On Mon, Nov 28, 2016 at 2:14 AM, Florian Gleixner <fl...@redflo.de> wrote:
> > Am 28.11.2016 um 00:00 schrieb Shawn Heisey:
> >>
> >> On 11/27/2016 12:51 PM, Florian Gleixner wrote:
> >>>
> >>> On 22.11.2016 14:54, Max Bridgewater wrote:
> >>>>
> >>>> test cases were exactly the same, the machines where exactly the same
> >>>> and heap settings exactly the same (Xms24g, Xmx24g). Requests were
> >>>> sent with
> >>>
> >>> Setting heap too large is a common error. Recent Solr use the
> >>> filesystem cache, so you don't have to set heap to the size of the
> >>> index. The avalible RAM has to be able to run the OS, run the jvm and
> >>> hold most of the index data in filesystem cache. If you have 32GB RAM
> >>> and a 20GB Index, then set -Xms never higher than 10GB. I personally
> >>> would set -Xms to 4GB and omit -Xmx
> >>
> >>
> >> In my mind, the Xmx setting is much more important than Xms.  Setting
> >> both to the same number avoids any need for Java to detect memory
> >> pressure before increasing the heap size, which can be helpful.
> >>
> >
> > From https://cwiki.apache.org/confluence/display/solr/JVM+Settings
> >
> > "The maximum heap size, set with -Xmx, is more critical. If the memory
> heap
> > grows to this size, object creation may begin to fail and throw
> > OutOfMemoryException. Setting this limit too low can cause spurious
> errors
> > in your application, but setting it too high can be detrimental as well."
> >
> > you are right, Xmx is more important. But setting Xms to Xmx will waste
> RAM,
> > that the OS can use to cache your index data. Setting Xmx can avoid
> problems
> > in some situations where solr can eat up your filesystem cache until the
> > next GC has been finished.
> >
> >> Without Xmx, Java is in control of the max heap size, and it may not
> >> make the correct choice.  It's important to know what your max heap is,
> >> because chances are excellent that the max heap *will* be reached.  Solr
> >> allocates a lot of memory to do its job.
> >>
> >
> > If you know the maximum size you ever will need, setting Xmx is good.
> >
> >
> >
> >
>

Re: Solr 6 Performance Suggestions

Posted by Erick Erickson <er...@gmail.com>.
bq: If you know the maximum size you ever will need, setting Xmx is good.

Not quite sure what you're getting at here. I pretty much guarantee that a
production system will eat up the default heap size, so not setting Xmx will
cause OOM errors pretty soon. Or did you mean Xms?

As far as setting Xms, there are differing opinions, mostly though since Solr
likes memory so much there's a lot of tuning to try to determine Xmx and
it's pretty much guaranteed that Java will need close to that amount of memory.
So setting Xms=Xmx is a minor optimization if that assumption is true.
It's arguable
how much practical difference it makes though.

Best,
Erick

On Mon, Nov 28, 2016 at 2:14 AM, Florian Gleixner <fl...@redflo.de> wrote:
> Am 28.11.2016 um 00:00 schrieb Shawn Heisey:
>>
>> On 11/27/2016 12:51 PM, Florian Gleixner wrote:
>>>
>>> On 22.11.2016 14:54, Max Bridgewater wrote:
>>>>
>>>> test cases were exactly the same, the machines where exactly the same
>>>> and heap settings exactly the same (Xms24g, Xmx24g). Requests were
>>>> sent with
>>>
>>> Setting heap too large is a common error. Recent Solr use the
>>> filesystem cache, so you don't have to set heap to the size of the
>>> index. The avalible RAM has to be able to run the OS, run the jvm and
>>> hold most of the index data in filesystem cache. If you have 32GB RAM
>>> and a 20GB Index, then set -Xms never higher than 10GB. I personally
>>> would set -Xms to 4GB and omit -Xmx
>>
>>
>> In my mind, the Xmx setting is much more important than Xms.  Setting
>> both to the same number avoids any need for Java to detect memory
>> pressure before increasing the heap size, which can be helpful.
>>
>
> From https://cwiki.apache.org/confluence/display/solr/JVM+Settings
>
> "The maximum heap size, set with -Xmx, is more critical. If the memory heap
> grows to this size, object creation may begin to fail and throw
> OutOfMemoryException. Setting this limit too low can cause spurious errors
> in your application, but setting it too high can be detrimental as well."
>
> you are right, Xmx is more important. But setting Xms to Xmx will waste RAM,
> that the OS can use to cache your index data. Setting Xmx can avoid problems
> in some situations where solr can eat up your filesystem cache until the
> next GC has been finished.
>
>> Without Xmx, Java is in control of the max heap size, and it may not
>> make the correct choice.  It's important to know what your max heap is,
>> because chances are excellent that the max heap *will* be reached.  Solr
>> allocates a lot of memory to do its job.
>>
>
> If you know the maximum size you ever will need, setting Xmx is good.
>
>
>
>

Re: Solr 6 Performance Suggestions

Posted by Florian Gleixner <fl...@redflo.de>.
Am 28.11.2016 um 00:00 schrieb Shawn Heisey:
> On 11/27/2016 12:51 PM, Florian Gleixner wrote:
>> On 22.11.2016 14:54, Max Bridgewater wrote:
>>> test cases were exactly the same, the machines where exactly the same
>>> and heap settings exactly the same (Xms24g, Xmx24g). Requests were
>>> sent with
>> Setting heap too large is a common error. Recent Solr use the
>> filesystem cache, so you don't have to set heap to the size of the
>> index. The avalible RAM has to be able to run the OS, run the jvm and
>> hold most of the index data in filesystem cache. If you have 32GB RAM
>> and a 20GB Index, then set -Xms never higher than 10GB. I personally
>> would set -Xms to 4GB and omit -Xmx
>
> In my mind, the Xmx setting is much more important than Xms.  Setting
> both to the same number avoids any need for Java to detect memory
> pressure before increasing the heap size, which can be helpful.
>

 From https://cwiki.apache.org/confluence/display/solr/JVM+Settings

"The maximum heap size, set with -Xmx, is more critical. If the memory 
heap grows to this size, object creation may begin to fail and throw 
OutOfMemoryException. Setting this limit too low can cause spurious 
errors in your application, but setting it too high can be detrimental 
as well."

you are right, Xmx is more important. But setting Xms to Xmx will waste 
RAM, that the OS can use to cache your index data. Setting Xmx can avoid 
problems in some situations where solr can eat up your filesystem cache 
until the next GC has been finished.

> Without Xmx, Java is in control of the max heap size, and it may not
> make the correct choice.  It's important to know what your max heap is,
> because chances are excellent that the max heap *will* be reached.  Solr
> allocates a lot of memory to do its job.
>

If you know the maximum size you ever will need, setting Xmx is good.





Re: Solr 6 Performance Suggestions

Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/27/2016 12:51 PM, Florian Gleixner wrote:
> On 22.11.2016 14:54, Max Bridgewater wrote:
>> test cases were exactly the same, the machines where exactly the same
>> and heap settings exactly the same (Xms24g, Xmx24g). Requests were
>> sent with 
> Setting heap too large is a common error. Recent Solr use the
> filesystem cache, so you don't have to set heap to the size of the
> index. The avalible RAM has to be able to run the OS, run the jvm and
> hold most of the index data in filesystem cache. If you have 32GB RAM
> and a 20GB Index, then set -Xms never higher than 10GB. I personally
> would set -Xms to 4GB and omit -Xmx

In my mind, the Xmx setting is much more important than Xms.  Setting
both to the same number avoids any need for Java to detect memory
pressure before increasing the heap size, which can be helpful.

Without Xmx, Java is in control of the max heap size, and it may not
make the correct choice.  It's important to know what your max heap is,
because chances are excellent that the max heap *will* be reached.  Solr
allocates a lot of memory to do its job.

Thanks,
Shawn


Re: Solr 6 Performance Suggestions

Posted by Florian Gleixner <fl...@redflo.de>.
On 22.11.2016 14:54, Max Bridgewater wrote:

> test cases were exactly the same, the machines where exactly the same and
> heap settings exactly the same (Xms24g, Xmx24g). Requests were sent with

Setting heap too large is a common error. Recent Solr use the filesystem
cache, so you don't have to set heap to the size of the index. The
avalible RAM has to be able to run the OS, run the jvm and hold most of
the index data in filesystem cache.
If you have 32GB RAM and a 20GB Index, then set -Xms never higher than
10GB. I personally would set -Xms to 4GB and omit -Xmx



Re: Solr 6 Performance Suggestions

Posted by Jerome Yang <je...@pivotal.io>.
Have you run IndexUpgrader?

Index Format Changes

Solr 6 has no support for reading Lucene/Solr 4.x and earlier indexes.  Be
sure to run the Lucene IndexUpgrader included with Solr 5.5 if you might
still have old 4x formatted segments in your index. Alternatively: fully
optimize your index with Solr 5.5 to make sure it consists only of one
up-to-date index segment.

Regards,
Jerome

On Tue, Nov 22, 2016 at 10:48 PM, Yonik Seeley <ys...@gmail.com> wrote:

> It depends highly on what your requests look like, and which ones are
> slower.
> If you're request mix is heterogeneous, find the types of requests
> that seem to have the largest slowdown and let us know what they look
> like.
>
> -Yonik
>
>
> On Tue, Nov 22, 2016 at 8:54 AM, Max Bridgewater
> <ma...@gmail.com> wrote:
> > I migrated an application from Solr 4 to Solr 6.  solrconfig.xml  and
> > schema.xml are sensibly the same. The JVM params are also pretty much
> > similar.  The indicces have each about 2 million documents. No particular
> > tuning was done to Solr 6 beyond the default settings. Solr 4 is running
> in
> > Tomcat 7.
> >
> > Early results seem to show Solr 4 outperforming Solr 6. The first shows
> an
> > average response time of 280 ms while the second averages at 430 ms. The
> > test cases were exactly the same, the machines where exactly the same and
> > heap settings exactly the same (Xms24g, Xmx24g). Requests were sent with
> > Jmeter with 50 concurrent threads for 2h.
> >
> > I know that this is not enough information to claim that Solr 4 generally
> > outperforms Solr 6. I also know that this pretty much depends on what the
> > application does. So I am not claiming anything general. All I want to do
> > is get some input before I start digging.
> >
> > What are some things I could tune to improve the numbers for Solr 6? Have
> > you guys experienced such discrepancies?
> >
> > Thanks,
> > Max.
>

Re: Solr 6 Performance Suggestions

Posted by Yonik Seeley <ys...@gmail.com>.
It depends highly on what your requests look like, and which ones are slower.
If you're request mix is heterogeneous, find the types of requests
that seem to have the largest slowdown and let us know what they look
like.

-Yonik


On Tue, Nov 22, 2016 at 8:54 AM, Max Bridgewater
<ma...@gmail.com> wrote:
> I migrated an application from Solr 4 to Solr 6.  solrconfig.xml  and
> schema.xml are sensibly the same. The JVM params are also pretty much
> similar.  The indicces have each about 2 million documents. No particular
> tuning was done to Solr 6 beyond the default settings. Solr 4 is running in
> Tomcat 7.
>
> Early results seem to show Solr 4 outperforming Solr 6. The first shows an
> average response time of 280 ms while the second averages at 430 ms. The
> test cases were exactly the same, the machines where exactly the same and
> heap settings exactly the same (Xms24g, Xmx24g). Requests were sent with
> Jmeter with 50 concurrent threads for 2h.
>
> I know that this is not enough information to claim that Solr 4 generally
> outperforms Solr 6. I also know that this pretty much depends on what the
> application does. So I am not claiming anything general. All I want to do
> is get some input before I start digging.
>
> What are some things I could tune to improve the numbers for Solr 6? Have
> you guys experienced such discrepancies?
>
> Thanks,
> Max.

Re: Solr 6 Performance Suggestions

Posted by Max Bridgewater <ma...@gmail.com>.
Thanks folks. It looks like the sweet spot where I get comparable results
is at 30 concurrent threads. It progressively degrades from there as I
increases the number of concurrent threads in the test script.

This made me think that something is configured in Tomcat ((Solr4) that is
not comparatively set in Solr 6. The only thing I found that would make
sense is the connector max number threads that we have set at 800 for
Tomcat. However, it jetty.xml, maxThreads is set to 50000. Not sure if
these two maxThreads have the same effect.

I thought about Yonik suggestion a little bit. Where I am scratching my
head is that if specific kind of queries where more expensive than others,
should this be reflected even at 30 concurrent threads?

Anyway, still digging.

On Wed, Nov 23, 2016 at 9:56 AM, Walter Underwood <wu...@wunderwood.org>
wrote:

> I recently ran benchmarks on 4.10.4 and 6.2.1 and found very little
> difference in query performance.
>
> This was with 8 million documents (homework problems) from production. I
> used query logs from
> production. The load is a constant number of requests per minute from 100
> threads. CPU usage
> is under 50% in order to avoid congestion. The benchmarks ran for 100
> minutes.
>
> Measuring median and 95th percentile, the times were within 10%. I think
> that is within the
> repeatability of the benchmark. A different number of GCs could make that
> difference.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>
> > On Nov 23, 2016, at 8:14 AM, Bram Van Dam <br...@intix.eu> wrote:
> >
> > On 22/11/16 15:34, Prateek Jain J wrote:
> >> I am not sure but I heard this in one of discussions, that you cant
> migrate directly from solr 4 to solr 6. It has to be incremental like solr
> 4 to solr 5 and then to solr 6. I might be wrong but is worth trying.
> >
> > Ideally the index needs to be upgraded using the IndexUpgrader.
> >
> > Something like this should do the trick:
> >
> > java -cp lucene-core-6.0.0.jar:lucene-backward-codecs-6.0.0.jar
> > org.apache.lucene.index.IndexUpgrader /path/to/index
> >
> > - Bram
>
>

Re: Solr 6 Performance Suggestions

Posted by Walter Underwood <wu...@wunderwood.org>.
I recently ran benchmarks on 4.10.4 and 6.2.1 and found very little difference in query performance.

This was with 8 million documents (homework problems) from production. I used query logs from
production. The load is a constant number of requests per minute from 100 threads. CPU usage
is under 50% in order to avoid congestion. The benchmarks ran for 100 minutes.

Measuring median and 95th percentile, the times were within 10%. I think that is within the
repeatability of the benchmark. A different number of GCs could make that difference.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 23, 2016, at 8:14 AM, Bram Van Dam <br...@intix.eu> wrote:
> 
> On 22/11/16 15:34, Prateek Jain J wrote:
>> I am not sure but I heard this in one of discussions, that you cant migrate directly from solr 4 to solr 6. It has to be incremental like solr 4 to solr 5 and then to solr 6. I might be wrong but is worth trying. 
> 
> Ideally the index needs to be upgraded using the IndexUpgrader.
> 
> Something like this should do the trick:
> 
> java -cp lucene-core-6.0.0.jar:lucene-backward-codecs-6.0.0.jar
> org.apache.lucene.index.IndexUpgrader /path/to/index
> 
> - Bram


Re: Solr 6 Performance Suggestions

Posted by Bram Van Dam <br...@intix.eu>.
On 22/11/16 15:34, Prateek Jain J wrote:
> I am not sure but I heard this in one of discussions, that you cant migrate directly from solr 4 to solr 6. It has to be incremental like solr 4 to solr 5 and then to solr 6. I might be wrong but is worth trying. 

Ideally the index needs to be upgraded using the IndexUpgrader.

Something like this should do the trick:

java -cp lucene-core-6.0.0.jar:lucene-backward-codecs-6.0.0.jar
org.apache.lucene.index.IndexUpgrader /path/to/index

 - Bram

RE: Solr 6 Performance Suggestions

Posted by Prateek Jain J <pr...@ericsson.com>.
I am not sure but I heard this in one of discussions, that you cant migrate directly from solr 4 to solr 6. It has to be incremental like solr 4 to solr 5 and then to solr 6. I might be wrong but is worth trying. 


Regards,
Prateek Jain

-----Original Message-----
From: Max Bridgewater [mailto:max.bridgewater@gmail.com] 
Sent: 22 November 2016 01:54 PM
To: solr-user@lucene.apache.org
Subject: Solr 6 Performance Suggestions

I migrated an application from Solr 4 to Solr 6.  solrconfig.xml  and schema.xml are sensibly the same. The JVM params are also pretty much similar.  The indicces have each about 2 million documents. No particular tuning was done to Solr 6 beyond the default settings. Solr 4 is running in Tomcat 7.

Early results seem to show Solr 4 outperforming Solr 6. The first shows an average response time of 280 ms while the second averages at 430 ms. The test cases were exactly the same, the machines where exactly the same and heap settings exactly the same (Xms24g, Xmx24g). Requests were sent with Jmeter with 50 concurrent threads for 2h.

I know that this is not enough information to claim that Solr 4 generally outperforms Solr 6. I also know that this pretty much depends on what the application does. So I am not claiming anything general. All I want to do is get some input before I start digging.

What are some things I could tune to improve the numbers for Solr 6? Have you guys experienced such discrepancies?

Thanks,
Max.