You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "bihan.chandu" <bi...@gmail.com> on 2014/05/29 07:36:08 UTC
Solr High GC issue
Hi All
I am Currently using solr 3.6.1 and my system handle lot of request .Now we
are facing High GC issue in system. Please find the memory parameters in my
solr system . Can some on help me to identify is there any relationship
between my memory parameters and GC issue.
MEM_ARGS="-Xms7936M -Xmx7936M -XX:NewSize=512M -XX:MaxNewSize=512M -Xss1024k
-XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSParallelRemarkEnabled -XX:+AggressiveOpts
-XX:LargePageSizeInBytes=2m -XX:+UseLargePages -XX:MaxTenuringThreshold=15
-XX:-UseAdaptiveSizePolicy -XX:PermSize=256M -XX:MaxPermSize=256M
-XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGC
-Xloggc:${GCLOG} -XX:-OmitStackTraceInFastThrow -XX:+DisableExplicitGC
-XX:-BindGCTaskThreadsToCPUs -verbose:gc -XX:StackShadowPages=20"
Thanks
Bihan
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-High-GC-issue-tp4138570.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr High GC issue
Posted by Erick Erickson <er...@gmail.com>.
BTW, memory usage is _much_ better in Solr 4.x, if it's at all an option
you might consider upgrading. See:
http://searchhub.org/2012/04/06/memory-comparisons-between-solr-3x-and-trunk/
Best,
Erick
On Thu, May 29, 2014 at 11:40 AM, Toke Eskildsen <te...@statsbiblioteket.dk>
wrote:
> bihan.chandu [bihan.chandu@gmail.com] wrote:
> > I am Currently using solr 3.6.1 and my system handle lot of request .Now
> we
> > are facing High GC issue in system.
>
> Maybe it would help to get an idea of what is causing all the allocations?
>
> - How many documents in your index?
> - How many queries/sec?
> - How long does a typical query take?
> - How many cores does your machine have?
> - How many documents are returned/query?
> - How much faceting do you perform? How many unique terms/facet field?
>
> - Toke Eskildsen
>
RE: Solr High GC issue
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
bihan.chandu [bihan.chandu@gmail.com] wrote:
> I am Currently using solr 3.6.1 and my system handle lot of request .Now we
> are facing High GC issue in system.
Maybe it would help to get an idea of what is causing all the allocations?
- How many documents in your index?
- How many queries/sec?
- How long does a typical query take?
- How many cores does your machine have?
- How many documents are returned/query?
- How much faceting do you perform? How many unique terms/facet field?
- Toke Eskildsen
Re: Solr High GC issue
Posted by Dmitry Kan <so...@gmail.com>.
@rulinma:
I've analyzed all your messages and noticed you keep sending useless "mark"
and "good". Are you trying to spam the mailing list? Could you please stop
doing this?
E-mail clients these days have bookmarking / staring in case you want to
save the "good" post for later reading or "mark" it otherwise.
Thanks,
Dmitry
On Wed, Jun 4, 2014 at 3:25 PM, rulinma <ru...@gmail.com> wrote:
> mark.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-High-GC-issue-tp4138570p4139785.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Dmitry Kan
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
Re: Solr High GC issue
Posted by rulinma <ru...@gmail.com>.
mark.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-High-GC-issue-tp4138570p4139785.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr High GC issue
Posted by Walter Underwood <wu...@wunderwood.org>.
Agreed, that is a LOT of options.
First, check the defaults and remove any flags that are setting something to the default. You can see all the flags and the default values with this command:
java -XX:+PrintFlagsFinal -version
For example, the default for ParallelGCThreads is 8, so you do not need to set that.
We set a fairly large new generation, about 1/4 of heap. 512 Meg is way too small. Solr will allocate a lot of objects that are only used to handle one HTTP request. You want all of those to fit in the new space, even when there are simultaneous requests. If new is not big enough, they will be allocated in tenured space and will cause more frequent major GCs. For an 8G heap, we use a 2G new size.
wunder
On May 29, 2014, at 7:20 AM, Otis Gospodnetic <ot...@gmail.com> wrote:
> Hi Bihan,
>
> That's a lot of parameters and without trying one can't really give you
> very specific and good advice. If I had to suggest something quickly I'd
> say:
>
> * go back to the basics - remove most of those params and stick with the
> basic ones. Look at GC and tune slowly by changing/adding params one at a
> time.
> * consider using G1 GC with the most recent Java7.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Thu, May 29, 2014 at 1:36 AM, bihan.chandu <bi...@gmail.com>wrote:
>
>> Hi All
>>
>> I am Currently using solr 3.6.1 and my system handle lot of request .Now we
>> are facing High GC issue in system. Please find the memory parameters in my
>> solr system . Can some on help me to identify is there any relationship
>> between my memory parameters and GC issue.
>>
>> MEM_ARGS="-Xms7936M -Xmx7936M -XX:NewSize=512M -XX:MaxNewSize=512M
>> -Xss1024k
>> -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
>> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+CMSParallelRemarkEnabled -XX:+AggressiveOpts
>> -XX:LargePageSizeInBytes=2m -XX:+UseLargePages -XX:MaxTenuringThreshold=15
>> -XX:-UseAdaptiveSizePolicy -XX:PermSize=256M -XX:MaxPermSize=256M
>> -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+PrintGCDetails
>> -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGC
>> -Xloggc:${GCLOG} -XX:-OmitStackTraceInFastThrow -XX:+DisableExplicitGC
>> -XX:-BindGCTaskThreadsToCPUs -verbose:gc -XX:StackShadowPages=20"
>>
>> Thanks
>> Bihan
>>
RE: Solr High GC issue
Posted by Boogie Shafer <bo...@ebrary.com>.
you will probably also want to get some better visibility into what is going on with your JVM and GC
easiest way is to enable some GC logging options. the following additional options will give you a good deal of information in gc logs
-Xloggc:$JETTY_LOGS/gc.log
-verbose:gc
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:+PrintClassHistogram
-XX:+PrintHeapAtGC
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
-XX:+PrintAdaptiveSizePolicy
-XX:+PrintTLAB
-XX:PrintFLSStatistics=1
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=5
-XX:GCLogFileSize=10m
you may find you have a particular portion of your heap which is undersized
using G1GC with the adaptive sizing is a very handy way to deal with the memory usage in solr which can be somewhat difficult to tune optimally using the traditional static ratios (what works well for data ingestion is probably not optimal for searching)
once you have a baseline of logs using your existing JVM sizings and the additional logging options above you might try switching from CMS to G1GC with adaptive sizing, and removing all the static tunings for tenuring and ratios and compare to a very minimal G1GC config
-XX:+UseG1GC
-XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=1000
--
configuring the JMX interface is another way to get real time views into what is going on using jconsole or jvisualvm tools
________________________________________
From: Otis Gospodnetic <ot...@gmail.com>
Sent: Thursday, May 29, 2014 07:20
To: solr-user@lucene.apache.org
Subject: Re: Solr High GC issue
Hi Bihan,
That's a lot of parameters and without trying one can't really give you
very specific and good advice. If I had to suggest something quickly I'd
say:
* go back to the basics - remove most of those params and stick with the
basic ones. Look at GC and tune slowly by changing/adding params one at a
time.
* consider using G1 GC with the most recent Java7.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Thu, May 29, 2014 at 1:36 AM, bihan.chandu <bi...@gmail.com>wrote:
> Hi All
>
> I am Currently using solr 3.6.1 and my system handle lot of request .Now we
> are facing High GC issue in system. Please find the memory parameters in my
> solr system . Can some on help me to identify is there any relationship
> between my memory parameters and GC issue.
>
> MEM_ARGS="-Xms7936M -Xmx7936M -XX:NewSize=512M -XX:MaxNewSize=512M
> -Xss1024k
> -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+CMSParallelRemarkEnabled -XX:+AggressiveOpts
> -XX:LargePageSizeInBytes=2m -XX:+UseLargePages -XX:MaxTenuringThreshold=15
> -XX:-UseAdaptiveSizePolicy -XX:PermSize=256M -XX:MaxPermSize=256M
> -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGC
> -Xloggc:${GCLOG} -XX:-OmitStackTraceInFastThrow -XX:+DisableExplicitGC
> -XX:-BindGCTaskThreadsToCPUs -verbose:gc -XX:StackShadowPages=20"
>
> Thanks
> Bihan
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-High-GC-issue-tp4138570.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr High GC issue
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Bihan,
That's a lot of parameters and without trying one can't really give you
very specific and good advice. If I had to suggest something quickly I'd
say:
* go back to the basics - remove most of those params and stick with the
basic ones. Look at GC and tune slowly by changing/adding params one at a
time.
* consider using G1 GC with the most recent Java7.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Thu, May 29, 2014 at 1:36 AM, bihan.chandu <bi...@gmail.com>wrote:
> Hi All
>
> I am Currently using solr 3.6.1 and my system handle lot of request .Now we
> are facing High GC issue in system. Please find the memory parameters in my
> solr system . Can some on help me to identify is there any relationship
> between my memory parameters and GC issue.
>
> MEM_ARGS="-Xms7936M -Xmx7936M -XX:NewSize=512M -XX:MaxNewSize=512M
> -Xss1024k
> -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+CMSParallelRemarkEnabled -XX:+AggressiveOpts
> -XX:LargePageSizeInBytes=2m -XX:+UseLargePages -XX:MaxTenuringThreshold=15
> -XX:-UseAdaptiveSizePolicy -XX:PermSize=256M -XX:MaxPermSize=256M
> -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGC
> -Xloggc:${GCLOG} -XX:-OmitStackTraceInFastThrow -XX:+DisableExplicitGC
> -XX:-BindGCTaskThreadsToCPUs -verbose:gc -XX:StackShadowPages=20"
>
> Thanks
> Bihan
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-High-GC-issue-tp4138570.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr High GC issue
Posted by "bihan.chandu" <bi...@gmail.com>.
Hi All
Thanks for the Suggestion. I will implement this changes and let us know the
update
Regards
Bihan
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-High-GC-issue-tp4138570p4138694.html
Sent from the Solr - User mailing list archive at Nabble.com.