You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by 胡一博 <hu...@163.com> on 2017/11/10 06:22:10 UTC
ygc problem on solr 5.5.1
Hello everyone!
I run a solr cloud on version 5.5.1.
Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep 10+seconds for several hours.Even after I trigger a fullgc ,the ygc still cost 10+seconds.
This happened seldom.
jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server VM(build 24.60-b09, mixed mode))
-XX:+CMSParallelRemarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+CMSFullGCsBeforeCompaction=1
-XX:+CMSInitiatingOccupancyFraction=80
-XX:ConcGCThreads=4
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:ParallelGCThreads=4
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-Xms30g
-Xmx30g
-Xss256k
Re: Re: ygc problem on solr 5.5.1
Posted by Samuel Tatipamula <sa...@gmail.com>.
Promotion failure usually means your old gen doesn't have enough space to
accommodate the incoming (promoted from young to old) object. In your case,
you have specified NewRatio as 3, which means, you have approximately
30*(3/4) = 22.5 GB old gen heap space. If a heap this big gets fragmented,
it would definitely trigger a stw GC to defragment it.
If you change NewRatio to 2, then heap size for old gen will reduce to 20
GB, but still it is huge.
I would recommend decreasing the heap size to a reasonable value (say 12?),
and apply NewRatio of 2, which I found to be working good for my use case.
Also, you have given CMSInitiatingOccupancyFraction as 80, which might be a
bit too late to start triggering CMS. I suggest a value b/w 70-75 for
bigger heaps.
Thanks.
On Fri, Nov 10, 2017 at 3:39 PM, 胡一博 <hu...@163.com> wrote:
> Thank you for your suggest!
>
>
> about the NewRatio param. I found some 'promotion failed' in the gc log.
> It trigger a stw gc instead of cms gc. If i change the NewRatio to 2,the
> promotion fail maybe appeard more frequenly. Is the 'promotion failed' is
> caused by some inappropriate use on solr cloud? .
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2017-11-10 16:46:22, "Samuel Tatipamula" <sa...@gmail.com>
> wrote:
> >Hi,
> >
> >There are a couple of things based on your configuration I can suggest.
> >
> >-XX:ConcGCThreads=4 - try removing this restriction on the threads.
> >
> >-Xms30g - you should re-consider this param, as 30 GB is huge heap size.
> >Instead, in SolrCloud, try spawning multiple instances if you have system
> >resources. I work on SolrCloud where each server handles 10k read rpm, and
> >found a 12 GB heap to be more than enough. Also, I found 1:2 ratio
> >(NewRatio=2) to be more optimal than 1:3.
> >
> >Thanks,
> >Samuel
> >
> >On Fri, Nov 10, 2017 at 11:52 AM, 胡一博 <hu...@163.com> wrote:
> >
> >> Hello everyone!
> >>
> >>
> >> I run a solr cloud on version 5.5.1.
> >> Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep
> >> 10+seconds for several hours.Even after I trigger a fullgc ,the ygc
> still
> >> cost 10+seconds.
> >> This happened seldom.
> >> jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server
> >> VM(build 24.60-b09, mixed mode))
> >> -XX:+CMSParallelRemarkEnabled
> >> -XX:+CMSScavengeBeforeRemark
> >> -XX:+ParallelRefProcEnabled
> >> -XX:+UseCMSInitiatingOccupancyOnly
> >> -XX:+UseConcMarkSweepGC
> >> -XX:+UseParNewGC
> >> -XX:+CMSFullGCsBeforeCompaction=1
> >> -XX:+CMSInitiatingOccupancyFraction=80
> >> -XX:ConcGCThreads=4
> >> -XX:MaxTenuringThreshold=8
> >> -XX:NewRatio=3
> >> -XX:ParallelGCThreads=4
> >> -XX:SurvivorRatio=4
> >> -XX:TargetSurvivorRatio=90
> >> -Xms30g
> >> -Xmx30g
> >> -Xss256k
>
Re:Re: ygc problem on solr 5.5.1
Posted by 胡一博 <hu...@163.com>.
Thank you for your suggest!
about the NewRatio param. I found some 'promotion failed' in the gc log. It trigger a stw gc instead of cms gc. If i change the NewRatio to 2,the promotion fail maybe appeard more frequenly. Is the 'promotion failed' is caused by some inappropriate use on solr cloud? .
At 2017-11-10 16:46:22, "Samuel Tatipamula" <sa...@gmail.com> wrote:
>Hi,
>
>There are a couple of things based on your configuration I can suggest.
>
>-XX:ConcGCThreads=4 - try removing this restriction on the threads.
>
>-Xms30g - you should re-consider this param, as 30 GB is huge heap size.
>Instead, in SolrCloud, try spawning multiple instances if you have system
>resources. I work on SolrCloud where each server handles 10k read rpm, and
>found a 12 GB heap to be more than enough. Also, I found 1:2 ratio
>(NewRatio=2) to be more optimal than 1:3.
>
>Thanks,
>Samuel
>
>On Fri, Nov 10, 2017 at 11:52 AM, 胡一博 <hu...@163.com> wrote:
>
>> Hello everyone!
>>
>>
>> I run a solr cloud on version 5.5.1.
>> Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep
>> 10+seconds for several hours.Even after I trigger a fullgc ,the ygc still
>> cost 10+seconds.
>> This happened seldom.
>> jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server
>> VM(build 24.60-b09, mixed mode))
>> -XX:+CMSParallelRemarkEnabled
>> -XX:+CMSScavengeBeforeRemark
>> -XX:+ParallelRefProcEnabled
>> -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+UseConcMarkSweepGC
>> -XX:+UseParNewGC
>> -XX:+CMSFullGCsBeforeCompaction=1
>> -XX:+CMSInitiatingOccupancyFraction=80
>> -XX:ConcGCThreads=4
>> -XX:MaxTenuringThreshold=8
>> -XX:NewRatio=3
>> -XX:ParallelGCThreads=4
>> -XX:SurvivorRatio=4
>> -XX:TargetSurvivorRatio=90
>> -Xms30g
>> -Xmx30g
>> -Xss256k
Re: ygc problem on solr 5.5.1
Posted by Samuel Tatipamula <sa...@gmail.com>.
Hi,
There are a couple of things based on your configuration I can suggest.
-XX:ConcGCThreads=4 - try removing this restriction on the threads.
-Xms30g - you should re-consider this param, as 30 GB is huge heap size.
Instead, in SolrCloud, try spawning multiple instances if you have system
resources. I work on SolrCloud where each server handles 10k read rpm, and
found a 12 GB heap to be more than enough. Also, I found 1:2 ratio
(NewRatio=2) to be more optimal than 1:3.
Thanks,
Samuel
On Fri, Nov 10, 2017 at 11:52 AM, 胡一博 <hu...@163.com> wrote:
> Hello everyone!
>
>
> I run a solr cloud on version 5.5.1.
> Sometime ,the ygc time would increase from 0.1s to 10+seconds and keep
> 10+seconds for several hours.Even after I trigger a fullgc ,the ygc still
> cost 10+seconds.
> This happened seldom.
> jvm params:(java version "1.7.0_60", Java HotSpot (TM) 64-Bit Server
> VM(build 24.60-b09, mixed mode))
> -XX:+CMSParallelRemarkEnabled
> -XX:+CMSScavengeBeforeRemark
> -XX:+ParallelRefProcEnabled
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC
> -XX:+CMSFullGCsBeforeCompaction=1
> -XX:+CMSInitiatingOccupancyFraction=80
> -XX:ConcGCThreads=4
> -XX:MaxTenuringThreshold=8
> -XX:NewRatio=3
> -XX:ParallelGCThreads=4
> -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -Xms30g
> -Xmx30g
> -Xss256k