You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Varun Sharma <va...@pinterest.com> on 2013/02/21 08:32:15 UTC
GC frequency
Hi,
I have a system tuned with new Gen 512M with a lot write load. The system
has 4 cores - ParNewGC and GCThreads is set to 4. I am using ConcMarkGC and
CMSInitiating fraction is set to 60 %. I am observing the 90th/99th
percentile of latency and see it highly correlated with GC pauses. There
are times when I have a GC pause of ~ 200 ms every 4 seconds - the tail
latency shoots up to 200 milliseconds for reads - most reads are being
served out of cache. Looking at the GC log and tail latency pattern, there
is direct correlation b/w the two. When the write load is low, and the GC
pauses are like 100-150 ms every 6 seconds, the tail latency improves.
After seeing this behaviour, I am intent on reducing the NewSize to 256M
but I risk 100 ms pauses pretty much every 1-2 seconds and perhaps higher
chance of promotion failures (MSLAB etc. is on). Does anyone know if
frequent young gen collections can be a problem ?
Thanks
Varun
答复: 答复: 答复: GC frequency
Posted by 谢良 <xi...@xiaomi.com>.
comments in line
________________________________________
发件人: Varun Sharma [varun@pinterest.com]
发送时间: 2013年2月21日 17:21
收件人: user@hbase.apache.org
主题: Re: 答复: 答复: GC frequency
This is a very interesting article indeed, it seems to say that for heap
size > 4-8GB - time of new gen collection could be dominated by size of the
heap rather than size of new generation. This is interesting and I have not
found any such guideline in hbase book etc.
[Liang Xie]: it's a jvm-related problem in real, not a hbase-special issue, IMHO:) so seems it's just fine that you could not find guide from hbase book
Do you think the overall heap size needs to go down in this case ?
[Liang Xie]: I don't think so, i am not a vm expert:) maybe you can ask help from hotspot-gc-use@openjdk.java.net list.
Per my understanding, if allocation rate is really too high(the load probably too high), then add more RS is a good choice:)
else, you should analyse your gc log and tweak vm option
On Thu, Feb 21, 2013 at 12:50 AM, 谢良 <xi...@xiaomi.com> wrote:
> Here is a good formula to estimate:
> http://blog.ragozin.info/2011/06/understanding-gc-pauses-in-jvm-hotspots.html
>
> Hope it helpful:)
> ________________________________________
> 发件人: Varun Sharma [varun@pinterest.com]
> 发送时间: 2013年2月21日 16:22
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: GC frequency
>
> What do you mean by normal size heap ? Here is JVM settings
>
> -Xms11480m -Xmx11480m -XX:NewSize=512m -XX:MaxNewSize=512m
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
> -XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=4
>
> I was told that for a 4 core machine, it typically takes 200ms to clean out
> 512m - now, the only thing that I am afraid with reducing the size of new
> gen is higher frequency and the chances of more frequent promotion
> failures.
>
> On Thu, Feb 21, 2013 at 12:10 AM, 谢良 <xi...@xiaomi.com> wrote:
>
> > Of course, you'll hit the nightmarish "CMS fragement" easier if NewSize
> > too low:)
> > Generally speaking, most of YGC should be less than 5ms for a normal size
> > heap.
> > maybe your load is too high or there're vm options be misconfigured ?
> > ________________________________________
> > 发件人: Varun Sharma [varun@pinterest.com]
> > 发送时间: 2013年2月21日 15:32
> > 收件人: user@hbase.apache.org
> > 主题: GC frequency
> >
> > Hi,
> >
> > I have a system tuned with new Gen 512M with a lot write load. The system
> > has 4 cores - ParNewGC and GCThreads is set to 4. I am using ConcMarkGC
> and
> > CMSInitiating fraction is set to 60 %. I am observing the 90th/99th
> > percentile of latency and see it highly correlated with GC pauses. There
> > are times when I have a GC pause of ~ 200 ms every 4 seconds - the tail
> > latency shoots up to 200 milliseconds for reads - most reads are being
> > served out of cache. Looking at the GC log and tail latency pattern,
> there
> > is direct correlation b/w the two. When the write load is low, and the GC
> > pauses are like 100-150 ms every 6 seconds, the tail latency improves.
> >
> > After seeing this behaviour, I am intent on reducing the NewSize to 256M
> > but I risk 100 ms pauses pretty much every 1-2 seconds and perhaps higher
> > chance of promotion failures (MSLAB etc. is on). Does anyone know if
> > frequent young gen collections can be a problem ?
> >
> > Thanks
> > Varun
> >
>
Re: 答复: 答复: GC frequency
Posted by Varun Sharma <va...@pinterest.com>.
This is a very interesting article indeed, it seems to say that for heap
size > 4-8GB - time of new gen collection could be dominated by size of the
heap rather than size of new generation. This is interesting and I have not
found any such guideline in hbase book etc.
Do you think the overall heap size needs to go down in this case ?
On Thu, Feb 21, 2013 at 12:50 AM, 谢良 <xi...@xiaomi.com> wrote:
> Here is a good formula to estimate:
> http://blog.ragozin.info/2011/06/understanding-gc-pauses-in-jvm-hotspots.html
>
> Hope it helpful:)
> ________________________________________
> 发件人: Varun Sharma [varun@pinterest.com]
> 发送时间: 2013年2月21日 16:22
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: GC frequency
>
> What do you mean by normal size heap ? Here is JVM settings
>
> -Xms11480m -Xmx11480m -XX:NewSize=512m -XX:MaxNewSize=512m
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
> -XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=4
>
> I was told that for a 4 core machine, it typically takes 200ms to clean out
> 512m - now, the only thing that I am afraid with reducing the size of new
> gen is higher frequency and the chances of more frequent promotion
> failures.
>
> On Thu, Feb 21, 2013 at 12:10 AM, 谢良 <xi...@xiaomi.com> wrote:
>
> > Of course, you'll hit the nightmarish "CMS fragement" easier if NewSize
> > too low:)
> > Generally speaking, most of YGC should be less than 5ms for a normal size
> > heap.
> > maybe your load is too high or there're vm options be misconfigured ?
> > ________________________________________
> > 发件人: Varun Sharma [varun@pinterest.com]
> > 发送时间: 2013年2月21日 15:32
> > 收件人: user@hbase.apache.org
> > 主题: GC frequency
> >
> > Hi,
> >
> > I have a system tuned with new Gen 512M with a lot write load. The system
> > has 4 cores - ParNewGC and GCThreads is set to 4. I am using ConcMarkGC
> and
> > CMSInitiating fraction is set to 60 %. I am observing the 90th/99th
> > percentile of latency and see it highly correlated with GC pauses. There
> > are times when I have a GC pause of ~ 200 ms every 4 seconds - the tail
> > latency shoots up to 200 milliseconds for reads - most reads are being
> > served out of cache. Looking at the GC log and tail latency pattern,
> there
> > is direct correlation b/w the two. When the write load is low, and the GC
> > pauses are like 100-150 ms every 6 seconds, the tail latency improves.
> >
> > After seeing this behaviour, I am intent on reducing the NewSize to 256M
> > but I risk 100 ms pauses pretty much every 1-2 seconds and perhaps higher
> > chance of promotion failures (MSLAB etc. is on). Does anyone know if
> > frequent young gen collections can be a problem ?
> >
> > Thanks
> > Varun
> >
>
答复: 答复: GC frequency
Posted by 谢良 <xi...@xiaomi.com>.
Here is a good formula to estimate: http://blog.ragozin.info/2011/06/understanding-gc-pauses-in-jvm-hotspots.html
Hope it helpful:)
________________________________________
发件人: Varun Sharma [varun@pinterest.com]
发送时间: 2013年2月21日 16:22
收件人: user@hbase.apache.org
主题: Re: 答复: GC frequency
What do you mean by normal size heap ? Here is JVM settings
-Xms11480m -Xmx11480m -XX:NewSize=512m -XX:MaxNewSize=512m
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
-XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=4
I was told that for a 4 core machine, it typically takes 200ms to clean out
512m - now, the only thing that I am afraid with reducing the size of new
gen is higher frequency and the chances of more frequent promotion failures.
On Thu, Feb 21, 2013 at 12:10 AM, 谢良 <xi...@xiaomi.com> wrote:
> Of course, you'll hit the nightmarish "CMS fragement" easier if NewSize
> too low:)
> Generally speaking, most of YGC should be less than 5ms for a normal size
> heap.
> maybe your load is too high or there're vm options be misconfigured ?
> ________________________________________
> 发件人: Varun Sharma [varun@pinterest.com]
> 发送时间: 2013年2月21日 15:32
> 收件人: user@hbase.apache.org
> 主题: GC frequency
>
> Hi,
>
> I have a system tuned with new Gen 512M with a lot write load. The system
> has 4 cores - ParNewGC and GCThreads is set to 4. I am using ConcMarkGC and
> CMSInitiating fraction is set to 60 %. I am observing the 90th/99th
> percentile of latency and see it highly correlated with GC pauses. There
> are times when I have a GC pause of ~ 200 ms every 4 seconds - the tail
> latency shoots up to 200 milliseconds for reads - most reads are being
> served out of cache. Looking at the GC log and tail latency pattern, there
> is direct correlation b/w the two. When the write load is low, and the GC
> pauses are like 100-150 ms every 6 seconds, the tail latency improves.
>
> After seeing this behaviour, I am intent on reducing the NewSize to 256M
> but I risk 100 ms pauses pretty much every 1-2 seconds and perhaps higher
> chance of promotion failures (MSLAB etc. is on). Does anyone know if
> frequent young gen collections can be a problem ?
>
> Thanks
> Varun
>
Re: 答复: GC frequency
Posted by Varun Sharma <va...@pinterest.com>.
What do you mean by normal size heap ? Here is JVM settings
-Xms11480m -Xmx11480m -XX:NewSize=512m -XX:MaxNewSize=512m
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60
-XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=4
I was told that for a 4 core machine, it typically takes 200ms to clean out
512m - now, the only thing that I am afraid with reducing the size of new
gen is higher frequency and the chances of more frequent promotion failures.
On Thu, Feb 21, 2013 at 12:10 AM, 谢良 <xi...@xiaomi.com> wrote:
> Of course, you'll hit the nightmarish "CMS fragement" easier if NewSize
> too low:)
> Generally speaking, most of YGC should be less than 5ms for a normal size
> heap.
> maybe your load is too high or there're vm options be misconfigured ?
> ________________________________________
> 发件人: Varun Sharma [varun@pinterest.com]
> 发送时间: 2013年2月21日 15:32
> 收件人: user@hbase.apache.org
> 主题: GC frequency
>
> Hi,
>
> I have a system tuned with new Gen 512M with a lot write load. The system
> has 4 cores - ParNewGC and GCThreads is set to 4. I am using ConcMarkGC and
> CMSInitiating fraction is set to 60 %. I am observing the 90th/99th
> percentile of latency and see it highly correlated with GC pauses. There
> are times when I have a GC pause of ~ 200 ms every 4 seconds - the tail
> latency shoots up to 200 milliseconds for reads - most reads are being
> served out of cache. Looking at the GC log and tail latency pattern, there
> is direct correlation b/w the two. When the write load is low, and the GC
> pauses are like 100-150 ms every 6 seconds, the tail latency improves.
>
> After seeing this behaviour, I am intent on reducing the NewSize to 256M
> but I risk 100 ms pauses pretty much every 1-2 seconds and perhaps higher
> chance of promotion failures (MSLAB etc. is on). Does anyone know if
> frequent young gen collections can be a problem ?
>
> Thanks
> Varun
>
答复: GC frequency
Posted by 谢良 <xi...@xiaomi.com>.
Of course, you'll hit the nightmarish "CMS fragement" easier if NewSize too low:)
Generally speaking, most of YGC should be less than 5ms for a normal size heap.
maybe your load is too high or there're vm options be misconfigured ?
________________________________________
发件人: Varun Sharma [varun@pinterest.com]
发送时间: 2013年2月21日 15:32
收件人: user@hbase.apache.org
主题: GC frequency
Hi,
I have a system tuned with new Gen 512M with a lot write load. The system
has 4 cores - ParNewGC and GCThreads is set to 4. I am using ConcMarkGC and
CMSInitiating fraction is set to 60 %. I am observing the 90th/99th
percentile of latency and see it highly correlated with GC pauses. There
are times when I have a GC pause of ~ 200 ms every 4 seconds - the tail
latency shoots up to 200 milliseconds for reads - most reads are being
served out of cache. Looking at the GC log and tail latency pattern, there
is direct correlation b/w the two. When the write load is low, and the GC
pauses are like 100-150 ms every 6 seconds, the tail latency improves.
After seeing this behaviour, I am intent on reducing the NewSize to 256M
but I risk 100 ms pauses pretty much every 1-2 seconds and perhaps higher
chance of promotion failures (MSLAB etc. is on). Does anyone know if
frequent young gen collections can be a problem ?
Thanks
Varun