Re: Asking about the trend of increasing latency, hbase spikes.

[Ted Yu <yu...@gmail.com>]

When triggerring full GC, better do in rolling fashion so that region
servers don't incur pause at the same time.

FYI

On Wed, Oct 7, 2015 at 8:24 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:

> Can you trigger full GC across the cluster? If slow queries will go down
> after that - you get the answer.
> GC s-t-w frequency and duration may increase steadily until full reset
> (full GC) is executed, I think
>
> -Vlad
>
> On Wed, Oct 7, 2015 at 6:35 PM, UI-JIN LIM <uj...@lgcns.com> wrote:
>
> > I have post images to hosting service and share again!!
> >
> > ---
> > Hi. This is Ui Jin, Lim in Korea, LG CNS
> >
> > We had setup and are operating hbase 0.98.13 on our customer, iptv
> Company.
> > Our customer claim about this spike. Following is our status.
> >
> > . We had been operating with BlockCache(heap) for 3 months. The ratio of
> > above 100ms has been reached 6.69% of total transaction.
> > The following graph show the trend of executions, above 100ms per minute,
> > at June 17th. (This is the response time of 2~3 Hbase transactions)
> > <Click to see Graph>
> >  http://i57.tinypic.com/mwcvtj.jpg
> >
> > - After Hbase restarting at June 18th, The spikes had been reduced
> > dramatically. But its ratio had been increasing continuously.
> >    <Click to see Graph>
> >   http://i57.tinypic.com/71pfee.jpg
> >
> > - We had adopted off-heap bucket cache at August 19th. But the increasing
> > does not disappear.
> > <Click to see Graph>
> > http://i61.tinypic.com/2aezazt.jpg
> > http://i58.tinypic.com/svm32x.jpg
> >
> > - We set HBase to log slow query which have above 10ms execution time.
> All
> > RegionServer logs show increasing its counts.
> > <Click to see Graph>
> > http://i59.tinypic.com/68u1ci.jpg
> >
> > What we concern is not the spike itself but its incremental trend.
> > There is no increment of transactions and dataset.
> > I ask your thoughts about this issue.
> >
> >
> > -----Original Message-----
> > From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
> > Sent: Thursday, October 08, 2015 9:50 AM
> > To: dev@hbase.apache.org
> > Subject: Re: Asking about the trend of increasing latency, hbase spikes.
> >
> > That is definitely, interesting graphs, but we do not see them.
> > Can you host them somewhere and post here the links?
> >
> > -Vlad
> >
> > On Wed, Oct 7, 2015 at 5:38 PM, Biju N <bi...@gmail.com> wrote:
> >
> > > Looks like the attachments are not viewable.
> > >
> > > On Wed, Oct 7, 2015 at 8:35 PM, UI-JIN LIM <uj...@lgcns.com> wrote:
> > >
> > > > Hi. This is Ui Jin, Lim in Korea, LG CNS
> > > >
> > > >
> > > >
> > > > We had setup and are operating hbase 0.98.13 on our customer, iptv
> > > > Company.
> > > >
> > > > Our customer claim about this spike. Following is our status.
> > > >
> > > >
> > > >
> > > > . We had been operating with BlockCache(heap) for 3 months. The
> > > > ratio of above 100ms has been reached 6.69% of total transaction.
> > > >
> > > >   The following graph show the trend of executions, above 100ms per
> > > > minute, at June 17th. (This is the response time of 2~3 Hbase
> > > > transactions)
> > > >
> > > > [image: cid:image001.jpg@01D1011A.5E93FF70]
> > > >
> > > >
> > > >
> > > > - After Hbase restarting at June 18th, The spikes had been reduced
> > > > dramatically. But its ratio had been increasing continuously.
> > > >
> > > > [image: cid:image002.png@01D1011A.5E93FF70]
> > > >
> > > >
> > > >
> > > > - We had adopted off-heap bucket cache at August 19th. But the
> > > > increasing does not disappear.
> > > >
> > > > [image: cid:image002.png@01D10103.82E488C0]
> > > >
> > > > [image: cid:image002.png@01D10103.82E488C0]
> > > >
> > > >
> > > >
> > > > - We set HBase to log slow query which have above 10ms execution
> time.
> > > All
> > > > RegionServer logs show increasing its counts.
> > > >
> > > >
> > > >
> > > > [image: cid:image005.png@01D1011A.5E93FF70]
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > What we concern is not the spike itself but its incremental trend.
> > > >
> > > > There is no increment of transactions and dataset.
> > > >
> > > > I ask your thoughts about this issue.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -------------------------------------------------------------
> > > >
> > > > *UI JIN, LIM*
> > > >
> > > > *DNA 3.0** - **LG CNS*
> > > >
> > > > Data Architecture Advisory
> > > >
> > > > Big Data Technical Support Team
> > > >
> > > > Mobile : +82) 010-8307-1210
> > > >
> > > >
> > > >
> > >
> >
>