You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ganesh V <ga...@gmail.com> on 2016/03/12 00:02:20 UTC
Hourly performance degradation on HBase cluster
Hello-
I am evaluating HBase for storage of time-series data and see periodic
degradation of the cluster happening at the start of the hour, every hour,
for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in the
logs during this time like so:
2016-03-01 06:09:35,151 INFO [WALProcedureStoreSyncThread]
wal.WALProcedureStore: Remove log:
hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log
In my tests, I'm pushing 1million data points (each about 200bytes in size)
into the cluster each minute at the same time using 5000 threads. I am also
querying 1000 of these data points each minute to test latency, errors, and
performance. The cluster runs absolutely fine except for those ~10mins at
the start of every hour when both reads and writes are affected. Could
someone suggest which part of HBase needs tuning? Thanks!
-Ganesh
Re: Hourly performance degradation on HBase cluster
Posted by Vladimir Rodionov <vl...@gmail.com>.
The problem is - periodic, with 1h interval, starts at the beginning of
each hour, performance degradation.
I would suspect periodic memstore flusher (flushes memstores every hour, by
default), but it does not explain
why it happens at the beginning of each hour? May be coincidence?
Grep log for "MemstoreFlusherChore".
-Vlad
On Fri, Mar 11, 2016 at 3:28 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> Master and RegionServer have nothing to do, to each other.
> the periodic rolling you see is on the master, and it just remove a file or
> two if it needs to.
>
> you are probably hitting GC or compactions on the Region Server which slow
> you down in the insert/scan perf.
> you can use a monitoring tool to visualize GC, or just add the -X option to
> print the GC in the log.
> also you can grep for compactions and flushes on the RS logs, to see what
> is happening around that time.
>
> Matteo
>
>
> On Fri, Mar 11, 2016 at 3:25 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> wrote:
>
> > HBASE-13202
> >
> > and Matteo Bertozzi can answer your question?
> >
> > -Vlad
> >
> >
> >
> >
> >
> > there is
> >
> >
> > On Fri, Mar 11, 2016 at 3:02 PM, Ganesh V <ga...@gmail.com> wrote:
> >
> > > Hello-
> > >
> > > I am evaluating HBase for storage of time-series data and see periodic
> > > degradation of the cluster happening at the start of the hour, every
> > hour,
> > > for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in
> > the
> > > logs during this time like so:
> > >
> > > 2016-03-01 06:09:35,151 INFO [WALProcedureStoreSyncThread]
> > > wal.WALProcedureStore: Remove log:
> > >
> >
> hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log
> > >
> > > In my tests, I'm pushing 1million data points (each about 200bytes in
> > size)
> > > into the cluster each minute at the same time using 5000 threads. I am
> > also
> > > querying 1000 of these data points each minute to test latency, errors,
> > and
> > > performance. The cluster runs absolutely fine except for those ~10mins
> at
> > > the start of every hour when both reads and writes are affected. Could
> > > someone suggest which part of HBase needs tuning? Thanks!
> > >
> > >
> > > -Ganesh
> > >
> >
>
Re: Hourly performance degradation on HBase cluster
Posted by Matteo Bertozzi <th...@gmail.com>.
Master and RegionServer have nothing to do, to each other.
the periodic rolling you see is on the master, and it just remove a file or
two if it needs to.
you are probably hitting GC or compactions on the Region Server which slow
you down in the insert/scan perf.
you can use a monitoring tool to visualize GC, or just add the -X option to
print the GC in the log.
also you can grep for compactions and flushes on the RS logs, to see what
is happening around that time.
Matteo
On Fri, Mar 11, 2016 at 3:25 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> HBASE-13202
>
> and Matteo Bertozzi can answer your question?
>
> -Vlad
>
>
>
>
>
> there is
>
>
> On Fri, Mar 11, 2016 at 3:02 PM, Ganesh V <ga...@gmail.com> wrote:
>
> > Hello-
> >
> > I am evaluating HBase for storage of time-series data and see periodic
> > degradation of the cluster happening at the start of the hour, every
> hour,
> > for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in
> the
> > logs during this time like so:
> >
> > 2016-03-01 06:09:35,151 INFO [WALProcedureStoreSyncThread]
> > wal.WALProcedureStore: Remove log:
> >
> hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log
> >
> > In my tests, I'm pushing 1million data points (each about 200bytes in
> size)
> > into the cluster each minute at the same time using 5000 threads. I am
> also
> > querying 1000 of these data points each minute to test latency, errors,
> and
> > performance. The cluster runs absolutely fine except for those ~10mins at
> > the start of every hour when both reads and writes are affected. Could
> > someone suggest which part of HBase needs tuning? Thanks!
> >
> >
> > -Ganesh
> >
>
Re: Hourly performance degradation on HBase cluster
Posted by Vladimir Rodionov <vl...@gmail.com>.
HBASE-13202
and Matteo Bertozzi can answer your question?
-Vlad
there is
On Fri, Mar 11, 2016 at 3:02 PM, Ganesh V <ga...@gmail.com> wrote:
> Hello-
>
> I am evaluating HBase for storage of time-series data and see periodic
> degradation of the cluster happening at the start of the hour, every hour,
> for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in the
> logs during this time like so:
>
> 2016-03-01 06:09:35,151 INFO [WALProcedureStoreSyncThread]
> wal.WALProcedureStore: Remove log:
> hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log
>
> In my tests, I'm pushing 1million data points (each about 200bytes in size)
> into the cluster each minute at the same time using 5000 threads. I am also
> querying 1000 of these data points each minute to test latency, errors, and
> performance. The cluster runs absolutely fine except for those ~10mins at
> the start of every hour when both reads and writes are affected. Could
> someone suggest which part of HBase needs tuning? Thanks!
>
>
> -Ganesh
>