You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Bi,
hongyu—mike <bo...@gmail.com> on 2014/09/04 08:29:29 UTC
HBase MOB performance
Hi all,
we store serialised hyperloglog object into hbase by use of coprocessor,
and the size distribution is below:
Row size (bytes):
min = 4279.00
max = 770757.00
mean = 67340.24
stddev = 153968.88
median = 14453.00
75% <= 63178.00
95% <= 336917.20
98% <= 761028.00
99% <= 767500.36
99.9% <= 770757.00
count = 827
the this value will be update 400 times every minute but the regionserver
where this table locate responsed slowly for other table's get request.
In my option, write mob/blob should not bother the performance of reading
other table's region in the same regionserver, only put pressure to hlog
fsync,flush and compaction; right?
thanks
ps:
hdfs blocksize=128MB
memstore size=128MB
max hlog=64
low/high water: 0.35/0.4
memstore percent:0.4
heap size=32GB
Re: HBase MOB performance
Posted by Andrey Stepachev <oc...@gmail.com>.
Yes, I'd start with jstack and moving to something like yjp/jmc later.
On Thu, Sep 4, 2014 at 5:44 AM, Bi,hongyu—mike <bo...@gmail.com> wrote:
> i keep the handler count as default(10 in 0.94.15)
>
> thanks for your reminding, i'll add regionserver's rpc monitor
> but how about the handler monitor? in other words, how to measure the busy
> degree of handler? jstack or else?
>
> thanks
>
>
> 2014-09-04 20:27 GMT+08:00 Andrey Stepachev <oc...@gmail.com>:
>
> > Hi Mike.
> >
> > Need to know how many handler you have and how many clients.
> > Can it happen, that you have all you handlers busy with writes?
> >
> >
> > On Wed, Sep 3, 2014 at 11:30 PM, Bi,hongyu—mike <bo...@gmail.com>
> wrote:
> >
> > > btw, i disable the block cache for the hyperloglog table to avoid the
> > cache
> > > pollution
> > >
> > >
> > > 2014-09-04 14:29 GMT+08:00 Bi,hongyu—mike <bo...@gmail.com>:
> > >
> > > > Hi all,
> > > >
> > > > we store serialised hyperloglog object into hbase by use of
> > coprocessor,
> > > > and the size distribution is below:
> > > > Row size (bytes):
> > > > min = 4279.00
> > > > max = 770757.00
> > > > mean = 67340.24
> > > > stddev = 153968.88
> > > > median = 14453.00
> > > > 75% <= 63178.00
> > > > 95% <= 336917.20
> > > > 98% <= 761028.00
> > > > 99% <= 767500.36
> > > > 99.9% <= 770757.00
> > > > count = 827
> > > >
> > > > the this value will be update 400 times every minute but the
> > regionserver
> > > > where this table locate responsed slowly for other table's get
> request.
> > > >
> > > > In my option, write mob/blob should not bother the performance of
> > reading
> > > > other table's region in the same regionserver, only put pressure to
> > hlog
> > > > fsync,flush and compaction; right?
> > > >
> > > > thanks
> > > >
> > > > ps:
> > > > hdfs blocksize=128MB
> > > > memstore size=128MB
> > > > max hlog=64
> > > > low/high water: 0.35/0.4
> > > > memstore percent:0.4
> > > >
> > > > heap size=32GB
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Andrey.
> >
>
--
Andrey.
Re: HBase MOB performance
Posted by Bi,
hongyu—mike <bo...@gmail.com>.
i keep the handler count as default(10 in 0.94.15)
thanks for your reminding, i'll add regionserver's rpc monitor
but how about the handler monitor? in other words, how to measure the busy
degree of handler? jstack or else?
thanks
2014-09-04 20:27 GMT+08:00 Andrey Stepachev <oc...@gmail.com>:
> Hi Mike.
>
> Need to know how many handler you have and how many clients.
> Can it happen, that you have all you handlers busy with writes?
>
>
> On Wed, Sep 3, 2014 at 11:30 PM, Bi,hongyu—mike <bo...@gmail.com> wrote:
>
> > btw, i disable the block cache for the hyperloglog table to avoid the
> cache
> > pollution
> >
> >
> > 2014-09-04 14:29 GMT+08:00 Bi,hongyu—mike <bo...@gmail.com>:
> >
> > > Hi all,
> > >
> > > we store serialised hyperloglog object into hbase by use of
> coprocessor,
> > > and the size distribution is below:
> > > Row size (bytes):
> > > min = 4279.00
> > > max = 770757.00
> > > mean = 67340.24
> > > stddev = 153968.88
> > > median = 14453.00
> > > 75% <= 63178.00
> > > 95% <= 336917.20
> > > 98% <= 761028.00
> > > 99% <= 767500.36
> > > 99.9% <= 770757.00
> > > count = 827
> > >
> > > the this value will be update 400 times every minute but the
> regionserver
> > > where this table locate responsed slowly for other table's get request.
> > >
> > > In my option, write mob/blob should not bother the performance of
> reading
> > > other table's region in the same regionserver, only put pressure to
> hlog
> > > fsync,flush and compaction; right?
> > >
> > > thanks
> > >
> > > ps:
> > > hdfs blocksize=128MB
> > > memstore size=128MB
> > > max hlog=64
> > > low/high water: 0.35/0.4
> > > memstore percent:0.4
> > >
> > > heap size=32GB
> > >
> > >
> > >
> >
>
>
>
> --
> Andrey.
>
Re: HBase MOB performance
Posted by Andrey Stepachev <oc...@gmail.com>.
Hi Mike.
Need to know how many handler you have and how many clients.
Can it happen, that you have all you handlers busy with writes?
On Wed, Sep 3, 2014 at 11:30 PM, Bi,hongyu—mike <bo...@gmail.com> wrote:
> btw, i disable the block cache for the hyperloglog table to avoid the cache
> pollution
>
>
> 2014-09-04 14:29 GMT+08:00 Bi,hongyu—mike <bo...@gmail.com>:
>
> > Hi all,
> >
> > we store serialised hyperloglog object into hbase by use of coprocessor,
> > and the size distribution is below:
> > Row size (bytes):
> > min = 4279.00
> > max = 770757.00
> > mean = 67340.24
> > stddev = 153968.88
> > median = 14453.00
> > 75% <= 63178.00
> > 95% <= 336917.20
> > 98% <= 761028.00
> > 99% <= 767500.36
> > 99.9% <= 770757.00
> > count = 827
> >
> > the this value will be update 400 times every minute but the regionserver
> > where this table locate responsed slowly for other table's get request.
> >
> > In my option, write mob/blob should not bother the performance of reading
> > other table's region in the same regionserver, only put pressure to hlog
> > fsync,flush and compaction; right?
> >
> > thanks
> >
> > ps:
> > hdfs blocksize=128MB
> > memstore size=128MB
> > max hlog=64
> > low/high water: 0.35/0.4
> > memstore percent:0.4
> >
> > heap size=32GB
> >
> >
> >
>
--
Andrey.
Re: HBase MOB performance
Posted by Bi,
hongyu—mike <bo...@gmail.com>.
btw, i disable the block cache for the hyperloglog table to avoid the cache
pollution
2014-09-04 14:29 GMT+08:00 Bi,hongyu—mike <bo...@gmail.com>:
> Hi all,
>
> we store serialised hyperloglog object into hbase by use of coprocessor,
> and the size distribution is below:
> Row size (bytes):
> min = 4279.00
> max = 770757.00
> mean = 67340.24
> stddev = 153968.88
> median = 14453.00
> 75% <= 63178.00
> 95% <= 336917.20
> 98% <= 761028.00
> 99% <= 767500.36
> 99.9% <= 770757.00
> count = 827
>
> the this value will be update 400 times every minute but the regionserver
> where this table locate responsed slowly for other table's get request.
>
> In my option, write mob/blob should not bother the performance of reading
> other table's region in the same regionserver, only put pressure to hlog
> fsync,flush and compaction; right?
>
> thanks
>
> ps:
> hdfs blocksize=128MB
> memstore size=128MB
> max hlog=64
> low/high water: 0.35/0.4
> memstore percent:0.4
>
> heap size=32GB
>
>
>