You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Gautam <ga...@gmail.com> on 2014/10/31 18:22:09 UTC
Increasing write throughput..
I'm trying to increase write throughput of our hbase cluster. we'r
currently doing around 7500 messages per sec per node. I think we have room
for improvement. Especially since the heap is under utilized and memstore
size doesn't seem to fluctuate much between regular and peak ingestion
loads.
We mainly have one large table that we write most of the data to. Other
tables are mainly opentsdb and some relatively small summary tables. This
table is read in batch once a day but otherwise is mostly serving writes
99% of the time. This large table has 1 CF and get's flushed at around
~128M fairly regularly like below..
{log}
2014-10-31 16:56:09,499 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~128.2 M/134459888, currentsize=879.5 K/900640
for region
msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
in 640ms, sequenceid=16861176169, compaction requested=true
{log}
Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
What i'v tried..
- turned of major compactions , and handling these manually.
- bumped up heap Xmx from 24G to 48 G
- hbase.hregion.memstore.flush.size = 512M
- lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
global heap has enough space to accommodate the default percentages.
- Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
128GB RAM.
There hasn't been any appreciable increase in write perf. Still hovering
around the 7500 per node write throughput number. The flushes still seem to
be hapenning at 128M (instead of the expected 512)
I'v attached a snapshot of the memstore size vs. flushQueueLen. the block
caches are utilizing the extra heap space but not the memstore. The flush
Queue lengths have increased which leads me to believe that it's flushing
way too often without any increase in throughput.
Please let me know where i should dig further. That's a long email, thanks
for reading through :-)
Cheers,
-Gautam.
Re: Increasing write throughput..
Posted by Qiang Tian <ti...@gmail.com>.
>From the code, "hbase.hregion.majorcompaction"=0 is not to turn off major
compaction...I did a search and find a good(but a little long) post on this
topic and performance tuning on write heavy workload:
http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html
Personally that the most important thing is to understand your workload
pattern. tune one knob at a time, monitor&measure your system(machine
resources, RS "metrics Dump" in web UI), and repeat.
ps I do not see you set hbase.regionserver.handler.count. by default it is
30.
On Sat, Nov 8, 2014 at 12:26 AM, Gautam <ga...@gmail.com> wrote:
> Thanks for the comment.
>
> ALTER 'msg', { MEMSTORE_FLUSHSIZE => '256MB' }
>
> Now I get ~256MB memstore flushes. Although this still hasn't
> increased the write throughput..
>
> This is the stable state request counts per RS :
> http://postimg.org/image/gbb5nf6d1
>
> Avg regions per node: ~100
>
> The table is constantly being major And minor compacted although i'v
> turned off major compactions (hbase.hregion.majorcompaction = 0)
>
> What should I look at next?
>
> Cheers,
> -Gautam.
>
>
>
>
> On Thu, Nov 6, 2014 at 1:45 AM, Qiang Tian <ti...@gmail.com> wrote:
> > just in case...did you set memstore flush size via HTableDescriptor?
> >
> >
> > On Thu, Nov 6, 2014 at 8:52 AM, Gautam <ga...@gmail.com> wrote:
> >
> >> Thanks Anoop, Ted for the replies. This helped me understand Hbase's
> >> write path a lot more.
> >>
> >> After going through the literature and your comments on "what triggers
> >> memstore flushes",
> >>
> >> Did the following :
> >>
> >> - Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
> >> - changed hbase.regionserver.maxlogs = 150 (default 32)
> >> - hbase.hregion.memstore.flush.size = 536870912 ( as before )
> >> - hbase.hstore.blockingStoreFiles = 120
> >> - merged tiny/empty regions and brought down regions to 30% for this
> >> table ( before: 1646 , after merge: ~600 )
> >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
> >> MAX_HEAP_SIZE = 48000M
> >>
> >>
> >> Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
> >> tables: [2]. The region count per RS is now around 100 (evenly
> >> distributed) and so are the requests per sec.
> >>
> >> Based on the memstore size math, the flush size should now be =
> >> 48000*0.4/100 = 192M ? I still consistently see the memstore flushes
> >> at ~128M.. it barely ever goes above that number. Also uploaded last
> >> 1000 lines of RS log after above settings + restart [3]
> >>
> >> Here's the verbatim hbase-site.xml [4]
> >>
> >>
> >> Cheers,
> >> -Gautam.
> >>
> >> ====
> >> [1] - postimg.org/image/t2cxb18sh
> >> [2] - postimg.org/image/v3zaz9571
> >> [3] - pastebin.com/HXK4s8zR
> >> [4] - pastebin.com/av9XxecY
> >>
> >>
> >>
> >> On Sun, Nov 2, 2014 at 5:46 PM, Anoop John <an...@gmail.com>
> wrote:
> >> > You have ~280 regions per RS.
> >> > And ur memstore size % is 40% and heap size 48GB
> >> > This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
> >> > considering the upper water mark alone)
> >> >
> >> > If u have to consider all 280 regions each with 512 MB heap you need
> much
> >> > more size of heap. And your writes are distributed to all regions
> >> right?
> >> >
> >> > So you will be seeing flushes because of global heap pressure.
> >> >
> >> > Increasing the xmx and flush size alone wont help. You need to
> consider
> >> > the regions# and writes
> >> >
> >> > When you tune this the next step will be to tune the HLog and its
> >> rolling.
> >> > That depends on your cell size as well.
> >> > By default when we reach 95% size of HDFS block size, we roll to a new
> >> HLog
> >> > file. And by default when we reach 32 Log files, we force flushes.
> FYI.
> >> >
> >> > -Anoop-
> >> >
> >> >
> >> > On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu <yu...@gmail.com> wrote:
> >> >
> >> >> Please read 9.7.7.2. MemStoreFlush under
> >> >> http://hbase.apache.org/book.html#regions.arch
> >> >>
> >> >> Cheers
> >> >>
> >> >> On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik <
> >> gautamkowshik@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > - Sorry bout the raw image upload, here’s the tsdb snapshot :
> >> >> > http://postimg.org/image/gq4nf96x9/
> >> >> > - Hbase version 98.1 (CDH 5.1 distro)
> >> >> > - hbase-site pastebin : http://pastebin.com/fEctQ3im
> >> >> > - this table ‘msg' has been pre-split with 240 regions and writes
> are
> >> >> > evenly distributed into 240 buckets. ( the bucket is a prefix to
> the
> >> row
> >> >> > key ) . These regions are well spread across the 8 RSs. Although
> over
> >> >> time
> >> >> > these 240 have split and now become 2440 .. each region server has
> >> ~280
> >> >> > regions.
> >> >> > - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb
> Al
> >> >> > - no hot regions from what i can tell.
> >> >> >
> >> >> > One of my main concerns was why even after setting the memstore
> flush
> >> >> size
> >> >> > to 512M is it still flushing at 128M. Is there a setting i’v
> missed ?
> >> I’l
> >> >> > try to get more details as i find them.
> >> >> >
> >> >> > Thanks and Cheers,
> >> >> > -Gautam.
> >> >> >
> >> >> > On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
> >> >> >
> >> >> > > What version of hbase (later versions have improvements in write
> >> >> > > throughput, especially when many writing threads). Post a
> pastebin
> >> of
> >> >> > > regionserver log in steadystate if you don't mind. About how
> many
> >> >> > writers
> >> >> > > going into server at a time? How many regions on server. All
> being
> >> >> > > written to at same rate or you have hotties?
> >> >> > > Thanks,
> >> >> > > St.Ack
> >> >> > >
> >> >> > > On Fri, Oct 31, 2014 at 10:22 AM, Gautam <
> gautamkowshik@gmail.com>
> >> >> > wrote:
> >> >> > >
> >> >> > >> I'm trying to increase write throughput of our hbase cluster.
> we'r
> >> >> > >> currently doing around 7500 messages per sec per node. I think
> we
> >> have
> >> >> > room
> >> >> > >> for improvement. Especially since the heap is under utilized and
> >> >> > memstore
> >> >> > >> size doesn't seem to fluctuate much between regular and peak
> >> ingestion
> >> >> > >> loads.
> >> >> > >>
> >> >> > >> We mainly have one large table that we write most of the data
> to.
> >> >> Other
> >> >> > >> tables are mainly opentsdb and some relatively small summary
> >> tables.
> >> >> > This
> >> >> > >> table is read in batch once a day but otherwise is mostly
> serving
> >> >> writes
> >> >> > >> 99% of the time. This large table has 1 CF and get's flushed at
> >> around
> >> >> > >> ~128M fairly regularly like below..
> >> >> > >>
> >> >> > >> {log}
> >> >> > >>
> >> >> > >> 2014-10-31 16:56:09,499 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HRegion:
> >> >> > >> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
> >> >> > K/900640
> >> >> > >> for region
> >> >> > >>
> >> >> >
> >> >>
> >>
> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
> >> >> > >> in 640ms, sequenceid=16861176169, compaction requested=true
> >> >> > >>
> >> >> > >> {log}
> >> >> > >>
> >> >> > >> Here's a pastebin of my hbase site :
> http://pastebin.com/fEctQ3im
> >> >> > >>
> >> >> > >> What i'v tried..
> >> >> > >> - turned of major compactions , and handling these manually.
> >> >> > >> - bumped up heap Xmx from 24G to 48 G
> >> >> > >> - hbase.hregion.memstore.flush.size = 512M
> >> >> > >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4)
> >> since
> >> >> the
> >> >> > >> global heap has enough space to accommodate the default
> >> percentages.
> >> >> > >> - Currently running Hbase 98.1 on an 8 node cluster that's
> scaled
> >> up
> >> >> to
> >> >> > >> 128GB RAM.
> >> >> > >>
> >> >> > >>
> >> >> > >> There hasn't been any appreciable increase in write perf. Still
> >> >> hovering
> >> >> > >> around the 7500 per node write throughput number. The flushes
> still
> >> >> > seem to
> >> >> > >> be hapenning at 128M (instead of the expected 512)
> >> >> > >>
> >> >> > >> I'v attached a snapshot of the memstore size vs. flushQueueLen.
> the
> >> >> > block
> >> >> > >> caches are utilizing the extra heap space but not the memstore.
> The
> >> >> > flush
> >> >> > >> Queue lengths have increased which leads me to believe that it's
> >> >> > flushing
> >> >> > >> way too often without any increase in throughput.
> >> >> > >>
> >> >> > >> Please let me know where i should dig further. That's a long
> email,
> >> >> > thanks
> >> >> > >> for reading through :-)
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >> Cheers,
> >> >> > >> -Gautam.
> >> >> > >>
> >> >> >
> >> >> >
> >> >>
> >>
> >>
> >>
> >> --
> >> "If you really want something in this life, you have to work for it.
> >> Now, quiet! They're about to announce the lottery numbers..."
> >>
>
>
>
> --
> "If you really want something in this life, you have to work for it.
> Now, quiet! They're about to announce the lottery numbers..."
>
Re: Increasing write throughput..
Posted by Gautam <ga...@gmail.com>.
Thanks for the comment.
ALTER 'msg', { MEMSTORE_FLUSHSIZE => '256MB' }
Now I get ~256MB memstore flushes. Although this still hasn't
increased the write throughput..
This is the stable state request counts per RS :
http://postimg.org/image/gbb5nf6d1
Avg regions per node: ~100
The table is constantly being major And minor compacted although i'v
turned off major compactions (hbase.hregion.majorcompaction = 0)
What should I look at next?
Cheers,
-Gautam.
On Thu, Nov 6, 2014 at 1:45 AM, Qiang Tian <ti...@gmail.com> wrote:
> just in case...did you set memstore flush size via HTableDescriptor?
>
>
> On Thu, Nov 6, 2014 at 8:52 AM, Gautam <ga...@gmail.com> wrote:
>
>> Thanks Anoop, Ted for the replies. This helped me understand Hbase's
>> write path a lot more.
>>
>> After going through the literature and your comments on "what triggers
>> memstore flushes",
>>
>> Did the following :
>>
>> - Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
>> - changed hbase.regionserver.maxlogs = 150 (default 32)
>> - hbase.hregion.memstore.flush.size = 536870912 ( as before )
>> - hbase.hstore.blockingStoreFiles = 120
>> - merged tiny/empty regions and brought down regions to 30% for this
>> table ( before: 1646 , after merge: ~600 )
>> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
>> MAX_HEAP_SIZE = 48000M
>>
>>
>> Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
>> tables: [2]. The region count per RS is now around 100 (evenly
>> distributed) and so are the requests per sec.
>>
>> Based on the memstore size math, the flush size should now be =
>> 48000*0.4/100 = 192M ? I still consistently see the memstore flushes
>> at ~128M.. it barely ever goes above that number. Also uploaded last
>> 1000 lines of RS log after above settings + restart [3]
>>
>> Here's the verbatim hbase-site.xml [4]
>>
>>
>> Cheers,
>> -Gautam.
>>
>> ====
>> [1] - postimg.org/image/t2cxb18sh
>> [2] - postimg.org/image/v3zaz9571
>> [3] - pastebin.com/HXK4s8zR
>> [4] - pastebin.com/av9XxecY
>>
>>
>>
>> On Sun, Nov 2, 2014 at 5:46 PM, Anoop John <an...@gmail.com> wrote:
>> > You have ~280 regions per RS.
>> > And ur memstore size % is 40% and heap size 48GB
>> > This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
>> > considering the upper water mark alone)
>> >
>> > If u have to consider all 280 regions each with 512 MB heap you need much
>> > more size of heap. And your writes are distributed to all regions
>> right?
>> >
>> > So you will be seeing flushes because of global heap pressure.
>> >
>> > Increasing the xmx and flush size alone wont help. You need to consider
>> > the regions# and writes
>> >
>> > When you tune this the next step will be to tune the HLog and its
>> rolling.
>> > That depends on your cell size as well.
>> > By default when we reach 95% size of HDFS block size, we roll to a new
>> HLog
>> > file. And by default when we reach 32 Log files, we force flushes. FYI.
>> >
>> > -Anoop-
>> >
>> >
>> > On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> >> Please read 9.7.7.2. MemStoreFlush under
>> >> http://hbase.apache.org/book.html#regions.arch
>> >>
>> >> Cheers
>> >>
>> >> On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik <
>> gautamkowshik@gmail.com>
>> >> wrote:
>> >>
>> >> > - Sorry bout the raw image upload, here’s the tsdb snapshot :
>> >> > http://postimg.org/image/gq4nf96x9/
>> >> > - Hbase version 98.1 (CDH 5.1 distro)
>> >> > - hbase-site pastebin : http://pastebin.com/fEctQ3im
>> >> > - this table ‘msg' has been pre-split with 240 regions and writes are
>> >> > evenly distributed into 240 buckets. ( the bucket is a prefix to the
>> row
>> >> > key ) . These regions are well spread across the 8 RSs. Although over
>> >> time
>> >> > these 240 have split and now become 2440 .. each region server has
>> ~280
>> >> > regions.
>> >> > - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
>> >> > - no hot regions from what i can tell.
>> >> >
>> >> > One of my main concerns was why even after setting the memstore flush
>> >> size
>> >> > to 512M is it still flushing at 128M. Is there a setting i’v missed ?
>> I’l
>> >> > try to get more details as i find them.
>> >> >
>> >> > Thanks and Cheers,
>> >> > -Gautam.
>> >> >
>> >> > On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
>> >> >
>> >> > > What version of hbase (later versions have improvements in write
>> >> > > throughput, especially when many writing threads). Post a pastebin
>> of
>> >> > > regionserver log in steadystate if you don't mind. About how many
>> >> > writers
>> >> > > going into server at a time? How many regions on server. All being
>> >> > > written to at same rate or you have hotties?
>> >> > > Thanks,
>> >> > > St.Ack
>> >> > >
>> >> > > On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com>
>> >> > wrote:
>> >> > >
>> >> > >> I'm trying to increase write throughput of our hbase cluster. we'r
>> >> > >> currently doing around 7500 messages per sec per node. I think we
>> have
>> >> > room
>> >> > >> for improvement. Especially since the heap is under utilized and
>> >> > memstore
>> >> > >> size doesn't seem to fluctuate much between regular and peak
>> ingestion
>> >> > >> loads.
>> >> > >>
>> >> > >> We mainly have one large table that we write most of the data to.
>> >> Other
>> >> > >> tables are mainly opentsdb and some relatively small summary
>> tables.
>> >> > This
>> >> > >> table is read in batch once a day but otherwise is mostly serving
>> >> writes
>> >> > >> 99% of the time. This large table has 1 CF and get's flushed at
>> around
>> >> > >> ~128M fairly regularly like below..
>> >> > >>
>> >> > >> {log}
>> >> > >>
>> >> > >> 2014-10-31 16:56:09,499 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegion:
>> >> > >> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
>> >> > K/900640
>> >> > >> for region
>> >> > >>
>> >> >
>> >>
>> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
>> >> > >> in 640ms, sequenceid=16861176169, compaction requested=true
>> >> > >>
>> >> > >> {log}
>> >> > >>
>> >> > >> Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
>> >> > >>
>> >> > >> What i'v tried..
>> >> > >> - turned of major compactions , and handling these manually.
>> >> > >> - bumped up heap Xmx from 24G to 48 G
>> >> > >> - hbase.hregion.memstore.flush.size = 512M
>> >> > >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4)
>> since
>> >> the
>> >> > >> global heap has enough space to accommodate the default
>> percentages.
>> >> > >> - Currently running Hbase 98.1 on an 8 node cluster that's scaled
>> up
>> >> to
>> >> > >> 128GB RAM.
>> >> > >>
>> >> > >>
>> >> > >> There hasn't been any appreciable increase in write perf. Still
>> >> hovering
>> >> > >> around the 7500 per node write throughput number. The flushes still
>> >> > seem to
>> >> > >> be hapenning at 128M (instead of the expected 512)
>> >> > >>
>> >> > >> I'v attached a snapshot of the memstore size vs. flushQueueLen. the
>> >> > block
>> >> > >> caches are utilizing the extra heap space but not the memstore. The
>> >> > flush
>> >> > >> Queue lengths have increased which leads me to believe that it's
>> >> > flushing
>> >> > >> way too often without any increase in throughput.
>> >> > >>
>> >> > >> Please let me know where i should dig further. That's a long email,
>> >> > thanks
>> >> > >> for reading through :-)
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> Cheers,
>> >> > >> -Gautam.
>> >> > >>
>> >> >
>> >> >
>> >>
>>
>>
>>
>> --
>> "If you really want something in this life, you have to work for it.
>> Now, quiet! They're about to announce the lottery numbers..."
>>
--
"If you really want something in this life, you have to work for it.
Now, quiet! They're about to announce the lottery numbers..."
Re: Increasing write throughput..
Posted by Qiang Tian <ti...@gmail.com>.
just in case...did you set memstore flush size via HTableDescriptor?
On Thu, Nov 6, 2014 at 8:52 AM, Gautam <ga...@gmail.com> wrote:
> Thanks Anoop, Ted for the replies. This helped me understand Hbase's
> write path a lot more.
>
> After going through the literature and your comments on "what triggers
> memstore flushes",
>
> Did the following :
>
> - Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
> - changed hbase.regionserver.maxlogs = 150 (default 32)
> - hbase.hregion.memstore.flush.size = 536870912 ( as before )
> - hbase.hstore.blockingStoreFiles = 120
> - merged tiny/empty regions and brought down regions to 30% for this
> table ( before: 1646 , after merge: ~600 )
> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
> MAX_HEAP_SIZE = 48000M
>
>
> Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
> tables: [2]. The region count per RS is now around 100 (evenly
> distributed) and so are the requests per sec.
>
> Based on the memstore size math, the flush size should now be =
> 48000*0.4/100 = 192M ? I still consistently see the memstore flushes
> at ~128M.. it barely ever goes above that number. Also uploaded last
> 1000 lines of RS log after above settings + restart [3]
>
> Here's the verbatim hbase-site.xml [4]
>
>
> Cheers,
> -Gautam.
>
> ====
> [1] - postimg.org/image/t2cxb18sh
> [2] - postimg.org/image/v3zaz9571
> [3] - pastebin.com/HXK4s8zR
> [4] - pastebin.com/av9XxecY
>
>
>
> On Sun, Nov 2, 2014 at 5:46 PM, Anoop John <an...@gmail.com> wrote:
> > You have ~280 regions per RS.
> > And ur memstore size % is 40% and heap size 48GB
> > This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
> > considering the upper water mark alone)
> >
> > If u have to consider all 280 regions each with 512 MB heap you need much
> > more size of heap. And your writes are distributed to all regions
> right?
> >
> > So you will be seeing flushes because of global heap pressure.
> >
> > Increasing the xmx and flush size alone wont help. You need to consider
> > the regions# and writes
> >
> > When you tune this the next step will be to tune the HLog and its
> rolling.
> > That depends on your cell size as well.
> > By default when we reach 95% size of HDFS block size, we roll to a new
> HLog
> > file. And by default when we reach 32 Log files, we force flushes. FYI.
> >
> > -Anoop-
> >
> >
> > On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Please read 9.7.7.2. MemStoreFlush under
> >> http://hbase.apache.org/book.html#regions.arch
> >>
> >> Cheers
> >>
> >> On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik <
> gautamkowshik@gmail.com>
> >> wrote:
> >>
> >> > - Sorry bout the raw image upload, here’s the tsdb snapshot :
> >> > http://postimg.org/image/gq4nf96x9/
> >> > - Hbase version 98.1 (CDH 5.1 distro)
> >> > - hbase-site pastebin : http://pastebin.com/fEctQ3im
> >> > - this table ‘msg' has been pre-split with 240 regions and writes are
> >> > evenly distributed into 240 buckets. ( the bucket is a prefix to the
> row
> >> > key ) . These regions are well spread across the 8 RSs. Although over
> >> time
> >> > these 240 have split and now become 2440 .. each region server has
> ~280
> >> > regions.
> >> > - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
> >> > - no hot regions from what i can tell.
> >> >
> >> > One of my main concerns was why even after setting the memstore flush
> >> size
> >> > to 512M is it still flushing at 128M. Is there a setting i’v missed ?
> I’l
> >> > try to get more details as i find them.
> >> >
> >> > Thanks and Cheers,
> >> > -Gautam.
> >> >
> >> > On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
> >> >
> >> > > What version of hbase (later versions have improvements in write
> >> > > throughput, especially when many writing threads). Post a pastebin
> of
> >> > > regionserver log in steadystate if you don't mind. About how many
> >> > writers
> >> > > going into server at a time? How many regions on server. All being
> >> > > written to at same rate or you have hotties?
> >> > > Thanks,
> >> > > St.Ack
> >> > >
> >> > > On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> I'm trying to increase write throughput of our hbase cluster. we'r
> >> > >> currently doing around 7500 messages per sec per node. I think we
> have
> >> > room
> >> > >> for improvement. Especially since the heap is under utilized and
> >> > memstore
> >> > >> size doesn't seem to fluctuate much between regular and peak
> ingestion
> >> > >> loads.
> >> > >>
> >> > >> We mainly have one large table that we write most of the data to.
> >> Other
> >> > >> tables are mainly opentsdb and some relatively small summary
> tables.
> >> > This
> >> > >> table is read in batch once a day but otherwise is mostly serving
> >> writes
> >> > >> 99% of the time. This large table has 1 CF and get's flushed at
> around
> >> > >> ~128M fairly regularly like below..
> >> > >>
> >> > >> {log}
> >> > >>
> >> > >> 2014-10-31 16:56:09,499 INFO
> >> > org.apache.hadoop.hbase.regionserver.HRegion:
> >> > >> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
> >> > K/900640
> >> > >> for region
> >> > >>
> >> >
> >>
> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
> >> > >> in 640ms, sequenceid=16861176169, compaction requested=true
> >> > >>
> >> > >> {log}
> >> > >>
> >> > >> Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
> >> > >>
> >> > >> What i'v tried..
> >> > >> - turned of major compactions , and handling these manually.
> >> > >> - bumped up heap Xmx from 24G to 48 G
> >> > >> - hbase.hregion.memstore.flush.size = 512M
> >> > >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4)
> since
> >> the
> >> > >> global heap has enough space to accommodate the default
> percentages.
> >> > >> - Currently running Hbase 98.1 on an 8 node cluster that's scaled
> up
> >> to
> >> > >> 128GB RAM.
> >> > >>
> >> > >>
> >> > >> There hasn't been any appreciable increase in write perf. Still
> >> hovering
> >> > >> around the 7500 per node write throughput number. The flushes still
> >> > seem to
> >> > >> be hapenning at 128M (instead of the expected 512)
> >> > >>
> >> > >> I'v attached a snapshot of the memstore size vs. flushQueueLen. the
> >> > block
> >> > >> caches are utilizing the extra heap space but not the memstore. The
> >> > flush
> >> > >> Queue lengths have increased which leads me to believe that it's
> >> > flushing
> >> > >> way too often without any increase in throughput.
> >> > >>
> >> > >> Please let me know where i should dig further. That's a long email,
> >> > thanks
> >> > >> for reading through :-)
> >> > >>
> >> > >>
> >> > >>
> >> > >> Cheers,
> >> > >> -Gautam.
> >> > >>
> >> >
> >> >
> >>
>
>
>
> --
> "If you really want something in this life, you have to work for it.
> Now, quiet! They're about to announce the lottery numbers..."
>
Re: Increasing write throughput..
Posted by Gautam <ga...@gmail.com>.
Thanks Anoop, Ted for the replies. This helped me understand Hbase's
write path a lot more.
After going through the literature and your comments on "what triggers
memstore flushes",
Did the following :
- Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
- changed hbase.regionserver.maxlogs = 150 (default 32)
- hbase.hregion.memstore.flush.size = 536870912 ( as before )
- hbase.hstore.blockingStoreFiles = 120
- merged tiny/empty regions and brought down regions to 30% for this
table ( before: 1646 , after merge: ~600 )
- lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
MAX_HEAP_SIZE = 48000M
Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
tables: [2]. The region count per RS is now around 100 (evenly
distributed) and so are the requests per sec.
Based on the memstore size math, the flush size should now be =
48000*0.4/100 = 192M ? I still consistently see the memstore flushes
at ~128M.. it barely ever goes above that number. Also uploaded last
1000 lines of RS log after above settings + restart [3]
Here's the verbatim hbase-site.xml [4]
Cheers,
-Gautam.
====
[1] - postimg.org/image/t2cxb18sh
[2] - postimg.org/image/v3zaz9571
[3] - pastebin.com/HXK4s8zR
[4] - pastebin.com/av9XxecY
On Sun, Nov 2, 2014 at 5:46 PM, Anoop John <an...@gmail.com> wrote:
> You have ~280 regions per RS.
> And ur memstore size % is 40% and heap size 48GB
> This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
> considering the upper water mark alone)
>
> If u have to consider all 280 regions each with 512 MB heap you need much
> more size of heap. And your writes are distributed to all regions right?
>
> So you will be seeing flushes because of global heap pressure.
>
> Increasing the xmx and flush size alone wont help. You need to consider
> the regions# and writes
>
> When you tune this the next step will be to tune the HLog and its rolling.
> That depends on your cell size as well.
> By default when we reach 95% size of HDFS block size, we roll to a new HLog
> file. And by default when we reach 32 Log files, we force flushes. FYI.
>
> -Anoop-
>
>
> On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Please read 9.7.7.2. MemStoreFlush under
>> http://hbase.apache.org/book.html#regions.arch
>>
>> Cheers
>>
>> On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik <ga...@gmail.com>
>> wrote:
>>
>> > - Sorry bout the raw image upload, here’s the tsdb snapshot :
>> > http://postimg.org/image/gq4nf96x9/
>> > - Hbase version 98.1 (CDH 5.1 distro)
>> > - hbase-site pastebin : http://pastebin.com/fEctQ3im
>> > - this table ‘msg' has been pre-split with 240 regions and writes are
>> > evenly distributed into 240 buckets. ( the bucket is a prefix to the row
>> > key ) . These regions are well spread across the 8 RSs. Although over
>> time
>> > these 240 have split and now become 2440 .. each region server has ~280
>> > regions.
>> > - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
>> > - no hot regions from what i can tell.
>> >
>> > One of my main concerns was why even after setting the memstore flush
>> size
>> > to 512M is it still flushing at 128M. Is there a setting i’v missed ? I’l
>> > try to get more details as i find them.
>> >
>> > Thanks and Cheers,
>> > -Gautam.
>> >
>> > On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
>> >
>> > > What version of hbase (later versions have improvements in write
>> > > throughput, especially when many writing threads). Post a pastebin of
>> > > regionserver log in steadystate if you don't mind. About how many
>> > writers
>> > > going into server at a time? How many regions on server. All being
>> > > written to at same rate or you have hotties?
>> > > Thanks,
>> > > St.Ack
>> > >
>> > > On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com>
>> > wrote:
>> > >
>> > >> I'm trying to increase write throughput of our hbase cluster. we'r
>> > >> currently doing around 7500 messages per sec per node. I think we have
>> > room
>> > >> for improvement. Especially since the heap is under utilized and
>> > memstore
>> > >> size doesn't seem to fluctuate much between regular and peak ingestion
>> > >> loads.
>> > >>
>> > >> We mainly have one large table that we write most of the data to.
>> Other
>> > >> tables are mainly opentsdb and some relatively small summary tables.
>> > This
>> > >> table is read in batch once a day but otherwise is mostly serving
>> writes
>> > >> 99% of the time. This large table has 1 CF and get's flushed at around
>> > >> ~128M fairly regularly like below..
>> > >>
>> > >> {log}
>> > >>
>> > >> 2014-10-31 16:56:09,499 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegion:
>> > >> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
>> > K/900640
>> > >> for region
>> > >>
>> >
>> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
>> > >> in 640ms, sequenceid=16861176169, compaction requested=true
>> > >>
>> > >> {log}
>> > >>
>> > >> Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
>> > >>
>> > >> What i'v tried..
>> > >> - turned of major compactions , and handling these manually.
>> > >> - bumped up heap Xmx from 24G to 48 G
>> > >> - hbase.hregion.memstore.flush.size = 512M
>> > >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since
>> the
>> > >> global heap has enough space to accommodate the default percentages.
>> > >> - Currently running Hbase 98.1 on an 8 node cluster that's scaled up
>> to
>> > >> 128GB RAM.
>> > >>
>> > >>
>> > >> There hasn't been any appreciable increase in write perf. Still
>> hovering
>> > >> around the 7500 per node write throughput number. The flushes still
>> > seem to
>> > >> be hapenning at 128M (instead of the expected 512)
>> > >>
>> > >> I'v attached a snapshot of the memstore size vs. flushQueueLen. the
>> > block
>> > >> caches are utilizing the extra heap space but not the memstore. The
>> > flush
>> > >> Queue lengths have increased which leads me to believe that it's
>> > flushing
>> > >> way too often without any increase in throughput.
>> > >>
>> > >> Please let me know where i should dig further. That's a long email,
>> > thanks
>> > >> for reading through :-)
>> > >>
>> > >>
>> > >>
>> > >> Cheers,
>> > >> -Gautam.
>> > >>
>> >
>> >
>>
--
"If you really want something in this life, you have to work for it.
Now, quiet! They're about to announce the lottery numbers..."
Re: Increasing write throughput..
Posted by Anoop John <an...@gmail.com>.
You have ~280 regions per RS.
And ur memstore size % is 40% and heap size 48GB
This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
considering the upper water mark alone)
If u have to consider all 280 regions each with 512 MB heap you need much
more size of heap. And your writes are distributed to all regions right?
So you will be seeing flushes because of global heap pressure.
Increasing the xmx and flush size alone wont help. You need to consider
the regions# and writes
When you tune this the next step will be to tune the HLog and its rolling.
That depends on your cell size as well.
By default when we reach 95% size of HDFS block size, we roll to a new HLog
file. And by default when we reach 32 Log files, we force flushes. FYI.
-Anoop-
On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu <yu...@gmail.com> wrote:
> Please read 9.7.7.2. MemStoreFlush under
> http://hbase.apache.org/book.html#regions.arch
>
> Cheers
>
> On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik <ga...@gmail.com>
> wrote:
>
> > - Sorry bout the raw image upload, here’s the tsdb snapshot :
> > http://postimg.org/image/gq4nf96x9/
> > - Hbase version 98.1 (CDH 5.1 distro)
> > - hbase-site pastebin : http://pastebin.com/fEctQ3im
> > - this table ‘msg' has been pre-split with 240 regions and writes are
> > evenly distributed into 240 buckets. ( the bucket is a prefix to the row
> > key ) . These regions are well spread across the 8 RSs. Although over
> time
> > these 240 have split and now become 2440 .. each region server has ~280
> > regions.
> > - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
> > - no hot regions from what i can tell.
> >
> > One of my main concerns was why even after setting the memstore flush
> size
> > to 512M is it still flushing at 128M. Is there a setting i’v missed ? I’l
> > try to get more details as i find them.
> >
> > Thanks and Cheers,
> > -Gautam.
> >
> > On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
> >
> > > What version of hbase (later versions have improvements in write
> > > throughput, especially when many writing threads). Post a pastebin of
> > > regionserver log in steadystate if you don't mind. About how many
> > writers
> > > going into server at a time? How many regions on server. All being
> > > written to at same rate or you have hotties?
> > > Thanks,
> > > St.Ack
> > >
> > > On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com>
> > wrote:
> > >
> > >> I'm trying to increase write throughput of our hbase cluster. we'r
> > >> currently doing around 7500 messages per sec per node. I think we have
> > room
> > >> for improvement. Especially since the heap is under utilized and
> > memstore
> > >> size doesn't seem to fluctuate much between regular and peak ingestion
> > >> loads.
> > >>
> > >> We mainly have one large table that we write most of the data to.
> Other
> > >> tables are mainly opentsdb and some relatively small summary tables.
> > This
> > >> table is read in batch once a day but otherwise is mostly serving
> writes
> > >> 99% of the time. This large table has 1 CF and get's flushed at around
> > >> ~128M fairly regularly like below..
> > >>
> > >> {log}
> > >>
> > >> 2014-10-31 16:56:09,499 INFO
> > org.apache.hadoop.hbase.regionserver.HRegion:
> > >> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
> > K/900640
> > >> for region
> > >>
> >
> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
> > >> in 640ms, sequenceid=16861176169, compaction requested=true
> > >>
> > >> {log}
> > >>
> > >> Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
> > >>
> > >> What i'v tried..
> > >> - turned of major compactions , and handling these manually.
> > >> - bumped up heap Xmx from 24G to 48 G
> > >> - hbase.hregion.memstore.flush.size = 512M
> > >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since
> the
> > >> global heap has enough space to accommodate the default percentages.
> > >> - Currently running Hbase 98.1 on an 8 node cluster that's scaled up
> to
> > >> 128GB RAM.
> > >>
> > >>
> > >> There hasn't been any appreciable increase in write perf. Still
> hovering
> > >> around the 7500 per node write throughput number. The flushes still
> > seem to
> > >> be hapenning at 128M (instead of the expected 512)
> > >>
> > >> I'v attached a snapshot of the memstore size vs. flushQueueLen. the
> > block
> > >> caches are utilizing the extra heap space but not the memstore. The
> > flush
> > >> Queue lengths have increased which leads me to believe that it's
> > flushing
> > >> way too often without any increase in throughput.
> > >>
> > >> Please let me know where i should dig further. That's a long email,
> > thanks
> > >> for reading through :-)
> > >>
> > >>
> > >>
> > >> Cheers,
> > >> -Gautam.
> > >>
> >
> >
>
Re: Increasing write throughput..
Posted by Ted Yu <yu...@gmail.com>.
Please read 9.7.7.2. MemStoreFlush under
http://hbase.apache.org/book.html#regions.arch
Cheers
On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik <ga...@gmail.com>
wrote:
> - Sorry bout the raw image upload, here’s the tsdb snapshot :
> http://postimg.org/image/gq4nf96x9/
> - Hbase version 98.1 (CDH 5.1 distro)
> - hbase-site pastebin : http://pastebin.com/fEctQ3im
> - this table ‘msg' has been pre-split with 240 regions and writes are
> evenly distributed into 240 buckets. ( the bucket is a prefix to the row
> key ) . These regions are well spread across the 8 RSs. Although over time
> these 240 have split and now become 2440 .. each region server has ~280
> regions.
> - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
> - no hot regions from what i can tell.
>
> One of my main concerns was why even after setting the memstore flush size
> to 512M is it still flushing at 128M. Is there a setting i’v missed ? I’l
> try to get more details as i find them.
>
> Thanks and Cheers,
> -Gautam.
>
> On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
>
> > What version of hbase (later versions have improvements in write
> > throughput, especially when many writing threads). Post a pastebin of
> > regionserver log in steadystate if you don't mind. About how many
> writers
> > going into server at a time? How many regions on server. All being
> > written to at same rate or you have hotties?
> > Thanks,
> > St.Ack
> >
> > On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com>
> wrote:
> >
> >> I'm trying to increase write throughput of our hbase cluster. we'r
> >> currently doing around 7500 messages per sec per node. I think we have
> room
> >> for improvement. Especially since the heap is under utilized and
> memstore
> >> size doesn't seem to fluctuate much between regular and peak ingestion
> >> loads.
> >>
> >> We mainly have one large table that we write most of the data to. Other
> >> tables are mainly opentsdb and some relatively small summary tables.
> This
> >> table is read in batch once a day but otherwise is mostly serving writes
> >> 99% of the time. This large table has 1 CF and get's flushed at around
> >> ~128M fairly regularly like below..
> >>
> >> {log}
> >>
> >> 2014-10-31 16:56:09,499 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> >> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
> K/900640
> >> for region
> >>
> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
> >> in 640ms, sequenceid=16861176169, compaction requested=true
> >>
> >> {log}
> >>
> >> Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
> >>
> >> What i'v tried..
> >> - turned of major compactions , and handling these manually.
> >> - bumped up heap Xmx from 24G to 48 G
> >> - hbase.hregion.memstore.flush.size = 512M
> >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
> >> global heap has enough space to accommodate the default percentages.
> >> - Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
> >> 128GB RAM.
> >>
> >>
> >> There hasn't been any appreciable increase in write perf. Still hovering
> >> around the 7500 per node write throughput number. The flushes still
> seem to
> >> be hapenning at 128M (instead of the expected 512)
> >>
> >> I'v attached a snapshot of the memstore size vs. flushQueueLen. the
> block
> >> caches are utilizing the extra heap space but not the memstore. The
> flush
> >> Queue lengths have increased which leads me to believe that it's
> flushing
> >> way too often without any increase in throughput.
> >>
> >> Please let me know where i should dig further. That's a long email,
> thanks
> >> for reading through :-)
> >>
> >>
> >>
> >> Cheers,
> >> -Gautam.
> >>
>
>
Re: Increasing write throughput..
Posted by Gautam Kowshik <ga...@gmail.com>.
- Sorry bout the raw image upload, here’s the tsdb snapshot : http://postimg.org/image/gq4nf96x9/
- Hbase version 98.1 (CDH 5.1 distro)
- hbase-site pastebin : http://pastebin.com/fEctQ3im
- this table ‘msg' has been pre-split with 240 regions and writes are evenly distributed into 240 buckets. ( the bucket is a prefix to the row key ) . These regions are well spread across the 8 RSs. Although over time these 240 have split and now become 2440 .. each region server has ~280 regions.
- last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
- no hot regions from what i can tell.
One of my main concerns was why even after setting the memstore flush size to 512M is it still flushing at 128M. Is there a setting i’v missed ? I’l try to get more details as i find them.
Thanks and Cheers,
-Gautam.
On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
> What version of hbase (later versions have improvements in write
> throughput, especially when many writing threads). Post a pastebin of
> regionserver log in steadystate if you don't mind. About how many writers
> going into server at a time? How many regions on server. All being
> written to at same rate or you have hotties?
> Thanks,
> St.Ack
>
> On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com> wrote:
>
>> I'm trying to increase write throughput of our hbase cluster. we'r
>> currently doing around 7500 messages per sec per node. I think we have room
>> for improvement. Especially since the heap is under utilized and memstore
>> size doesn't seem to fluctuate much between regular and peak ingestion
>> loads.
>>
>> We mainly have one large table that we write most of the data to. Other
>> tables are mainly opentsdb and some relatively small summary tables. This
>> table is read in batch once a day but otherwise is mostly serving writes
>> 99% of the time. This large table has 1 CF and get's flushed at around
>> ~128M fairly regularly like below..
>>
>> {log}
>>
>> 2014-10-31 16:56:09,499 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5 K/900640
>> for region
>> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
>> in 640ms, sequenceid=16861176169, compaction requested=true
>>
>> {log}
>>
>> Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
>>
>> What i'v tried..
>> - turned of major compactions , and handling these manually.
>> - bumped up heap Xmx from 24G to 48 G
>> - hbase.hregion.memstore.flush.size = 512M
>> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
>> global heap has enough space to accommodate the default percentages.
>> - Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
>> 128GB RAM.
>>
>>
>> There hasn't been any appreciable increase in write perf. Still hovering
>> around the 7500 per node write throughput number. The flushes still seem to
>> be hapenning at 128M (instead of the expected 512)
>>
>> I'v attached a snapshot of the memstore size vs. flushQueueLen. the block
>> caches are utilizing the extra heap space but not the memstore. The flush
>> Queue lengths have increased which leads me to believe that it's flushing
>> way too often without any increase in throughput.
>>
>> Please let me know where i should dig further. That's a long email, thanks
>> for reading through :-)
>>
>>
>>
>> Cheers,
>> -Gautam.
>>
Re: Increasing write throughput..
Posted by Ted Yu <yu...@gmail.com>.
Gautum:
bq. I'v attached a snapshot of the memstore size vs. flushQueueLen
That didn't go through.
Consider using third-party site.
Answers to Stack's questions would help us get more clue.
Cheers
On Fri, Oct 31, 2014 at 10:47 AM, Stack <st...@duboce.net> wrote:
> What version of hbase (later versions have improvements in write
> throughput, especially when many writing threads). Post a pastebin of
> regionserver log in steadystate if you don't mind. About how many writers
> going into server at a time? How many regions on server. All being
> written to at same rate or you have hotties?
> Thanks,
> St.Ack
>
> On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com> wrote:
>
> > I'm trying to increase write throughput of our hbase cluster. we'r
> > currently doing around 7500 messages per sec per node. I think we have
> room
> > for improvement. Especially since the heap is under utilized and memstore
> > size doesn't seem to fluctuate much between regular and peak ingestion
> > loads.
> >
> > We mainly have one large table that we write most of the data to. Other
> > tables are mainly opentsdb and some relatively small summary tables. This
> > table is read in batch once a day but otherwise is mostly serving writes
> > 99% of the time. This large table has 1 CF and get's flushed at around
> > ~128M fairly regularly like below..
> >
> > {log}
> >
> > 2014-10-31 16:56:09,499 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > Finished memstore flush of ~128.2 M/134459888, currentsize=879.5 K/900640
> > for region
> >
> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
> > in 640ms, sequenceid=16861176169, compaction requested=true
> >
> > {log}
> >
> > Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
> >
> > What i'v tried..
> > - turned of major compactions , and handling these manually.
> > - bumped up heap Xmx from 24G to 48 G
> > - hbase.hregion.memstore.flush.size = 512M
> > - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
> > global heap has enough space to accommodate the default percentages.
> > - Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
> > 128GB RAM.
> >
> >
> > There hasn't been any appreciable increase in write perf. Still hovering
> > around the 7500 per node write throughput number. The flushes still seem
> to
> > be hapenning at 128M (instead of the expected 512)
> >
> > I'v attached a snapshot of the memstore size vs. flushQueueLen. the block
> > caches are utilizing the extra heap space but not the memstore. The flush
> > Queue lengths have increased which leads me to believe that it's flushing
> > way too often without any increase in throughput.
> >
> > Please let me know where i should dig further. That's a long email,
> thanks
> > for reading through :-)
> >
> >
> >
> > Cheers,
> > -Gautam.
> >
>
Re: Increasing write throughput..
Posted by Stack <st...@duboce.net>.
What version of hbase (later versions have improvements in write
throughput, especially when many writing threads). Post a pastebin of
regionserver log in steadystate if you don't mind. About how many writers
going into server at a time? How many regions on server. All being
written to at same rate or you have hotties?
Thanks,
St.Ack
On Fri, Oct 31, 2014 at 10:22 AM, Gautam <ga...@gmail.com> wrote:
> I'm trying to increase write throughput of our hbase cluster. we'r
> currently doing around 7500 messages per sec per node. I think we have room
> for improvement. Especially since the heap is under utilized and memstore
> size doesn't seem to fluctuate much between regular and peak ingestion
> loads.
>
> We mainly have one large table that we write most of the data to. Other
> tables are mainly opentsdb and some relatively small summary tables. This
> table is read in batch once a day but otherwise is mostly serving writes
> 99% of the time. This large table has 1 CF and get's flushed at around
> ~128M fairly regularly like below..
>
> {log}
>
> 2014-10-31 16:56:09,499 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5 K/900640
> for region
> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
> in 640ms, sequenceid=16861176169, compaction requested=true
>
> {log}
>
> Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im
>
> What i'v tried..
> - turned of major compactions , and handling these manually.
> - bumped up heap Xmx from 24G to 48 G
> - hbase.hregion.memstore.flush.size = 512M
> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
> global heap has enough space to accommodate the default percentages.
> - Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
> 128GB RAM.
>
>
> There hasn't been any appreciable increase in write perf. Still hovering
> around the 7500 per node write throughput number. The flushes still seem to
> be hapenning at 128M (instead of the expected 512)
>
> I'v attached a snapshot of the memstore size vs. flushQueueLen. the block
> caches are utilizing the extra heap space but not the memstore. The flush
> Queue lengths have increased which leads me to believe that it's flushing
> way too often without any increase in throughput.
>
> Please let me know where i should dig further. That's a long email, thanks
> for reading through :-)
>
>
>
> Cheers,
> -Gautam.
>