You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Anthony Nguyen <an...@gmail.com> on 2015/10/06 20:36:51 UTC
hbase.bucketcache.percentage.in.combinedcache with ioengine= on HBase 0.98
Hi all,
I'm running HBase 0.98, with the complicated percentage.in.combinedcache
setting. What's the best way to set the percentage.in.combinedcache setting
when using a file on SSD storage as the location for ioengine? The SSD
drive is dedicated for the cache. I don't want to modify the LruBlockCache
settings at all, keeping it available for index + bloom and just add the L2
SSD cache for storage for the data blocks (not the raw L1+L2 method).
My confusion is in how I should best modify the percentage.in.combinedcache
setting. HBASE-11520 makes things much clearer, but unfortunately it isn't
in 0.98. To me, it doesn't make much sense to have L1 be set to
hbase.bucketcache.size (100G) * (1.0 -
hbase.bucketcache.percentage.in.combinedcache (default=0.9) = 1G when I
want the L1 to be the same as it was prior to bucketcache
(hfile.block.cache.size * heap).
Also, I don't want to have any direct addressing of memory since my offheap
is completely on SSD. Does this mean I shouldn't change
XX:MaxDirectMemorySize?
Could someone sanity check these settings for me? Thanks!
30 GB RAM
~120 GB SSD
HBASE_REGIONSERVER_OPTS -Xms17G -Xmx17G
XX:MaxDirectMemorySize <- Don't touch since we're not using direct byte
buffers, right?
<property>
<name>hbase.bucketcache.ioengine</name>
<value>file:/mnt/foo/bar</value>
</property>
<property>
<name>hbase.bucketcache.percentage.in.combinedcache</name>
<value>0.9</value>
</property>
<property>
<name>hfile.block.cache.size</name>
<value>0.6</value>
</property>
<property>
<name>hbase.bucketcache.size</name>
<value>102400</value>
</property>
Re: hbase.bucketcache.percentage.in.combinedcache with ioengine= on HBase 0.98
Posted by Nick Dimiduk <nd...@gmail.com>.
I compared "offheap" vs file:tmpfs in the "showdown" blog post I linked
previously. For that test, they were comparable.
Configuring MaxDirectMemory is only required for "offheap", not the other
configurations. In fact, if you use both tmpfs and MaxDirectMemory, you'll
probably run out of memory very quickly (or start swapping?) as JVM tries
to allocate the memory you already mounted to tmpfs.
On Tue, Oct 6, 2015 at 1:29 PM, donmai <do...@gmail.com> wrote:
> Thanks Nick! Unfortunately, I'm stuck with 0.98. Perhaps it's something I
> should look into backporting if all else fails...
>
> Would also bigger great if anyone had any insight in the differences
> between offheap and file:/ in terms of configuration. When RAM to tmpfs was
> used as the workaround for the RS issue, was MaxDirectMemory size still
> set?
>
> Thanks!
> On Oct 6, 2015 4:03 PM, "Nick Dimiduk" <nd...@apache.org> wrote:
>
> > Configuring offheap cache in 0.98 is a huge PITA. There are some example
> > configs [0] linked from my old blog post [1], maybe that's a starting
> > point? I also worked with our docs team to get this explained [2] in a
> > somewhat useful way, using an algorithm/formula as the basis of the
> > explanation. That algorithm was eventually implemented into Ambari's
> Stack
> > Advisor script [3], so if you're really feeling brave you can try to
> > extract it from there.
> >
> > But really, you should upgrade to 1.x and enjoy Stack's nice work on
> making
> > it simpler to configure.
> >
> > Good luck,
> > -n
> >
> > [0]: https://github.com/ndimiduk/perf_blockcache
> > [1]: http://www.n10k.com/blog/blockcache-showdown/
> > [2]:
> >
> >
> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_Sys_Admin_Guides/content/ref-db219cd6-c586-49c1-bc56-c9c1c5475276.1.html
> > [3]:
> >
> >
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py#L552-L579
> >
> > On Tue, Oct 6, 2015 at 11:36 AM, Anthony Nguyen <
> > anthony.an.nguyen@gmail.com
> > > wrote:
> >
> > > Hi all,
> > >
> > > I'm running HBase 0.98, with the complicated
> percentage.in.combinedcache
> > > setting. What's the best way to set the percentage.in.combinedcache
> > setting
> > > when using a file on SSD storage as the location for ioengine? The SSD
> > > drive is dedicated for the cache. I don't want to modify the
> > LruBlockCache
> > > settings at all, keeping it available for index + bloom and just add
> the
> > L2
> > > SSD cache for storage for the data blocks (not the raw L1+L2 method).
> > >
> > > My confusion is in how I should best modify the
> > percentage.in.combinedcache
> > > setting. HBASE-11520 makes things much clearer, but unfortunately it
> > isn't
> > > in 0.98. To me, it doesn't make much sense to have L1 be set to
> > > hbase.bucketcache.size (100G) * (1.0 -
> > > hbase.bucketcache.percentage.in.combinedcache (default=0.9) = 1G when I
> > > want the L1 to be the same as it was prior to bucketcache
> > > (hfile.block.cache.size * heap).
> > >
> > > Also, I don't want to have any direct addressing of memory since my
> > offheap
> > > is completely on SSD. Does this mean I shouldn't change
> > > XX:MaxDirectMemorySize?
> > >
> > > Could someone sanity check these settings for me? Thanks!
> > >
> > > 30 GB RAM
> > > ~120 GB SSD
> > > HBASE_REGIONSERVER_OPTS -Xms17G -Xmx17G
> > > XX:MaxDirectMemorySize <- Don't touch since we're not using direct byte
> > > buffers, right?
> > >
> > > <property>
> > > <name>hbase.bucketcache.ioengine</name>
> > > <value>file:/mnt/foo/bar</value>
> > > </property>
> > > <property>
> > > <name>hbase.bucketcache.percentage.in.combinedcache</name>
> > > <value>0.9</value>
> > > </property>
> > >
> > > <property>
> > > <name>hfile.block.cache.size</name>
> > > <value>0.6</value>
> > > </property>
> > >
> > > <property>
> > > <name>hbase.bucketcache.size</name>
> > > <value>102400</value>
> > > </property>
> > >
> >
>
Re: hbase.bucketcache.percentage.in.combinedcache with ioengine= on HBase 0.98
Posted by donmai <do...@gmail.com>.
Thanks Nick! Unfortunately, I'm stuck with 0.98. Perhaps it's something I
should look into backporting if all else fails...
Would also bigger great if anyone had any insight in the differences
between offheap and file:/ in terms of configuration. When RAM to tmpfs was
used as the workaround for the RS issue, was MaxDirectMemory size still set?
Thanks!
On Oct 6, 2015 4:03 PM, "Nick Dimiduk" <nd...@apache.org> wrote:
> Configuring offheap cache in 0.98 is a huge PITA. There are some example
> configs [0] linked from my old blog post [1], maybe that's a starting
> point? I also worked with our docs team to get this explained [2] in a
> somewhat useful way, using an algorithm/formula as the basis of the
> explanation. That algorithm was eventually implemented into Ambari's Stack
> Advisor script [3], so if you're really feeling brave you can try to
> extract it from there.
>
> But really, you should upgrade to 1.x and enjoy Stack's nice work on making
> it simpler to configure.
>
> Good luck,
> -n
>
> [0]: https://github.com/ndimiduk/perf_blockcache
> [1]: http://www.n10k.com/blog/blockcache-showdown/
> [2]:
>
> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_Sys_Admin_Guides/content/ref-db219cd6-c586-49c1-bc56-c9c1c5475276.1.html
> [3]:
>
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py#L552-L579
>
> On Tue, Oct 6, 2015 at 11:36 AM, Anthony Nguyen <
> anthony.an.nguyen@gmail.com
> > wrote:
>
> > Hi all,
> >
> > I'm running HBase 0.98, with the complicated percentage.in.combinedcache
> > setting. What's the best way to set the percentage.in.combinedcache
> setting
> > when using a file on SSD storage as the location for ioengine? The SSD
> > drive is dedicated for the cache. I don't want to modify the
> LruBlockCache
> > settings at all, keeping it available for index + bloom and just add the
> L2
> > SSD cache for storage for the data blocks (not the raw L1+L2 method).
> >
> > My confusion is in how I should best modify the
> percentage.in.combinedcache
> > setting. HBASE-11520 makes things much clearer, but unfortunately it
> isn't
> > in 0.98. To me, it doesn't make much sense to have L1 be set to
> > hbase.bucketcache.size (100G) * (1.0 -
> > hbase.bucketcache.percentage.in.combinedcache (default=0.9) = 1G when I
> > want the L1 to be the same as it was prior to bucketcache
> > (hfile.block.cache.size * heap).
> >
> > Also, I don't want to have any direct addressing of memory since my
> offheap
> > is completely on SSD. Does this mean I shouldn't change
> > XX:MaxDirectMemorySize?
> >
> > Could someone sanity check these settings for me? Thanks!
> >
> > 30 GB RAM
> > ~120 GB SSD
> > HBASE_REGIONSERVER_OPTS -Xms17G -Xmx17G
> > XX:MaxDirectMemorySize <- Don't touch since we're not using direct byte
> > buffers, right?
> >
> > <property>
> > <name>hbase.bucketcache.ioengine</name>
> > <value>file:/mnt/foo/bar</value>
> > </property>
> > <property>
> > <name>hbase.bucketcache.percentage.in.combinedcache</name>
> > <value>0.9</value>
> > </property>
> >
> > <property>
> > <name>hfile.block.cache.size</name>
> > <value>0.6</value>
> > </property>
> >
> > <property>
> > <name>hbase.bucketcache.size</name>
> > <value>102400</value>
> > </property>
> >
>
Re: hbase.bucketcache.percentage.in.combinedcache with ioengine= on HBase 0.98
Posted by Nick Dimiduk <nd...@apache.org>.
Configuring offheap cache in 0.98 is a huge PITA. There are some example
configs [0] linked from my old blog post [1], maybe that's a starting
point? I also worked with our docs team to get this explained [2] in a
somewhat useful way, using an algorithm/formula as the basis of the
explanation. That algorithm was eventually implemented into Ambari's Stack
Advisor script [3], so if you're really feeling brave you can try to
extract it from there.
But really, you should upgrade to 1.x and enjoy Stack's nice work on making
it simpler to configure.
Good luck,
-n
[0]: https://github.com/ndimiduk/perf_blockcache
[1]: http://www.n10k.com/blog/blockcache-showdown/
[2]:
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_Sys_Admin_Guides/content/ref-db219cd6-c586-49c1-bc56-c9c1c5475276.1.html
[3]:
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py#L552-L579
On Tue, Oct 6, 2015 at 11:36 AM, Anthony Nguyen <anthony.an.nguyen@gmail.com
> wrote:
> Hi all,
>
> I'm running HBase 0.98, with the complicated percentage.in.combinedcache
> setting. What's the best way to set the percentage.in.combinedcache setting
> when using a file on SSD storage as the location for ioengine? The SSD
> drive is dedicated for the cache. I don't want to modify the LruBlockCache
> settings at all, keeping it available for index + bloom and just add the L2
> SSD cache for storage for the data blocks (not the raw L1+L2 method).
>
> My confusion is in how I should best modify the percentage.in.combinedcache
> setting. HBASE-11520 makes things much clearer, but unfortunately it isn't
> in 0.98. To me, it doesn't make much sense to have L1 be set to
> hbase.bucketcache.size (100G) * (1.0 -
> hbase.bucketcache.percentage.in.combinedcache (default=0.9) = 1G when I
> want the L1 to be the same as it was prior to bucketcache
> (hfile.block.cache.size * heap).
>
> Also, I don't want to have any direct addressing of memory since my offheap
> is completely on SSD. Does this mean I shouldn't change
> XX:MaxDirectMemorySize?
>
> Could someone sanity check these settings for me? Thanks!
>
> 30 GB RAM
> ~120 GB SSD
> HBASE_REGIONSERVER_OPTS -Xms17G -Xmx17G
> XX:MaxDirectMemorySize <- Don't touch since we're not using direct byte
> buffers, right?
>
> <property>
> <name>hbase.bucketcache.ioengine</name>
> <value>file:/mnt/foo/bar</value>
> </property>
> <property>
> <name>hbase.bucketcache.percentage.in.combinedcache</name>
> <value>0.9</value>
> </property>
>
> <property>
> <name>hfile.block.cache.size</name>
> <value>0.6</value>
> </property>
>
> <property>
> <name>hbase.bucketcache.size</name>
> <value>102400</value>
> </property>
>