You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Amit Shah <am...@gmail.com> on 2016/03/30 16:59:47 UTC
Leveraging As Much Memory As Possible
Hi,
I am trying to configure my hbase (version 1.0) phoenix (version - 4.6)
cluster to utilize as much memory as possible on the server hardware. We
have an OLAP workload that allows users to perform interactive analysis
over huge sets of data. While reading about hbase configuration I came
across two configs
1. Hbase bucket cache
<http://blog.asquareb.com/blog/2014/11/24/how-to-leverage-large-physical-memory-to-improve-hbase-read-performance>
(off heap) which looks like a good option to bypass garbage collection.
2. Hadoop pinned hdfs blocks
<http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/>
(max locked memory) concept that loads the hdfs blocks in memory, but given
that hbase is configured with short circuit reads I assume this config may
not be of much help. Instead it would be right to increase hbase region
server heap memory. Is my understanding right?
We use HBase with Phoenix.
Kindly let me know your thoughts or suggestions on any more options that I
should explore
Thanks,
Amit.
Re: Leveraging As Much Memory As Possible
Posted by "Ding,Dongchao" <di...@baidu.com>.
For the first question
http://${rs-host}:${rs-info-port}/rs-status#bc_l2, There is detail info
about L2 Cache（bucket cache）here.
在 16/3/31 14:07， "Amit Shah" <am...@gmail.com> 写入:
>Thanks Ted and Anoop for the replies. A couple of follow up questions
>
>1. I could see the bucket cache and on heap memory stats reflected on
>hbase
>region server UI <http://i.imgur.com/4uwaWuC.png>, what are ways to
>monitor
>and see if its getting utilized?
>2. We are using phoenix as a sql layer above hbase. Are there any phoenix
>configurations that I could benefit in leveraging memory?
>
>Thanks,
>Amit.
>
>On Thu, Mar 31, 2016 at 10:44 AM, Anoop John <an...@gmail.com>
>wrote:
>
>> Ya having HBase side cache will be a better choice rather than HDFS
>> cache IMO. Yes u r correct... You might not want to give a very
>> large size for the heap. You can make use of the off heap BucketCache.
>>
>> -Anoop-
>>
>> On Thu, Mar 31, 2016 at 4:35 AM, Ted Yu <yu...@gmail.com> wrote:
>> > For #1, please see the top two blogs @ https://blogs.apache.org/hbase/
>> >
>> > FYI
>> >
>> > On Wed, Mar 30, 2016 at 7:59 AM, Amit Shah <am...@gmail.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> I am trying to configure my hbase (version 1.0) phoenix (version -
>>4.6)
>> >> cluster to utilize as much memory as possible on the server
>>hardware. We
>> >> have an OLAP workload that allows users to perform interactive
>>analysis
>> >> over huge sets of data. While reading about hbase configuration I
>>came
>> >> across two configs
>> >>
>> >> 1. Hbase bucket cache
>> >> <
>> >>
>>
>>http://blog.asquareb.com/blog/2014/11/24/how-to-leverage-large-physical-m
>>emory-to-improve-hbase-read-performance
>> >> >
>> >> (off heap) which looks like a good option to bypass garbage
>>collection.
>> >> 2. Hadoop pinned hdfs blocks
>> >> <
>> http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/>
>> >> (max locked memory) concept that loads the hdfs blocks in memory, but
>> given
>> >> that hbase is configured with short circuit reads I assume this
>>config
>> may
>> >> not be of much help. Instead it would be right to increase hbase
>>region
>> >> server heap memory. Is my understanding right?
>> >>
>> >> We use HBase with Phoenix.
>> >> Kindly let me know your thoughts or suggestions on any more options
>> that I
>> >> should explore
>> >>
>> >> Thanks,
>> >> Amit.
>> >>
>>
Re: Leveraging As Much Memory As Possible
Posted by Ted Yu <yu...@gmail.com>.
For #2, I did a quick search in Phoenix code base for bucket cache - I
didn't find match.
Maybe ask on Phoenix mailing list.
On Wed, Mar 30, 2016 at 11:07 PM, Amit Shah <am...@gmail.com> wrote:
> Thanks Ted and Anoop for the replies. A couple of follow up questions
>
> 1. I could see the bucket cache and on heap memory stats reflected on hbase
> region server UI <http://i.imgur.com/4uwaWuC.png>, what are ways to
> monitor
> and see if its getting utilized?
> 2. We are using phoenix as a sql layer above hbase. Are there any phoenix
> configurations that I could benefit in leveraging memory?
>
> Thanks,
> Amit.
>
> On Thu, Mar 31, 2016 at 10:44 AM, Anoop John <an...@gmail.com>
> wrote:
>
> > Ya having HBase side cache will be a better choice rather than HDFS
> > cache IMO. Yes u r correct... You might not want to give a very
> > large size for the heap. You can make use of the off heap BucketCache.
> >
> > -Anoop-
> >
> > On Thu, Mar 31, 2016 at 4:35 AM, Ted Yu <yu...@gmail.com> wrote:
> > > For #1, please see the top two blogs @ https://blogs.apache.org/hbase/
> > >
> > > FYI
> > >
> > > On Wed, Mar 30, 2016 at 7:59 AM, Amit Shah <am...@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I am trying to configure my hbase (version 1.0) phoenix (version -
> 4.6)
> > >> cluster to utilize as much memory as possible on the server hardware.
> We
> > >> have an OLAP workload that allows users to perform interactive
> analysis
> > >> over huge sets of data. While reading about hbase configuration I came
> > >> across two configs
> > >>
> > >> 1. Hbase bucket cache
> > >> <
> > >>
> >
> http://blog.asquareb.com/blog/2014/11/24/how-to-leverage-large-physical-memory-to-improve-hbase-read-performance
> > >> >
> > >> (off heap) which looks like a good option to bypass garbage
> collection.
> > >> 2. Hadoop pinned hdfs blocks
> > >> <
> > http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/>
> > >> (max locked memory) concept that loads the hdfs blocks in memory, but
> > given
> > >> that hbase is configured with short circuit reads I assume this config
> > may
> > >> not be of much help. Instead it would be right to increase hbase
> region
> > >> server heap memory. Is my understanding right?
> > >>
> > >> We use HBase with Phoenix.
> > >> Kindly let me know your thoughts or suggestions on any more options
> > that I
> > >> should explore
> > >>
> > >> Thanks,
> > >> Amit.
> > >>
> >
>
Re: Leveraging As Much Memory As Possible
Posted by Amit Shah <am...@gmail.com>.
Thanks Ted and Anoop for the replies. A couple of follow up questions
1. I could see the bucket cache and on heap memory stats reflected on hbase
region server UI <http://i.imgur.com/4uwaWuC.png>, what are ways to monitor
and see if its getting utilized?
2. We are using phoenix as a sql layer above hbase. Are there any phoenix
configurations that I could benefit in leveraging memory?
Thanks,
Amit.
On Thu, Mar 31, 2016 at 10:44 AM, Anoop John <an...@gmail.com> wrote:
> Ya having HBase side cache will be a better choice rather than HDFS
> cache IMO. Yes u r correct... You might not want to give a very
> large size for the heap. You can make use of the off heap BucketCache.
>
> -Anoop-
>
> On Thu, Mar 31, 2016 at 4:35 AM, Ted Yu <yu...@gmail.com> wrote:
> > For #1, please see the top two blogs @ https://blogs.apache.org/hbase/
> >
> > FYI
> >
> > On Wed, Mar 30, 2016 at 7:59 AM, Amit Shah <am...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I am trying to configure my hbase (version 1.0) phoenix (version - 4.6)
> >> cluster to utilize as much memory as possible on the server hardware. We
> >> have an OLAP workload that allows users to perform interactive analysis
> >> over huge sets of data. While reading about hbase configuration I came
> >> across two configs
> >>
> >> 1. Hbase bucket cache
> >> <
> >>
> http://blog.asquareb.com/blog/2014/11/24/how-to-leverage-large-physical-memory-to-improve-hbase-read-performance
> >> >
> >> (off heap) which looks like a good option to bypass garbage collection.
> >> 2. Hadoop pinned hdfs blocks
> >> <
> http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/>
> >> (max locked memory) concept that loads the hdfs blocks in memory, but
> given
> >> that hbase is configured with short circuit reads I assume this config
> may
> >> not be of much help. Instead it would be right to increase hbase region
> >> server heap memory. Is my understanding right?
> >>
> >> We use HBase with Phoenix.
> >> Kindly let me know your thoughts or suggestions on any more options
> that I
> >> should explore
> >>
> >> Thanks,
> >> Amit.
> >>
>
Re: Leveraging As Much Memory As Possible
Posted by Anoop John <an...@gmail.com>.
Ya having HBase side cache will be a better choice rather than HDFS
cache IMO. Yes u r correct... You might not want to give a very
large size for the heap. You can make use of the off heap BucketCache.
-Anoop-
On Thu, Mar 31, 2016 at 4:35 AM, Ted Yu <yu...@gmail.com> wrote:
> For #1, please see the top two blogs @ https://blogs.apache.org/hbase/
>
> FYI
>
> On Wed, Mar 30, 2016 at 7:59 AM, Amit Shah <am...@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to configure my hbase (version 1.0) phoenix (version - 4.6)
>> cluster to utilize as much memory as possible on the server hardware. We
>> have an OLAP workload that allows users to perform interactive analysis
>> over huge sets of data. While reading about hbase configuration I came
>> across two configs
>>
>> 1. Hbase bucket cache
>> <
>> http://blog.asquareb.com/blog/2014/11/24/how-to-leverage-large-physical-memory-to-improve-hbase-read-performance
>> >
>> (off heap) which looks like a good option to bypass garbage collection.
>> 2. Hadoop pinned hdfs blocks
>> <http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/>
>> (max locked memory) concept that loads the hdfs blocks in memory, but given
>> that hbase is configured with short circuit reads I assume this config may
>> not be of much help. Instead it would be right to increase hbase region
>> server heap memory. Is my understanding right?
>>
>> We use HBase with Phoenix.
>> Kindly let me know your thoughts or suggestions on any more options that I
>> should explore
>>
>> Thanks,
>> Amit.
>>
Re: Leveraging As Much Memory As Possible
Posted by Ted Yu <yu...@gmail.com>.
For #1, please see the top two blogs @ https://blogs.apache.org/hbase/
FYI
On Wed, Mar 30, 2016 at 7:59 AM, Amit Shah <am...@gmail.com> wrote:
> Hi,
>
> I am trying to configure my hbase (version 1.0) phoenix (version - 4.6)
> cluster to utilize as much memory as possible on the server hardware. We
> have an OLAP workload that allows users to perform interactive analysis
> over huge sets of data. While reading about hbase configuration I came
> across two configs
>
> 1. Hbase bucket cache
> <
> http://blog.asquareb.com/blog/2014/11/24/how-to-leverage-large-physical-memory-to-improve-hbase-read-performance
> >
> (off heap) which looks like a good option to bypass garbage collection.
> 2. Hadoop pinned hdfs blocks
> <http://blog.cloudera.com/blog/2014/08/new-in-cdh-5-1-hdfs-read-caching/>
> (max locked memory) concept that loads the hdfs blocks in memory, but given
> that hbase is configured with short circuit reads I assume this config may
> not be of much help. Instead it would be right to increase hbase region
> server heap memory. Is my understanding right?
>
> We use HBase with Phoenix.
> Kindly let me know your thoughts or suggestions on any more options that I
> should explore
>
> Thanks,
> Amit.
>